# https://katama.io/
#
# Every crawler may read every page. The one path closed is /api/, the enquiry
# form endpoint, which only answers a POST and has nothing to read.
#
# Answer engines are named as well as covered by the wildcard, because being
# readable by them is the point of the AEO work we sell. A crawler follows only
# the most specific group that names it, so the named crawlers share the one
# group with the wildcard and every rule applies to all of them at once. Add a
# new crawler as a User-agent line inside this group, never as a group of its
# own, or it will stop seeing Disallow: /api/. The Disallow comes first because
# older parsers take the first rule that matches, while Google and RFC 9309 take
# the longest; in this order both readings agree.
#
#   OpenAI       GPTBot (training), OAI-SearchBot (ChatGPT search),
#                ChatGPT-User (fetches a page a user asked about)
#   Anthropic    ClaudeBot, Claude-SearchBot, Claude-User
#   Perplexity   PerplexityBot, Perplexity-User
#   Google       Googlebot (Search and AI Overviews), Google-Extended (Gemini)
#   Microsoft    Bingbot (Bing and Copilot, and a source for ChatGPT search)
#   Apple        Applebot, Applebot-Extended (Apple Intelligence)
#   Meta         meta-externalagent, meta-externalfetcher (Meta AI)
#   Amazon       Amazonbot
#   DuckDuckGo   DuckAssistBot
#   Mistral      MistralAI-User
#   Common Crawl CCBot, the open corpus most language models are trained on
#
# A plain language summary of the site for language models is at
# https://katama.io/llms.txt

User-agent: *
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Google-Extended
User-agent: Bingbot
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: CCBot
Disallow: /api/
Allow: /

Sitemap: https://katama.io/sitemap.xml