robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot and Google-Extended

AI companies publish the user agents their crawlers use, and robots.txt is how you tell each one what it may fetch. The catch is that one company can run several crawlers with different jobs, so a single rule rarely does what you intend.

Search, training and user-triggered agents

• Search crawlers fetch pages to answer questions with live results, for example OAI-SearchBot, ClaudeBot and PerplexityBot.
• Training crawlers collect pages for future models, for example GPTBot and CCBot.
• User-triggered agents fetch a page because a person asked about it, for example ChatGPT-User, Claude-User and Perplexity-User.
• Google-Extended is not a crawler at all: it is a control token that tells Google whether content may be used for Gemini models.

A sensible starting point

Most businesses that want to be recommended by AI assistants allow the search crawlers and make a deliberate decision about training. A minimal example:

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Decide on training separately
User-agent: GPTBot
Disallow: /private/

Sitemap: https://www.example.com/sitemap.xml

Common mistakes

• A blanket "User-agent: * / Disallow: /" left over from a staging site.
• Blocking every AI crawler to stop training, and with it disappearing from AI search answers.
• Forgetting the Sitemap line, so crawlers find new pages slowly.