robots.txt for AI crawlers: the seven agents worth naming
Name each AI agent explicitly in robots.txt instead of relying on your wildcard group. Training crawlers and answer-time crawlers are different agents: blocking Google-Extended or OAI-SearchBot removes you from answers, while blocking GPTBot only affects training data collection.
The agents, and what each one controls
GPTBot collects data for model training. OAI-SearchBot fetches pages at answer time for ChatGPT search — block it and you disappear from ChatGPT's cited results even though your training exposure is unchanged.
ClaudeBot serves Anthropic's crawling. PerplexityBot fetches for Perplexity's index. Google-Extended is a control for Gemini and AI Overviews grounding, separate from Googlebot. Applebot-Extended governs Apple Intelligence. Bingbot still feeds Copilot.
Why implicit allow is not good enough
If you have no named group, these agents fall back to your wildcard rules. That works right up until someone tightens the wildcard group to stop a scraper and takes four assistants offline with it. We have watched that happen twice on sites where nobody noticed for a quarter.
Explicit groups also document intent. The next engineer reading the file knows the decision was deliberate.
A starting file
Allow all seven on your public content. Disallow the paths you would not want quoted: checkout, account, search results, staging. Add a Sitemap: line — several AI crawlers use it as their discovery entry point.
Then verify. Fetch your own robots.txt and check each group parses the way you think it does; a stray blank line inside a group is the most common cause of a rule that never applies.
Want the version of this measured against your own site? Run a free audit.