Three Kinds of AI Bot
"AI crawler" covers a few very different jobs, and most companies now run a separate bot for each one.
Training crawlers collect pages to train future models. Blocking them keeps your writing out of the next training run, and it doesn't affect whether an assistant can find or cite you today.
Search crawlers build the index an AI search product answers from, like ChatGPT search or Perplexity. Blocking these is closer to blocking Googlebot: your pages stop showing up as sources.
User-triggered fetchers visit a page because a person asked an assistant to read it, or the assistant decided to open a link while answering. These behave more like a browser than a crawler, and OpenAI and Perplexity both say their user-triggered fetchers may not follow robots.txt for that reason. Anthropic says Claude-User does follow it.
So the useful question is which of those jobs you're fine with.
The User Agents
These are the tokens you'll put in robots.txt, grouped by company. Names change now and then, so it's worth checking each company's own documentation before you ship a rule.
| User agent | Company | What it's for |
|---|---|---|
GPTBot | OpenAI | Training |
OAI-SearchBot | OpenAI | ChatGPT search results |
ChatGPT-User | OpenAI | Pages a ChatGPT user asks it to open |
ClaudeBot | Anthropic | Training |
Claude-SearchBot | Anthropic | Claude's search results |
Claude-User | Anthropic | Pages a Claude user asks it to open |
PerplexityBot | Perplexity | Perplexity's search index |
Perplexity-User | Perplexity | Pages fetched while answering a user |
Google-Extended | Whether Gemini may train on your pages or use them to ground its answers (a control token, not a separate crawler) | |
Applebot-Extended | Apple | Whether Apple may use pages Applebot crawled for AI training (also a control token) |
meta-externalagent | Meta | Training and AI features |
Amazonbot | Amazon | Amazon's products, and training Amazon's AI models |
Amzn-SearchBot | Amazon | Search in Amazon products like Alexa |
Bytespider | ByteDance | Training (and widely reported to ignore robots.txt) |
CCBot | Common Crawl | An open web archive that a lot of models train on |
On Contrastly, Bytespider alone made about half of all the AI crawler fetches, which is a lot for a bot that's widely reported to ignore robots.txt.
Google-Extended confuses a lot of people. Googlebot still does the crawling, and Google-Extended only tells Google whether that crawl may be used to train Gemini and to ground answers in the Gemini apps. Blocking it doesn't remove you from Google Search, and it doesn't keep you out of AI Overviews either, because those are part of Search and come from the regular Googlebot crawl. Blocking Googlebot to keep out of AI would remove you from search entirely.
You can see which of these a site allows with the robots.txt Checker. It tests any path against any user agent, so you know exactly what a rule does before you deploy it.
Rules You Can Copy
robots.txt groups rules by user agent. A bot follows the most specific group that names it and ignores the rest, and anything not named falls back to the * group.
Allow AI search, block training
This is the setup I'd recommend for most sites that want traffic. Your pages can still be found and cited by AI search, and they stay out of training sets.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Listing several User-agent lines above one set of rules applies those rules to all of them, which keeps the file short. Blocking Google-Extended also keeps your pages out of the answers Gemini grounds in search results, so leave it off the list if you'd rather show up there.
Block all the AI bots
If you don't want AI products using your pages at all, add the search and user bots to the list:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
There's a trade-off here. AI answers are becoming a real source of visits, and I wrote about how much that's already changed search in Nobody Clicks Anymore. Blocking the search bots means giving that up.
Keep bots out of one section
Rules can be per path, too. This lets everything crawl the site but keeps AI bots out of a paid archive:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: OAI-SearchBot
Disallow: /archive/
User-agent: *
Allow: /
What robots.txt Can't Do
robots.txt is a request, and well-behaved crawlers honor it. Nothing enforces it.
On Contrastly, 213 of the 216 visits claiming to be Common Crawl's CCBot came from somewhere else entirely. So did 41% of the visits claiming to be OpenAI's OAI-SearchBot, along with 15% of the PerplexityBot visits and 11% of the ClaudeBot ones. Someone else borrowed the name. A rule for ClaudeBot does nothing about an impostor, because the impostor isn't following robots.txt in the first place.
To stop a bot for real, you have to do it at the server or the CDN:
# nginx: refuse a user agent outright
if ($http_user_agent ~* "Bytespider") {
return 403;
}
Cloudflare has a one-click setting to block known AI crawlers, and it verifies bots by where they connect from, so impostors get caught too. Other CDNs and bot management services offer something similar.
robots.txt also only affects future crawling. Pages already collected for training stay collected.
Content Signals
There's a newer way to say how pages may be used, as opposed to which bots may fetch them. Cloudflare's Content Signals add a line to robots.txt with three separate yes or no answers:
Content-Signal: search=yes, ai-input=yes, ai-train=no
search covers ordinary search results, ai-input covers using a page to answer a question in the moment, and ai-train covers training. Cloudflare launched it in September 2025, and it's still unclear how much each company honors it. It's still free to add alongside your user-agent rules, and this site has one.
Check Your Work
After you change robots.txt:
- Open
https://yoursite.com/robots.txtin a browser to make sure the live file is the one you edited. - Run it through the robots.txt Checker and test a few paths against the bots you care about.
- Make sure
Googlebotcan still reach everything you want in search. - Run the AI Agent Readiness Check if you want to see the whole picture, including markdown versions of your pages and llms.txt.
Then give it a week or two and watch your logs or analytics. The bots that respect the file will taper off, and anything still showing up under a blocked name is one to deal with at the server.