Esc

    ↑ ↓ to moveEnter to openEsc to close

    Guide

    robots.txt for AI Crawlers

    Which AI bots visit your site, and the robots.txt rules to let in only the ones you want.

    On Contrastly, a photography site I run, AI crawlers made almost 12,000 fetches in 90 days, and some of them weren't who they claimed to be. Here's who these bots are and how to handle them.

    Three Kinds of AI Bot

    "AI crawler" covers a few very different jobs, and most companies now run a separate bot for each one.

    Training crawlers collect pages to train future models. Blocking them keeps your writing out of the next training run, and it doesn't affect whether an assistant can find or cite you today.

    Search crawlers build the index an AI search product answers from, like ChatGPT search or Perplexity. Blocking these is closer to blocking Googlebot: your pages stop showing up as sources.

    User-triggered fetchers visit a page because a person asked an assistant to read it, or the assistant decided to open a link while answering. These behave more like a browser than a crawler, and OpenAI and Perplexity both say their user-triggered fetchers may not follow robots.txt for that reason. Anthropic says Claude-User does follow it.

    So the useful question is which of those jobs you're fine with.

    The User Agents

    These are the tokens you'll put in robots.txt, grouped by company. Names change now and then, so it's worth checking each company's own documentation before you ship a rule.

    User agentCompanyWhat it's for
    GPTBotOpenAITraining
    OAI-SearchBotOpenAIChatGPT search results
    ChatGPT-UserOpenAIPages a ChatGPT user asks it to open
    ClaudeBotAnthropicTraining
    Claude-SearchBotAnthropicClaude's search results
    Claude-UserAnthropicPages a Claude user asks it to open
    PerplexityBotPerplexityPerplexity's search index
    Perplexity-UserPerplexityPages fetched while answering a user
    Google-ExtendedGoogleWhether Gemini may train on your pages or use them to ground its answers (a control token, not a separate crawler)
    Applebot-ExtendedAppleWhether Apple may use pages Applebot crawled for AI training (also a control token)
    meta-externalagentMetaTraining and AI features
    AmazonbotAmazonAmazon's products, and training Amazon's AI models
    Amzn-SearchBotAmazonSearch in Amazon products like Alexa
    BytespiderByteDanceTraining (and widely reported to ignore robots.txt)
    CCBotCommon CrawlAn open web archive that a lot of models train on

    On Contrastly, Bytespider alone made about half of all the AI crawler fetches, which is a lot for a bot that's widely reported to ignore robots.txt.

    Google-Extended confuses a lot of people. Googlebot still does the crawling, and Google-Extended only tells Google whether that crawl may be used to train Gemini and to ground answers in the Gemini apps. Blocking it doesn't remove you from Google Search, and it doesn't keep you out of AI Overviews either, because those are part of Search and come from the regular Googlebot crawl. Blocking Googlebot to keep out of AI would remove you from search entirely.

    You can see which of these a site allows with the robots.txt Checker. It tests any path against any user agent, so you know exactly what a rule does before you deploy it.

    Rules You Can Copy

    robots.txt groups rules by user agent. A bot follows the most specific group that names it and ignores the rest, and anything not named falls back to the * group.

    Allow AI search, block training

    This is the setup I'd recommend for most sites that want traffic. Your pages can still be found and cited by AI search, and they stay out of training sets.

    Text
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Google-Extended
    User-agent: Applebot-Extended
    User-agent: meta-externalagent
    User-agent: Amazonbot
    User-agent: CCBot
    User-agent: Bytespider
    Disallow: /
    
    User-agent: *
    Allow: /
    
    Sitemap: https://example.com/sitemap.xml
    

    Listing several User-agent lines above one set of rules applies those rules to all of them, which keeps the file short. Blocking Google-Extended also keeps your pages out of the answers Gemini grounds in search results, so leave it off the list if you'd rather show up there.

    Block all the AI bots

    If you don't want AI products using your pages at all, add the search and user bots to the list:

    Text
    User-agent: GPTBot
    User-agent: OAI-SearchBot
    User-agent: ChatGPT-User
    User-agent: ClaudeBot
    User-agent: Claude-SearchBot
    User-agent: Claude-User
    User-agent: PerplexityBot
    User-agent: Perplexity-User
    User-agent: Google-Extended
    User-agent: Applebot-Extended
    User-agent: meta-externalagent
    User-agent: Amazonbot
    User-agent: Amzn-SearchBot
    User-agent: CCBot
    User-agent: Bytespider
    Disallow: /
    

    There's a trade-off here. AI answers are becoming a real source of visits, and I wrote about how much that's already changed search in Nobody Clicks Anymore. Blocking the search bots means giving that up.

    Keep bots out of one section

    Rules can be per path, too. This lets everything crawl the site but keeps AI bots out of a paid archive:

    Text
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: PerplexityBot
    User-agent: OAI-SearchBot
    Disallow: /archive/
    
    User-agent: *
    Allow: /
    

    What robots.txt Can't Do

    robots.txt is a request, and well-behaved crawlers honor it. Nothing enforces it.

    On Contrastly, 213 of the 216 visits claiming to be Common Crawl's CCBot came from somewhere else entirely. So did 41% of the visits claiming to be OpenAI's OAI-SearchBot, along with 15% of the PerplexityBot visits and 11% of the ClaudeBot ones. Someone else borrowed the name. A rule for ClaudeBot does nothing about an impostor, because the impostor isn't following robots.txt in the first place.

    To stop a bot for real, you have to do it at the server or the CDN:

    Nginx
    # nginx: refuse a user agent outright
    if ($http_user_agent ~* "Bytespider") {
        return 403;
    }
    

    Cloudflare has a one-click setting to block known AI crawlers, and it verifies bots by where they connect from, so impostors get caught too. Other CDNs and bot management services offer something similar.

    robots.txt also only affects future crawling. Pages already collected for training stay collected.

    Content Signals

    There's a newer way to say how pages may be used, as opposed to which bots may fetch them. Cloudflare's Content Signals add a line to robots.txt with three separate yes or no answers:

    Text
    Content-Signal: search=yes, ai-input=yes, ai-train=no
    

    search covers ordinary search results, ai-input covers using a page to answer a question in the moment, and ai-train covers training. Cloudflare launched it in September 2025, and it's still unclear how much each company honors it. It's still free to add alongside your user-agent rules, and this site has one.

    Check Your Work

    After you change robots.txt:

    1. Open https://yoursite.com/robots.txt in a browser to make sure the live file is the one you edited.
    2. Run it through the robots.txt Checker and test a few paths against the bots you care about.
    3. Make sure Googlebot can still reach everything you want in search.
    4. Run the AI Agent Readiness Check if you want to see the whole picture, including markdown versions of your pages and llms.txt.

    Then give it a week or two and watch your logs or analytics. The bots that respect the file will taper off, and anything still showing up under a blocked name is one to deal with at the server.