robots.txt looks like the simplest file on your site, and it's surprisingly easy to get wrong.
One stray Disallow: / and search engines stop crawling the whole thing. Now that there's a whole list of AI crawlers to make decisions about too, I wanted a quick way to see exactly who's allowed to read what.
How the Rules Work
Crawlers look for the group of rules with their own name first, and only fall back to the User-agent: * group if there isn't one. That catches people out, because a crawler with its own group ignores everything in the * group.
Inside a group, the most specific rule wins, meaning the longest path that matches. If an Allow and a Disallow are the same length, the Allow wins. * matches anything, and $ marks the end of the URL.
This tool follows those rules from RFC 9309, which is the standard Google and most other crawlers follow.
Things robots.txt Can't Do
It doesn't hide pages. A blocked page can still show up in search results if other sites link to it. Use a noindex meta tag for that, and leave the page crawlable so search engines can see the tag.
It isn't enforced. Well-behaved crawlers follow it, and the rest don't have to. Anything private needs a login.
Google ignores Crawl-delay and noindex. Bing and some others honor Crawl-delay, but Google stopped reading noindex in robots.txt back in 2019.
Frequently Asked Questions
Should I block AI crawlers?
That's your call, and robots.txt lets you be specific about it. Blocking the training crawlers (like GPTBot and ClaudeBot) keeps your pages out of future training data. Blocking the search and user crawlers keeps you out of AI answers and stops assistants from reading your page when someone asks about it. I let all of them in on this site, and the AI agent readiness check shows where any site stands.
What happens if my robots.txt is missing?
Crawlers treat a missing robots.txt (a 404) as permission to crawl everything. A server error is different, and Google treats a robots.txt that returns 500 errors as a reason to stop crawling your site for a while.
What's the size limit?
Google reads the first 500 KiB and ignores the rest. You'll never get close to that with a normal file.