# robots.txt checker

Load a site's robots.txt or paste your own to find mistakes and test what each crawler is allowed to read. There's a builder at the bottom if you'd rather start fresh.

This tool looks up the address you give it (a page, a domain, or an email address, depending on the tool) from this site's server and checks what comes back. Use it in a browser at https://joncphillips.com/tools/robots-txt-checker/, or add ?url= and the address to that link to run it straight away.

robots.txt looks like the simplest file on your site, and it's surprisingly easy to get wrong.

One stray `Disallow: /` and search engines stop crawling the whole thing. Now that there's a whole list of AI crawlers to make decisions about too, I wanted a quick way to see exactly who's allowed to read what.

## How the Rules Work

Crawlers look for the group of rules with their own name first, and only fall back to the `User-agent: *` group if there isn't one. That catches people out, because a crawler with its own group ignores everything in the `*` group.

Inside a group, the most specific rule wins, meaning the longest path that matches. If an `Allow` and a `Disallow` are the same length, the `Allow` wins. `*` matches anything, and `$` marks the end of the URL.

This tool follows those rules from RFC 9309, which is the standard Google and most other crawlers follow.

## Things robots.txt Can't Do

**It doesn't hide pages.** A blocked page can still show up in search results if other sites link to it. Use a `noindex` meta tag for that, and leave the page crawlable so search engines can see the tag.

**It isn't enforced.** Well-behaved crawlers follow it, and the rest don't have to. Anything private needs a login.

**Google ignores `Crawl-delay` and `noindex`.** Bing and some others honor `Crawl-delay`, but Google stopped reading `noindex` in robots.txt back in 2019.

## Frequently Asked Questions

### Should I block AI crawlers?

That's your call, and robots.txt lets you be specific about it. Blocking the training crawlers (like GPTBot and ClaudeBot) keeps your pages out of future training data. Blocking the search and user crawlers keeps you out of AI answers and stops assistants from reading your page when someone asks about it. I let all of them in on this site, and the [AI agent readiness check](https://joncphillips.com/tools/agent-readiness-check/) shows where any site stands.

### What happens if my robots.txt is missing?

Crawlers treat a missing robots.txt (a 404) as permission to crawl everything. A server error is different, and Google treats a robots.txt that returns 500 errors as a reason to stop crawling your site for a while.

### What's the size limit?

Google reads the first 500 KiB and ignores the rest. You'll never get close to that with a normal file.

## Preflight.sh

A robots.txt that still says Disallow: / from staging is one of the launch mistakes Preflight catches before you deploy. https://preflight.sh/

More free tools: https://joncphillips.com/tools/
