AI agent readiness check

A growing share of the visits to your site come from agents reading on behalf of someone else. This shows you what they see when they land on your page.

The page is fetched from this site's server, so it has to be public. Try it on .

About this tool

I wrote about this shift in An Audience of Agents, and it changed how I think about my own site.

Some of the visitors here are AI search engines, some are assistants fetching a link that someone pasted into a chat, and some are coding tools reading docs. None of them see the page the way you do.

What It Checks

It fetches your page twice. The first time is the way a browser would, and the second time it asks for markdown the way Claude Code and Cursor do. Then it reads your robots.txt and your llms.txt.

From there it checks how much of the page shows up without JavaScript and whether you offer a markdown version. It also works out what each of the well-known AI crawlers is allowed to read, and whether you've said how your content may be used.

This site does everything the check looks for, so if you want an example, try it on joncphillips.com.

Training, Search, and User Crawlers

AI companies run separate crawlers for separate jobs, and robots.txt lets you treat each one differently.

Training crawlers like GPTBot and ClaudeBot collect pages to train models. Google-Extended and Applebot-Extended work a little differently, since they aren't crawlers at all. They're rules that tell Google and Apple whether the pages their regular crawlers fetch may be used for training, and blocking them doesn't affect your search rankings.

Search crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot build the indexes that AI search answers come from, and they're how you end up getting cited.

User agents like ChatGPT-User and Claude-User fetch a page because a person asked about it right then.

A lot of people block training and allow the other two, and that's a reasonable middle ground. The check flags it when a site blocks the search or user crawlers, since that keeps the page out of AI answers entirely, and I doubt that's what most people are going for.

Frequently Asked Questions

Is llms.txt a standard?

Not yet. It's a proposal from llmstxt.org for a markdown file at /llms.txt with the site's name, a one-line summary, and lists of links to the pages worth reading. None of the big AI companies have said their crawlers use it, so the check treats it as a nice-to-have. It's handy for coding agents and docs, though, and it's cheap to add.

What's Content-Signal?

It's a line in robots.txt, proposed by Cloudflare, that says how your content can be used once it's fetched. For example, Content-Signal: search=yes, ai-input=yes, ai-train=no allows search and AI answers but asks not to be used for training. It's a stated preference, and crawlers don't have to honor it.

Do I need to serve markdown?

You don't need to, but it helps agents a lot. A markdown version drops the navigation, scripts, and styling, so an agent gets your content in a fraction of the tokens. You can link one with <link rel="alternate" type="text/markdown">, answer requests that send Accept: text/markdown, or do both like I do here.

Run a website and spotted JCPToolsBot in your logs? It's what fetches pages and records for this tool. Here's what it does and how to block it.

Also by me

Runlight

Once agents are reading your site, you'll probably want to know about it. Runlight is my open source analytics tool that installs like a package and keeps the numbers in your own database.

Visit runlight.sh

More free tools

All tools