# AI agent readiness check

A growing share of the visits to your site come from agents reading on behalf of someone else. This shows you what they see when they land on your page.

This tool looks up the address you give it (a page, a domain, or an email address, depending on the tool) from this site's server and checks what comes back. Use it in a browser at https://joncphillips.com/tools/agent-readiness-check/, or add ?url= and the address to that link to run it straight away.

I wrote about this shift in [An Audience of Agents](https://joncphillips.com/an-audience-of-agents/), and it changed how I think about my own site.

Some of the visitors here are AI search engines, some are assistants fetching a link that someone pasted into a chat, and some are coding tools reading docs. None of them see the page the way you do.

## What It Checks

It fetches your page twice. The first time is the way a browser would, and the second time it asks for markdown the way Claude Code and Cursor do. Then it reads your `robots.txt` and your `llms.txt`.

From there it checks how much of the page shows up without JavaScript and whether you offer a markdown version. It also works out what each of the well-known AI crawlers is allowed to read, and whether you've said how your content may be used.

**This site does everything the check looks for**, so if you want an example, try it on joncphillips.com.

## Training, Search, and User Crawlers

AI companies run separate crawlers for separate jobs, and robots.txt lets you treat each one differently.

**Training crawlers** like GPTBot and ClaudeBot collect pages to train models. Google-Extended and Applebot-Extended work a little differently, since they aren't crawlers at all. They're rules that tell Google and Apple whether the pages their regular crawlers fetch may be used for training, and blocking them doesn't affect your search rankings.

**Search crawlers** like OAI-SearchBot, Claude-SearchBot, and PerplexityBot build the indexes that AI search answers come from, and they're how you end up getting cited.

**User agents** like ChatGPT-User and Claude-User fetch a page because a person asked about it right then.

A lot of people block training and allow the other two, and that's a reasonable middle ground. The check flags it when a site blocks the search or user crawlers, since that keeps the page out of AI answers entirely, and I doubt that's what most people are going for.

## Frequently Asked Questions

### Is llms.txt a standard?

Not yet. It's a proposal from llmstxt.org for a markdown file at `/llms.txt` with the site's name, a one-line summary, and lists of links to the pages worth reading. None of the big AI companies have said their crawlers use it, so the check treats it as a nice-to-have. It's handy for coding agents and docs, though, and it's cheap to add.

### What's Content-Signal?

It's a line in `robots.txt`, proposed by Cloudflare, that says how your content can be used once it's fetched. For example, `Content-Signal: search=yes, ai-input=yes, ai-train=no` allows search and AI answers but asks not to be used for training. It's a stated preference, and crawlers don't have to honor it.

### Do I need to serve markdown?

You don't need to, but it helps agents a lot. A markdown version drops the navigation, scripts, and styling, so an agent gets your content in a fraction of the tokens. You can link one with `<link rel="alternate" type="text/markdown">`, answer requests that send `Accept: text/markdown`, or do both like I do here.

## Runlight

Once agents are reading your site, you'll probably want to know about it. Runlight is my open source analytics tool that installs like a package and keeps the numbers in your own database. https://runlight.sh/

More free tools: https://joncphillips.com/tools/
