JCPToolsBot

If you found JCPToolsBot in your logs, it's the fetcher behind the free tools on this site. It only visits when a person types an address into one of them, and here's how it behaves.

The Bot

When It Visits

JCPToolsBot never crawls on its own. It shows up when someone uses one of my tools on a page of your site, like the social card preview or the accessibility checker, and it fetches what that tool needs. That's usually the page itself, along with things any browser or search engine would read anyway, such as your robots.txt, your sitemap, your favicon, and your social image. It never goes looking for files you didn't publish. The launch check does ask for one made-up address, to see whether your missing pages answer with a 404, so that's what a single odd 404 in your logs is.

It only sends GET and HEAD requests, and it stops reading a response once it hits a size or time limit. The header checker sends a regular browser's user agent by default, because the point of that tool is to show what a visitor gets, and it can be switched to JCPToolsBot's. The same limits apply either way, and it never claims to be a search engine or any other crawler.

How Often

It doesn't visit much. Each person gets a small number of checks an hour, and any one site can only be checked about twenty times an hour by everyone combined. Requests that look like they're probing a site for weaknesses, rather than pointing at a page, are refused before anything is sent.

How to Block It

Add a group for it to your robots.txt, and it will stop fetching the paths you disallow:

Text
User-agent: JCPToolsBot
Disallow: /

It only follows a group that names it. Rules for every crawler (User-agent: *) don't apply, since people often use these tools to check pages that keep crawlers out on purpose, and the bot only ever acts for a person who asked.

If you'd rather not touch your robots.txt, or you want your whole domain left alone, including the DNS and email checks, send me a note and I'll add it to the blocklist. That covers every subdomain too.

Something Looks Wrong

If you're seeing traffic from JCPToolsBot that doesn't match what's described here, I want to know about it. Get in touch with the time and the paths it hit, and I'll look into it right away. I keep a short log of checks for two weeks for exactly this reason.