FREE TOOL · NO SIGNUP · NO ACCOUNT

AI crawler access check

Not whether a site shows up in AI answers — why it can’t. Point it at a prospect’s domain and you get the cause and the fix in about two seconds.

It reads robots.txt properly, then probes the homepage with each bot’s own user-agent to catch the CDN and firewall blocks that robots.txt does not show.

Add your name and logo

Both are optional and both live in the URL, not in a database — they put your name on the page so it can be forwarded to a prospect. https logo URLs only.

Four kinds of bot, and blocking each one costs something different

Most crawler checkers print one flat list of user-agents, which tells you nothing. Blocking GPTBot and blocking OAI-SearchBot have completely different consequences, two of the twenty AI agents are not crawlers at all, and the fourth group below is the one whose consequence you can see with your own eyes.

Retrieval and user-triggered

These are the agents that fetch a page in order to answer someone's question. Blocking one of these removes you from that product's answers.

OAI-SearchBot · ChatGPT-User · Claude-SearchBot · Claude-User · PerplexityBot · Perplexity-User · Bingbot · Googlebot

Training crawlers

Blocking these is a policy choice. It keeps your content out of model training and costs little visibility.

GPTBot · ClaudeBot · CCBot · Bytespider · Meta-ExternalAgent · Meta-ExternalFetcher · anthropic-ai · cohere-ai · MistralAI-User · Amazonbot

Control tokens

These are not crawlers. They do not identify an agent that visits your site — they change what a vendor may do with content its ordinary crawler already fetched.

Google-Extended · Applebot-Extended

Social and link-preview crawlers

These fetch your page to build the preview card when someone pastes a link. If one is blocked, links to your site paste as plain text.

Twitterbot · facebookexternalhit · Slackbot-LinkExpanding · Pinterestbot

Blocking Google-Extended does not remove you from AI Overviews or AI Mode. Google’s documentation says Google-Extended has no separate user-agent string of its own — crawling is done by Googlebot, and the token only changes what Google may do with content it already has. AI Overviews and AI Mode are generated from the Search index. Blocking Googlebot is what takes you out of them.

Where this comes from, in Google’s own words — along with three other things about AI bots that turned out to be wrong.

What it actually does

  1. Reads robots.txt the way the RFC says to

    Longest user-agent match, not first match. Most specific rule by path length, with Allow winning ties. An empty Disallow means allow everything. A 5xx means we cannot determine — never that you are blocked.

  2. Probes the homepage with each bot's user-agent

    robots.txt can permit a bot while Cloudflare or a WAF rule refuses it at the edge. That is the most common accidental block and it is invisible to every robots.txt checker.

  3. Checks what a crawler finds once it is in

    llms.txt, structured data, whether an h1 and a last-updated date exist, and whether the homepage content is in the HTML at all or arrives only after JavaScript runs.

  4. Checks whether a link to the site pastes with a preview

    The crawlers behind the card on X, Meta's apps, Slack and Pinterest — plus the tags they build it from, including whether og:image is absolute and actually resolves, and whether og:url names the host that answered.

  5. Generates the fix from what is wrong here

    Not a template. It carries over the rules your `*` group already had, and it tells you when appending to robots.txt cannot fix the problem at all.

Where this list of 24 agents came fromChecked 2026-08-10

Read from each vendor’s own crawler documentation on 2026-08-10. These tokens get renamed and retired, and third-hand lists go stale quietly, so here is exactly what those pages say:

  • OpenAI documents OAI-SearchBot and GPTBot as independent settings: a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to stay out of training. Sites opted out of OAI-SearchBot 'will not be shown in ChatGPT search answers'.
  • OpenAI documents ChatGPT-User as user-initiated and adds: 'Because these actions are initiated by a user, robots.txt rules may not apply.' It also states ChatGPT-User is not used to determine whether content appears in Search.
  • Perplexity documents Perplexity-User with: 'Since a user requested the fetch, this fetcher generally ignores robots.txt rules.' PerplexityBot, by contrast, is the one they recommend allowing in order to appear in results.
  • Anthropic documents three distinct agents: ClaudeBot (training), Claude-User (live fetch for a user's question) and Claude-SearchBot (indexing for search). Blocking the latter two is what costs visibility.
  • Google's crawler documentation states that Google-Extended 'doesn't have a separate HTTP request user agent string' — crawling is done with Google's existing user agents. It is a use-preference token, not an agent that visits. This is why blocking it cannot remove a site from AI Overviews or AI Mode: those are generated from the Search index Googlebot builds.
  • Two of the eight retrieval agents (ChatGPT-User, Perplexity-User) are documented by their own vendors as possibly or generally ignoring robots.txt. We still report the rule, because it is what the site is asking for — but we say on the page that the request may not be honoured.
  • Slack states plainly that it does not honour robots.txt: 'we do not currently honor robots.txt files', explaining that it acts on behalf of a user pasting a link rather than as a crawler. Every third-party list we checked describes Slackbot as respecting robots.txt. It is the clearest example of why these strings are read from the vendor.
  • Meta documents one agent, facebookexternalhit, for previews across 'Facebook, Instagram, or Messenger'. WhatsApp is commonly listed as covered by the same agent; Meta's page does not name it, so we do not either.
  • Meta also notes the agent 'might bypass robots.txt when performing security or integrity checks', so an allowed rule is not a guarantee of a fetch and a Disallow is not a guarantee of absence.
  • X documents the agent as Twitterbot with a version (Twitterbot/1.0) and gives `User-agent: Twitterbot` as the robots.txt exception to write. It states the crawler respects the robots.txt specification, that a blocked page shows no card at all, and that a blocked image URL costs the thumbnail only.
  • Pinterest publishes two strings, 'Pinterest/0.2' and 'Mozilla/5.0 (compatible; Pinterestbot/1.0; …)', plus a mobile variant. We probe with the Pinterestbot form because that is the token a robots.txt group would name.

Four link-preview crawlers are deliberately absent, because no vendor publishes their user-agent — LinkedIn, Discord, Telegram, Reddit. Every string on offer for those traces back to a third-party bot directory, and a verdict is only as good as the string it was measured with.

This page fails its own build if that check goes more than 90 days without being redone.

SIMPLYREPORT

The monthly version of this sits alongside GA4 and Search Console, in one branded report for your client.

SimplyReport pulls the data and writes the analysis; you send one link. Folding AI search visibility into the same report is what we are building next — it is not measuring anything yet. Built for solo consultants. Closed beta, from $29/month.

Other free tools: the GA4 channel group that catches AI referral traffic · read the provenance record attached to an image