Google-Extended isn't a crawler, and three other things I had wrong about AI bots

I set out to build a small thing: point it at a domain, tell me whether the crawlers behind AI answers can reach it. I assumed the hard part would be parsing robots.txt. I assumed the list of bots was a solved problem — pick one of the many published lists, done.

The list turned out to be the hard part. Not because the information is hidden, but because almost every list circulating online disagrees with what the companies themselves publish. Below are the four places that gap costs you something real. Every one links to the vendor's own page.

1. Blocking Google-Extended does not take you out of AI Overviews

This is the most expensive misconception, because people act on it.

Google's crawler documentation is unambiguous: Google-Extended has no separate HTTP request user agent string of its own. It is not a crawler. Nothing arrives at your server identifying itself that way. It is a use-preference token — crawling is still done by Google's ordinary user agents, and the token changes what Google may do with content it already fetched.

AI Overviews and AI Mode are generated from the Search index. The Search index is built by Googlebot. So if your goal is to stay out of AI Overviews, blocking Google-Extended does not do it — blocking Googlebot does, along with everything else Googlebot is doing for you.

The inverse is worth stating too: allowing Google-Extended is not a permission you are granting a crawler that visits. It is a policy setting on data Google already has.

Source: Google's common crawlers documentation

2. Three of the agents you'd write rules for say they may not follow them

Robots.txt is a request. That is by design. But three of the agents most people write rules for are documented by their own vendors as possibly or generally ignoring those rules, and the reasoning is the same in all three cases: the fetch was triggered by a person, so the vendor does not treat it as crawling.

OpenAI, on ChatGPT-User — the agent that fetches a page when someone asks ChatGPT to read it. OpenAI's page states that because these actions are initiated by a user, robots.txt rules may not apply. It also says ChatGPT-User is not used to determine whether content appears in Search, and directs you to OAI-SearchBot for managing that. OpenAI's bot documentation

Perplexity, on Perplexity-User — their documentation says that since a user requested the fetch, this fetcher generally ignores robots.txt rules. PerplexityBot is the one they recommend allowing if you want to appear in results. Perplexity's bot guide

Slack — states plainly that it does not currently honour robots.txt files, explaining that it acts on behalf of a user pasting a link rather than as a crawler. Almost every third-party list I checked describes Slackbot as respecting robots.txt. This one is the clearest case in the whole exercise for reading the vendor rather than the summary. Slack's robots page

None of this makes writing the rule pointless. It is still what your site is asking for, and it is what an honest actor will honour. But a checker that prints "blocked" for these three is telling you about a request, not an outcome, and the difference matters when you're explaining to a client why they show up in ChatGPT anyway.

3. Robots.txt permitting a bot doesn't mean the bot gets in

This one has nothing to do with documentation and everything to do with what actually happens on a request.

Your robots.txt can say Allow while your CDN or firewall refuses the request at the edge. Bot-management rules, rate limiting, a WAF ruleset someone enabled two years ago — any of them can block a crawler your robots.txt welcomes, and none of them appear in robots.txt. Every robots.txt checker will tell you everything is fine.

The only way to find out is to make the request with the bot's own user-agent and see what comes back. That is a genuinely different question from what the rules say, and I'd argue it's the more useful one.

Caveat, and it's a real one: a request from my server with a spoofed user-agent is not what a verified bot experiences. Real bots arrive from published IP ranges, and some sites treat them differently on that basis. A probe catches blanket blocks, which is the common case. It cannot tell you what OpenAI's infrastructure sees. That gap can't be closed by trying harder — only stated.

4. Four link-preview crawlers have no documented user-agent at all

LinkedIn, Discord, Telegram and Reddit appear on every AI-and-social crawler list you'll find. Usually with the same strings, in the same order, which is the tell.

I went looking for the vendor page for each. I did not find one. Every source led to a bot directory, which cited another bot directory, and eventually the trail simply stops. Somebody wrote a string down once and it propagated.

That's not proof the strings are wrong. It's the absence of any way to know they're right — and a verdict about whether a bot is blocked is only ever as good as the string it was measured with. So the checker leaves all four out, says so on the page, and there's a test that keeps them out.

If they're ever added, the honest route is to measure it: paste a real link through each platform and read the server log. That's the only source that isn't someone else's guess.

The four that did ship — Twitterbot, facebookexternalhit, Slackbot-LinkExpanding and Pinterestbot — are the ones where the vendor publishes the string themselves.

Two smaller things worth knowing

Meta's agent may bypass robots.txt during integrity checks. Their own documentation notes it. So an Allow is not a guarantee of a fetch, and a Disallow is not a guarantee of absence. Meta's crawler documentation

WhatsApp is widely listed as covered by facebookexternalhit. Meta's page names Facebook, Instagram and Messenger. It does not name WhatsApp. It may well be covered — but I'm not going to assert it on the strength of everyone else asserting it.

What I'd actually do with this

If you manage sites for clients, the three checks worth running are: is Googlebot allowed (that's your AI Overviews answer, not Google-Extended), do the retrieval agents get an actual 200 when they ask, and does a link to the site paste with a preview card.

The rest is policy. Blocking training crawlers is a legitimate choice that costs little visibility. Blocking retrieval agents removes you from those products' answers. Those are different decisions and most lists present them as one.

I built a free checker that runs all of this — no signup, no account, and it generates the fix from what's actually wrong rather than from a template. It also has a section on what it deliberately doesn't check, which took longer to write than the tool did.

Every claim above was read from the vendor's own page. If one of them has changed since, tell me and I'll fix it — that's happened twice already on this site, both times because I went back to the source.