Cloudflare's 15 September AI bot defaults, and the part that could cost you Google rankings

If your site is not behind Cloudflare, this post is not about you. Vercel, Netlify, Fastly, a plain nginx box, Shopify, WordPress on shared hosting — none of them are changing anything on 15 September. You can stop here.

If your site is behind Cloudflare, there is one thing worth ten minutes before that date, and it is not the thing most of the coverage has led with.

What changed, and when

On 1 July 2026 Cloudflare replaced its single "block AI bots" switch with three separate categories. From its announcement, the definitions are:

The second date is 15 September 2026, when new defaults take effect. Cloudflare's changelog entry states it plainly: "Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: Bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed."

Two limits in that sentence are worth reading slowly, because a lot of secondary coverage has dropped both.

It is scoped to new domains. The bot documentation describes the change as updated defaults for new domains, not a setting that rewrites itself on zones already running. You will find posts elsewhere claiming the defaults also reach any existing zone that never saved an explicit preference. I cannot find that claim in Cloudflare's own material, and the one sentence that gets read that way — that existing customers can opt out of the new defaults before 15 September — is doing a lot of work in those retellings. If your zone is already on Cloudflare, plan on checking the setting rather than on being changed by it.

It is scoped to pages that display ads. Cloudflare gives its reasoning in the announcement: "An ad is a signal that a website owner meant for a person to land there and see it — something monetizable that fuels the business." A documentation site or a services site with no ad units is not what these defaults are aimed at.

So: a narrower change than the headlines suggest, and if you are not onboarding a new domain it may not touch you at all. But the defaults are not the only thing that changes on the 15th, and the second change is the one that can cost you rankings. It is next.

On the 15th, blocking Training starts blocking Googlebot

Googlebot does not crawl for one purpose. It crawls for Search, and Google's AI features are fed from the same index. The same is true of Applebot and BingBot.

Until now those crawlers have not been caught by a Training block. That changes on the same date. From the announcement:

Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners.

And the consequence, spelled out in the same post: "Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)."

Read that with the category names in place. From 15 September, turning off Training turns off Googlebot. Not Google's AI features — Googlebot, the crawler your organic rankings depend on.

This one is not scoped to new domains. It applies to any zone whose owner has selected to block Training — including anyone who switched on the old, single "Block AI bots" toggle at some point in the last two years and has not thought about it since. That toggle is deprecating on 15 September and, per the same documentation, mixed-purpose crawlers "will also be blocked by all configurations to block AI training, including the legacy 'Block AI bots' option."

There is no combination of these three switches that keeps Googlebot and blocks Training. That is the trade, and it is the whole decision.

The one lever that changes the answer is scope rather than category: Cloudflare's mitigation options are "Block (on all pages)" and "Block on pages with ads", so a Training block confined to ad pages only costs you Googlebot on ad pages. If your site carries no ads, that scope blocks nothing at all — which is worth knowing before you reach for the category switches.

The failure mode here is not "my content stays out of AI training." It is "I lost organic Google rankings while trying to stay out of AI training." That is the whole point of this post.

Why the three categories are not interchangeable

They answer different questions, and the difference is the reason splitting one switch into three was worth doing at all.

Retrieval and user-triggered crawlers decide whether you appear in AI answers. When someone asks an assistant a question and it goes and fetches pages to answer with, that fetch is what puts you in the answer — with a link, in front of a person who is looking for what you sell. Blocking those crawlers removes you from that surface. It does not remove anything from a model.

Training crawlers decide whether your content contributes to a model's weights. That is a legitimate thing to have an opinion about, and plenty of publishers have one. But it is a separate question, on a separate timescale, with separate consequences.

Blocking the first to achieve the second is the expensive mistake, and it is a common one because for two years there was one switch and it did both.

What to actually do

Open the Cloudflare dashboard and go to Security Settings → Configure AI bot policies. Three things are worth reading before you change anything.

Which of the three categories are currently blocked. If Training is among them — or if the legacy "Block AI bots" toggle is still on — that is the configuration that starts catching Googlebot on the 15th.

Whether the block is set to all pages or only pages with ads. "Block (on all pages)" and "Block on pages with ads" are different answers to a different question from the category switches, and this is the setting that decides how much of your site a Training block actually reaches.

Whether the zone serves ads at all. If it does not, the new September defaults are irrelevant to it, and an ads-scoped block of your own is doing nothing — which is useful to know either way.

Then make a decision and save it, whatever the decision is. Not "unblock everything," not "block everything" — record a deliberate choice, so that what your site does is something you chose. If you want out of model training and you also want your rankings, you now know those two wants are in direct tension, and you can decide which side of it you are on with that fact in front of you.

The danger here is inheriting a default, not making a decision.

One caveat about numbers

I have not put a figure in this post for how many sites have AI bot blocking switched on, or how many of those are inadvertently blocking Googlebot. I have seen several such figures quoted in the last month and could not trace any of them to a source that measured it. If you see one, check whether it links to something.

If you want to see what your robots.txt and your edge are currently doing to each crawler, the crawler access check reads both and tells you which rule is responsible.

All posts