Cloudflare flipped your crawler defaults

Cloudflare's new crawler defaults are a policy layer that sits above your robots.txt and decides, by bot category, which automated clients reach your pages. As of September 15, 2026, that layer ships with training and agent crawlers blocked on ad-monetized pages, and the block applies to sites whose owners never opened the setting.

This matters for AI visibility because the categories are not clean. A single choice labelled "block training" can remove a crawler that also feeds an answer engine, and in one case it can remove Googlebot.

#What the categories actually are

Cloudflare splits crawler behavior into three configurable buckets. Its own announcement defines them as search (collecting or indexing content to answer questions about it later), agent (acting in real time on a person's behalf), and training (taking content to train or fine-tune a model). There are further categories in the taxonomy for things like ads verification and link previews, but those three are what most site owners will see as toggles.

The new default keeps search allowed and blocks training and agent traffic on pages that carry ads. Cloudflare's stated reasoning is that an ad signals the page was meant for a person to land on and see it.

That reasoning is coherent. The problem is what happens when a crawler belongs to more than one bucket.

#Most restrictive wins, and Googlebot is mixed-use

This is the part worth reading twice. Crawlers that combine purposes, which Cloudflare names as including Googlebot, Applebot and BingBot, are resolved under the most restrictive applicable rule. Cloudflare states that such mixed-use crawlers "will be blocked by customers who have selected to block Training."

So a site owner who reasonably decides they do not want their content training models, and flips the training toggle off, can take Googlebot off their ad-bearing pages at the same time. Not the AI Overviews layer specifically. Googlebot.

The same logic runs the other way for answer engines. Several of the crawlers that retrieve pages to compose a cited answer are operated by companies that also run training crawlers, and where the operator has not cleanly separated the two, a training block catches the retrieval path as well. That is the entire point of the deadline: Cloudflare is pressuring AI companies to split search from agent use and training so that publishers can say yes to one and no to the other. Until an operator does that split, you cannot say yes to one and no to the other, and the default resolves against you.

For anyone tracking citations, this is a silent failure mode. Your content did not get worse. Your schema did not break. A category rule you never looked at decided a retrieval bot does not get a 200.

#Who this applies to

The defaults are not retroactive across the whole network. Per TechCrunch's reporting, they apply to new Cloudflare customers, to new sites set up by existing customers, and to all existing free customers. Cloudflare's own post frames the change as applying to new domains onboarding to the platform, with existing customers able to adjust settings first.

Read that list carefully, because two groups get caught without noticing. Free-tier customers are in scope regardless of how long the site has existed. And existing paid customers who add a new property inherit the new default on that property while their older properties keep the old one, which means two sites on the same account can behave differently.

Scale makes this worth checking even if you think it does not apply to you. Cloudflare says it powers more than 20% of web domains, so a meaningful share of the pages an answer engine considers for any given query are sitting behind this policy.

#How to check, in about five minutes

The setting lives in AI Crawl Control rather than in your DNS or page rules. Cloudflare's AI Crawl Control documentation covers the dashboard, and the manage AI crawlers page covers the per-crawler controls specifically.

Three things to look at:

First, find the blanket "block AI bots" toggle and confirm what it covers on your account. It is a broad control, and a broad control is exactly what you do not want here, because it does not distinguish a bot that retrieves your page to cite it from a bot that ingests your page to train on it.

Second, open the crawlers table and look at the category assigned to each bot hitting you. The operator grouping view makes this faster, since it clusters crawlers under the company that runs them. You are looking for any crawler you want cited by that is sitting in a blocked category.

Third, check whether your pages are classified as ad-monetized, because that classification is what activates the new default. A site with no ad units is in a different position than a publisher running programmatic inventory.

#What to actually set

There is no universally correct answer, and anyone who tells you there is has not read the category list. The honest framing is a trade you have to make deliberately.

If referral traffic and citations matter more to you than training exclusion, allow the search and agent categories and accept that some operators will not cleanly separate training from retrieval. If you have a licensing position or a legal reason to refuse training, block it and accept that you will lose retrieval from operators who have not split their crawlers, and verify separately that Googlebot still reaches your ad pages.

What you should not do is leave it on a default you have not read, then spend a quarter wondering why your citation counts drifted down. In our own crawl work, access failures are the least interesting reason to lose a citation and the most annoying to diagnose after the fact, because nothing in the content gives you a clue. The page looks fine. It just never got fetched.

Go look at the toggle.