The crawl-to-refer ratio is not a verdict

The crawl-to-refer ratio is the number of pages an AI platform's crawler fetches from your site for every one visit that platform sends back. It is a cost measurement. It is not a visibility measurement, and on its own it cannot tell you whether to block a bot.

That distinction got more urgent once the metric started shipping inside products. Cloudflare publishes the ratio on Radar, and Microsoft added an AI Scrape-to-Referral Ratio card to Clarity's Bot Analytics dashboard, reported by Semrush in August 2026 (Semrush). When a number appears in a dashboard next to a bot name, teams start reading it as a verdict on that bot. It is not built to carry that weight.

#How the number is built

Cloudflare's definition is public and purely arithmetic. It divides total HTML requests from the user agents associated with a platform by total HTML requests whose Referer header contains a hostname associated with that same platform, then normalizes to a single referral (Cloudflare).

Both halves of that fraction are weaker than the clean output suggests.

#The denominator misses referrals it cannot see

Cloudflare says this itself. Traffic referred by Claude's native app arrives with no Referer header, and the same is likely true of other native apps, so the published ratios may overstate the real imbalance by an amount Cloudflare cannot quantify (Cloudflare).

The undercount is not an edge case. AI platforms frequently pass no referrer at all, which lands their visits in analytics as direct traffic (Semrush). We wrote about that failure mode on its own in why AI traffic still shows up as Direct.

The important property is that the error runs one way. Every referral the header does not record inflates the ratio. Nothing deflates it. A metric with one-directional measurement error is fine for spotting extremes and unfit for close calls.

#The numerator mixes purposes you cannot separate

Cloudflare sorts crawler purpose into training, search, user action, and undeclared, where undeclared covers crawlers whose operators publish nothing about intent (Cloudflare). For July 1 to 28, 2025, it attributed close to 80% of AI bot crawling to training (Cloudflare).

Its 2026 reporting uses different boundaries. Training sits at 52% of crawler requests as of June 2026, up from 22% in spring 2025, and a mixed-use category now holds more than 36% of activity (Cloudflare). Those two series are not a trend line, because the category definitions moved underneath them. Read each as a snapshot of how Cloudflare classified traffic at the time.

The operationally significant part is the mixed-use bucket. For more than a third of crawler activity, the user agent does not tell you whether a given fetch is filling a training corpus or answering a live user query. Cloudflare states the resulting bind plainly: content owners are pushed to choose between staying discoverable in an agentic web and handing over their most valuable content for nothing (Cloudflare).

That is exactly the tradeoff the ratio cannot arbitrate. Blocking a mixed-use agent because its ratio looks extractive also removes you from the retrieval path that produces citations. If you want purposes handled separately, the instrument is the purpose layer in robots.txt, not the ratio. We covered Content Signals and RSL in the purpose layer of robots.txt.

#The number is window-specific, and the window is usually missing

Anthropic's ClaudeBot is the most-quoted figure in this space, so track what has been published for it.

Cloudflare reported 70,900:1 for the week of June 19 to 26, 2025 (Cloudflare). Working from SEOmator's GEO Data Report and Cloudflare Radar, Digital Applied lists 23,951:1 for Q1 2026 and 11,122:1 for the week of May 25 to June 1, 2026 (Digital Applied). Nobori puts it near 4,580:1 in June 2026 and falling to 1,917:1 in July (Nobori). SEOmator's own rolling 28-day window ending July 21, 2026 gives 2,237:1 (SEOmator).

Some of that decline is real. Assistants that added web search and visible source lists do generate clicks that a pure training crawler never did. But a factor-of-four move inside a single month is not a behavior change, and Nobori says as much: the window, the denominator, and the level of aggregation all move the figure independently of anything the bot did (Nobori).

The ordering flips too. In the SEOmator window ending July 21, 2026, Mistral leads at 3,389:1 with Anthropic behind it at 2,237:1 (SEOmator), while the June Cloudflare Radar cut Nobori uses puts Anthropic on top (Nobori). Same network, different slices, different rankings.

This is the denominator problem we described in why AI citation studies disagree, wearing different clothes. A ratio quoted without its window is a rhetorical device, not a measurement.

#What the ratio is genuinely good for

It is a solid cost signal. Origin bandwidth, cache-miss rate, and crawl load on expensive dynamic routes are real line items, and if one operator is pulling your entire catalog nightly, this metric is a reasonable way to notice. Rate limiting in response to measured load is proportionate and reversible.

It is a poor visibility signal, and not only because of measurement error. An AI answer can build awareness without ever sending a click (Semrush). For an assistant that answers in place, near-zero referrals is the expected steady state rather than a failure. A metric that scores an uncited crawl and a well-cited answer that produced no click as the same loss cannot separate an invisible brand from a dominant one.

#Measure the three questions separately

There are three distinct questions hiding inside "should I block this crawler," and one ratio cannot answer them.

Cost is a server question. Take crawl volume per operator from your own access logs, not from a network-wide average, and act on load rather than on principle.

Visibility is a citation question. Run your prompt set against the assistants directly and measure citation share, and where you can, how much of the answer your page actually shaped. Your referral log is downstream of that, and lossy.

Access is a policy question. Purpose-scoped permissions let training and retrieval get different answers, which is the only way to keep the discoverability half of the mixed-use tradeoff while declining the other half.

The crawl-to-refer ratio answers how much a bot is taking. It does not answer whether that bot is worth allowing. Only the second question has a business answer, and it is not in the ratio.