Your AI bot policy now needs three answers

An AI bot policy is the set of rules deciding which automated clients may fetch your pages, and since July 1, 2026 it has stopped being one decision. Cloudflare now splits AI traffic into three categories, Search, Agent, and Training, each with its own allow or block setting. Teams still asking "should we block the AI bots" are giving one answer to a question that now has three parts, and the three parts have genuinely different consequences.

#The three categories describe different relationships

Cloudflare's definitions are short and worth reading literally. Search is "any behavior that collects or indexes your content, so it can answer questions about it later". Agent is "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now". Training is "a crawler taking your content to train or fine-tune a model". Those are quoted from the July 1 announcement.

Read them side by side and the difference is not technical, it is who benefits and when. A search crawler is building an index you want to be in. A training crawler is taking content into a model whose output may never point back at you. An agent is a person, right now, waiting on an answer that involves your page.

This taxonomy is not something Cloudflare invented in isolation. OpenAI already publishes three purpose-separated tokens in its bot documentation: OAI-SearchBot, "used to surface websites in search results in ChatGPT's search features"; ChatGPT-User, "used for certain user actions in ChatGPT and Custom GPTs"; and GPTBot, which crawls to make its foundation models "more useful and safe". Search, agent, training. The same three buckets, declared by the operator at the user-agent level.

#The default changes on September 15

The part worth putting in your calendar is the default. Cloudflare's post states that on September 15, 2026 it will set new defaults for each of the three classifications: for all new domains onboarding to Cloudflare, Training and Agent will be blocked by default on pages that display ads, while Search remains allowed by default.

Two qualifications matter before anyone panics. It applies to new domains onboarding, not a retroactive flip of every existing zone. And the block is scoped to ad-monetized pages, which is a coherent position for publishers whose revenue model an agent visit bypasses. Given that Cloudflare says more than 20% of web domains sit behind it, even a new-domains-only default shapes a meaningful slice of the crawlable web over time.

The risk for a B2B SaaS site is inheriting a publisher's default without a publisher's business model. If you launch a marketing site on a new zone in October and never look at the setting, you may ship with agent access off. Nothing breaks, no error appears in your logs, and the failure mode is silence.

#Agent traffic is small, and that is the point

Volume is a bad proxy for value here. Cloudflare's breakdown of AI crawler traffic by purpose found that training accounted for nearly 80% of AI bot crawling, while the User action and Undeclared purposes together came in under 5% of AI bot traffic. That analysis covered a July and August 2025 window and used Cloudflare's earlier category names, so treat the exact figures as a snapshot rather than a current reading.

The shape of the finding is what carries. Training crawls arrive in bulk, on their own schedule, with no user attached. Agent fetches are rare, and each one is triggered by a specific person who asked a specific question. Blocking training costs you a share of a corpus. Blocking agents costs you the visit, at the exact moment someone tried to reach you.

That asymmetry is why the two switches should not move together, even though the September default moves them together for ad-monetized pages. Most SaaS marketing sites, docs sites, and support centers have no ad revenue to protect. For them, Agent is the category that most resembles ordinary traffic, and the one where a block does the most damage per request.

#Purpose claims only work if identity does

A category system based on declared purpose is only as trustworthy as the identity behind the declaration. A user-agent string is a claim anyone can copy, and IP allowlists do not scale to operators you have never heard of.

That gap is what Web Bot Auth addresses. Under Cloudflare's implementation, an operator generates an Ed25519 key, publishes the public key as a JWKS at /.well-known/http-message-signatures-directory, and signs each request with Signature-Input, Signature, and Signature-Agent headers. The signature carries a short expires value, with the documentation suggesting a minute is often sufficient, which limits replay. The result is a cryptographic answer to "who is this", not a guess.

This is heading toward a real standard rather than a vendor feature. The IETF has an active Web Bot Authentication working group with an approved charter, scoped to authenticating automated clients and conveying operator information to sites. Its stated scope covers search crawlers, archivers, link checkers, AI training crawlers, and AI agents, and explicitly excludes bot reputation tracking. Standardizing identity is in scope; deciding whether a given bot deserves access stays with you.

Cloudflare has already folded the two concepts together. Its documentation notes that as of July 1, 2026, signed agents are classified as Verified bots, with the distinction now expressed as a Direct versus Intermediary metadata field tracking who operates the bot. The earlier signed agents announcement listed participants including OpenAI's ChatGPT agent, Block's Goose, Browserbase, and Anchor Browser, all cases where an agent runs on behalf of an end user rather than a company.

#What to check this month

Four things, in order of how quickly they bite.

Check which zone settings you actually have. If your marketing site sits behind Cloudflare, look at the AI traffic settings per zone and record the current Search, Agent, and Training values. Do this before September 15 so you know whether a later change was yours.

Decide the three answers separately, in writing. Search is nearly always allow. Training is a genuine business decision about licensing and control. Agent deserves an explicit allow unless you sell ad impressions, and even then the block should be scoped to the ad-monetized pages rather than your docs and pricing.

Separate your logs by purpose, not by "AI". Group requests by the declared token so OAI-SearchBot, ChatGPT-User, and GPTBot land in different columns. A single "AI bot hits" line hides the only movement that predicts lost visits.

Re-test access from outside your network. Our crawler checks robots.txt and live fetchability per URL for exactly this reason: policy stated in a config panel and policy observed at the edge drift apart, and the gap only shows up when you request the page the way a bot does.

The one-switch era is over. The teams that come out ahead are not the ones who block the most or the least, but the ones who can say which of the three they chose, and why.