Your AI crawler rules run on the honor system

Bot verification is the step where your server proves that a request claiming to be GPTBot or OAI-SearchBot actually came from OpenAI, and most AI-visibility workflows skip it entirely.

That gap sits underneath a lot of GEO work. Teams write robots.txt rules per crawler, build edge rules to enforce them, and then read crawl logs as evidence of AI attention. All three steps rest on a header that any client can set to any value.

#What the rule is actually matching

OpenAI documents three separate agents with three separate jobs. GPTBot crawls content for training foundation models, OAI-SearchBot surfaces sites in ChatGPT's search features, and ChatGPT-User handles user-initiated fetches. The split is the whole basis of the common posture: disallow GPTBot, allow OAI-SearchBot, stay eligible for citations without donating a training corpus.

The posture is sound. The enforcement is not. A robots.txt directive is a request addressed to a well-behaved client, and a user-agent string is self-reported text. Nothing in either mechanism is checked. The IETF architecture draft behind cryptographic bot authentication says this plainly, noting that user-agent strings can be spoofed and are overloaded, because an agent may need to present as a browser while still differentiating its traffic (draft-meunier-web-bot-auth-architecture).

#The failure runs in both directions

Impersonation inbound is the obvious half. Anything can send GPTBot in a header and inherit whatever allowance you granted OpenAI.

The less obvious half is that honest clients are the only ones your rules reliably reach. Cloudflare published a case study of exactly this asymmetry: after customers disallowed Perplexity in robots.txt and added firewall rules, Cloudflare found a generic browser user agent impersonating Chrome on macOS, rotating IPs and ASNs, across tens of thousands of domains and millions of requests per day. Cloudflare de-listed Perplexity from its verified bots program and added heuristics to block the undeclared traffic.

Whatever you conclude about that specific dispute, the mechanism generalizes. A per-crawler block is enforceable against operators who declare themselves and want to stay declared. It is a suggestion to everyone else.

#The fix that exists, and how far along it is

Web Bot Auth is the standards-track answer. It builds on RFC 9421, HTTP Message Signatures, a Proposed Standard published in February 2024. Instead of naming itself, a client signs its request with a private key and hands the server the means to check.

Cloudflare's May 2025 write-up lays out the request shape. A Signature-Input header carries the validity window, the key ID, and a tag marking the purpose as web-bot-auth. A Signature-Agent header points at where the signer's public keys live. The implementation notes in Cloudflare's bot verification docs are strict about the details: host the key directory at /.well-known/http-message-signatures-directory, serve it over HTTPS, quote the Signature-Agent value as a structured string, and include that header in the signed component list so it cannot be swapped in transit.

The same write-up explains why the older options lose. IP allowlists go stale and blur ownership on shared cloud ranges. Pre-shared secrets do not scale past a handful of named partnerships, because every site would need its own token and its own rotation story.

One caution on maturity. The architecture document is still an individual Internet-Draft rather than an adopted working group product. Treat signatures as a signal you can accept today, not a requirement you can impose.

#The part that matters for citations

Signing arrived on the agent side first. Cloudflare's signed agents launch named ChatGPT agent, Goose from Block, Browserbase, and Anchor Browser as the initial cohort. Every one of those is a browsing agent working through a task for a specific person.

The crawlers that determine whether you are eligible to be cited are not in that cohort. OpenAI's own crawler documentation still directs site operators to published IP address lists for verification, with no mention of message signatures. Google is in the same place: its guidance on confirming a crawler is genuinely Googlebot offers a forward-confirmed reverse DNS lookup against googlebot.com, google.com, or googleusercontent.com, or a CIDR match against published JSON IP ranges. Cryptographic signatures do not appear there either.

So the population you can now verify cheaply is the one browsing on behalf of one user, and the population that feeds retrieval and citation is still the one you have to verify the old way. That is backwards from what a GEO team would want, and it is the current state.

#What to do with this

Verify before you count. A log line becomes a dashboard row only after a forward-confirmed reverse DNS lookup or a CIDR match against the operator's published list. Both operators above document the method. An unverified user-agent count is a count of claims.

Keep crawler rows and agent rows apart. They are verified differently, they arrive for different reasons, and they mean different things. A signed agent fetch is one person mid-task. An OAI-SearchBot fetch is index maintenance. Averaging them produces a number that answers no question.

Treat robots.txt as policy, not as a control. It documents your intent and binds compliant operators, which is genuinely useful. If a directive carries commercial weight, back it with an edge rule that does not depend on the client's cooperation.

Accept signatures, do not yet demand them. Where a valid Signature-Agent and signature are present, that request needs no IP heuristics. Where they are absent, fall back rather than block, because absence currently means unadopted far more often than it means dishonest.

Do not read crawls as citations. Verification tightens the denominator, not the conclusion. A verified fetch proves an engine looked. Whether it quoted you is a separate measurement, taken from answers rather than from logs.

The honest summary is that AI-visibility reporting inherited an access-control model built for a smaller and more polite web. Signatures fix it eventually. Until the citation-relevant crawlers sign, verification is manual, and skipping it means your crawler dashboard is measuring what clients say about themselves.