Your crawler allowlist trusts a string

An AI crawler allowlist is a rule that grants access based on a user-agent string, and a user-agent string is a claim the client makes about itself, not a fact your server checked. Anyone can type GPTBot. The header costs nothing to forge, and the whole point of an allowlist is that the name inside it buys lighter treatment.

That gap has stopped being theoretical. Once enough sites started writing crawler names into robots.txt, WAF rules and rate-limit exemptions, the names became worth stealing.

#What the forged traffic looks like

HUMAN Security sampled two weeks of traffic that identified itself as a well known AI crawler and checked each request against the vendor's own published network information. About one request in eighteen was fake, which works out to 5.7% of all traffic labelled as AI crawler traffic and more than two million spoofed requests a day on average (HUMAN Security).

The distribution is uneven. Meta-ExternalAgent was the most impersonated name in that sample at 16.4 million spoofed requests, ahead of ChatGPT-User at 7.9 million, while PerplexityBot had the highest impersonation rate of the set, with close to 2.4% of requests claiming that name turning out to be fraudulent (HUMAN Security).

Some of that is ordinary scraping wearing a costume. Some of it is not. GreyNoise tracked a scanning cluster between July 28 and August 23, 2026 that forged six crawler names belonging to OpenAI, Anthropic, Google and Perplexity, plus two Amazon ones, across 824 addresses on 795 distinct /24 networks, and used the traffic to request .env files, cloud access keys, private keys and password stores (Help Net Security). None of the 824 addresses matched any of those vendors' published IP ranges.

One detail from that report is worth keeping. The forged traffic never asked for robots.txt, while genuine Anthropic crawler traffic in the same window had robots.txt as its single most requested path, at 12% of its total (Help Net Security). Real crawlers read the rules. Impostors skip straight to the payload.

#Two ways to actually check

There are exactly two methods in production today, and they are at different stages of maturity.

The first is published IP ranges. Most of the major operators now list the networks their crawlers run from as machine-readable JSON, so you can verify a request by source address instead of by header. OpenAI publishes a separate file per bot, including gptbot.json for GPTBot and equivalents for OAI-SearchBot, OAI-AdsBot and ChatGPT-User, and its own bot documentation tells operators to allow requests from those ranges. This works, it is available now, and it is the reason GreyNoise could say with confidence that all 824 scanning addresses were impostors.

It also has a real ceiling. The lists change, so you inherit a refresh job. They only cover crawlers that publish them. And they fit a world where a bot is one company running one fleet, which is not the shape of a user-directed agent browsing on someone's behalf from a residential connection.

The second method is cryptographic. Web Bot Auth applies RFC 9421 HTTP message signatures to bot traffic: the client signs each request with an Ed25519 key, names its key directory in a Signature-Agent header alongside Signature and Signature-Input, and publishes the matching public key set at /.well-known/http-message-signatures-directory on that domain (Cloudflare). Your edge fetches the key, verifies the signature, and checks the created and expires parameters so a captured signature cannot be replayed. Identity stops being a claim and becomes arithmetic.

This is no longer a single vendor's idea. The IETF chartered a WebBotAuth working group to standardise it, co-chaired by David Schinazi and Rifaat Shekh-Yusef, with its main specifications targeted for April 2026 and a best-current-practice document on key and lifecycle management due that August. Cloudflare folded message signatures into its Verified Bots programme, and its signed agents launch in August 2025 covered OpenAI's ChatGPT agent, Block's Goose, Browserbase and Anchor Browser. AWS WAF added Web Bot Auth support on November 21, 2025 and now allows verified signed bots by default for CloudFront distributions (AWS).

Coverage is still partial on the crawler side. OpenAI's bot docs describe IP ranges and say nothing about signatures, so for the classic content crawlers, address-based verification remains the practical check. Signed requests are arriving first from the agent end of the spectrum, where a fixed IP list was never going to work anyway.

#This is a measurement problem too

Security teams see spoofing as an access-control issue. For anyone reporting on AI visibility, it is also a data quality issue, and the error runs in the direction people least want to notice.

If your AI crawler report is built by grepping user agents out of access logs, every forged request lands in it as a real one. Your GPTBot volume goes up. Your coverage looks healthier. The spoof rate quoted above is a network-wide average across many sites, so a domain being actively probed can sit well above it, and the inflation lands on exactly the crawler names an executive summary highlights.

The fix is the same instrument either way. Verify the source before you count the hit, then read the verified traffic by status code rather than by request volume, because a forged request and a 403 have the same value to you: none. It is also worth separating the question of which bots you allow from the question of which bots you can identify. A bot policy needs distinct answers for training, for retrieval and for user-directed fetches, and none of those answers mean anything if the identity underneath them is unverified.

#What to do now

Pull the published IP lists for the crawlers you care about, verify your logged AI traffic against them, and find out what share of your own AI bot volume is real. That number is yours, not an industry average, and it is the only version that should go in a report.

Then make sure your edge is not stripping Signature, Signature-Input or Signature-Agent headers before they reach anything that could check them, and confirm /.well-known/ paths are served rather than caught by a catch-all rule. If you run behind a CDN that already supports Web Bot Auth, the verification is largely a configuration decision rather than an engineering project.

The underlying shift is small and worth stating plainly. For thirty years, bots identified themselves and sites decided how much to believe them. That arrangement is being replaced by one where the bot proves who it is. Until the transition finishes, treat every crawler name in your logs as an assertion, and check it before you act on it or report it.