Read your AI crawler log by status code

An AI crawler log audit is the practice of grouping AI bot requests by the HTTP status code you returned, not by request count, because a request answered with 403 or 404 consumed the same slice of a crawler's attention as a successful one and gave back nothing.

Most teams read their bot traffic the other way around. The dashboard says GPTBot made twelve thousand requests last week, the number is going up, and that gets filed as good news. It is compatible with total invisibility. Volume tells you a crawler is interested in your domain. Only the status split tells you whether it left with anything.

#The network baseline

There is now a public number to measure yourself against. An analysis of Cloudflare's Radar crawler dataset over the 28 days ending July 19, 2026 found that verified AI bots received a 200 on 73.0% of their requests, a 403 on 13.5%, a 404 on 4.1% and a 429 on 1.0% (SEOmator). Roughly one request in four never reached a page.

Across all crawler traffic, not just the AI subset, the picture is worse: 45.9% returned 200, 20.6% returned 403, 8.0% returned 301, 7.8% returned 404 and 6.3% returned 429, which rolls up to 2xx 47.8%, 4xx 35.7%, 3xx 14.3% and 5xx 2.2% (SEOmator). Non-AI bots, the SEO tools and monitors and scrapers, got a 200 only 33.3% of the time against 29.2% 403s, which is what deliberate blocking looks like in aggregate.

Two caveats before you use those figures as a target. Radar classifies verified crawlers only, so unverified scrapers wearing browser user agents sit in a different bucket. And the numbers are network-wide, mixing publishers who block AI bots on purpose with sites that never meant to. Your own site is the comparison that matters. Cloudflare exposes a response status widget for AI bots on its Radar AI Insights page, groupable by individual code or by 2xx/3xx/4xx/5xx category, if you want the current shape of the distribution (Cloudflare).

#A 403 is almost always your decision, not theirs

The 403 share is the line worth staring at, because it is the one most often produced by something nobody on the content team knows is running.

Being a verified bot does not buy an exemption from your security stack. Cloudflare's own documentation is explicit that verified bots are excluded from default bot configurations, and that every customer still configures their own AI bot policy on top (Cloudflare). A custom WAF rule, a rate-limiting rule, a country block, a managed challenge, or an "AI bots" toggle somebody flipped during a scraping scare will all return a 403 to a crawler your robots.txt explicitly welcomes.

The ordering is what makes this invisible. robots.txt is a directive the crawler reads after your edge has already decided whether to answer it. If the edge returns 403, the crawler never sees your Allow line, your sitemap reference, or your content. Your robots.txt tester will keep reporting that everything is fine, because it is testing a file that is no longer being consulted.

This is ordinary, not exotic. While researching this post we tried to fetch Cloudflare's public Radar AI Insights page and got back a 403 for our trouble. Edge rules do not check whether the requester was someone you wanted.

#404s and redirects are an inventory problem

The 4.1% 404 rate for verified AI bots (SEOmator) has a different cause mix than the one you are used to from search crawlers. Assistants follow URLs recovered from training data, from old citations, and sometimes from pattern-matched guesses, which means a meaningful share of AI bot 404s are for URLs that were never valid or have not been valid for years. We wrote separately about URLs Google requests that you never created and about citations that break during a site move.

Redirects deserve their own line. Combined 301s and 302s accounted for 12.6% of crawler requests in the same dataset (SEOmator). A redirect is not a failure, but every hop is an extra round trip against a fetch budget, and chains of them are pure overhead. Collapse multi-hop chains to a single 301 and point internal links at final URLs.

The 429 rate for verified AI bots was 1.0%, against 8.8% for non-AI bots (SEOmator). Small in aggregate, but rate limits are usually configured per source, so a large site can be quietly capping one specific crawler well below what it would otherwise fetch.

#How to run the audit

Start by verifying identity, because a user agent string is free to type. OpenAI publishes separate IP lists for each of its agents: GPTBot for training crawls, OAI-SearchBot for ChatGPT search, and ChatGPT-User for user-initiated fetches, at gptbot.json, searchbot.json and chatgpt-user.json. Anthropic publishes a single allowlist covering ClaudeBot, Claude-User and Claude-SearchBot at claude.com/crawling/bots.json. Filter your logs to requests whose source IP appears in those lists, then discard the rest for this exercise.

From there, four passes:

1. Group by status code per bot. Anything materially under the 73.0% network success rate (SEOmator) is worth explaining, and the explanation is usually in your own configuration.
2. Work the 403s first. Check them at the edge, not in robots.txt: WAF custom rules, bot management settings, rate limits, geo rules, and any managed ruleset that fires on non-browser user agents. Add an explicit allow rule for the verified bots you want, ordered above the blocking rules.
3. Then the 3xx. Find chains longer than one hop and the internal links feeding them.
4. Then the 404s. Sort by request volume. A URL an assistant keeps asking for is a citation you already earned and are currently answering with nothing, which is the cheapest fix on this list.

Recheck after every CDN or WAF change. Edge configuration drifts, and nothing in your content workflow will tell you when it does.

The number to keep on the dashboard is the weekly 200 rate per verified AI bot. Request volume answers whether crawlers are interested. The 200 rate answers whether that interest produced anything you can be cited for.