The AI-Readiness Report: the top 1,000 sites, three signals

We took the Tranco top 1,000 hostnames and, for each, fetched /robots.txt, /llms.txt, and the homepage exactly as a non-JavaScript AI crawler would: raw HTML, no rendering. 561 served a reachable homepage (the rest are CDN, DNS, and cloud endpoints that do not host a browsable site). For each reachable site we recorded three things: which major AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider) are blocked at the root; whether an llms.txt exists and is spec-shaped (a title, sections, and links); and whether the homepage emits JSON-LD structured data.

We call a site "fully AI-ready" when it clears all three bars: it does not block the AI crawlers, it publishes a valid llms.txt, and it emits JSON-LD an engine can read. Just 10.7% of the top 1,000 clear it. The signals rarely line up: 60% are open to AI crawlers but publish no llms.txt, leaving the engine to guess which pages matter, and a smaller 1.4% publish an llms.txt yet still block an AI crawler, inviting engines to a door they have locked.

Block rates split sharply by category:

The pattern is the story of the AI-content fight in one table: publishers and social platforms are pulling up the drawbridge; SaaS and developer tools are throwing it open. SaaS is the only category that is broadly AI-ready (40%), which tracks with the incentive: those companies want to be the answer an AI engine gives.

Block rates per crawler, across all 561 reachable sites:

The most-blocked bots are the broad training and scraper crawlers (CCBot, Bytespider), then the model crawlers (ClaudeBot, GPTBot). The dedicated search-citation bots are blocked least: OAI-SearchBot, the crawler that puts you in ChatGPT search results, is the least-blocked of all at 10.9%. Operators are more willing to be cited than to be trained on.

That "cited, not trained" split is not spread evenly: it is almost entirely a publisher move. Crossing category against crawler shows the high-blocking categories treat the bots very differently, while everyone else applies one flat policy:

News, social, and media all show the same shape: they block the scraper and training crawlers hard, then leave the citation crawler far more open. News blocks Bytespider and ClaudeBot at 67% but OAI-SearchBot at just 37%: a 30-point gap. Media blocks the scrapers at 40% but the citation bot at only 10%. The publisher position is coherent: do not train on my archive, but do cite me in search, because a citation sends a reader back. The other categories draw no such line. E-commerce, SaaS, developer, and reference sites set a single rule and point it at every crawler, so their rows are flat: whatever they decide about GPTBot, they decide about OAI-SearchBot too.

Only 14.3% publish a valid llms.txt (16.8% publish one at all; the rest are malformed or soft-404s). The gap between "open to AI" and "publishes an llms.txt" is the biggest missed opportunity in the data: 60% of sites let the crawlers in but never hand them a map. Publishing one takes a few minutes. Validate yours free or generate one from your sitemap.

Being cited by AI starts with being readable by it: let the crawlers in, tell them where the answers are (llms.txt), and mark up the page (JSON-LD). It is necessary, not sufficient, since authority still decides who actually gets cited. But with only 10.7% of the top of the web clearing that bar, most of your competitors have not, which makes it one of the cheapest edges available. Check your site free.