The State of llms.txt — May 2026

We fetched /llms.txt and /llms-full.txt from each of the 10,000 hostnames in the Tranco top-10K list (the academic alternative to the deprecated Alexa list). For each hit we recorded: HTTP status, content-type, byte size, number of H2 sections, number of link entries, whether the H1 + blockquote summary were present, and whether a companion /llms-full.txt returned 200. We deliberately did not crawl past the index file — the point of llms.txt is that the index is the thing.

Three categories stand out:

The pattern is consistent with every prior emerging web standard (RSS, sitemap.xml, robots.txt): developer-adjacent sites adopt first, mainstream catches up over the following 18–36 months once their CMS plugins ship default support.

The median file is 1.8 KB with 14 link entries. The mean is 4.7 KB — pulled up by a long tail of large docs hubs (Anthropic, Vercel, Supabase, Mintlify) that ship 40–80 KB files. The smallest valid file we found is 142 bytes (an H1, a 1-line blockquote summary, and a 3-entry product section). 87% of files use exactly the structure the spec recommends. 9% bolt on non-standard H2 sections like ## Brand assets or ## API. 4% are technically malformed — usually a missing blockquote summary or H2 entries without a list underneath.

Only 11% of sites with an /llms.txt also publish /llms-full.txt — the optional "full content" companion. Of those, the median size is 412 KB and the largest is 38 MB (a vendor doc hub). This is the file AI engines actually retrieve when grounding answers; sites shipping it report a 2–4× citation rate uplift in our customer interviews, but the cost is non-trivial (it has to be regenerated when content changes).

We can confirm three engines fetch /llms.txt regularly: OAI-SearchBot (OpenAI), PerplexityBot, and ClaudeBot (Anthropic). Google-Extended and Applebot-Extended fetch it occasionally but do not appear to use it as a primary index — they continue to rely on sitemap.xml. The signal is unambiguous from server log analysis across our customer base: enabling /llms.txt produces a measurable lift in citations from ChatGPT, Perplexity, and Claude within ~14 days, but no measurable lift in Google AI Overviews over the same window.

If you are a developer-tooling or AI-adjacent SaaS, you are behind your peers if you do not yet ship /llms.txt. If you are a mainstream SaaS, you are early — and being early matters here, because AI engines cache and prefer the first canonical index they encounter for a brand. The cost of shipping a v1 is low: a curated list of your 10–30 most important URLs with one-sentence summaries. Ship it, monitor citations for 30 days, then decide whether /llms-full.txt is worth the maintenance cost.

We built a free llms.txt generator that takes a sitemap and outputs a draft file, and a free validator that checks the result against the spec.