Why your llms.txt is probably broken (and how to tell)

llms.txt is the single highest-leverage file you can ship for AI visibility right now. It's also the file that most sites ship and then forget about, which is why the typical llms.txt we audit ranges from "barely useful" to "actively misleading."

Across the audits we run, most sites have at least one issue in their llms.txt that meaningfully reduced its value to the AI engines that consume it. Here are the seven we see most, in order of how often we see them.

#1. Broken links in the body

Almost every audit catches this. The llms.txt lists 20 URLs as canonical resources. Three of them 404. Two redirect through a chain to an old marketing domain. One points to a draft preview URL the PM accidentally pasted in.

AI engines treat broken links in llms.txt as a strong negative quality signal. The fix is to put the file behind a CI check that asserts every link returns 200, the same way you'd assert your sitemap.

#2. Missing one-line summary at the top

The llms.txt spec calls for a short summary as the first paragraph after the H1. Most files we audit either skip it entirely or use a boilerplate corporate line that adds zero information.

A good summary tells the engine what your site does in one sentence. "Crawlmind audits websites for AI visibility, grading them on llms.txt presence, schema markup, robots-txt posture, and direct citation tracking" is the right shape. "We're a leading provider of solutions for the modern enterprise" is not.

#3. No section structure

llms.txt uses Markdown sections with ## headers. Each section groups related URLs. Files with no sections (just a flat list of 50 URLs) lose the contextual grouping that makes the file useful.

The minimum structure: one ## Core section with your most important pages, one ## Optional section with supplementary content. That's it. Two headings is enough.

#4. Off-host links polluting the canonical list

The spec is clear: links in the main body should be on the same host as the llms.txt file. Off-host links (your CDN, your help center on a subdomain, your GitHub repo) belong in an ## Optional section if anywhere.

We see this constantly. Site example.com/llms.txt lists pages on docs.example.com, blog.example.com, and github.com/example in the main body. The engines either ignore the off-host links or penalize the file for spec violation.

#5. Stale URLs from a previous site structure

Site relaunch in 2024. New IA, new URLs. The llms.txt was updated once during the relaunch and never touched again. Six months later, half the URLs in it redirect to other URLs because subsequent content migrations weren't reflected.

The fix is to treat llms.txt as a generated artifact, not a hand- maintained file. Build it from the same source of truth that generates your sitemap. Most static-site generators have a plugin.

#6. Duplicate or near-duplicate entries

The same page listed twice with slightly different paths (/pricing and /pricing/). Or the same page with and without a trailing slash. Or the same page on the canonical domain and the www subdomain.

Each duplicate is a small quality hit. Normalize before writing the file.

#7. The file is served from the wrong location

Specifically: it lives at /.well-known/llms.txt or /docs/llms.txt or some other path. The spec is unambiguous: llms.txt lives at the root, exactly like robots.txt. Anywhere else and most consumers won't find it.

#How to audit your own

Three things to check today:

1. Does https://yourdomain.com/llms.txt return 200 with Content-Type starting with text/?
2. Does every link in it currently return 200?
3. Is there a clear section structure with a one-line summary?

If your llms.txt passes all three, you're already ahead of most of the sites we audit. If it doesn't, the fixes are cheap and the upside is real. We've seen sites pick up meaningfully more weekly Perplexity citations in the weeks after fixing their llms.txt. The correlation isn't proven causation, but the cost of fixing the file is also basically zero, so the expected value is straightforward.