We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

← All posts

Your PDF arrives with none of its signals

Crawlmind Engineering··6 min read

A PDF is a page that AI engines can read but that arrives stripped of almost every signal an HTML page uses to earn a citation: no schema block, no meta description, no in-document robots directive, and no reliable heading structure. The text usually survives. The context around the text does not.

This matters because PDFs tend to hold the good stuff. Benchmark reports, product datasheets, pricing tables, technical manuals, original research. The pages a B2B company is proudest of are frequently the ones it published as a download, and those are exactly the pages it then wonders why no assistant quotes.

#What a PDF actually stores

The gap starts below the file format's reputation. A PDF's content stream is a list of drawing instructions. As the Overleaf writeup on tagged PDF internals puts it, the stream holds a sequence of operators that place text or graphics at a particular page location. A heading is not stored as a heading. It is stored as some glyphs, at some coordinates, in some font size. Nothing in the file says "this line is the title of the section below it."

Tagged PDF exists to fix that. It adds a structure layer that marks content as a heading, paragraph, list, or table, and it declares the intended reading order. It was introduced in 2001, in PDF 1.4, which is late in the format's life and long after most PDF-producing tools had settled their defaults.

An untagged PDF leaves an extractor guessing. The same source notes that content can sit in the stream in a different sequence than the one a reader is meant to follow, which is how a two-column layout gets read straight across, and how a table gets flattened row-wise when it was meant to be read down. Word boundaries can be ambiguous too, because some engines position text with spacing rather than space characters.

Run that output into a retrieval pipeline and the damage compounds. Retrieval works on chunks, and a chunk drawn from a mis-ordered extraction is a chunk that says something the document never said. That is not a citation problem yet. It is a comprehension problem that becomes a citation problem.

#The signals that have nowhere to live

Even a perfectly tagged PDF is missing the layer GEO actually runs on.

Structured data is the clearest case. Google's introduction to structured data describes JSON-LD as a script tag embedded in the head or body of an HTML page, with Microdata and RDFa both expressed as HTML tag attributes. All three formats assume an HTML document. A PDF has no head, no body, and no tag attributes. There is no place to declare that this document is a TechArticle, who the author is, what organization published it, or when it was last revised. Every entity signal you would normally hand an engine has to be inferred from the prose.

Robots control is the second case. Google's robots meta tag documentation is explicit that to block indexing of non-HTML resources such as PDF files, you use the X-Robots-Tag response header instead, because a robots meta tag is not possible in those files. That is a server configuration, not a page-level edit. Teams that manage indexing through a CMS field usually have no equivalent lever for the files sitting in their asset directory, which is why old PDFs outlive the pages that linked to them.

Canonicalization is the third. Google supports a rel="canonical" HTTP response header precisely so you can point a non-HTML file at its canonical URL, and the documentation frames it around publishing the same content in multiple formats. Almost nobody sets it. So the whitepaper PDF and the HTML landing page describing the whitepaper compete as two separate documents covering one topic, and the engine picks whichever it likes.

#Why this shows up as a visibility problem

Put the three together and a PDF is a document with content but no claims about itself. It cannot say who wrote it. It cannot say when it changed. It cannot say which URL is the real one. It cannot say what type of thing it is.

An answer engine assembling a response has to decide whether a candidate source is relevant, current, and attributable. A PDF makes the first question harder, because the extracted text may be scrambled. It makes the second nearly impossible, because there is no dateModified to read and the visible date, if there is one, is often just the cover art. It makes the third awkward, because the author may appear nowhere except a footer.

The result is rarely a hard exclusion. PDFs do get cited. The result is that a PDF loses close contests it should win, and the loss is invisible from the inside, because the document ranks fine as a download and the team never sees the answer it was passed over for.

There is a second-order cost as well. Links inside a PDF are inert as an internal linking strategy for most teams, because the PDF sits outside the site structure the rest of the pages were planned around. A downloadable report that cites five of your own pages does not distribute authority the way an article that cites the same five pages does.

#What to do instead

The fix is not to delete the PDF. It is to stop treating the PDF as the publication.

Publish the HTML first, and treat the PDF as a rendering of it. Give the content a real page with headings, a dateModified, an author block, and a schema type, then offer the PDF as an option for people who want to print or circulate it. The content is identical; only the primary address changes.

If the PDF must be the canonical form, for example because it is a signed or paginated document, then set the canonical HTTP header so the engine knows which URL to treat as authoritative, and publish an HTML summary at that URL with the document's key claims in text. The summary is what gets quoted. The PDF is what gets downloaded.

Tag the PDF if you produce it from a tool that can. Export from a source document with real heading styles rather than manually enlarged text, and check the reading order before you ship. This is the same work accessibility requires, which means it is often already on someone's list and can simply be pulled forward.

Set X-Robots-Tag deliberately on your PDF directory rather than by accident. Decide which files should be indexed and which are archive clutter, then encode that decision in the server config where it is visible to the next engineer.

Then check what your own PDFs actually extract to. Open one, select all, paste into a plain text editor, and read the result. If the columns interleave or the table collapses into nonsense, that is roughly what a retrieval pipeline sees, and no amount of writing quality downstream will repair it.

The underlying point is one this blog keeps arriving at from different directions. Content is not the same thing as a document an engine can use. A PDF is the clearest example, because the gap between how good it looks and how little it declares about itself is the widest of any format still in common use.

Related field notes

Share or discuss

Field notes in your inbox

New posts, no spam. Roughly monthly. Unsubscribe with one click.