What Gets Lost Converting Documents to Text for an LLM

A PDF is not text. Neither is a Word file or a spreadsheet. Something has to walk the container and decide what the words were, and that step is where a document quietly stops meaning what it meant.

The usual way to measure this is to score a parser against ground truth on a pile of real pages and report an edit distance. That tells you one tool is a few points better than another. It does not tell you which structure broke, and it does not tell you whether the loss was the format's fault or the library's. Those are different problems with different fixes, and only one of them is worth swapping a library over.

So we built a corpus where every file isolates one structure, ran it through the reader we ship, and ran the PDFs through a second reader that shares no code with ours. Three results change what you do with a document before you paste it.

The question this page answers: when a document becomes LLM context, which structures survive, and where one does not, is the loss forced by the format or chosen by the reader?

The second half is what makes the first half actionable. A structure that every reader loses is a property of the file format, and no amount of library shopping will get it back. A structure that another reader recovers on the same bytes is a decision, and decisions can be changed.

Document extraction is well benchmarked and the benchmarks answer a different question.

Two things are missing from all of it.

Nothing isolates a structure. These studies score whole real pages, so a merged cell, a running header and a two-column layout all land in one edit-distance number. When that number moves you cannot say what moved it. Our corpus goes the other way: one file, one structure, one predicate.

The largest benchmark excludes the furniture. OmniDocBench states its ignore rule plainly: headers, footers, page numbers and page footnotes "participate in matching but are excluded from metric calculations", because tools handle them inconsistently. Captions are excluded for the same reason. That is a fair choice for ranking parsers against each other, and it means the benchmark is silent on a structure we measured doing real damage. A 40-page report that interleaves its running header into every page seam is not a small scoring artifact to the model reading it.

And nobody separates the format from the library. This is the finding that changes what you do next, and it needs two readers on identical bytes to say anything at all.

Twenty-seven documents, twelve formats, each file written to isolate one structure:

Every fixture is generated, written with the docx, exceljs, pptxgenjs and pdfkit libraries plus PIL for the scanned pages. No real corpus was available, and a real one could not be published anyway: the directory is gitignored precisely because a document dropped there is somebody's. What is published is the generator and this measurement, not the files.

That is the sample's main weakness and it is not a small one. A merged cell written by the docx library and a merged cell written by Word are not guaranteed to be the same XML. Findings that could plausibly differ on a file Word itself produced are marked in the results.

The corpus is deliberately not weighted toward what people extract most. It is weighted toward structures that are hard, which is a different thing and the right thing for a floor check.

Every file goes through the path the product actually uses. Nothing here reimplements extraction.

1. 1

   Route from the bytes, never the name

   routeBytes reads the leading bytes and decides what the file is. A renamed .docx is still extracted, an extensionless PDF is still read, and a plain zip is told apart from the OOXML container that shares its signature. The file extension is never consulted.

2. 2

   Extract through the shipped parser registry

   The router names a parser and the registry runs it. This is the same registry the CLI uses, so the text measured here is the text a bundle would carry.

3. 3

   Score each structure as a predicate

   Every defect from the earlier reading is written as a test over the extracted text. A rerun reports FIXED or STILL BROKEN. Nobody re-reads a paragraph and decides.

4. 4

   Read the PDFs again with an unrelated library

   The same bytes go through pypdf 6.14.2, which shares no code with our stack. Running the same predicate against both outputs is what turns a defect into an attribution.

Two details in that fourth step matter more than they look.

A predicate that hunts for a defect can pass for the wrong reason. If a check looks for the broken output and the corpus is regenerated with different wording, the check finds nothing and reports FIXED while nothing was fixed. Every such check therefore carries an anchor: text the fixture must produce for the check to mean anything. A missing anchor reports SIGNAL ABSENT and is never counted as a pass.

A predicate written for our output shape does not transfer. Our reader renders a recovered table with pipes. No other library does, so running that predicate against pypdf would score output formatting rather than whether data survived. That check is marked not compared, and the honest consequence is that the table finding below has no attribution.

Routing

Twenty-seven of 27 files routed correctly from their leading bytes. No file was misidentified.

Two produced no text, and both are correct behaviour rather than failure. The encrypted PDF is named as password protected, which is something a person can act on, instead of disappearing under a generic parse error. The fully scanned PDF has no text layer at all, so on this path it yields nothing.

Re-testing our own extraction

The earlier reading found 16 defects and checked one correctness property that has never broken. Re-run against the current build:

docx-3.2 carries the generated-file caveat: Word's merge XML differs slightly from what the docx library emits, so that one is worth re-checking against a file Word wrote.

docx-3.1 is listed apart from the rest on purpose. It is a standing guard, not a defect that got fixed, and folding it into the headline would inflate the count by one. A tracked deletion has never reached a bundle, and the check exists so that stays true.

Where the loss belongs

Four PDF findings have predicates that read only the text, so they can be run against both readers on identical bytes. This is the part no benchmark reports.

The third row is the practical one. On three pages of prose with a running header and a Page n of 3 footer, pypdf pours the header into the text three times and the footer three times. Ours emits the header once and no footers, and marks the page boundaries instead. On a 40-page report that is the difference between 80 lines of furniture spliced through your sentences and none.

The experiment that separates a column model from luck

Reading order on a two-column page is the classic extraction failure, and it is usually reported as a single pass or fail. That is not enough to tell a reader that understands columns from one that got lucky, because a PDF's content stream often happens to be written column by column already.

So the corpus contains the same visual layout twice. One file writes the content stream column by column. The other writes it line by line across the gutter, which is what a reader following the stream would produce if it had no column model.

Ours finishes the left column through line 18 before the right column starts. pypdf interleaves every line. Both readers handle the naturally ordered file, so a test using only that file would have scored them equal and told you nothing.

This is the whole argument for structure-isolating fixtures in one place. The manipulation is free, it takes one extra file, and it converts a pass into evidence.

When the comparison refuses to answer

The footnote-marker row reports SIGNAL ABSENT rather than a verdict, and the reason is itself a result. On the typography fixture, pypdf's character mapping breaks against the embedded font: its own output reads Claim needin' support, with letters substituted for punctuation throughout. Our reader recovers that page's ligatures, curly quotes, en dash and accented characters intact.

A comparison against a reference whose own text is corrupted proves nothing, so the harness says so instead of scoring a false pass. Most extraction comparisons do not make that distinction, and it is the difference between "we fixed it" and "we cannot see it from here".

The three that are still broken

Spreadsheet dates arrive as serial numbers. A schedule whose due dates read 46037 and 46203 is not wrong-looking enough for a model to reject and not right enough to use. This is the oldest open defect here.

A superscript footnote marker sorts above the line it annotates. Lines are ordered by vertical position and a superscript sits a few points higher, so the marker lands on its own line before the claim. Minor next to the others, and the same root cause the column work addressed.

A page with no text layer disappears without a note. A document whose page one carries real text and whose page two is a raster image extracts page one, marks the page boundary, and says nothing about the rest. The document is not empty, so it never reaches the "no extractable text" path.

That last one is worth being precise about, because it is the only finding here where our own layers disagree. The silence is in the shared engine: a note is raised for a page that failed to load, never for one that loaded carrying no text. The browser's OCR can recover the missing page, and only if the reader accepts it. We are not promising a fix date; emitting the note needs page-level text accounting the underlying library does not currently expose.

Check the furniture before you trust a long document. The losses that hurt most are not exotic. They are running headers spliced into prose, a footnote detached from its claim, a hyperlink reduced to its anchor text. None of them look like errors in the output. All of them change what the document says.

A silent loss is worse than a loud one. Every defect above that we ranked BROKEN shares one property: the output stays plausible. APAC 980 980 reads as a complete row. A date reading 46037 reads as a number someone meant. A part-scanned report reads as a whole report. A model has no way to know.

If a structure matters to your question, verify it survived. None of these checks needs a tool, and each one is faster than reading the document again.

Do not assume a library swap fixes reading order. Both readers here handle a naturally ordered two-column page and only one handles an adversarial one. The published rankings are averages over real pages, and the page you care about is not the average.

• Every fixture is generated. Real documents are written by software with its own habits, and Word's merge XML is not the docx library's. docx-3.2 is flagged for exactly this reason. Treat every result here as a floor check on hard structures, not as a survey of documents in the wild.
• One file per structure. A structure that passes here passes on one instance of it. This design buys attribution and pays for it in breadth.
• The independent reader covers PDF only. pypdf reads PDFs. The docx, xlsx, pptx and rtf findings therefore have no attribution column at all, and could be format limits or our own choices. We do not know which, and say so rather than guessing.
• One comparison is unattributable even within PDF. The table finding's predicate reads our own pipe rendering, which no other library emits.
• This is the Node path. The browser adds an OCR layer that recovers scanned pages when the reader accepts it. Scan results here are floors.
• The severity labels are ours. BROKEN means the bundle says something false or loses data invisibly, DEGRADED means it is poorer than it should be but not misleading. That is a judgment, it is published so it can be argued with, and a different reasonable line would move rows between them.
• This measures what survives, not what helps. Whether a model answers better from a document whose footnotes are attached is a separate experiment that needs a model in the loop. It is not this one.

# The corpus, written by real writer libraries. Gitignored output, tracked generator.
cd packages/core/tests/fixtures/real && node generate.mjs

# The independent reader.
pip install pypdf
python3 packages/cli/scripts/extract-pdf-with-pypdf.py

# Every structure, every predicate, both readers.
pnpm --filter @fileconcat/cli measure-extraction

The last command prints both tables on this page: the per-file extraction and the findings with their attribution column. Run it without the pypdf step and the comparison column reports "not run" rather than assuming anything. How the same extraction behaves in day-to-day use, including what a parse failure does to a run, is in the CLI usage docs.

One number here cannot be reproduced from the current build, and it is named where it appears: the defect states as they stood in August describe builds that no longer exist. What the command reproduces is the right-hand column, which is the claim this page actually makes.

What gets lost when you convert a PDF to text?

Structure, mostly. Text in a PDF is positioned glyphs with no notion of a column, a cell or a heading, so a reader has to infer all of it. In our measurement the recoverable-but-commonly-lost structures were column reading order, table cell positions, running headers and footers, and footnote attachment. The words themselves usually survive; what they belonged to often does not.

Is a document extraction failure the library's fault or the format's?

Both happen, and you cannot tell without a second reader. On identical bytes, our reader and pypdf both read a naturally ordered two-column page correctly, but only ours read a page whose content stream was written across the gutter. That is a library difference. A table's cell boundaries, by contrast, do not exist anywhere in a PDF, so no reader can recover them reliably.

Should I just upload the PDF to the model instead?

Sometimes. Anthropic's documentation says each page is converted to an image and its text extracted, and both are sent, which is why charts and diagrams can be asked about. It costs more: their Bedrock documentation puts text-only extraction at roughly 1,000 tokens for a three-page PDF against roughly 7,000 for the full visual mode. Encrypted PDFs are rejected outright. Extracting to text first is cheaper and lets you see exactly what the model will see, which is the point of checking the furniture.

Why do my spreadsheet dates come out as numbers?

Because a date in a spreadsheet is stored as a number of days and its date-ness lives in the cell's format, not its value. Our reader currently passes the raw serial through, which is the one BROKEN defect in this measurement that has been open longest.

Does a scanned PDF work?

In the browser, yes: pages with no text layer are rendered and read with OCR that runs on your machine, offered rather than automatic. In the CLI, no. And a document that is only partly scanned currently loses its image-only pages with no note in either, which is one of the three open defects above.

How do I check whether my document survived extraction?

Open the extracted text and search for the specific thing you plan to ask about: the figure, the footnote, the link target. Every defect ranked BROKEN here produces output that looks plausible, so plausibility is not evidence.

• Adhikari, N. S. and Agarwal, S. A Comparative Study of PDF Parsing Tools Across Diverse Document Categories. arXiv:2410.09871, 2024. Ten parsers over the DocLayNet dataset across six document categories, scored with F1, BLEU-4 and local alignment.
• Ouyang, L. et al. OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations. CVPR 2025, arXiv:2412.07626. 981 pages over nine page types; normalized edit distance for text and reading order, TEDS for tables; headers, footers, page numbers and footnotes excluded from scoring.
• Anthropic. PDF support. Claude Platform documentation, read 2026-09-07. Page-to-image plus text extraction, 32 MB request ceiling, 600 page maximum, encrypted files unsupported.
• pypdf, version 6.14.2, used as the independent reader.
• FileConcat, How Many Tokens Is a Codebase? The companion measurement, on what reaches the model rather than what survives conversion.