mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-24 10:08:00 -06:00
Tanish MankuandClaude Opus 5
50cbf5fa19
`cargo xtask text-oracle` reported one number: word F1 against `pdftotext`. F1 is a bag of words. It catches a character read wrong, a glyph not read at all, a font we cannot map -- and it is blind to order, so a page emitted back to front scores a perfect 1.000. Reading order is most of what text extraction has left to get wrong: a column read into the middle of its neighbour (#2387), a figure's labels threaded through a paragraph (#2586), a form's value parted from its label. The hole is not hypothetical. Reviewing #2431, a change that moves text and alters nothing else, **word F1 called 26 of the 29 files whose output changed identical**, while scoring the word sequence separated 18 improvements from 6 regressions and named the shape of the regressions (forms: `prefilled_f1040.pdf` 0.672 -> 0.481, a checkbox's number parted from its label). The harness could not see the only thing that change does. So a second score per file: the longest common subsequence of the two word sequences, as a share of the longer one. It shows up in the first three corpus files -- `Test-plusminus.pdf` scores F1 1.000 and reading order 0.875, the same forty words in a title block that we read down the column and `pdftotext` reads across. Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages is plenty of signal; the run says how many files that bit, so the cap cannot quietly become the measurement. It reports; it does not gate, and like the pixel score it is a *disagreement* with Poppler rather than a verdict -- in the #2431 measurement the same split scored as a win on one file and a loss on another purely because `pdftotext` makes it in one place and not the other. The function's doc comment, `crates/oracle/README.md` and `docs/gaps.md` gap 12 all say so. `words` now takes the sequence rather than re-splitting the text, so both scores read the same tokens. `docs/file-format-parity.md` claimed plain-text export was good "(word-F1 0.98)" in a row whose own column is reading order; it now says the two are scored separately. Verified: `cargo xtask text-oracle --limit 60` end to end; `cargo test -p xtask` 49 passed; fmt and clippy clean. The unit test was watched failing with the score blinded to order (sorting both sides made it 1.000 where it must be below 0.7). The corpus-wide first reading is not in this commit: the machine is shared with a parallel build and the full sweep was stopped twice for memory. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>