mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-31 21:48:09 -06:00
fix(oracle): say what the render-oracle score does and does not mean
Investigating the worst file of the first 250 turned the headline finding around. bug1552113.pdf scores 84%, and the cause is Poppler: it paints the link's `/Border [0 0 112]` as a 112-unit band across a 250x50 page, hiding the two lines of text. The fixture is named "this text should be visible", and in our rendering it is. The oracle is not ground truth, and the top of its worst list needs a person to look at both images before it is read as a bug of ours. That led to a plausible change that the measurement rejected. Poppler synthesises appearances for annotations that have none, where we draw only what the document carries -- `/AP`-less link borders are already recorded as not covered under view.render-annotation-appearances -- so hiding annotations looked like the way to measure page content alone. It is worse: it then scores Poppler for every appearance we do draw. Over the same 250 files, shown gives mean 2.16%, p90 5.35% and 25 files within 0.1%; hidden gives 2.18%, 5.55% and 22, with annotation-tx3.pdf jumping to 45.64% and annotation-tx2.pdf to 18.79%. The two biases nearly cancel. So the default is unchanged and `--hide-annotations` is an opt-in for explaining a single file, with the numbers written down so the experiment is not repeated. Verified that the default reproduces the merged baseline exactly and that the flag moves it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
f988cf87ed
commit
5e08aa8719
@@ -221,6 +221,18 @@ update the numbers.
|
||||
CI, and a first reading of a borrowed renderer is a baseline to improve on, not a bar to clear.
|
||||
The worst results cluster on JBIG2 `bitmap-*` files (`hayro-jbig2` carries 3 local patches) and
|
||||
on ZapfDingbats, which points the renderer work at something specific for the first time.
|
||||
- **Read the score as a disagreement, not a verdict (2026-10-14).** The worst file of the 250,
|
||||
`bug1552113.pdf` at 84%, turned out to be Poppler's defect: it paints the link's `/Border [0 0
|
||||
112]` as a 112-unit band over the two lines of text, and the fixture is named "this text should
|
||||
be visible". Ours is the readable rendering. Anything at the top of the list needs a person to
|
||||
look at both images before it is treated as a bug of ours.
|
||||
- **Annotations are shown, and that was measured rather than argued.** Poppler synthesises an
|
||||
appearance for an annotation that has none, where we draw only what the document carries (link
|
||||
borders without `/AP` are recorded as not covered under `view.render-annotation-appearances`), so
|
||||
showing them scores us for a known gap. Hiding them is worse, because it then scores Poppler for
|
||||
every appearance we *do* draw: over the same 250 files, shown gives mean 2.16% / p90 5.35% / 25
|
||||
files within 0.1%, hidden gives 2.18% / 5.55% / 22, with `annotation-tx3.pdf` jumping to 45.64%.
|
||||
The biases nearly cancel; `--hide-annotations` tells the two apart for one file at a time.
|
||||
- **Evidence:** text extraction (word-F1 0.98 against `pdftotext`) and now rendering (above); qpdf
|
||||
checks written output on every CI run since `PDFCRAFT_ORACLES_REQUIRED`. Acrobat captures must
|
||||
stay local in `plan/acrobat/` (clean-room, `AGENTS.md` §1.1).
|
||||
|
||||
Reference in New Issue
Block a user