mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-10-30 13:08:48 -06:00
aiongg
9526363152
* Build each crate's integration tests as one binary Every tests/*.rs file is its own executable that links the whole crate graph. Move each crate's test files into tests/it/ as modules of one tests/it/main.rs, for apps/pdfcraft, apps/pdfcraft-cli, automation, cos, crypt, engine, render, sign and ui-egui. ui-egui's ui_group binary and the support modules move into tests/it/ too. Crates with a single test file keep it. The test executables go from 150 to 52, and every crate runs the same number of tests as before. ui-egui perf keeps its own binary (own process): its frame-time budgets would charge each measurement for the other tests running beside it. What the move needs: - include_str!/include_bytes! and #[path] paths gain one "../". - ui-egui: one GPU lock in tests/it/main.rs replaces the five per-file ones (content, document_links, editing, fill_sign, view), so one process creates one wgpu device at a time. The drop-line test in ui.rs and the screenshot blocks behind PDFCRAFT_*_SHOT(S) and CRAFT_UI_DOCKING_DIR, which render without holding it, take it too. - pdfcraft winit_wayland_scroll: the vendored winit source it compiles names crate::event, so that shim lives in main.rs. - parity evidence names the new paths. * Name the one-binary test paths in docs and comments - READMEs, docs, ATTRIBUTION and source comments name tests/it/<file>.rs. - The oracle README's example filters with --test it encryption::qpdf_agrees. - pdfcraft's GPU lock comment points at the UI tests' shared lock in crates/ui-egui/tests/it/main.rs.
126 lines
6.5 KiB
Markdown
126 lines
6.5 KiB
Markdown
# pdfcraft-oracle
|
||
|
||
The external programs PdfCraft checks its own output against, and one rule: **a missing oracle is
|
||
either a declared skip or a failure, never a silent pass.**
|
||
|
||
- **Layer:** test support (`Class::Testkit` in `xtask/src/layers.rs`). A dev-dependency only, never
|
||
compiled into the product, and so usable even from the standalone crates (`cos`, `geom`,
|
||
`filters`, `crypt`, `arlington`) whose own tests need it.
|
||
- No panicking path of its own. The `or_skip!` macro expands its `assert!` into the caller, which
|
||
is test code.
|
||
|
||
## Why it exists
|
||
|
||
Every test that wanted an oracle used to probe for it by hand and decide on its own what to do when
|
||
it was missing. The common choice — print a line and return — reports as a **pass**, so a suite
|
||
with no oracles installed looked exactly like one in which every oracle agreed.
|
||
|
||
That was not hypothetical. No CI runner installed `qpdf`, so `cos/tests/it/encryption.rs::qpdf_agrees`
|
||
had never executed on a runner, while `README.md` said saved output was "independently checked with
|
||
qpdf". For R2–R4 that test is the *only* check of owner-password authentication (Algorithm 7) that
|
||
does not go through our own code.
|
||
|
||
## API
|
||
|
||
```rust
|
||
// In a test: bind the oracle, or return after saying why it was skipped.
|
||
let qpdf = pdfcraft_oracle::or_skip!(pdfcraft_oracle::qpdf());
|
||
let out = qpdf.command().arg("--check").arg(&path).output()?;
|
||
|
||
// Elsewhere (xtask), handle the absence yourself.
|
||
match pdfcraft_oracle::pdftotext() {
|
||
Ok(tool) => { /* tool.command() */ }
|
||
Err(absent) => bail!("{absent}"),
|
||
}
|
||
```
|
||
|
||
| Probe | Tool | Used for |
|
||
|---|---|---|
|
||
| `qpdf()` | qpdf | `--check` structural validation; the independent owner-password oracle |
|
||
| `pdftoppm()` | Poppler | page pixels, for comparing against our own raster output |
|
||
| `pdftotext()` | Poppler | text extraction (`cargo xtask text-oracle` scores word F1 and reading order against it) |
|
||
|
||
Each returns `Result<Oracle, Absent>`. `Oracle::command()` hands back a fresh `Command`;
|
||
`Absent` renders as an actionable sentence naming the install command.
|
||
|
||
## The gate
|
||
|
||
`PDFCRAFT_ORACLES_REQUIRED` (any value, including `0` — the same convention as
|
||
`CRAFT_FONTS_REQUIRED`) turns absence from a skip into a failure. CI sets it on the runners that
|
||
install the tools, so a skip notice there becomes a red build.
|
||
|
||
```
|
||
$ cargo test -p pdfcraft-cos --test it encryption::qpdf_agrees
|
||
test result: ok. 1 passed # qpdf absent: skipped, with a notice on stderr
|
||
|
||
$ PDFCRAFT_ORACLES_REQUIRED=1 cargo test -p pdfcraft-cos --test it encryption::qpdf_agrees
|
||
oracle `qpdf` is required but is not on PATH (program not found): PDFCRAFT_ORACLES_REQUIRED is
|
||
set, so this is a failure rather than a skip (apt install qpdf / brew install qpdf)
|
||
test result: FAILED. 1 failed
|
||
```
|
||
|
||
`or_skip!` expands to a `return`, so it belongs in a test returning `()`; a test returning `Result`
|
||
should match on the probe directly.
|
||
|
||
## Comparing pixels (`pixels`)
|
||
|
||
Rendering needs more than "did the tool run". A raw pixel-for-pixel difference is useless here:
|
||
two correct renderers disagree on every glyph edge, so a strict comparison calls a perfect page
|
||
badly wrong and tells you nothing about whether a shading is missing.
|
||
|
||
`pixels::compare` counts a pixel as differing only when it differs from **every** pixel in the
|
||
other image's 3x3 neighbourhood by more than a tolerance. A one-pixel shift or a differently
|
||
weighted edge finds a near match next door and is forgiven; a glyph that is not there, or a region
|
||
painted the wrong colour, has no near match and is counted. The test runs in both directions and
|
||
reports the worse result, so neither argument order flatters the blurrier renderer.
|
||
|
||
`decode_pgm` reads what `pdftoppm -gray` writes; `decode_pam_gray` reads what
|
||
`pdfcraft-cli render --out x.pam` writes, compositing premultiplied alpha over white first so a
|
||
transparent margin is not scored as black.
|
||
|
||
`cargo xtask render-oracle` drives it over the corpus. It reports; it does not gate.
|
||
|
||
A high score is a *disagreement*, not a verdict on us. The worst file of the first 250,
|
||
`bug1552113.pdf` at 84%, is Poppler painting a link's `/Border [0 0 112]` as a band over the text
|
||
the fixture exists to show; ours is the readable rendering. Look at both images before believing
|
||
the number. `--hide-annotations` separates annotation differences from page content, though the
|
||
default shows them: hiding is slightly worse overall, because it then scores Poppler for every
|
||
appearance we do draw.
|
||
|
||
**Poppler is not a usable oracle for JBIG2.** The 96 `bitmap-*.pdf` corpus fixtures encode one
|
||
picture through every corner of the format, so a correct decoder renders them all identically. We
|
||
render 1 distinct image across all 96; Poppler renders 8, drawing a blank page for seven of them.
|
||
`crates/render/tests/it/jbig2_invariants.rs` pins that invariant, and needs no oracle to do it.
|
||
|
||
## Comparing text
|
||
|
||
`cargo xtask text-oracle` scores two different things against `pdftotext`, because one number
|
||
hides the other.
|
||
|
||
**Word F1** is a bag of words: how much of what each side extracted the other also extracted,
|
||
with multiplicity. It catches a character read wrong, a glyph not read at all, a font we cannot
|
||
map. It is blind to *order* — a page emitted back to front scores 1.000.
|
||
|
||
**Reading order** is the longest common subsequence of the two word sequences, as a share of the
|
||
longer one. It catches the failures F1 cannot see: a column read into the middle of its
|
||
neighbour, a figure's labels threaded through a paragraph, a form's value parted from its label.
|
||
Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages
|
||
is plenty of signal; the run says how many files that bit.
|
||
|
||
The second score exists because of a measurement. Reviewing #2431, a change that moves text and
|
||
alters nothing else, word F1 called 26 of the 29 files whose output changed *identical*, while
|
||
the order score separated 18 improvements from 6 regressions — the regressions on forms, where
|
||
a checkbox's number was parted from its label.
|
||
|
||
Read it the same way as the pixel score, and then once more: Poppler's reading order is a second
|
||
opinion, not the truth, and the same split can score as a win on one file and a loss on another
|
||
purely because `pdftotext` happens to make it in one place and not the other.
|
||
|
||
## Still to do
|
||
|
||
- Acrobat itself as the reference, rather than other open renderers.
|
||
- Oracles that are not external processes (hayro, `lopdf`) are ordinary dev-dependencies and are
|
||
not routed through this crate.
|
||
- Two renderers sometimes round the same page box to different pixel sizes, and those files are
|
||
counted as unmeasurable rather than compared.
|