pdfcraft/crates/oracle/README.md

pdfcraft-oracle

The external programs PdfCraft checks its own output against, and one rule: a missing oracle is either a declared skip or a failure, never a silent pass.

• Layer: test support (Class::Testkit in xtask/src/layers.rs). A dev-dependency only, never compiled into the product, and so usable even from the standalone crates (cos, geom, filters, crypt, arlington) whose own tests need it.
• No panicking path of its own. The or_skip! macro expands its assert! into the caller, which is test code.

Why it exists

Every test that wanted an oracle used to probe for it by hand and decide on its own what to do when it was missing. The common choice — print a line and return — reports as a pass, so a suite with no oracles installed looked exactly like one in which every oracle agreed.

That was not hypothetical. No CI runner installed qpdf, so cos/tests/encryption.rs::qpdf_agrees had never executed on a runner, while README.md said saved output was "independently checked with qpdf". For R2–R4 that test is the only check of owner-password authentication (Algorithm 7) that does not go through our own code.

API

// In a test: bind the oracle, or return after saying why it was skipped.
let qpdf = pdfcraft_oracle::or_skip!(pdfcraft_oracle::qpdf());
let out = qpdf.command().arg("--check").arg(&path).output()?;

// Elsewhere (xtask), handle the absence yourself.
match pdfcraft_oracle::pdftotext() {
    Ok(tool) => { /* tool.command() */ }
    Err(absent) => bail!("{absent}"),
}

Probe	Tool	Used for
qpdf()	qpdf	--check structural validation; the independent owner-password oracle
pdftoppm()	Poppler	page pixels, for comparing against our own raster output
pdftotext()	Poppler	text extraction (cargo xtask text-oracle scores word F1 against it)

Each returns Result<Oracle, Absent>. Oracle::command() hands back a fresh Command; Absent renders as an actionable sentence naming the install command.

The gate

PDFCRAFT_ORACLES_REQUIRED (any value, including 0 — the same convention as CRAFT_FONTS_REQUIRED) turns absence from a skip into a failure. CI sets it on the runners that install the tools, so a skip notice there becomes a red build.

$ cargo test -p pdfcraft-cos --test encryption qpdf_agrees
test result: ok. 1 passed            # qpdf absent: skipped, with a notice on stderr

$ PDFCRAFT_ORACLES_REQUIRED=1 cargo test -p pdfcraft-cos --test encryption qpdf_agrees
oracle `qpdf` is required but is not on PATH (program not found): PDFCRAFT_ORACLES_REQUIRED is
set, so this is a failure rather than a skip (apt install qpdf / brew install qpdf)
test result: FAILED. 1 failed

or_skip! expands to a return, so it belongs in a test returning (); a test returning Result should match on the probe directly.

Comparing pixels (pixels)

Rendering needs more than "did the tool run". A raw pixel-for-pixel difference is useless here: two correct renderers disagree on every glyph edge, so a strict comparison calls a perfect page badly wrong and tells you nothing about whether a shading is missing.

pixels::compare counts a pixel as differing only when it differs from every pixel in the other image's 3x3 neighbourhood by more than a tolerance. A one-pixel shift or a differently weighted edge finds a near match next door and is forgiven; a glyph that is not there, or a region painted the wrong colour, has no near match and is counted. The test runs in both directions and reports the worse result, so neither argument order flatters the blurrier renderer.

decode_pgm reads what pdftoppm -gray writes; decode_pam_gray reads what pdfcraft-cli render --out x.pam writes, compositing premultiplied alpha over white first so a transparent margin is not scored as black.

cargo xtask render-oracle drives it over the corpus. It reports; it does not gate.

A high score is a disagreement, not a verdict on us. The worst file of the first 250, bug1552113.pdf at 84%, is Poppler painting a link's /Border [0 0 112] as a band over the text the fixture exists to show; ours is the readable rendering. Look at both images before believing the number. --hide-annotations separates annotation differences from page content, though the default shows them: hiding is slightly worse overall, because it then scores Poppler for every appearance we do draw.

Poppler is not a usable oracle for JBIG2. The 96 bitmap-*.pdf corpus fixtures encode one picture through every corner of the format, so a correct decoder renders them all identically. We render 1 distinct image across all 96; Poppler renders 8, drawing a blank page for seven of them. crates/render/tests/jbig2_invariants.rs pins that invariant, and needs no oracle to do it.

Still to do

• Acrobat itself as the reference, rather than other open renderers.
• Oracles that are not external processes (hayro, lopdf) are ordinary dev-dependencies and are not routed through this crate.
• Two renderers sometimes round the same page box to different pixel sizes, and those files are counted as unmeasurable rather than compared.