diff --git a/crates/oracle/README.md b/crates/oracle/README.md index 7d2996b77b..04eea193aa 100644 --- a/crates/oracle/README.md +++ b/crates/oracle/README.md @@ -38,7 +38,7 @@ match pdfcraft_oracle::pdftotext() { |---|---|---| | `qpdf()` | qpdf | `--check` structural validation; the independent owner-password oracle | | `pdftoppm()` | Poppler | page pixels, for comparing against our own raster output | -| `pdftotext()` | Poppler | text extraction (`cargo xtask text-oracle` scores word F1 against it) | +| `pdftotext()` | Poppler | text extraction (`cargo xtask text-oracle` scores word F1 and reading order against it) | Each returns `Result`. `Oracle::command()` hands back a fresh `Command`; `Absent` renders as an actionable sentence naming the install command. @@ -92,6 +92,30 @@ picture through every corner of the format, so a correct decoder renders them al render 1 distinct image across all 96; Poppler renders 8, drawing a blank page for seven of them. `crates/render/tests/jbig2_invariants.rs` pins that invariant, and needs no oracle to do it. +## Comparing text + +`cargo xtask text-oracle` scores two different things against `pdftotext`, because one number +hides the other. + +**Word F1** is a bag of words: how much of what each side extracted the other also extracted, +with multiplicity. It catches a character read wrong, a glyph not read at all, a font we cannot +map. It is blind to *order* — a page emitted back to front scores 1.000. + +**Reading order** is the longest common subsequence of the two word sequences, as a share of the +longer one. It catches the failures F1 cannot see: a column read into the middle of its +neighbour, a figure's labels threaded through a paragraph, a form's value parted from its label. +Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages +is plenty of signal; the run says how many files that bit. + +The second score exists because of a measurement. Reviewing #2431, a change that moves text and +alters nothing else, word F1 called 26 of the 29 files whose output changed *identical*, while +the order score separated 18 improvements from 6 regressions — the regressions on forms, where +a checkbox's number was parted from its label. + +Read it the same way as the pixel score, and then once more: Poppler's reading order is a second +opinion, not the truth, and the same split can score as a win on one file and a loss on another +purely because `pdftotext` happens to make it in one place and not the other. + ## Still to do - Acrobat itself as the reference, rather than other open renderers. diff --git a/docs/architecture.md b/docs/architecture.md index a226cc8de6..aa91c36c4e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -101,7 +101,7 @@ Every user-facing feature is reachable headlessly (`AGENTS.md` §3): `cargo xtask ci` runs fmt, clippy (`-D warnings`), tests, the layering check, the WASM check, `xtask assets` (asset licences and hashes), `xtask parity` (checklist claims must cite real commands, tools and tests) and `xtask deny` (dependency licences). Other tasks: `xtask fuzz`, `xtask corpus` and `xtask check` (robustness -sweep over the pinned pdf.js corpus), `xtask text-oracle` (word F1 against `pdftotext`), +sweep over the pinned pdf.js corpus), `xtask text-oracle` (word F1 and reading order against `pdftotext`), `xtask demo-pdf`, `xtask screenshots`, `xtask models` (OCR models), `xtask version`. Releases are built by `.github/workflows/release.yml` ([releasing.md](releasing.md)). diff --git a/docs/file-format-parity.md b/docs/file-format-parity.md index 9a5f4d20d4..d7230c0962 100644 --- a/docs/file-format-parity.md +++ b/docs/file-format-parity.md @@ -86,7 +86,7 @@ features total). | PowerPoint (`.pptx`) | yes | slides per page: title from largest text, body text, images, tables (no slide overflow reflow / normAutofit; first page sets presentation dimensions) | basic | `create.export-pptx` partial | | RTF | yes | yes | basic | `create.export-rtf` | | HTML (single page, multiple pages) | yes | single file with headings, paragraphs, images, tables | basic | `create.export-html` partial | -| Plain text | yes | yes (reading order) | good (word-F1 0.98 vs `pdftotext`) | `create.export-text`, `xtask text-oracle` | +| Plain text | yes | yes (reading order) | good (word-F1 0.98 vs `pdftotext`; reading order is scored separately, because F1 cannot see it) | `create.export-text`, `xtask text-oracle` | | Accessible text (from tags) | yes | no | none | `create.export-accessible-text` planned | | PNG, JPEG, TIFF | yes | yes | good | `create.export-png`, `create.export-jpeg-tiff` | | JPEG 2000 | yes | no | none | `create.export-jpeg2000` planned | diff --git a/docs/gaps.md b/docs/gaps.md index c006e48d29..75d6fa4f2b 100644 --- a/docs/gaps.md +++ b/docs/gaps.md @@ -247,6 +247,17 @@ update the numbers. every appearance we *do* draw: over the same 250 files, shown gives mean 2.16% / p90 5.35% / 25 files within 0.1%, hidden gives 2.18% / 5.55% / 22, with `annotation-tx3.pdf` jumping to 45.64%. The biases nearly cancel; `--hide-annotations` tells the two apart for one file at a time. +- **The text score was blind to reading order (2026-10-15).** Word F1 is a bag of words, so a page + emitted back to front scores a perfect 1.000 — and reading order is most of what text extraction + has left to get wrong (#2387, #2586). `cargo xtask text-oracle` now reports a second score: the + longest common subsequence of the two word sequences over the longer one, both cut to 3,000 + words. The hole was not hypothetical. Measuring #2431, a change that moves text and alters + nothing else, **word F1 called 26 of the 29 files whose output changed identical**, where the + order score separated 18 improvements from 6 regressions and named their shape: forms, where a + checkbox's number was parted from its label (`prefilled_f1040.pdf` 0.672 → 0.481). Read it as a + disagreement, like the pixel score: the same split scored as a win on one file and a loss on + another purely because `pdftotext` makes it in one place and not the other. The corpus-wide + first reading has not been taken yet. - **Evidence:** text extraction (word-F1 0.98 against `pdftotext`) and now rendering (above); qpdf checks written output on every CI run since `PDFCRAFT_ORACLES_REQUIRED`. Acrobat captures must stay local in `plan/acrobat/` (clean-room, `AGENTS.md` §1.1). diff --git a/xtask/src/gates.rs b/xtask/src/gates.rs index 6591233e91..8e7db8cf08 100644 --- a/xtask/src/gates.rs +++ b/xtask/src/gates.rs @@ -196,8 +196,13 @@ pub fn check(args: &[String]) -> anyhow::Result<()> { } /// `cargo xtask text-oracle [--limit N]`: compare `pdfcraft-cli text` with poppler's `pdftotext` -/// (external oracle process, never linked) over the corpus. Reports word-level F1 per file and the -/// median. Plan target (M2): median ≥ 0.97. +/// (external oracle process, never linked) over the corpus. +/// +/// Two scores per file, because one hides the other. **Word F1** ([`f1`]) is a bag of words: it +/// catches a character read wrong, a glyph not read at all, a font we cannot map, and is blind to +/// order. **Reading order** ([`order_score`]) catches what F1 cannot see — a column read into the +/// middle of its neighbour, a figure's labels threaded through a paragraph. Reports both; gates +/// neither. Plan target (M2): median F1 ≥ 0.97. pub fn text_oracle(args: &[String]) -> anyhow::Result<()> { let limit: usize = args.iter().position(|a| a == "--limit").and_then(|i| args.get(i + 1)).and_then(|v| v.parse().ok()).unwrap_or(usize::MAX); let pdftotext = match pdfcraft_oracle::pdftotext() { @@ -211,8 +216,11 @@ pub fn text_oracle(args: &[String]) -> anyhow::Result<()> { files.sort(); let cli = release_cli(); let mut scores = Vec::new(); + let mut orders = Vec::new(); let mut worst: Vec<(f64, String)> = Vec::new(); + let mut worst_order: Vec<(f64, String)> = Vec::new(); let mut not_permitted = 0usize; + let mut truncated = 0usize; for f in files.iter().take(limit) { // A document may forbid copying its text, and we honour that: `pdfcraft-cli text` refuses // and says how to override it with the owner password. pdftotext ignores the flag and @@ -225,29 +233,49 @@ pub fn text_oracle(args: &[String]) -> anyhow::Result<()> { let ours = run_capture(Command::new(&cli).arg("text").arg(f), 20); let theirs = run_capture(pdftotext.command().arg("-q").arg(f).arg("-"), 20); let (Some(ours), Some(theirs)) = (ours, theirs) else { continue }; - let (a, b) = (words(&ours), words(&theirs)); + let (sa, sb) = (word_sequence(&ours), word_sequence(&theirs)); + let (a, b) = (words(&sa), words(&sb)); if b.len() < 20 { continue; // image-only or near-empty per the oracle } let score = f1(&a, &b); + let (order, cut) = order_score(&sa, &sb); + truncated = truncated.saturating_add(usize::from(cut)); + let name = f.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default(); scores.push(score); - worst.push((score, f.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default())); + orders.push(order); + worst.push((score, name.clone())); + worst_order.push((order, name)); } if scores.is_empty() { bail!("no comparable files"); } - scores.sort_by(f64::total_cmp); + let summary = |v: &mut Vec| { + v.sort_by(f64::total_cmp); + let mean = v.iter().sum::() / v.len() as f64; + (v[v.len() / 2], mean, v[v.len() / 10]) + }; + let compared = scores.len(); + let (median, mean, p10) = summary(&mut scores); + let (omedian, omean, op10) = summary(&mut orders); worst.sort_by(|a, b| a.0.total_cmp(&b.0)); - let median = scores[scores.len() / 2]; - let mean = scores.iter().sum::() / scores.len() as f64; - println!("text oracle: {} files with text; median F1 {:.3}, mean {:.3}, p10 {:.3}", scores.len(), median, mean, scores[scores.len() / 10]); + worst_order.sort_by(|a, b| a.0.total_cmp(&b.0)); + println!("text oracle: {compared} files with text; median F1 {median:.3}, mean {mean:.3}, p10 {p10:.3}"); + println!( + " reading order: median {omedian:.3}, mean {omean:.3}, p10 {op10:.3}{}", + if truncated > 0 { format!(" ({truncated} cut to {MAX_ORDER_WORDS} words)") } else { String::new() } + ); if not_permitted > 0 { println!(" {not_permitted} skipped: the document forbids copying its text, which we honour and pdftotext does not"); } - println!("lowest:"); + println!("lowest F1:"); for (s, n) in worst.iter().take(12) { println!(" {s:.3} {n}"); } + println!("lowest reading order:"); + for (s, n) in worst_order.iter().take(12) { + println!(" {s:.3} {n}"); + } Ok(()) } @@ -416,14 +444,57 @@ fn run_capture_bytes(cmd: &mut Command, timeout_s: u64) -> Option> { Some(out) } -fn words(s: &str) -> BTreeMap { +/// The words of `s`, in the order they were extracted. +fn word_sequence(s: &str) -> Vec { + s.split(|c: char| !c.is_alphanumeric()).filter(|w| !w.is_empty()).map(str::to_lowercase).collect() +} + +fn words(seq: &[String]) -> BTreeMap { let mut m = BTreeMap::new(); - for w in s.split(|c: char| !c.is_alphanumeric()).filter(|w| !w.is_empty()) { - *m.entry(w.to_lowercase()).or_insert(0) += 1; + for w in seq { + *m.entry(w.clone()).or_insert(0) += 1; } m } +/// Both word sequences are cut to this length before [`order_score`] compares them: the +/// comparison is quadratic, and three pages of prose is plenty of signal about reading order. +const MAX_ORDER_WORDS: usize = 3000; + +/// How much of the longer word sequence reads in the same order in both, and whether either side +/// was cut to [`MAX_ORDER_WORDS`] first. +/// +/// [`f1`] is a bag of words and cannot see order at all: a page emitted back to front scores a +/// perfect 1.000 against the page read properly. Reading order is most of what extraction has +/// left to get +/// wrong — a column read into the middle of its neighbour, a figure's labels threaded through a +/// paragraph, a form's value parted from its label — so it needs a number of its own. Reviewing +/// #2431, a change that moves text and alters nothing else, F1 called 26 of the 29 files whose +/// output changed identical, while this separated 18 improvements from 6 regressions. +/// +/// The score is the length of the longest common subsequence over the longer side. Like every +/// other oracle number here it is a *disagreement* with Poppler, not a verdict: the same split +/// scores as a win on one file and a loss on another purely because `pdftotext` happens to make +/// it in one place and not the other. +fn order_score(a: &[String], b: &[String]) -> (f64, bool) { + let cut = a.len() > MAX_ORDER_WORDS || b.len() > MAX_ORDER_WORDS; + let a = a.get(..a.len().min(MAX_ORDER_WORDS)).unwrap_or(a); + let b = b.get(..b.len().min(MAX_ORDER_WORDS)).unwrap_or(b); + let (n, m) = (a.len(), b.len()); + if n == 0 || m == 0 { + return (0.0, cut); + } + // Two rows of the usual table: after row `i`, `prev[j]` answers `a[..i]` against `b[..j]`. + let (mut prev, mut cur) = (vec![0u32; m + 1], vec![0u32; m + 1]); + for x in a { + for (j, y) in b.iter().enumerate() { + cur[j + 1] = if x == y { prev[j].saturating_add(1) } else { cur[j].max(prev[j + 1]) }; + } + std::mem::swap(&mut prev, &mut cur); + } + (f64::from(prev[m]) / n.max(m) as f64, cut) +} + fn f1(a: &BTreeMap, b: &BTreeMap) -> f64 { let common: usize = a.iter().map(|(w, n)| (*n).min(*b.get(w).unwrap_or(&0))).sum(); let (na, nb): (usize, usize) = (a.values().sum(), b.values().sum()); @@ -436,7 +507,42 @@ fn f1(a: &BTreeMap, b: &BTreeMap) -> f64 { #[cfg(test)] mod tests { - use super::root; + use super::{f1, order_score, root, word_sequence, words}; + + /// The two scores answer different questions, and the point of adding the second one is the + /// first row: F1 cannot tell a page from the same page read backwards. + #[test] + fn reading_order_sees_what_word_f1_cannot() { + let score = |ours: &str, theirs: &str| { + let (a, b) = (word_sequence(ours), word_sequence(theirs)); + (f1(&words(&a), &words(&b)), order_score(&a, &b).0) + }; + + // Two columns, read as columns and read interleaved: every word is there either way. + let columns = "left one left two left three right one right two right three"; + let woven = "left one right one left two right two left three right three"; + let (same_words, out_of_order) = score(woven, columns); + assert_eq!(same_words, 1.0, "F1 is a bag of words and cannot see the difference"); + assert!(out_of_order < 0.7, "reading order can: {out_of_order}"); + + // It is not blind to the thing F1 measures, either: a missing word costs both. + let (f, o) = score("left one left two left", columns); + assert!(f < 0.9 && o < 0.9, "{f} {o}"); + + // Identical text, and the degenerate inputs. + assert_eq!(score(columns, columns), (1.0, 1.0)); + assert_eq!(score("", columns).1, 0.0); + assert_eq!(score(columns, "").1, 0.0); + assert_eq!(order_score(&[], &[]), (0.0, false)); + + // Case and punctuation are not order: both sides are folded the same way. + assert_eq!(score("Left, one. LEFT two!", "left one left two"), (1.0, 1.0)); + + // Over the cap, the comparison still answers, and says that it was cut. + let long: Vec = (0..super::MAX_ORDER_WORDS + 10).map(|i| i.to_string()).collect(); + let (s, cut) = order_score(&long, &long); + assert!((s - 1.0).abs() < 1e-9 && cut, "{s} {cut}"); + } // Issue #211: Finder lists the app under Open With > Recommended only when the // bundle claims PDFs by UTI (#176).