feat(oracle): score reading order, which word F1 cannot see

`cargo xtask text-oracle` reported one number: word F1 against `pdftotext`. F1 is a bag of
words. It catches a character read wrong, a glyph not read at all, a font we cannot map -- and
it is blind to order, so a page emitted back to front scores a perfect 1.000. Reading order is
most of what text extraction has left to get wrong: a column read into the middle of its
neighbour (#2387), a figure's labels threaded through a paragraph (#2586), a form's value parted
from its label.

The hole is not hypothetical. Reviewing #2431, a change that moves text and alters nothing else,
**word F1 called 26 of the 29 files whose output changed identical**, while scoring the word
sequence separated 18 improvements from 6 regressions and named the shape of the regressions
(forms: `prefilled_f1040.pdf` 0.672 -> 0.481, a checkbox's number parted from its label). The
harness could not see the only thing that change does.

So a second score per file: the longest common subsequence of the two word sequences, as a share
of the longer one. It shows up in the first three corpus files -- `Test-plusminus.pdf` scores F1
1.000 and reading order 0.875, the same forty words in a title block that we read down the
column and `pdftotext` reads across.

Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages
is plenty of signal; the run says how many files that bit, so the cap cannot quietly become the
measurement. It reports; it does not gate, and like the pixel score it is a *disagreement* with
Poppler rather than a verdict -- in the #2431 measurement the same split scored as a win on one
file and a loss on another purely because `pdftotext` makes it in one place and not the other.
The function's doc comment, `crates/oracle/README.md` and `docs/gaps.md` gap 12 all say so.

`words` now takes the sequence rather than re-splitting the text, so both scores read the same
tokens. `docs/file-format-parity.md` claimed plain-text export was good "(word-F1 0.98)" in a row
whose own column is reading order; it now says the two are scored separately.

Verified: `cargo xtask text-oracle --limit 60` end to end; `cargo test -p xtask` 49 passed; fmt
and clippy clean. The unit test was watched failing with the score blinded to order (sorting both
sides made it 1.000 where it must be below 0.7). The corpus-wide first reading is not in this
commit: the machine is shared with a parallel build and the full sweep was stopped twice for
memory.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

crates/oracle/README.md

@@ -38,7 +38,7 @@ match pdfcraft_oracle::pdftotext() {
 |---|---|---|
 | `qpdf()` | qpdf | `--check` structural validation; the independent owner-password oracle |
 | `pdftoppm()` | Poppler | page pixels, for comparing against our own raster output |
-| `pdftotext()` | Poppler | text extraction (`cargo xtask text-oracle` scores word F1 against it) |
+| `pdftotext()` | Poppler | text extraction (`cargo xtask text-oracle` scores word F1 and reading order against it) |
 
 Each returns `Result<Oracle, Absent>`. `Oracle::command()` hands back a fresh `Command`;
 `Absent` renders as an actionable sentence naming the install command.
@@ -92,6 +92,30 @@ picture through every corner of the format, so a correct decoder renders them al
 render 1 distinct image across all 96; Poppler renders 8, drawing a blank page for seven of them.
 `crates/render/tests/jbig2_invariants.rs` pins that invariant, and needs no oracle to do it.
 
+## Comparing text
+
+`cargo xtask text-oracle` scores two different things against `pdftotext`, because one number
+hides the other.
+
+**Word F1** is a bag of words: how much of what each side extracted the other also extracted,
+with multiplicity. It catches a character read wrong, a glyph not read at all, a font we cannot
+map. It is blind to *order* — a page emitted back to front scores 1.000.
+
+**Reading order** is the longest common subsequence of the two word sequences, as a share of the
+longer one. It catches the failures F1 cannot see: a column read into the middle of its
+neighbour, a figure's labels threaded through a paragraph, a form's value parted from its label.
+Both sides are cut to the first 3,000 words, because the comparison is quadratic and three pages
+is plenty of signal; the run says how many files that bit.
+
+The second score exists because of a measurement. Reviewing #2431, a change that moves text and
+alters nothing else, word F1 called 26 of the 29 files whose output changed *identical*, while
+the order score separated 18 improvements from 6 regressions — the regressions on forms, where
+a checkbox's number was parted from its label.
+
+Read it the same way as the pixel score, and then once more: Poppler's reading order is a second
+opinion, not the truth, and the same split can score as a win on one file and a loss on another
+purely because `pdftotext` happens to make it in one place and not the other.
+
 ## Still to do
 
 - Acrobat itself as the reference, rather than other open renderers.

docs/architecture.md

@@ -101,7 +101,7 @@ Every user-facing feature is reachable headlessly (`AGENTS.md` §3):
 `cargo xtask ci` runs fmt, clippy (`-D warnings`), tests, the layering check, the WASM check,
 `xtask assets` (asset licences and hashes), `xtask parity` (checklist claims must cite real commands,
 tools and tests) and `xtask deny` (dependency licences). Other tasks: `xtask fuzz`, `xtask corpus` and `xtask check` (robustness
-sweep over the pinned pdf.js corpus), `xtask text-oracle` (word F1 against `pdftotext`),
+sweep over the pinned pdf.js corpus), `xtask text-oracle` (word F1 and reading order against `pdftotext`),
 `xtask demo-pdf`, `xtask screenshots`, `xtask models` (OCR models), `xtask version`. Releases are
 built by `.github/workflows/release.yml` ([releasing.md](releasing.md)).
 

docs/file-format-parity.md

@@ -86,7 +86,7 @@ features total).
 | PowerPoint (`.pptx`) | yes | slides per page: title from largest text, body text, images, tables (no slide overflow reflow / normAutofit; first page sets presentation dimensions) | basic | `create.export-pptx` partial |
 | RTF | yes | yes | basic | `create.export-rtf` |
 | HTML (single page, multiple pages) | yes | single file with headings, paragraphs, images, tables | basic | `create.export-html` partial |
-| Plain text | yes | yes (reading order) | good (word-F1 0.98 vs `pdftotext`) | `create.export-text`, `xtask text-oracle` |
+| Plain text | yes | yes (reading order) | good (word-F1 0.98 vs `pdftotext`; reading order is scored separately, because F1 cannot see it) | `create.export-text`, `xtask text-oracle` |
 | Accessible text (from tags) | yes | no | none | `create.export-accessible-text` planned |
 | PNG, JPEG, TIFF | yes | yes | good | `create.export-png`, `create.export-jpeg-tiff` |
 | JPEG 2000 | yes | no | none | `create.export-jpeg2000` planned |

docs/gaps.md

@@ -247,6 +247,17 @@ update the numbers.
   every appearance we *do* draw: over the same 250 files, shown gives mean 2.16% / p90 5.35% / 25
   files within 0.1%, hidden gives 2.18% / 5.55% / 22, with `annotation-tx3.pdf` jumping to 45.64%.
   The biases nearly cancel; `--hide-annotations` tells the two apart for one file at a time.
+- **The text score was blind to reading order (2026-10-15).** Word F1 is a bag of words, so a page
+  emitted back to front scores a perfect 1.000 — and reading order is most of what text extraction
+  has left to get wrong (#2387, #2586). `cargo xtask text-oracle` now reports a second score: the
+  longest common subsequence of the two word sequences over the longer one, both cut to 3,000
+  words. The hole was not hypothetical. Measuring #2431, a change that moves text and alters
+  nothing else, **word F1 called 26 of the 29 files whose output changed identical**, where the
+  order score separated 18 improvements from 6 regressions and named their shape: forms, where a
+  checkbox's number was parted from its label (`prefilled_f1040.pdf` 0.672 → 0.481). Read it as a
+  disagreement, like the pixel score: the same split scored as a win on one file and a loss on
+  another purely because `pdftotext` makes it in one place and not the other. The corpus-wide
+  first reading has not been taken yet.
 - **Evidence:** text extraction (word-F1 0.98 against `pdftotext`) and now rendering (above); qpdf
   checks written output on every CI run since `PDFCRAFT_ORACLES_REQUIRED`. Acrobat captures must
   stay local in `plan/acrobat/` (clean-room, `AGENTS.md` §1.1).

xtask/src/gates.rs

@@ -196,8 +196,13 @@ pub fn check(args: &[String]) -> anyhow::Result<()> {
 }
 
 /// `cargo xtask text-oracle [--limit N]`: compare `pdfcraft-cli text` with poppler's `pdftotext`
-/// (external oracle process, never linked) over the corpus. Reports word-level F1 per file and the
-/// median. Plan target (M2): median ≥ 0.97.
+/// (external oracle process, never linked) over the corpus.
+///
+/// Two scores per file, because one hides the other. **Word F1** ([`f1`]) is a bag of words: it
+/// catches a character read wrong, a glyph not read at all, a font we cannot map, and is blind to
+/// order. **Reading order** ([`order_score`]) catches what F1 cannot see — a column read into the
+/// middle of its neighbour, a figure's labels threaded through a paragraph. Reports both; gates
+/// neither. Plan target (M2): median F1 ≥ 0.97.
 pub fn text_oracle(args: &[String]) -> anyhow::Result<()> {
     let limit: usize = args.iter().position(|a| a == "--limit").and_then(|i| args.get(i + 1)).and_then(|v| v.parse().ok()).unwrap_or(usize::MAX);
     let pdftotext = match pdfcraft_oracle::pdftotext() {
@@ -211,8 +216,11 @@ pub fn text_oracle(args: &[String]) -> anyhow::Result<()> {
     files.sort();
     let cli = release_cli();
     let mut scores = Vec::new();
+    let mut orders = Vec::new();
     let mut worst: Vec<(f64, String)> = Vec::new();
+    let mut worst_order: Vec<(f64, String)> = Vec::new();
     let mut not_permitted = 0usize;
+    let mut truncated = 0usize;
     for f in files.iter().take(limit) {
         // A document may forbid copying its text, and we honour that: `pdfcraft-cli text` refuses
         // and says how to override it with the owner password. pdftotext ignores the flag and
@@ -225,29 +233,49 @@ pub fn text_oracle(args: &[String]) -> anyhow::Result<()> {
         let ours = run_capture(Command::new(&cli).arg("text").arg(f), 20);
         let theirs = run_capture(pdftotext.command().arg("-q").arg(f).arg("-"), 20);
         let (Some(ours), Some(theirs)) = (ours, theirs) else { continue };
-        let (a, b) = (words(&ours), words(&theirs));
+        let (sa, sb) = (word_sequence(&ours), word_sequence(&theirs));
+        let (a, b) = (words(&sa), words(&sb));
         if b.len() < 20 {
             continue; // image-only or near-empty per the oracle
         }
         let score = f1(&a, &b);
+        let (order, cut) = order_score(&sa, &sb);
+        truncated = truncated.saturating_add(usize::from(cut));
+        let name = f.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default();
         scores.push(score);
-        worst.push((score, f.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default()));
+        orders.push(order);
+        worst.push((score, name.clone()));
+        worst_order.push((order, name));
     }
     if scores.is_empty() {
         bail!("no comparable files");
     }
-    scores.sort_by(f64::total_cmp);
+    let summary = |v: &mut Vec<f64>| {
+        v.sort_by(f64::total_cmp);
+        let mean = v.iter().sum::<f64>() / v.len() as f64;
+        (v[v.len() / 2], mean, v[v.len() / 10])
+    };
+    let compared = scores.len();
+    let (median, mean, p10) = summary(&mut scores);
+    let (omedian, omean, op10) = summary(&mut orders);
     worst.sort_by(|a, b| a.0.total_cmp(&b.0));
-    let median = scores[scores.len() / 2];
-    let mean = scores.iter().sum::<f64>() / scores.len() as f64;
-    println!("text oracle: {} files with text; median F1 {:.3}, mean {:.3}, p10 {:.3}", scores.len(), median, mean, scores[scores.len() / 10]);
+    worst_order.sort_by(|a, b| a.0.total_cmp(&b.0));
+    println!("text oracle: {compared} files with text; median F1 {median:.3}, mean {mean:.3}, p10 {p10:.3}");
+    println!(
+        "  reading order: median {omedian:.3}, mean {omean:.3}, p10 {op10:.3}{}",
+        if truncated > 0 { format!(" ({truncated} cut to {MAX_ORDER_WORDS} words)") } else { String::new() }
+    );
     if not_permitted > 0 {
         println!("  {not_permitted} skipped: the document forbids copying its text, which we honour and pdftotext does not");
     }
-    println!("lowest:");
+    println!("lowest F1:");
     for (s, n) in worst.iter().take(12) {
         println!("  {s:.3}  {n}");
     }
+    println!("lowest reading order:");
+    for (s, n) in worst_order.iter().take(12) {
+        println!("  {s:.3}  {n}");
+    }
     Ok(())
 }
 
@@ -416,14 +444,57 @@ fn run_capture_bytes(cmd: &mut Command, timeout_s: u64) -> Option<Vec<u8>> {
     Some(out)
 }
 
-fn words(s: &str) -> BTreeMap<String, usize> {
+/// The words of `s`, in the order they were extracted.
+fn word_sequence(s: &str) -> Vec<String> {
+    s.split(|c: char| !c.is_alphanumeric()).filter(|w| !w.is_empty()).map(str::to_lowercase).collect()
+}
+
+fn words(seq: &[String]) -> BTreeMap<String, usize> {
     let mut m = BTreeMap::new();
-    for w in s.split(|c: char| !c.is_alphanumeric()).filter(|w| !w.is_empty()) {
-        *m.entry(w.to_lowercase()).or_insert(0) += 1;
+    for w in seq {
+        *m.entry(w.clone()).or_insert(0) += 1;
     }
     m
 }
 
+/// Both word sequences are cut to this length before [`order_score`] compares them: the
+/// comparison is quadratic, and three pages of prose is plenty of signal about reading order.
+const MAX_ORDER_WORDS: usize = 3000;
+
+/// How much of the longer word sequence reads in the same order in both, and whether either side
+/// was cut to [`MAX_ORDER_WORDS`] first.
+///
+/// [`f1`] is a bag of words and cannot see order at all: a page emitted back to front scores a
+/// perfect 1.000 against the page read properly. Reading order is most of what extraction has
+/// left to get
+/// wrong — a column read into the middle of its neighbour, a figure's labels threaded through a
+/// paragraph, a form's value parted from its label — so it needs a number of its own. Reviewing
+/// #2431, a change that moves text and alters nothing else, F1 called 26 of the 29 files whose
+/// output changed identical, while this separated 18 improvements from 6 regressions.
+///
+/// The score is the length of the longest common subsequence over the longer side. Like every
+/// other oracle number here it is a *disagreement* with Poppler, not a verdict: the same split
+/// scores as a win on one file and a loss on another purely because `pdftotext` happens to make
+/// it in one place and not the other.
+fn order_score(a: &[String], b: &[String]) -> (f64, bool) {
+    let cut = a.len() > MAX_ORDER_WORDS || b.len() > MAX_ORDER_WORDS;
+    let a = a.get(..a.len().min(MAX_ORDER_WORDS)).unwrap_or(a);
+    let b = b.get(..b.len().min(MAX_ORDER_WORDS)).unwrap_or(b);
+    let (n, m) = (a.len(), b.len());
+    if n == 0 || m == 0 {
+        return (0.0, cut);
+    }
+    // Two rows of the usual table: after row `i`, `prev[j]` answers `a[..i]` against `b[..j]`.
+    let (mut prev, mut cur) = (vec![0u32; m + 1], vec![0u32; m + 1]);
+    for x in a {
+        for (j, y) in b.iter().enumerate() {
+            cur[j + 1] = if x == y { prev[j].saturating_add(1) } else { cur[j].max(prev[j + 1]) };
+        }
+        std::mem::swap(&mut prev, &mut cur);
+    }
+    (f64::from(prev[m]) / n.max(m) as f64, cut)
+}
+
 fn f1(a: &BTreeMap<String, usize>, b: &BTreeMap<String, usize>) -> f64 {
     let common: usize = a.iter().map(|(w, n)| (*n).min(*b.get(w).unwrap_or(&0))).sum();
     let (na, nb): (usize, usize) = (a.values().sum(), b.values().sum());
@@ -436,7 +507,42 @@ fn f1(a: &BTreeMap<String, usize>, b: &BTreeMap<String, usize>) -> f64 {
 
 #[cfg(test)]
 mod tests {
-    use super::root;
+    use super::{f1, order_score, root, word_sequence, words};
+
+    /// The two scores answer different questions, and the point of adding the second one is the
+    /// first row: F1 cannot tell a page from the same page read backwards.
+    #[test]
+    fn reading_order_sees_what_word_f1_cannot() {
+        let score = |ours: &str, theirs: &str| {
+            let (a, b) = (word_sequence(ours), word_sequence(theirs));
+            (f1(&words(&a), &words(&b)), order_score(&a, &b).0)
+        };
+
+        // Two columns, read as columns and read interleaved: every word is there either way.
+        let columns = "left one left two left three right one right two right three";
+        let woven = "left one right one left two right two left three right three";
+        let (same_words, out_of_order) = score(woven, columns);
+        assert_eq!(same_words, 1.0, "F1 is a bag of words and cannot see the difference");
+        assert!(out_of_order < 0.7, "reading order can: {out_of_order}");
+
+        // It is not blind to the thing F1 measures, either: a missing word costs both.
+        let (f, o) = score("left one left two left", columns);
+        assert!(f < 0.9 && o < 0.9, "{f} {o}");
+
+        // Identical text, and the degenerate inputs.
+        assert_eq!(score(columns, columns), (1.0, 1.0));
+        assert_eq!(score("", columns).1, 0.0);
+        assert_eq!(score(columns, "").1, 0.0);
+        assert_eq!(order_score(&[], &[]), (0.0, false));
+
+        // Case and punctuation are not order: both sides are folded the same way.
+        assert_eq!(score("Left, one. LEFT two!", "left one left two"), (1.0, 1.0));
+
+        // Over the cap, the comparison still answers, and says that it was cut.
+        let long: Vec<String> = (0..super::MAX_ORDER_WORDS + 10).map(|i| i.to_string()).collect();
+        let (s, cut) = order_score(&long, &long);
+        assert!((s - 1.0).abs() < 1e-9 && cut, "{s} {cut}");
+    }
 
     // Issue #211: Finder lists the app under Open With > Recommended only when the
     // bundle claims PDFs by UTI (#176).