diff --git a/ROADMAP.md b/ROADMAP.md index bacdc6e33..34c4c1893 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -189,6 +189,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md - **2026-10-20 (M6, calculation order page):** Prepare a form ▸ Calculation order, a page of its own, lists the calculated fields in the order they calculate with their formulas and values, moved with Up/Down arrows (as Acrobat's Set Field Calculation Order), Alt+↑/↓ or by dragging. A card says what is wrong with the order and holds Fix order; a loop has a card of its own that opens one of its formulas. On the page, each calculated field shows its step, and arrows show what the field in focus reads and what reads it. Estimates unchanged. - **2026-10-20 (M6, calculation order):** The order a form's fields calculate in (`/CO`) is read, checked, fixed and set headlessly with `form_calculation_order`: it says which fields read one calculated later (they show stale values), which read each other in a loop, and which the PDF's order leaves out; Fix order moves only what must move, in one undo step, and the form recalculates. Editing a formula no longer moves its field to the end of the order, and a field the order lists twice calculates once. Estimates unchanged. - **2026-10-20 (M2, #3574):** Forms without Resources use page Properties for ActualText and layer visibility. Synthetic tests cover boundaries, nested/shared forms, and save/reopen. Estimates unchanged. +- **2026-10-20 (M2, PR #3348 review):** Tracking samples retain short repeated punctuation (up to four glyphs), keeping tracked ellipses intact and searchable while longer TOC leaders still preserve title word spaces. Synthetic regressions cover isolated and surrounded ellipses. Estimates unchanged. - **2026-10-19 (M6, renames keep formulas):** Renaming a field renames it in every calculation that reads it, in the same undo step: `AFSimple_Calculate` field lists, simplified field notation (the comment and the JavaScript after it) and `getField` calls naming it in one string literal; the rest of each script is kept as written. A sum used to drop the renamed field, notation read it as 0 and a script kept a stale value. Field Properties says how many formulas changed and `form_set_props` lists them as `formulas_updated`. Names built at run time and other events' scripts are not rewritten. Estimates unchanged. - **2026-10-19 (M7, #3591):** A background added to a page that already has one is painted over the old one instead of going under it and staying hidden. Behind the content, backgrounds come first, in the order added, and then behind-content watermarks, so a watermark added after a background no longer disappears. Synthetic regressions in `crates/edit` check the order. Estimates unchanged. @@ -210,6 +211,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md - **2026-10-18 (M5, PR #1971 review):** Nested FDF annotation values that identify a source `/Page` or `/Pages` dictionary become null at every copy depth. A synthetic three-page PDF saved and reopened before export verifies that destination actions cannot pull source page content into the FDF, while unrelated nested metadata and shared references remain intact. FDF import semantics, copied-graph limits and estimates are unchanged. - **2026-10-15 (M5, PR #1971 refresh):** Nested FDF annotation values retain shared or cyclic copied graphs. Discovery borrows direct values and records annotation positions with lazy array owners; retained entries are budgeted before cloning. Synthetic stress fixtures include repeated aliases and saved-PDF cache eviction. Limits cover the copied graph; source parsing and page discovery retain their existing limits, with no total source-memory bound. FDF import semantics and estimates are unchanged. - **2026-10-13 (M5, nested FDF values):** Comment export copies and renumbers nested indirect values, retaining geometry, colours, borders and shared graphs. Graph/data limits return an error before the destination is written; synthetic saved-PDF and automation exchange regressions. Estimates unchanged. +- **2026-10-18 (M2, #3086):** Text extraction estimates letter tracking without repeated punctuation gaps so dotted TOC leaders do not swallow word spaces. Synthetic PDF regressions cover leader lengths, phrase search and selection; strongly tracked text stays intact. Estimates unchanged. - **2026-10-18 (M6, #2892):** Push-button normal labels are editable per widget through Options and form_set_props, independently of field names; rendering, undo/redo and incremental/full save/reopen are covered. Estimates unchanged. diff --git a/contributors/people.toml b/contributors/people.toml index f7cb39292..2d660829c 100644 --- a/contributors/people.toml +++ b/contributors/people.toml @@ -153,3 +153,9 @@ sync_github_name = true real_name = "Eric Kaden" display_name = "Eric Kaden" sync_github_name = false + +[people.MatrixMagician] +sync_github_name = true + +[people.Lcoding15] +real_name = "Livio Oswald" diff --git a/crates/a11y/src/headings.rs b/crates/a11y/src/headings.rs index c7e911fde..537ed908d 100644 --- a/crates/a11y/src/headings.rs +++ b/crates/a11y/src/headings.rs @@ -263,7 +263,9 @@ pub(crate) fn mcid_texts(doc: &Document, page: &Dict) -> HashMap { tm = tlm; } let pieces: Vec<&Object> = match op.op.as_slice() { - b"TJ" => op.operands.first().and_then(Object::as_array).map(|a| a.iter().collect()).unwrap_or_default(), + // A surplus array (`[(A)] [(B)] TJ`) is malformed, but the operator still takes the + // one immediately before it, as the renderer and every other reader here do (#3569). + b"TJ" => op.operands.last().and_then(Object::as_array).map(|a| a.iter().collect()).unwrap_or_default(), _ => op.operands.last().into_iter().collect(), }; for piece in pieces { diff --git a/crates/a11y/src/tests.rs b/crates/a11y/src/tests.rs index fcd681efc..d4e5e8820 100644 --- a/crates/a11y/src/tests.rs +++ b/crates/a11y/src/tests.rs @@ -926,6 +926,16 @@ fn an_untagged_document_has_no_headings() { assert!(headings(&pdf(&objs, "")).is_empty()); } +/// `[(FIRSTARR)] [(LASTARR)] TJ` is malformed (a surplus array operand), but the operator still +/// draws only the one immediately before it, as the renderer and every other reader agree (#3569) +/// -- not the first, which is never drawn at all. +#[test] +fn a_heading_with_a_tj_surplus_array_reads_the_one_the_page_draws() { + let doc = heading_page(&["BT /F1 20 Tf 72 700 Td [(FIRSTARR)] [(LASTARR)] TJ ET".into()]); + let titles: Vec = headings(&doc).into_iter().map(|h| h.title).collect(); + assert_eq!(titles, ["LASTARR"]); +} + /// The text of every `"…"` literal with a space in it, outside comments and byte strings. fn sentences(source: &str) -> Vec { let b = source.as_bytes(); diff --git a/crates/edit/src/tests.rs b/crates/edit/src/tests.rs index 0fd635aeb..5c5c92e22 100644 --- a/crates/edit/src/tests.rs +++ b/crates/edit/src/tests.rs @@ -5966,3 +5966,38 @@ fn equal_size_rows_need_the_tighter_tolerance_even_without_overlap() { let blocks = text::text_blocks(&doc, 0).unwrap(); assert_eq!(blocks.len(), 2, "two separate rows stay two paragraphs, not one: {blocks:?}"); } + +/// A `TJ` with a surplus array operand (`[(FIRSTARR)] [(LASTARR)] TJ`) is malformed, but the +/// operator still takes the one immediately before it, as the renderer, MuPDF and pdftotext do +/// (#3569) -- not the first, which this reader took before, showing a word the page never draws. +#[test] +fn a_tj_with_a_surplus_array_reads_the_last_one() { + let doc = text_page("BT /F1 24 Tf 20 100 Td [(FIRSTARR)] [(LASTARR)] TJ ET"); + let lines = text::text_lines(&doc, 0).unwrap(); + assert_eq!(lines.len(), 1); + assert_eq!(lines[0].text, "LASTARR", "the operand immediately before TJ, as the page draws: {lines:?}"); +} + +/// A producer wrote `0.0001 Tc` on one word of a line (Word through macOS Quartz). That spacing +/// can't be seen, so the word stays on its line; spacing that can be seen still keeps runs apart. +#[test] +fn a_word_with_invisible_extra_spacing_stays_on_its_line() { + let page = |spacing: &str| { + text_page(&format!( + "BT /F1 12 Tf 1 0 0 1 72 700 Tm (for company ) Tj ET \ + BT /F1 12 Tf 1 0 0 1 141.36 700 Tm (A) Tj ET \ + BT {spacing} Tc /F1 12 Tf 1 0 0 1 149.364 700 Tm (EROSOL) Tj ET" + )) + }; + let blocks = text::text_blocks(&page("0.0001"), 0).unwrap(); + assert_eq!(blocks.iter().map(|b| b.text.as_str()).collect::>(), ["for company AEROSOL"]); + + let mut doc = page("0.0001"); + text::rewrite_block(&mut doc, 0, 0, Some("for company AEROSOL LLC"), &BlockStyle::default()).unwrap(); + let lines = text::text_lines(&reopen(&doc), 0).unwrap(); + assert_eq!(lines.iter().map(|l| l.text.as_str()).collect::>(), ["for company AEROSOL LLC"]); + assert!((lines[0].rect[0] - 72.0).abs() < 0.01, "{:?}", lines[0].rect); + + let blocks = text::text_blocks(&page("0.5"), 0).unwrap(); + assert_eq!(blocks.iter().map(|b| b.text.as_str()).collect::>(), ["for company A", "EROSOL"]); +} diff --git a/crates/edit/src/text.rs b/crates/edit/src/text.rs index eea1ef3dd..c6e6982aa 100644 --- a/crates/edit/src/text.rs +++ b/crates/edit/src/text.rs @@ -1085,7 +1085,11 @@ fn interpret( let Some((name, m)) = ts.font.clone() else { continue }; // Pieces: strings, and TJ number adjustments. let pieces: Vec = match op.op.as_slice() { - b"TJ" => op.operands.first().and_then(Object::as_array).cloned().unwrap_or_default(), + // A surplus array (`[(A)] [(B)] TJ`) is malformed, but the operator still + // takes the one immediately before it, as the renderer and every other + // reader here do (#3569) -- not the first, which a reader counting operands + // from the front of a surplus list would otherwise take. + b"TJ" => op.operands.last().and_then(Object::as_array).cloned().unwrap_or_default(), _ => op.operands.last().cloned().into_iter().collect(), }; // Tc is already included in the pen advance; TJ may encode tracking instead. @@ -1384,13 +1388,18 @@ pub fn text_lines(doc: &Document, page: usize) -> Result, EditErro /// Joining independent text objects is safe only when they use the same text and graphics /// transforms and spacing. In particular, a nearby rotated label isn't part of this line. fn compatible_word(line: &TextLine, shown: &Shown) -> bool { - let same = |a: &[f64], b: &[f64]| a.iter().zip(b).all(|(a, b)| (a - b).abs() < 1e-6); + let within = |a: &[f64], b: &[f64], tolerance: f64| a.iter().zip(b).all(|(a, b)| (a - b).abs() < tolerance); + let same = |a: &[f64], b: &[f64]| within(a, b, 1e-6); let (a, b) = (&line.origin.state, TextState::of(&shown.state)); + // Spacing that differs by less than a thousandth of an em, the unit glyph widths are given in, + // can't be seen. Some producers still write it (`0.0001 Tc` on one word of a line). + let em = shown.state.size.abs() / 1000.0; line.scope == shown.scope && line.paint == shown.paint && same(&line.origin.tm[..4], &shown.tm.0[..4]) && same(&line.origin.ctm, &shown.state.ctm.0) - && same(&[a.char_spacing, a.word_spacing, a.scale, a.rise], &[b.char_spacing, b.word_spacing, b.scale, b.rise]) + && within(&[a.char_spacing, a.word_spacing], &[b.char_spacing, b.word_spacing], em.max(1e-6)) + && same(&[a.scale, a.rise], &[b.scale, b.rise]) && a.fill.len() == b.fill.len() && a.fill.iter().zip(&b.fill).all(|(a, b)| a.op == b.op && a.operands == b.operands) } diff --git a/crates/fonts/src/pdf.rs b/crates/fonts/src/pdf.rs index f6e81b4b8..0aaed6939 100644 --- a/crates/fonts/src/pdf.rs +++ b/crates/fonts/src/pdf.rs @@ -1154,7 +1154,8 @@ pub fn glyph_unicode(name: &str) -> Option { chars.next().filter(|_| chars.len() == 0) } -/// Code → Unicode for a font: its ToUnicode CMap, else (simple fonts) its encoding. +/// Code → Unicode for a font: its ToUnicode CMap (for a composite font, also the name +/// `/Identity-H` or `/Identity-V`, read as code = Unicode), else (simple fonts) its encoding. fn unicode_map(doc: &Document, font: &Dict, composite: bool) -> Unicode { let mut out = Unicode::default(); if !composite { @@ -1217,14 +1218,44 @@ fn unicode_map(doc: &Document, font: &Dict, composite: bool) -> Unicode { } } } - if let Some(Object::Stream(s)) = font.get(b"ToUnicode").map(|t| doc.resolve(t)).as_deref() - && let Ok(data) = s.decoded() - { - parse_to_unicode(&data, &mut out); + match font.get(b"ToUnicode").map(|t| doc.resolve(t)).as_deref() { + Some(Object::Stream(s)) => { + if let Ok(data) = s.decoded() { + parse_to_unicode(&data, &mut out); + } + } + // `/ToUnicode /Identity-H` is not a stream, as ISO 32000-2 §9.10.3 requires, but CAD + // exporters (WSCAD among them) write it for composite fonts whose two-byte codes are the + // UTF-16 code units. Acrobat shows them as code = Unicode, and so does the renderer's + // text layer; without it the text could be seen but not edited or found. The same rule + // (code = Unicode, no controls or lone surrogates) is in the vendored renderer, + // `Type0Font::char_code_to_unicode` in vendor/hayro-interpret/src/font/cid.rs: keep both alike. + Some(Object::Name(n)) if composite && matches!(n.as_slice(), b"Identity-H" | b"Identity-V") => { + parse_to_unicode(&identity_to_unicode(), &mut out); + } + _ => {} } out } +/// A ToUnicode CMap mapping each two-byte code to the same UTF-16 code unit, leaving out the +/// surrogates (which are not characters on their own) and the controls (C0, DEL and C1, as the +/// renderer's text layer does). No range crosses a last-byte boundary (§9.10.3). +fn identity_to_unicode() -> Vec { + let mut ranges = vec![(0x20u32, 0x7Eu32), (0xA0, 0xFF)]; + ranges.extend((0x01u32..=0xFF).filter(|h| !(0xD8..=0xDF).contains(h)).map(|h| (h << 8, (h << 8) | 0xFF))); + let mut cmap = String::from("1 begincodespacerange <0000> endcodespacerange\n"); + // At most 100 entries per section (§9.7.5.2 limits for CMap files). + for chunk in ranges.chunks(100) { + cmap.push_str(&format!("{} beginbfrange\n", chunk.len())); + for (lo, hi) in chunk { + cmap.push_str(&format!("<{lo:04X}> <{hi:04X}> <{lo:04X}>\n")); + } + cmap.push_str("endbfrange\n"); + } + cmap.into_bytes() +} + #[cfg(test)] mod tests { use pdfcraft_cos::{Dict, Document, Object, Stream}; @@ -1531,6 +1562,41 @@ mod tests { assert_eq!(m.decode(&[0, 1, 0, 7, 0, 2]), "M á"); } + /// WSCAD writes `/ToUnicode /Identity-H` (a name, not a CMap stream) on composite fonts whose + /// codes are UTF-16 code units: the codes read as the same Unicode, as in Acrobat. Controls + /// and lone surrogates map to nothing, and a simple font's name is not a ToUnicode map. + #[test] + fn identity_to_unicode_names_read_codes_as_unicode() { + let doc = Document::new_empty(); + for name in ["Identity-H", "Identity-V"] { + let f = font( + &mut Document::new_empty(), + vec![ + ("Subtype", Object::name("Type0")), + ("BaseFont", Object::name("RobotoMono-Regular")), + ("Encoding", Object::name("Identity-H")), + ("DescendantFonts", Object::Array(vec![Object::Dict(Dict::new())])), + ("ToUnicode", Object::name(name)), + ], + ); + let m = Metrics::from_dict(&doc, &f); + assert_eq!(m.decode(&[0, b'Z', 0, 0xFC, 0, b'r', 0x20, 0xAC, 0x4E, 0x2D]), "Zür€中", "{name}"); + assert_eq!(m.text_of(0x0001), None); + assert_eq!(m.text_of(0x007F), None, "DEL is a control"); + assert_eq!(m.text_of(0x0085), None, "a C1 control"); + assert_eq!(m.text_of(0x00A0).as_deref(), Some("\u{A0}"), "the range resumes after C1"); + assert_eq!(m.text_of(0x007E).as_deref(), Some("~")); + assert_eq!(m.text_of(0xD800), None); + assert_eq!(m.text_of(0xFFFD).as_deref(), Some("\u{FFFD}")); + } + let simple = font( + &mut Document::new_empty(), + vec![("Subtype", Object::name("TrueType")), ("BaseFont", Object::name("Arial")), ("ToUnicode", Object::name("Identity-H"))], + ); + let m = Metrics::from_dict(&doc, &simple); + assert_eq!(m.text_of(0x41).as_deref(), Some("A"), "the encoding still applies"); + } + /// Chrome's subsets leave the glyphs of `/DW` width out of `/W` (#1443): "s" (CID 0x56) is /// still in the font, but nothing is when the font gives no default width to point at. #[test] diff --git a/crates/redact/src/interp.rs b/crates/redact/src/interp.rs index 415382b23..c9a404b93 100644 --- a/crates/redact/src/interp.rs +++ b/crates/redact/src/interp.rs @@ -900,7 +900,10 @@ pub(crate) fn process(doc: &mut Document, scope: &mut Scope<'_>, streams: &[Vec< tm = tlm; } let items: Vec = match o { - b"TJ" => op.operands.first().and_then(Object::as_array).cloned().unwrap_or_default(), + // A surplus array (`[(A)] [(B)] TJ`) is malformed, but the operator still + // takes the one immediately before it, as the renderer and every other + // reader here do (#3569), so a redaction search matches what is drawn. + b"TJ" => op.operands.last().and_then(Object::as_array).cloned().unwrap_or_default(), _ => op.operands.last().cloned().into_iter().collect(), }; // Vertical-writing text is laid out horizontally by `show`, so its glyphs are diff --git a/crates/redact/src/tests.rs b/crates/redact/src/tests.rs index 8bc5ec141..d8c4b0948 100644 --- a/crates/redact/src/tests.rs +++ b/crates/redact/src/tests.rs @@ -2186,3 +2186,13 @@ fn unused_descriptions_in_shared_or_inherited_resources_are_retired() { } mod property_retirement; + +/// `[(FIRSTARR)] [(LASTARR)] TJ` is malformed (a surplus array operand), but the operator still +/// draws only the one immediately before it (LASTARR, 7 glyphs, 35 pt wide from x = 10) -- not +/// the first (FIRSTARR, 8 glyphs), which is never drawn at all, as the renderer agrees (#3569). +#[test] +fn a_tj_surplus_arrays_undrawn_glyphs_are_not_seen_under_a_mark() { + let mut doc = one_page(b"BT /F1 10 Tf 10 100 Td [(FIRSTARR)] [(LASTARR)] TJ ET", "", vec![]); + assert_eq!(under(&mut doc, 0, &[[45.0, 95.0, 50.0, 110.0]]), 0, "nothing is drawn past LASTARR's 35pt width"); + assert_eq!(under(&mut doc, 0, &[[10.0, 95.0, 45.0, 110.0]]), 7, "LASTARR's 7 glyphs, the one the page draws"); +} diff --git a/crates/render/src/raster.rs b/crates/render/src/raster.rs index 4fd044086..62f2ed99c 100644 --- a/crates/render/src/raster.rs +++ b/crates/render/src/raster.rs @@ -7966,4 +7966,36 @@ trailer << /Root 1 0 R >> // #1201: /DefaultGray, /DefaultRGB and /DefaultCMYK (vendored hayro-interpret patch 38). mod default_colour_spaces; + + /// #3109 / #3116: clipping-only text (`7 Tr`) installs a clip for subsequent painting + /// but draws no pixels itself. + #[test] + fn test_clip_only_text_raster_preserves_clipping_without_painting() { + let pdf_bytes = |content: &str| { + Arc::new( + format!( + "%PDF-1.7\n1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj\n2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >> endobj\n3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 200 100] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >> endobj\n4 0 obj << /Length {} >> stream\n{content}\nendstream endobj\n5 0 obj << /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >> endobj\ntrailer << /Root 1 0 R >>\n%%EOF", + content.len() + ) + .into_bytes(), + ) + }; + let render = |content: &str| { + let mut r = PageRenderer::new(pdf_bytes(content), RenderConfig::default()); + let p = r.render(RenderRequest { page: 0, kind: RequestKind::Pixels, tile: None, scale: 1.0, tag: 0, ..Default::default() }); + assert!(p.error.is_none(), "{content}: {:?}", p.error); + p + }; + let white = [255, 255, 255, 255]; + + // The text alone paints nothing. + let p = render("BT /F1 40 Tf 7 Tr 20 60 Td (Hi) Tj ET"); + assert!(p.rgba.as_chunks::<4>().0.iter().all(|c| *c == white), "7 Tr alone leaves the page blank"); + + // A fill afterwards shows only through the text-shaped clip. + let p = render("q BT /F1 40 Tf 7 Tr 20 60 Td (Hi) Tj ET 0 0 1 rg 0 0 200 100 re f Q"); + let pixels = p.rgba.as_chunks::<4>().0; + assert!(pixels.contains(&[0, 0, 255, 255]), "fill shows through the 7 Tr clip"); + assert!(pixels.contains(&white), "the clip is text-shaped, not the whole page"); + } } diff --git a/crates/render/src/text.rs b/crates/render/src/text.rs index 6784f14eb..1e989714f 100644 --- a/crates/render/src/text.rs +++ b/crates/render/src/text.rs @@ -38,6 +38,22 @@ impl TextFilter { mod search; +/// Whether `(x, y)` lies in or on the convex quadrilateral `quad`, whichever way it winds. A +/// quad with a NaN corner, or no area, holds nothing. +fn quad_contains(quad: &[[f32; 2]; 4], x: f32, y: f32) -> bool { + let (mut positive, mut negative) = (false, false); + for (k, a) in quad.iter().enumerate() { + let b = quad[(k + 1) % 4]; + let cross = (b[0] - a[0]) * (y - a[1]) - (b[1] - a[1]) * (x - a[0]); + positive |= cross > 0.0; + negative |= cross < 0.0; + if cross.is_nan() { + return false; + } + } + positive != negative +} + /// One glyph on the page. #[derive(Clone, Debug, PartialEq)] pub struct TextGlyph { @@ -160,7 +176,26 @@ impl PageText { } /// Index of the glyph nearest to a view-space point (for selection anchors). + /// + /// A glyph whose drawn em box (`quad`) holds the point wins. Skewed or turned text has bounding + /// boxes wider than the glyphs, so the box of the letter before overlaps this one and would + /// otherwise be found first (#3693). Failing that, the nearest bounding box wins, and a tie + /// goes to the glyph whose centre is closer. pub fn nearest(&self, x: f32, y: f32) -> Option { + let centre_distance = |g: &TextGlyph| { + let (cx, cy) = (g.quad.iter().map(|q| q[0]).sum::() / 4.0, g.quad.iter().map(|q| q[1]).sum::() / 4.0); + (cx - x).powi(2) + (cy - y).powi(2) + }; + let inside = self + .glyphs + .iter() + .enumerate() + .filter(|(_, g)| quad_contains(&g.quad, x, y)) + .min_by(|a, b| centre_distance(a.1).total_cmp(¢re_distance(b.1))) + .map(|(i, _)| i); + if inside.is_some() { + return inside; + } self.glyphs .iter() .enumerate() @@ -180,10 +215,10 @@ impl PageText { 0.0 }; // Prefer the same line: vertical distance weighs more. - (i, dx + dy * 3.0) + (i, dx + dy * 3.0, centre_distance(g)) }) - .min_by(|a, b| a.1.total_cmp(&b.1)) - .map(|(i, _)| i) + .min_by(|a, b| a.1.total_cmp(&b.1).then(a.2.total_cmp(&b.2))) + .map(|(i, _, _)| i) } /// The word around glyph `i` as its first and last glyph (double-click): it stops at word @@ -549,6 +584,13 @@ impl TextDevice { } } +/// Whether `PDFCRAFT_TEXT_DEBUG` is set. `draw_glyph` runs once per glyph and `getenv` scans the +/// whole environment under a lock, so read it once. +fn text_debug() -> bool { + static DEBUG: std::sync::OnceLock = std::sync::OnceLock::new(); + *DEBUG.get_or_init(|| std::env::var_os("PDFCRAFT_TEXT_DEBUG").is_some()) +} + impl<'a> Device<'a> for TextDevice { fn set_soft_mask(&mut self, _: Option>) {} fn set_blend_mode(&mut self, _: BlendMode) {} @@ -567,7 +609,7 @@ impl<'a> Device<'a> for TextDevice { Some(hayro::hayro_interpret::hayro_cmap::BfString::String(s)) => s, None if replaced => String::new(), None => { - if std::env::var_os("PDFCRAFT_TEXT_DEBUG").is_some() { + if text_debug() { eprintln!( "NOUNICODE {} {:?}", if matches!(glyph, Glyph::Type3(_)) { "t3" } else { "ol" }, @@ -583,7 +625,7 @@ impl<'a> Device<'a> for TextDevice { // Glyph space → view space. Glyph space uses 1000 units per em for outline glyphs; the em // box spans descender (−200) to ascender (800), the advance gives the width. let t = transform * glyph_transform; - if std::env::var_os("PDFCRAFT_TEXT_DEBUG").is_some() { + if text_debug() { let kind = if matches!(glyph, Glyph::Type3(_)) { "t3" } else { "ol" }; eprintln!("{kind} {text:?} transform={:?} glyph_transform={:?}", transform.as_coeffs(), glyph_transform.as_coeffs()); } @@ -1420,13 +1462,15 @@ pub fn layout(glyphs: Vec) -> PageText { fn arrange(glyphs: &[TextGlyph]) -> (Vec, Vec, Vec) { // Drop "fake bold" duplicates: the same character redrawn at (almost) the same place. A redraw // also overlaps most of its own width: a narrow letter in condensed text (80 Tz and below) sits - // closer than a fifth of its height to its double, as the second "l" of "Alle" (#1467). + // closer than a fifth of its height to its double, as the second "l" of "Alle" (#1467), and + // OCR layers squeeze words to 20 Tz and below (#2457). Only a zero-width mark, which has no + // width to overlap, is matched by its height alone. let mut keep = vec![true; glyphs.len()]; for i in 1..glyphs.len() { let g = &glyphs[i]; let h = (g.rect[3] - g.rect[1]).max(0.1); let w = g.rect[2] - g.rect[0]; - let dx = if w > 0.05 * h { (h * 0.2).min(w * 0.5) } else { h * 0.2 }; + let dx = if w > 0.0 { (h * 0.2).min(w * 0.5) } else { h * 0.2 }; for j in (i.saturating_sub(4)..i).rev() { let p = &glyphs[j]; if keep[j] && p.text == g.text && (p.rect[0] - g.rect[0]).abs() < dx && (p.rect[1] - g.rect[1]).abs() < h * 0.2 { @@ -1599,8 +1643,25 @@ fn arrange(glyphs: &[TextGlyph]) -> (Vec, Vec, Vec) { place_glyphs(&glyphs, &height, &segs[si], &mut sc); let h = segs[si].h; // Word gaps relative to this line's typical letter gap (handles tracking). + // Long runs of repeated punctuation belong to leaders, not letter tracking: a TOC leader + // must not raise the threshold above the word gaps in its title (#3086). sc.gaps.clear(); - sc.gaps.extend(sc.placed.windows(2).map(|w| glyphs[w[1]].rect[0] - glyphs[w[0]].rect[2])); + sc.gaps.extend(sc.placed.windows(2).enumerate().filter_map(|(w, pair)| { + let &[p, c] = pair else { return None }; + let (p, c) = (glyphs.get(p)?, glyphs.get(c)?); + if p.text == c.text && p.text.chars().all(|ch| !ch.is_alphanumeric() && !ch.is_whitespace()) { + // Keep short tracked punctuation, including an ellipsis with a final period. + // Treat five or more matching glyphs as a leader. Three neighbours on each + // side suffice to classify this pair, bounding work even on long runs. + let same = |&&i: &&usize| glyphs.get(i).is_some_and(|g| g.text == p.text); + let before = sc.placed.get(..w).unwrap_or_default().iter().rev().take(3).take_while(same).count(); + let after = sc.placed.get(w.saturating_add(2)..).unwrap_or_default().iter().take(3).take_while(same).count(); + if before + after >= 3 { + return None; + } + } + Some(c.rect[0] - p.rect[2]) + })); sc.gaps.sort_by(f32::total_cmp); let typical = sc.gaps.get(sc.gaps.len() / 3).copied().unwrap_or(0.0).max(0.0); let threshold = (typical + h * WORD_GAP).max(h * WORD_GAP); @@ -1703,6 +1764,37 @@ mod tests { } } + /// WSCAD writes `/ToUnicode /Identity-H`, a name rather than a CMap stream, on Type 0 fonts + /// whose two-byte codes are UTF-16 code units. The codes read as that Unicode text (as in + /// Acrobat), so the page's text can be selected and found; before, it had none. Controls (here a + /// DEL and a C1 control at the line ends) are not text. + #[test] + fn identity_to_unicode_names_give_text() { + use lopdf::{Document, Object, Stream, dictionary}; + for name in ["Identity-H", "Identity-V"] { + let mut doc = Document::with_version("1.4"); + let cid = doc.add_object(dictionary! { + "Type" => "Font", "Subtype" => "CIDFontType2", "BaseFont" => "RobotoMono-Regular", + "CIDSystemInfo" => dictionary! { "Registry" => Object::string_literal("Adobe"), "Ordering" => Object::string_literal("Identity"), "Supplement" => 0 }, + "DW" => 600, "CIDToGIDMap" => "Identity" + }); + let font = doc.add_object(dictionary! { "Type" => "Font", "Subtype" => "Type0", "BaseFont" => "RobotoMono-Regular", "Encoding" => "Identity-H", "DescendantFonts" => vec![Object::Reference(cid)], "ToUnicode" => name }); + let operators = "BT /F1 12 Tf 1 0 0 1 20 250 Tm <004400650063006B0062006C006100740074007F> Tj 1 0 0 1 20 200 Tm <005A00FC0072006900630068002020AC0085> Tj ET"; + let content = doc.add_object(Stream::new(dictionary! {}, operators.as_bytes().to_vec())); + let pages = doc.new_object_id(); + let page = doc.add_object(dictionary! { "Type" => "Page", "Parent" => pages, "MediaBox" => vec![Object::Integer(0), Object::Integer(0), Object::Integer(300), Object::Integer(300)], "Resources" => dictionary! {"Font" => dictionary! {"F1" => font}}, "Contents" => content }); + doc.objects.insert(pages, dictionary! {"Type" => "Pages", "Kids" => vec![Object::Reference(page)], "Count" => 1}.into()); + let catalog = doc.add_object(dictionary! {"Type" => "Catalog", "Pages" => pages}); + doc.trailer.set("Root", catalog); + let mut bytes = Vec::new(); + doc.save_to(&mut bytes).unwrap(); + let pdf = Pdf::new(std::sync::Arc::new(bytes)).unwrap(); + let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap(); + assert_eq!(text.plain_text(), "Deckblatt\nZürich €", "{name}"); + assert_eq!(text.find("zürich"), vec![9..15], "{name}: glyphs 9 to 14"); + } + } + /// One page per text matrix, each showing `codes` in Helvetica whose `/ToUnicode` maps the /// codes in `unicode` (a `bfchar` body) to several characters. fn ligature_pages(unicode: &str, codes: &str, matrices: &[&str]) -> Pdf { @@ -1869,6 +1961,84 @@ mod tests { } } + /// Issue #3109: text drawn in clipping mode (`7 Tr`) sets the clip for a subsequent fill, + /// shading or image without painting the glyphs directly, and must still reach `TextDevice` + /// with the same boxes and search ranges as modes 0–6. + #[test] + fn clipping_mode_text_is_extracted_and_searchable_like_other_rendering_modes() { + let fill_through = " 0 0 1 rg 0 0 300 300 re f"; + let mut pages: Vec = + (0..=7).map(|mode| format!("q BT /F1 24 Tf {mode} Tr 72 150 Td () Tj [(Hello) -200 (World)] TJ ET{fill_through} Q")).collect(); + // Page 8: /ActualText wrapping 7 Tr clip-only text. + pages.push(format!("q /Span <> BDC BT /F1 24 Tf 7 Tr 72 150 Td (Glyphs) Tj ET EMC{fill_through} Q")); + // Page 9: 0 Tr underpainting followed by 7 Tr at the same position (outline/gradient overlay) deduplicates. + pages.push(format!("BT /F1 24 Tf 0 Tr 72 150 Td (Hello World) Tj ET q BT /F1 24 Tf 7 Tr 72 150 Td (Hello World) Tj ET{fill_through} Q")); + // Page 10: 90° turned 7 Tr text. + pages.push(format!("q BT /F1 24 Tf 7 Tr 0 1 -1 0 150 100 Tm (Hello World) Tj ET{fill_through} Q")); + // Page 11: 7 Tr without any fill-through (visually blank, still extractable). + pages.push("BT /F1 24 Tf 7 Tr 72 150 Td (Hello World) Tj ET".into()); + + let refs: Vec<&str> = pages.iter().map(String::as_str).collect(); + let pdf = marked_pages(&refs); + let control = marked_text(&pdf, 0); + assert_eq!(control.plain_text(), "Hello World"); + let control_rects = control.line_rects(0..5); + assert_eq!(control_rects.len(), 1); + for mode in 0..=7 { + let text = marked_text(&pdf, mode); + assert_eq!(text.plain_text(), "Hello World", "mode {mode} Tr"); + assert_eq!(text.find("hello"), vec![0..5], "mode {mode} Tr"); + assert_eq!(text.find_opts("Hello", true, true), vec![0..5], "mode {mode} Tr"); + assert_eq!( + text.find_with(|chars| chars.starts_with(&['H', 'e', 'l', 'l', 'o']).then_some(0..5).into_iter().collect()), + vec![0..5], + "mode {mode} Tr" + ); + assert_eq!(text.line_rects(0..5), control_rects, "mode {mode} Tr"); + assert_eq!(text.glyphs, control.glyphs, "mode {mode} Tr"); + } + let actual = marked_text(&pdf, 8); + assert_eq!(actual.plain_text(), "Hello World"); + assert_eq!(actual.find("Hello"), vec![0..1]); + + let deduped = marked_text(&pdf, 9); + assert_eq!(deduped.plain_text(), "Hello World"); + + let turned = marked_text(&pdf, 10); + assert_eq!(turned.plain_text(), "Hello World"); + assert_eq!(turned.find("World"), vec![6..11]); + + let unfilled = marked_text(&pdf, 11); + assert_eq!(unfilled.plain_text(), "Hello World"); + + // Visual check: page 7 (`7 Tr` + blue fill-through) paints blue only inside the clipped + // glyph outlines, whereas page 11 (`7 Tr` alone) paints no ink at all. + let mut renderer = crate::PageRenderer::new(std::sync::Arc::new(pdf.data().as_ref().to_vec()), crate::RenderConfig::default()); + let filled_px = renderer.render(crate::RenderRequest { page: 7, kind: crate::RequestKind::Pixels, scale: 1.0, ..Default::default() }); + assert!(filled_px.error.is_none()); + let blue_pixels = filled_px.rgba.as_chunks::<4>().0.iter().filter(|p| p[0] < 32 && p[1] < 32 && p[2] > 200).count(); + let white_pixels = filled_px.rgba.as_chunks::<4>().0.iter().filter(|p| p[..3] == [255, 255, 255]).count(); + assert!(blue_pixels > 50 && white_pixels > 80_000, "blue={blue_pixels}, white={white_pixels}"); + + let unfilled_px = renderer.render(crate::RenderRequest { page: 11, kind: crate::RequestKind::Pixels, scale: 1.0, ..Default::default() }); + assert!(unfilled_px.error.is_none()); + assert!(unfilled_px.rgba.as_chunks::<4>().0.iter().all(|p| p[..3] == [255, 255, 255])); + } + + #[test] + fn clipping_mode_filter_keeps_titles_and_excludes_ocr() { + let pdf = marked_pages(&[ + "q BT /F1 24 Tf 7 Tr 30 200 Td (Clipped heading) Tj ET 0 0 1 rg 0 0 300 300 re f Q BT /F1 12 Tf 3 Tr 30 100 Td (Hidden OCR) Tj ET", + "q /Span <> BDC BT /F1 24 Tf 7 Tr 30 200 Td (Clipped heading) Tj ET EMC 0 0 1 rg 0 0 300 300 re f Q", + ]); + assert!(marked_text(&pdf, 0).plain_text().contains("Hidden OCR")); + for (page, expected) in [(0, "Clipped heading"), (1, "Accessible heading")] { + let text = extract_page_filtered(&pdf, page, &InterpreterSettings::default(), TextFilter { images: true, ..Default::default() }).unwrap(); + assert_eq!(text.plain_text(), expected); + assert!(!text.find("heading").is_empty()); + } + } + /// One page per `(encoding, content)`, in Helvetica (`/F1`). fn encoded_pages(pages: &[(lopdf::Object, &[u8])]) -> Pdf { use lopdf::{Document, Object, Stream, dictionary}; @@ -1923,6 +2093,64 @@ mod tests { assert_eq!(texts(2), ["a", "\u{ad}", "b", "\u{a0}", "c"], "a named glyph keeps its glyph list value"); } + /// #3169, #3170: glyph names outside the Adobe Glyph List still read as text. Ligature + /// (`f_i`) and suffixed (`hyphen.case`) names go by the glyph list specification's full + /// mapping. A name that only numbers the glyph (`c251` small caps, `C39`, `C174`, `C39.sc`) + /// reads as nothing, not as a guess from the code (review on #3239: with `/C1` at 233, + /// StandardEncoding's 233 would read `Ø` for a glyph drawn as `é`), and so do a ZapfDingbats + /// name (`a12`) and a name with no meaning left (`foo`). + #[test] + fn glyph_names_outside_the_glyph_list_still_read_as_text() { + use lopdf::{Object, dictionary}; + let name = |n: &str| Object::Name(n.as_bytes().to_vec()); + let differences = Object::Dictionary(dictionary! { + "Type" => "Encoding", + "Differences" => vec![ + Object::Integer(30), name("f_i"), name("f_l"), Object::Integer(2), name("hyphen.case"), + Object::Integer(39), name("C39"), Object::Integer(97), name("c251"), Object::Integer(174), name("C174"), + Object::Integer(88), name("a12"), name("foo"), name("uni00660069"), + Object::Integer(61), name("C39.sc"), Object::Integer(233), name("C1"), + ], + }); + // One glyph per line: the test font has none of these glyphs, so they have no width, and + // on one line a later `fi` would sit on the first and be dropped as its fake-bold copy. + let mut content = b"BT /F1 20 Tf 20 290 Td".to_vec(); + for code in *b"\x1e\x1f\x02'a\xaeXYZ=\xe9" { + content.extend_from_slice(b" ("); + content.push(code); + content.extend_from_slice(b") Tj 0 -25 Td"); + } + content.extend_from_slice(b" ET"); + let pdf = encoded_pages(&[(differences, &content[..])]); + let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap().plain_text(); + let lines: Vec<&str> = text.lines().map(str::trim).filter(|l| !l.is_empty()).collect(); + assert_eq!(lines, ["fi", "fl", "-", "fi"], "{text:?}"); + } + + /// Review on #3239: the glyph list specification's `uni` and `u` forms take uppercase hex + /// digits only, and map a name with lowercase ones to nothing ("uni20ac" in its examples). + /// Names upstream's own lookup can't read (several groups, a suffix) show it: lowercase reads + /// as nothing, uppercase as its characters. A bare `uni20ac` or `u1040c` is still read by + /// upstream's lenient `unicode_from_name`, outside this mapping. + #[test] + fn glyph_names_with_lowercase_hex_read_as_nothing() { + use lopdf::{Object, dictionary}; + let name = |n: &str| Object::Name(n.as_bytes().to_vec()); + let differences = Object::Dictionary(dictionary! { + "Type" => "Encoding", + "Differences" => vec![Object::Integer(65), name("uni20ac0308"), name("uni20AC0308"), name("u1040c.alt"), name("u1040C.alt"), name("f_uni20ac"), name("f_uni20AC")], + }); + let pdf = encoded_pages(&[( + differences, + &b"BT /F1 20 Tf 20 180 Td (A) Tj 0 -25 Td (B) Tj 0 -25 Td (C) Tj 0 -25 Td (D) Tj 0 -25 Td (E) Tj 0 -25 Td (F) Tj ET"[..], + )]); + // One glyph per line, so each name's text stays its own (and a name read as nothing + // leaves no line). + let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap().plain_text(); + let lines: Vec<&str> = text.lines().map(str::trim).filter(|l| !l.is_empty()).collect(); + assert_eq!(lines, ["\u{20ac}\u{308}", "\u{1040c}", "f", "f\u{20ac}"], "{text:?}"); + } + #[test] fn narrow_ligatures_keep_every_letter() { // Split evenly, this "ffi" put its two "f"s so close that the second was dropped as a @@ -2052,6 +2280,62 @@ mod tests { TextGlyph { text: t.into(), rect, quad: [[x0, y0], [x1, y0], [x1, rect[3]], [x0, rect[3]]], direction: [1.0, 0.0] } } + /// A glyph whose em box is `w` × `h` at `origin`, advancing along `dir` with its up edge along + /// `up` (a unit step of each), as a sheared or turned text matrix draws it. + fn tilted(t: &str, origin: [f32; 2], dir: [f32; 2], up: [f32; 2], w: f32, h: f32) -> TextGlyph { + let p = |a: f32, b: f32| [origin[0] + dir[0] * a + up[0] * b, origin[1] + dir[1] * a + up[1] * b]; + let quad = [p(0.0, 0.0), p(w, 0.0), p(w, h), p(0.0, h)]; + let xs = quad.map(|q| q[0]); + let ys = quad.map(|q| q[1]); + let rect = [ + xs.iter().copied().fold(f32::MAX, f32::min), + ys.iter().copied().fold(f32::MAX, f32::min), + xs.iter().copied().fold(f32::MIN, f32::max), + ys.iter().copied().fold(f32::MIN, f32::max), + ]; + TextGlyph { text: t.into(), rect, quad, direction: dir } + } + + /// #3693: pointing at the middle of a glyph in sheared or turned text finds that glyph, not + /// the one before it whose bounding box overlaps it. + #[test] + fn nearest_finds_the_glyph_under_the_pointer_in_sheared_and_turned_text() { + let s = std::f32::consts::FRAC_1_SQRT_2; + // (writing direction, up edge) in view space (y down): upright, sheared 0.35, turned 45°. + for (dir, up) in [([1.0, 0.0], [0.0, -1.0]), ([1.0, 0.0], [0.35, -1.0]), ([s, -s], [-s, -s])] { + let mut advance = 0.0; + let glyphs: Vec = "film in the list of tiny jilt flit" + .chars() + .map(|c| { + let w = if "filjt ".contains(c) { 3.0 } else { 6.0 }; + let g = tilted(&c.to_string(), [50.0 + dir[0] * advance, 300.0 + dir[1] * advance], dir, up, w, 14.0); + advance += w; + g + }) + .collect(); + let t = layout(glyphs.clone()); + assert_eq!(t.glyphs.len(), glyphs.len(), "layout kept every glyph"); + for (i, g) in glyphs.iter().enumerate() { + let cx = g.quad.iter().map(|q| q[0]).sum::() / 4.0; + let cy = g.quad.iter().map(|q| q[1]).sum::() / 4.0; + assert_eq!(t.nearest(cx, cy), Some(i), "dir {dir:?} up {up:?}: glyph {i} ({})", g.text); + } + // Past either end of the line, the end glyph is still the nearest. + let (first, last) = (&glyphs[0], &glyphs[glyphs.len() - 1]); + assert_eq!(t.nearest(first.quad[0][0] - dir[0] * 8.0, first.quad[0][1] - dir[1] * 8.0), Some(0)); + assert_eq!(t.nearest(last.quad[1][0] + dir[0] * 8.0, last.quad[1][1] + dir[1] * 8.0), Some(glyphs.len() - 1)); + } + } + + #[test] + fn a_degenerate_or_nan_quad_holds_no_point() { + assert!(!quad_contains(&[[1.0, 1.0]; 4], 1.0, 1.0)); + assert!(!quad_contains(&[[0.0, 0.0], [1.0, 0.0], [f32::NAN, 1.0], [0.0, 1.0]], 0.5, 0.5)); + assert!(quad_contains(&[[0.0, 0.0], [4.0, 0.0], [4.0, 4.0], [0.0, 4.0]], 4.0, 2.0)); + // Wound the other way. + assert!(quad_contains(&[[0.0, 0.0], [0.0, 4.0], [4.0, 4.0], [4.0, 0.0]], 2.0, 2.0)); + } + #[test] fn lines_words_and_search() { // "Hello world" on line 1, "Next" on line 2. @@ -2175,6 +2459,49 @@ mod tests { } } + /// #2457: OCR text layers squeeze each word into its box down to 20 Tz and below, where a + /// narrow letter is less than a twentieth of its height wide. Doubled letters, and letters + /// repeated two or three places on, are still letters there, not redraws. + #[test] + fn ocr_condensed_double_letters_are_kept() { + // Helvetica advances, in thousandths of an em. + let adv = |c: char| match c { + 'M' => 833.0, + 'K' | 'B' => 667.0, + 'W' => 944.0, + 'I' | 't' | 'f' => 278.0, + 'i' | 'l' => 222.0, + 'm' => 833.0, + 'r' => 333.0, + 's' => 500.0, + _ => 556.0, + }; + for scale in [0.2, 0.1, 0.05] { + for word in ["Hallo", "Mitte", "Wille", "Kommission", "Illustration", "still", "Brunnen", "Kaffee"] { + let mut v = Vec::new(); + let mut x = 10.0; + for c in word.chars() { + let w = adv(c) / 100.0 * scale; + v.push(g(&c.to_string(), x, 10.0, x + w)); + x += w; + } + assert_eq!(layout(v.clone()).plain_text(), word, "at {scale} horizontal scaling"); + // Each letter drawn twice at the same place is still one letter. + let bold: Vec = v.iter().flat_map(|c| [c.clone(), c.clone()]).collect(); + assert_eq!(layout(bold).plain_text(), word, "fake bold at {scale} horizontal scaling"); + // And redrawn a quarter of its own width to the right (the match reaches half of it). + let shifted: Vec = v + .iter() + .flat_map(|c| { + let d = (c.rect[2] - c.rect[0]) * 0.25; + [c.clone(), g(&c.text, c.rect[0] + d, 10.0, c.rect[2] + d)] + }) + .collect(); + assert_eq!(layout(shifted).plain_text(), word, "shifted fake bold at {scale} horizontal scaling"); + } + } + } + #[test] fn two_columns_read_left_then_right_even_if_drawn_interleaved() { let mut v = Vec::new(); @@ -2195,6 +2522,92 @@ mod tests { assert_eq!(layout(v).plain_text(), "CHAPTER FOUR"); } + #[test] + fn toc_leaders_do_not_hide_word_gaps() { + use lopdf::{Document, Object, Stream, dictionary}; + for dots in [0, 5, 10, 20, 40, 60] { + for (words, expected) in [ + ("(Was)-333(ist)-333(OTRS?)", "Was ist OTRS?"), + ("(Das)-333(OTRS)-333(Web-Interface)", "Das OTRS Web-Interface"), + ("(Neuen)-333(Zugang)-333(beantragen)", "Neuen Zugang beantragen"), + ] { + // The issue's LaTeX TJ spacing, without embedding the corpus PDF or a font asset. + let mut doc = Document::with_version("1.7"); + let font = doc.add_object(dictionary! { "Type" => "Font", "Subtype" => "Type1", "BaseFont" => "Times-Roman" }); + let operators = format!("BT /F1 10 Tf 10 100 Td [{words}-441{}-1663(7)] TJ ET", "-500(.)".repeat(dots)); + let content = doc.add_object(Stream::new(dictionary! {}, operators.into_bytes())); + let pages = doc.new_object_id(); + let page = doc.add_object(dictionary! { + "Type" => "Page", "Parent" => pages, + "MediaBox" => vec![0.into(), 0.into(), 800.into(), Object::Integer(200)], + "Resources" => dictionary! { "Font" => dictionary! { "F1" => font } }, "Contents" => content + }); + doc.objects.insert(pages, dictionary! { "Type" => "Pages", "Kids" => vec![Object::Reference(page)], "Count" => 1 }.into()); + let catalog = doc.add_object(dictionary! { "Type" => "Catalog", "Pages" => pages }); + doc.trailer.set("Root", catalog); + let mut bytes = Vec::new(); + doc.save_to(&mut bytes).unwrap(); + let pdf = Pdf::new(std::sync::Arc::new(bytes)).unwrap(); + let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap(); + assert_eq!(text.plain_text(), format!("{expected} {}7", ". ".repeat(dots)), "{dots} leader dots"); + let hits = text.find_opts(expected, true, true); + assert_eq!(hits.len(), 1, "{expected:?}, {dots} leader dots"); + assert_eq!(text.text_of(hits[0].clone()), expected); + assert_eq!(text.line_rects(hits[0].clone()).len(), 1); + assert_eq!(text.find(&expected.to_lowercase()), hits); + assert!(text.find(&expected.replace(' ', "")).is_empty()); + } + } + } + + #[test] + fn short_tracked_ellipses_remain_searchable() { + for dots in [3, 4] { + let mut v = Vec::new(); + for i in 0..dots { + let x = 10.0 + i as f32 * 4.0; + v.push(g(".", x, 10.0, x + 2.0)); + } + let ellipsis = ".".repeat(dots); + let text = layout(v.clone()); + assert_eq!(text.plain_text(), ellipsis); + assert_eq!(text.find(&ellipsis), vec![0..dots]); + v.insert(0, g("A", 0.0, 10.0, 6.0)); + let x = 12.0 + dots as f32 * 4.0; + v.push(g("B", x, 10.0, x + 6.0)); + let text = layout(v); + let expected = format!("A {ellipsis} B"); + assert_eq!(text.plain_text(), expected); + assert_eq!(text.find(&expected), vec![0..dots + 2]); + } + } + + #[test] + fn leaders_preserve_tracking_in_letters_numbers_and_punctuation() { + for advance in [9.0, 12.0] { + for leader in [".", "-", "…"] { + let mut v = Vec::new(); + word(&mut v, "CHAPTER-1", 10.0, 10.0, advance); + let x = 10.0 + 9.0 * advance + 5.0; + word(&mut v, "1.4", x, 10.0, advance); + let mut x = x + 3.0 * advance + 5.0; + for _ in 0..60 { + v.push(g(leader, x, 10.0, x + 2.5)); + x += 11.0; + } + let text = layout(v); + assert_eq!(text.plain_text(), format!("CHAPTER-1 1.4 {}", vec![leader; 60].join(" "))); + assert_eq!(text.find("chapter-1 1.4"), vec![0..12]); + } + } + // Decimal punctuation and repeated letters/digits still use their original tracking estimate. + for value in ["1.4", "1111", "llll"] { + let mut v = Vec::new(); + word(&mut v, value, 10.0, 10.0, 12.0); + assert_eq!(layout(v).plain_text(), value); + } + } + #[test] fn rtl_runs_come_out_in_logical_order_and_cjk_has_no_spaces() { let mut v = Vec::new(); diff --git a/docs/gaps.md b/docs/gaps.md index 2682d090a..3c74282af 100644 --- a/docs/gaps.md +++ b/docs/gaps.md @@ -1,5 +1,6 @@ # Where PdfCraft falls short of Acrobat Pro +> **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-19 · **Change:** minor (gap 2: composite fonts with `/ToUnicode /Identity-H` read as Unicode, hayro-interpret patch 44; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (gap 12: reading order is scored, not just word F1) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-14 · **Last updated:** 2026-10-14 · **Change:** minor (M1: raw-deflate fallback preserves decoding-limit errors; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-12 · **Last updated:** 2026-10-13 · **Change:** minor (gap 14: Vietnamese catalog shipped; Hindi/Indonesian/Korean and Arabic RTL remain) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) @@ -382,6 +383,7 @@ update the numbers. | Date | Change | Summary | |---|---|---| +| 2026-10-19 | minor | Gap 2: a Type 0 font whose `/ToUnicode` is the name `/Identity-H` or `/Identity-V` (WSCAD) reads its codes as Unicode, so such pages have text to select, find and edit (hayro-interpret patch 44, `view.render-cid-fonts`). Estimates unchanged. | | 2026-10-15 | minor | Gap 12: the text oracle scores reading order as well as word F1, which is a bag of words and cannot see order; the first measurement separated 18 improvements from 6 regressions in a change F1 called unchanged on 26 of 29 files. Estimates unchanged. | | 2026-10-14 | minor | M1: raw-deflate fallback preserves decoding-limit errors in strict and tolerant decoding; synthetic filter and COS stream regressions. Estimates unchanged. | | 2026-10-15 | patch | Gap 24: record localization and installers as partial (unusable Chinese, Telugu and Arabic; unsigned Linux packages) | diff --git a/docs/roadmap.md b/docs/roadmap.md index 5db284cf4..201f90126 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -5,7 +5,11 @@ > **Last reviewed:** 2026-10-20 | **Last updated:** 2026-10-20 | **Change:** minor (M2, #3574: named form properties retain page origin and own dictionary boundaries; estimates unchanged) > **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** minor (M3, #3541/#3620: comment hover follows individual marked lines and original annotation paint order; estimates unchanged) +> **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** patch (PR #3348 review: retain short tracked punctuation in gap estimates; estimates unchanged) + > **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** minor (M5, PR #1971: bounded nested FDF export excludes source page/page-tree dictionaries; refreshed against upstream 4db6ac4a; estimates unchanged) +> **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** patch (M2 #3086: dotted TOC leaders no longer hide word spaces; estimates unchanged) + > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M14/#2383 GPU-limit candidate; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-20 - **Last updated:** 2026-10-20 - **Change:** minor (M6, #3502: form detection uses the existing content-stream boundary policy; estimates unchanged) > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M7: the Edit text caret is as tall as the row it is on; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) @@ -123,6 +127,7 @@ total matches the parity estimate. | M0 | Skeleton: workspace, xtask gates, CI | 15–30 | 98% | 1–2 | GitHub workflow, `deny.toml` (licence audit of every dependency), parity checklist (826 features, `xtask parity`) done. Missing: remaining crate stubs, testkit/oracle crates | | M1 | COS: filters, crypt, parser, xref, writer | 120–200 | 80% | 20–40 | Done:
- filters and crypt: every standard-security revision R2–R6 (RC4, AES-128/256), SASLprep, permissions, creating encryption;
- cos: parse and repair (including invalid xref object-number ranges), decoded-stream limits including unfiltered data, decrypt on load, re-encrypt on save, incremental and full writing.
Corpus: open/edit/save passes on 958 files, and all 7 password-protected files open.
Full saves now pack objects into compressed object streams. An unchanged incremental save of a classic table writes a zero-length cross-reference subsection instead of a section with no subsections (#1339). A full rewrite keeps unknown trailer entries, including objects reachable only from them (#1338). Fuzzing runs nightly (`xtask fuzz`). Raw-deflate fallback preserves decoding-limit errors, including through tolerant COS stream decoding. Missing: ≥ 250 tests, own image codecs | | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. Forms without Resources use page Properties for ActualText and layer visibility; own dictionaries stop fallback. Oversized missing inline-image samples, including header and terminator overhead, no longer pin the content window (#3107 / PR #3438); normal large streamed images retain deterministic sample boundaries. Radial luminosity fades have synthetic coverage for the existing #2132 mask guard, including per-pixel references, antialiased clips, previews, scales and tiles; no additional production fix was needed. | +| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. Dotted TOC leaders no longer inflate letter tracking or hide word spaces (#3086); phrase search and selection are covered. Short tracked ellipses retain their gap samples and remain searchable. | | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. The renderer keeps its one-point crop-size floor in physical points after UserUnit scaling, with regression coverage for crop origins, rotations and f32 reciprocal rounding. Existing unit policy and nearly-zero fallback are retained. | | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings | | M3 | Viewer app (native + web) | 80–150 | 90% | 10–20 | Acrobat-style shell, find, select, panels, tiles, web build, UI control channel for agents (opt-in). #3808: the Pages inspector offers a 50–150% thumbnail slider and buttons, also available through `ui.set`. Missing: 60 fps test on a 500-page document, snapshot tests of every panel
Done since: fit visible, document title in the window; Linux middle-button scrolling; shared three-state theme preferences in the toolbar, View menu and control channel; numeric Linux portal events and idle wakeups completing #881, with native appearance inheritance retained (#446); late browser startup downloads preserve the current document and Save target (#171). Selected-text Copy in Edit and the canvas context menu is covered by focused regressions and native clipboard checks (#1225); document copy restrictions apply to the floating action too. #1364 candidate: absolute local GoToR links ask before target access; headless callers explicitly opt in. Native-opening candidate: chosen tools survive preparation and password retries; startup intake waits for pending preparation. Page-label navigation displays logical labels, retains the current page on unchanged submission and refreshes focused same-document replacements; clean file-watcher reload preserves the chosen scrolling mode, preventing a Single Page reader from jumping to page 1 during replacement. Synthetic control regressions cover these behaviors. #3149: page-list changes discard render demand based on old geometry; actual raster dimensions are checked against the GPU limit before upload. Synthetic crash and continued-rendering regressions cover the replacement path. Save-prompt shortcuts yield to dialogs above them; stacked link-consent UI regression covers Escape and Return. | diff --git a/parity/acrobat-features.toml b/parity/acrobat-features.toml index 9cad0f053..e062012fa 100644 --- a/parity/acrobat-features.toml +++ b/parity/acrobat-features.toml @@ -782,8 +782,8 @@ milestone = "M2" status = "partial" commands = [] tools = [] -evidence = ["crates/render/src/raster.rs::huge_cid_width_ranges_terminate", "crates/render/src/raster.rs::non_embedded_japanese_cid_fonts_draw_with_a_craft_fonts_face", "crates/render/src/raster.rs::mincho_substitute_covers_half_width_katakana"] -notes = "CID-to-Unicode tables are M2.4; a non-embedded Adobe-Japan1 font is drawn in a craft-fonts Japanese face (Mincho or Gothic by name; BIZ UDMincho for Mincho, which covers half-width katakana) when built with craft-fonts; non-embedded GB1, CNS1 and Korea1 fonts still draw nothing (no allowed face)" +evidence = ["crates/render/src/raster.rs::huge_cid_width_ranges_terminate", "crates/render/src/raster.rs::non_embedded_japanese_cid_fonts_draw_with_a_craft_fonts_face", "crates/render/src/raster.rs::mincho_substitute_covers_half_width_katakana", "crates/render/src/text.rs::identity_to_unicode_names_give_text", "crates/fonts/src/pdf.rs::identity_to_unicode_names_read_codes_as_unicode"] +notes = "CID-to-Unicode tables are M2.4; a non-embedded Adobe-Japan1 font is drawn in a craft-fonts Japanese face (Mincho or Gothic by name; BIZ UDMincho for Mincho, which covers half-width katakana) when built with craft-fonts; non-embedded GB1, CNS1 and Korea1 fonts still draw nothing (no allowed face); a `/ToUnicode` given as the name `/Identity-H` or `/Identity-V` (WSCAD) reads codes as Unicode for selection, search and editing, without control characters" [[feature]] id = "view.render-image-codecs" @@ -1394,8 +1394,8 @@ milestone = "M3" status = "shipped" commands = ["edit.find"] tools = ["text_find"] -evidence = ["crates/ui-egui/tests/ui.rs::find_counts_matches_across_pages", "crates/ui-egui/tests/ui.rs::find_reports_no_matches", "crates/automation/tests/automation.rs::open_inspect_render_and_find", "crates/render/src/text.rs::identity_v_columns_are_searchable_in_reading_order"] -notes = "Horizontal and WMode 1 vertical text are grouped independently, including a horizontal heading above right-to-left vertical columns. Complex interleaved flows still use a geometric reading-order heuristic." +evidence = ["crates/ui-egui/tests/ui.rs::find_counts_matches_across_pages", "crates/ui-egui/tests/ui.rs::find_reports_no_matches", "crates/automation/tests/automation.rs::open_inspect_render_and_find", "crates/render/src/text.rs::identity_v_columns_are_searchable_in_reading_order", "crates/render/src/text.rs::toc_leaders_do_not_hide_word_gaps", "crates/render/src/text.rs::short_tracked_ellipses_remain_searchable"] +notes = "Horizontal and WMode 1 vertical text are grouped independently, including a horizontal heading above right-to-left vertical columns. Dotted TOC leaders preserve word spaces for phrase search (#3086), while short tracked ellipses remain intact and searchable; sparse lines and condensed OCR gaps below the existing height-based threshold remain limitations. Complex interleaved flows still use a geometric reading-order heuristic." [[feature]] id = "view.find-next-previous" diff --git a/vendor/README.md b/vendor/README.md index 56e4eadae..eb4768163 100644 --- a/vendor/README.md +++ b/vendor/README.md @@ -17,7 +17,7 @@ Other resource categories retain their existing parent-chain lookup. | Crate | Version | Licence | Patches | Test | |---|---|---|---|---| -| hayro-interpret | 0.7.0 | Apache-2.0 OR MIT | (1) select `/AP /N` appearance states via `/AS` (checkboxes, radios, stateful icons) and honour the NoView flag; (2) Type3 Unicode fallback to `/Encoding` glyph names, then printable ASCII; (3) `Type3Glyph::advance_width`; (4) `InterpreterSettings::ocg_overrides` for viewer layer toggles; (5) soft masks whose group lacks `/CS` were dropped (Chrome gradient text); (6) paint nesting cap `MAX_PAINT_NESTING` for tiling patterns and Type 3 glyphs (self-referencing content overflowed the stack); (7) CID width ranges in `/W` and `/W2` clamped to `MAX_CID` (a `0 4294967295 w` range hung); (8) `InterpreterSettings::hide_comments` skips markup annotations but keeps widgets and links (View ▸ Hide all comments); (9) `image_size_ok`: images and masks over 2^28 pixels are not decoded (a fuzzed `/W 4294967295` stencil mask allocated 4 GiB); (10) `MAX_NESTED_PAINTS`: nested paints (forms, Type 3 glyphs and tiling patterns inside one another) share a per-page budget of 50,000 (a glyph showing eight glyphs of its own font, sixteen deep, hung); (11) depends on skrifa 0.47 instead of 0.42 (upstream read-fonts fixed a Type 1 tokenizer that recursed through runs of numbers: a fuzzed font hung or overflowed the stack), with the one API change (`Charset::glyph_id` returns `Option`); (12) annotation appearances whose `/BBox` or `/Matrix` collapses to a line or a point are skipped (mapping them onto `/Rect` divided by zero and handed the device infinite or NaN matrices); (13) a Highlight annotation without a usable normal appearance is drawn from its `/QuadPoints`, `/C` and `/CA` with Multiply blending, as pdfcraft-annot draws highlights (it was not drawn at all); (14) simple-font `/Widths` (`read_widths`): entries only for codes 0–255 (one entry per code below `/FirstChar` was allocated: `4294967295` asked for 34 GB), an entry that isn't a number leaves its code missing instead of ending the array, and a `/LastChar` below `/FirstChar` gives no widths instead of discarding the font for a standard one; (15) the appearance box is mapped onto `/Rect` corner to corner: the translation now accounts for the scale (an appearance whose box didn't start at the origin and differed in size from `/Rect` was drawn shifted); (16) a clip set by `W`/`W*` takes effect after whichever path-painting operator ends the path (`S`, `f`, `B`, … as well as `n`; ISO 32000-2 §8.5.4): it was applied only on `n`, so `re W* S` clipped nothing and the clip stayed pending for a later, unrelated `n`; (17) `InterpreterSettings::cancelled`: once set, interpretation stops before the next content operator (a render pool sets it for every worker when dropped, after an edit or on closing, and for one worker when the watchdog gives up on its render, so renders whose answers nobody can receive stop instead of running to the end); (18) `MAX_PAGE_CONTENT`: the content a page holds at once (its contents, and the forms, tiling patterns and Type 3 glyph procedures being painted) stays within 512 MiB (`decode_content`; once it runs out, `InterpreterWarning::ContentTruncated` is reported (once per page by pdfcraft-render) so a truncated page is not a clean render; each level gives its bytes back when it finishes): forms nest fifty deep and each level kept its content while the next ran, so a form painting itself around a stream inflating to 256 MiB would have held over 12 GB; and Type 0 sampled functions take at most `MAX_SAMPLE_ENTRIES` (2^21) table points and `MAX_SAMPLE_VALUES` (2^23) samples, read no further than `/Size` asks (256 MiB of 1-bit samples asked for over 8 GB); (19) mesh shadings (types 4–7) are sampled only inside the area being drawn (`ShadingPattern::encode_within`, which the vendored hayro renderer calls with that area plus a pixel), within 8 visits per pixel of it plus 4 per triangle (`TriangleSampler`), and patches are triangulated one at a time: points far off the page were sampled over their whole bounding box, up to 65536 × 65536 entries (a fuzzed 147-byte patch stream; a synthetic page passed 6 GB in six seconds), ten thousand overlapping triangles ran for minutes, and every patch became 722 triangles before any limit; meshes hold at most `MAX_MESH_PATCHES` (2^16) patches and `MAX_MESH_TRIANGLES` (2^20) triangles, a lattice needs two to 2^19 vertices per row (`/VerticesPerRow 0` pushed empty rows for ever; one huge row read the whole stream), and a page samples at most `MAX_PAGE_MESH_TRIANGLES` (2^25) mesh triangles in all (a mesh painted a thousand times was triangulated a thousand times); each cut logs a warning; (20) PDF function construction is limited to 64 levels and 10,000 total nodes with cycle detection; calculator parsing and evaluation share procedure depth/work budgets, integer division/bitshift/rotation/index are panic-free, and fixed-size stack errors propagate; (21) a Type 0 font drawn with a substitute that isn't CID-keyed (it isn't embedded, or its program doesn't load) draws `.notdef` for a CID of an Adobe character collection (Japan1, GB1, CNS1, Korea1) that it can't find by Unicode: hayro took the CID for a glyph index of the substitute, which numbers its glyphs its own way, so the character was drawn as an unrelated one (𠮷 as "Ｑ" in BIZ UDPGothic); (22) a Type 0 font drawn with a substitute finds a character given with a variation selector (`/ToUnicode`, or Adobe-Japan1-UCS2, which maps the JIS X 0208 forms of the kanji JIS X 0213:2004 redrew that way: CID 1247 is 噂 U+E0100) through the substitute's variation sequences, else as the character alone: such a CID found no glyph and was drawn as an unrelated one; (23) backport of hayro 0.8's `gradient` module: `EncodedShadingPattern::as_svg_gradient` turns an axial or radial shading into stops of an sRGB gradient (within 0.01 per channel, a stop at every stitching-function bound via `Function::stitching_bounds`) when a native gradient draws it exactly (extended past the box drawn in; a radial one with no background colour); (24) `DecodedImageCache`: decoded image pixels shared across renders through the interpreter cache within a 64 MiB budget (an image-heavy page decoded its thousands of images again per tile and per scale), keyed by every decode input (`DecodedImageKey`: the stream's place and dictionary bytes, its object, mask/stencil/raster output, interpolation, decode target size, preview or final); images under a transfer function and inline images are not cached, an image over the budget is never kept, and a poisoned lock decodes uncached; (25) `Device::actual_text` hands the device a marked content sequence's `/ActualText` (inline or named in `/Properties`), so text extraction keeps the Thai, Indic and Arabic clusters Chrome marks this way (their shaped tone marks and vowels have no Unicode of their own); (26) non-embedded simple fonts drawn with a standard-font stand-in build the Latin Extended-A letters the stand-in lacks (`/Gbreve`, `/Idotaccent`, `/Scedilla`, `/aogonek`, `/ccaron`…) from the base letter and accent it has (`StandardKind::composite_glyph`, `ACCENTED_LETTERS`): PDFium's Foxit stand-ins hold only the Standard/WinAnsi/Mac Roman glyphs, so iText letters with Times New Roman unembedded showed blanks for Ğ, İ, Ş, ğ and ş; (27) an embedded TrueType font without a base encoding reads its codes as WinAnsi for text extraction (`EmbeddedKind::code_to_unicode_name`), matching pdftotext's output, instead of Standard: OCR text layers of scans write WinAnsi codes with no `/Encoding`, and Standard has no ü, ö or ß and reads é as Ø (glyph selection unchanged); (28) an image whose pixels hold at most 16 bits of samples (an 8-bit Separation or DeviceN image, a 1-, 2- or 4-bit gray or indexed one) is converted once per possible pixel value through a lookup table instead of once per pixel (`rgb_via_lookup`): a spot-colour tint transform is often a PostScript calculator program, run for every pixel, so a fillable character sheet's 3304 × 4105 DeviceN background took four seconds and 430 MB to draw a page; each entry is converted on its own (`convert_each`), so a tint transform giving the wrong number of components for a value the image never uses cannot shift the colours of the values it does; (29) `Device::begin_annotations`/`end_annotations` bracket the page's annotations, so a device can tell comment paints from document paints (reading colours recolour the document, never a comment, #1548); (30) the Annex D.2 alias codes read as the ink they draw: WinAnsiEncoding 0xA0 and 0xAD, and MacRomanEncoding 0xCA, carry the glyph names `space` and `hyphen` -- the names those tables also give 0x20 and 0x2D -- not `nbspace` and `sfthyphen`, whose glyph list values U+00A0 and U+00AD are invisible formatting characters. Extraction hid drawn ink behind them, so search and redaction matched neither (`XiaoBiaoSong.pdf` in the pdf.js corpus drew 104 soft hyphens; across the 989 files the soft hyphens we extract fall from 117 to 13 and the non-breaking spaces from 30 to 26, what remains coming from `/ToUnicode` maps, which say so explicitly). The base table only, in the text direction only: a `/Differences` name keeps its glyph list value and glyph lookup is untouched, both as pdftotext does; (31) a Type 0 sampled function keeps its samples in one flat array indexed by strides (`Type0::entry`) instead of a hash map keyed by each grid point: the map was built again, one hashed insert per sample, every time a shading using the function was painted (a 2,097,152-sample ramp painted 30 times took 38 s; `bug1721218` in the pdf.js corpus spent most of a 1.5 s page on it); the interpolation is unchanged, so results are bit for bit the same, a sample missing from a short stream still gives no value, and the maximum sample value is computed without overflowing at 32 bits per sample; (32) the decoded-image cache (24) keys a decode by the size the image is drawn at only when a JPX stream decodes it or its soft mask or mask (`decode_reads_target`): JPX is the only filter that decodes a reduced resolution for a size, so a JPEG or Flate scan was decoded again at every zoom step and kept once per zoom level; one decode now serves every size; `InterpreterCache::decoded_image_decodes` counts the cache's decodes; (33) Coons and tensor-product patches copy their vertices' colours less: each point of a patch's 20 × 20 triangle grid is interpolated once (`generate_patch_triangles`; each cell interpolated its four corners itself, 1,444 interpolations a patch for 400 points), a cell's second triangle takes the vertices its first copied, and the mesh sampler moves a patch's triangles into device space instead of copying each vertex (`TriangleSampler::sample_owned`); the same values, 10–16% faster on mesh-gradient pages; (34) `color::tint_transform_runs`: how many colours the calling thread has run through a Separation or DeviceN tint transform (counted once per batch in `convert_f32`), so a test counts patch (28)'s once-per-sample-value runs instead of timing them (its 10 s deadline failed on a busy FreeBSD VM in CI); (35) `EncodedShadingPattern::extent`: where a mesh shading (types 4–7) can paint at all, the pixels its triangles touched when it was encoded (`None` for the types that paint everywhere, and when the shading has a `/Background`), so the renderer samples a texture there alone and skips a mesh with nothing on the canvas; (36) a mesh shading's pixel whose corner no triangle holds strictly (on an edge two triangles share, or on the mesh's side) takes the triangle that covers its centre, edges included (`Triangle::covers`): testing only the corner left a white seam along every shared edge, and the colour is still taken at the corner; (38) the `/DefaultGray`, `/DefaultRGB` and `/DefaultCMYK` colour spaces of a resource dictionary (ISO 32000-2 §8.6.5.6) replace the device space content selects: with `g`/`G`, `rg`/`RG`, `k`/`K` and `cs`/`CS` (by name, or through a `/ColorSpace` resource that is a device space), and as the `/ColorSpace` of an image XObject, an inline image or a shading painted by `sh` or as a shading pattern (`DefaultColorSpaces`, looked up once per content window in the resources and those they inherit, so a form XObject without a default of its own draws in its page's, as Poppler and MuPDF draw it); a default that is a Lab, Indexed, Pattern or device space, has another number of components or is no colour space is ignored and device colour is drawn as before; device spaces inside other spaces (an Indexed base, a Separation or DeviceN alternate, a Pattern's underlying space), shading dictionaries, masks and transparency group spaces are unchanged, and the decoded-image cache (24) keys a decode by the default it was drawn in. hayro drew device colour as it was, so pages that calibrate it this way (common in PDF/A) came out in the wrong colours; (39) knockout groups (#2971): `Device::set_knockout` right after a `/K true` group is pushed; `Device::knockout_image` for an unmasked image before its group is pushed, when `Device::in_knockout_group`: the device clears its placement rectangle only if it will draw the image (on the canvas, not a preview's tiny image, and it decodes), decoded at its draw size so the draw reuses that decode; and, when `Device::knockout_erase_wanted`, a nested group's content drawn once between `begin_knockout_erase` and `end_knockout_erase` before the group itself, which then always gets a layer of its own; (40) an optional content membership dictionary's visibility expression (`/VE`: `/And`, `/Or`, `/Not`, nested up to 32 deep) decides its visibility ahead of `/OCGs` and `/P` (ISO 32000-2 §8.11.2.2); it was ignored, so content under it stayed visible with its layers hidden; a malformed expression, including one naming an object the document doesn't list in `/OCProperties /OCGs`, falls back to `/OCGs` and `/P` (#2986); (41) a transparency group that is neither isolated nor knockout, entered at full opacity with no soft mask and Normal blending, is drawn straight onto the page instead of into a layer of its own (§11.4.7: its elements composite with the backdrop; the layer made a `/BM /Multiply` highlight appearance an opaque block, #1968); (42) text in a Type 3 font is drawn filled whatever the text rendering mode, except mode 3 (invisible): only that mode has any effect on Type 3 fonts (ISO 32000-1 §9.3.6). Stroking ran the glyph procedure a second time, so `2 Tr` made bitmap fonts look bold, `1 Tr` painted an uncoloured glyph in the stroking colour, and `7 Tr` hid it; (43) Lab colour spaces honour their own `/WhitePoint`: L\*a\*b\* goes to XYZ relative to that white, is Bradford-adapted to D65 and converted to sRGB (ISO 32000-2 §8.6.5.4) instead of through `LAB.icc` as if every white were D50 (a D65 blue came out 45 levels too red; the profile is removed); components normalised to [0, 1] (`manual_scale`, an `/Indexed` palette over Lab) map onto L\* 0..100 and the a\*/b\* `/Range`; (46) an image XObject's group applies the fill opacity, so the fill alpha is reset to 1 before a stencil mask takes the fill colour as its paint, and a pattern paint drops the opacity it captured at `scn` (`Pattern::without_opacity`) (it was applied twice: a mask at `ca 0.5` was drawn at a quarter, #2975); (47) `path_as_rect`: a five-element path is a rectangle only when its four points are distinct, each step from one to the next runs along a single axis, and the last returns to the first. The length test admits one MoveTo and four LineTo as well as one MoveTo, three LineTo and a ClosePath, and asking only that every corner of the bounding box was visited says nothing about the order or the steps between, so a five-point polyline whose ends reached the corners was painted as that box -- a fourth side the path does not have -- and a clip of that shape was discarded; the coordinates are compared exactly, since the matrix is applied after this check and a `cm` of 10,000 turns `is_nearly_equal`'s 256th of a unit into tens of device units (#3327); (48) an `/ICCBased` colour space is cached by what its profile is read from (`IccProfileKey`): the stream object it names, looked up without reading the stream, and then the stream's dictionary and its own bytes (decrypted, not decoded), so a byte-for-byte copy of a profile in another object is read once, and a profile stream written directly in its array, which reports the object holding the array as every other stream written there does, is told apart by its bytes. hayro keyed it by the stream dictionary's bytes alone, so two different profiles with the same `/N` and `/Length` shared whichever was read first, and colours in the second were drawn through the first. `InterpreterCache::icc_profile_reads` counts the profiles read | (1) `raster::tests::widget_appearance_states`; (2, 3) `xtask text-oracle` on corpus/pdfjs issue918; (4) `raster::tests::layer_override_hides_content`; (5) `raster::tests::soft_mask_without_group_colour_space_is_applied`; (6) `raster::tests::self_referencing_tiling_pattern_terminates`, `raster::tests::self_referencing_type3_glyph_terminates`; (7) `raster::tests::huge_cid_width_ranges_terminate`; (8) `raster::tests::hiding_comments_keeps_fields`; (9) `raster::tests::absurd_mask_and_fax_sizes_are_refused_before_decoding`; (10) `raster::tests::type3_glyphs_that_fan_out_into_themselves_finish`; (11) `raster::tests::type1_font_programs_with_long_runs_of_numbers_load_quickly`; (12) `raster::tests::degenerate_annotation_appearances_are_skipped`; (13) `raster::tests::highlights_without_an_appearance_are_drawn`; (14) `raster::tests::malformed_simple_font_widths_keep_later_widths_and_the_font`, `raster::tests::huge_simple_font_first_char_terminates`; (15) `raster::tests::annotation_appearance_box_maps_onto_rect`; (16) `raster::tests::clipping_path_applies_after_any_painting_operator`; (17) `raster::tests::dropping_the_pool_cancels_its_renders`, `raster::tests::the_watchdog_stops_the_render_it_gives_up_on`; (18) `raster::tests::page_contents_and_nested_forms_share_one_budget`, `raster::tests::content_painted_again_and_again_keeps_drawing`, `raster::tests::sampled_functions_with_billions_of_samples_are_refused`; (19) `raster::tests::mesh_shadings_far_off_the_page_render`, `raster::tests::many_overlapping_mesh_triangles_finish`, `raster::tests::mesh_shadings_with_many_patches_or_no_rows_finish`; (20) `raster::tests::function_limits_calculator_arithmetic_and_depth`, `raster::tests::function_limits_public_construction_work_and_depth`, `raster::tests::function_limits_stitching_cycles_are_refused_and_shared_children_work`, `raster::tests::function_limits_invalid_calculator_keeps_the_page_renderable`; (21) `raster::tests::cids_a_substitute_lacks_are_not_drawn_as_other_glyphs`; (26) `raster::tests::accented_letters_the_standard_stand_ins_lack_are_built_from_their_parts`; (22) `raster::tests::variation_sequences_draw_their_kanji_in_a_substitute`; (23) `raster::tests::axial_and_radial_shadings_draw_as_native_gradients`; (24) `raster::tests::tiles_match_full_render`, `raster::tests::skipping_paths_off_the_canvas_changes_no_pixel` (both decode through the cache), `raster::tests::decoded_images_with_identical_dictionaries_keep_their_own_pixels`, `raster::tests::decoded_images_under_a_transfer_function_are_not_shared`; (25) `text::tests::actual_text_replaces_the_glyphs_it_marks_in_every_writing_direction`, `text::tests::actual_text_follows_the_pdf_rules_for_marked_content`; (27) `text::tests::truetype_without_an_encoding_reads_its_codes_as_win_ansi`; (28) `raster::tests::spot_colour_images_run_their_tint_transform_once_per_sample_value`, `raster::tests::images_converted_through_a_table_keep_every_samples_colour`, `raster::tests::a_malformed_tint_for_an_unused_sample_keeps_the_used_colours`; (29) `raster::tests::reading_colors_keep_highlight_annotations_readable`; (30) `text::tests::the_annex_d2_alias_codes_read_as_an_ordinary_hyphen_and_space`; (31) `raster::tests::shadings_with_large_sampled_functions_paint_quickly`, `raster::tests::sampled_functions_interpolate_their_grid_as_before`; (32) `raster::tests::decoded_images_serve_every_size_they_are_drawn_at`; (33) `raster::tests::patch_meshes_keep_their_corner_colours`; (34) `raster::tests::spot_colour_images_run_their_tint_transform_once_per_sample_value`; (35) `mesh_extent_tests::{a_mesh_paints_its_triangles_alone, a_mesh_keeps_the_graphics_state_alpha, a_mesh_off_the_canvas_paints_nothing}` (`cargo test -p hayro`); (36) `raster::tests::mesh_triangles_sharing_an_edge_leave_no_seam`; (38) `raster::tests::default_colour_spaces::device_rgb_operators_draw_in_default_rgb`, `raster::tests::default_colour_spaces::device_cmyk_operators_draw_in_default_cmyk`, `raster::tests::default_colour_spaces::device_gray_operators_draw_in_default_gray`, `raster::tests::default_colour_spaces::device_images_draw_in_the_default_colour_spaces`, `raster::tests::default_colour_spaces::device_shadings_draw_in_the_default_colour_spaces`, `raster::tests::default_colour_spaces::an_image_drawn_with_and_without_a_default_keeps_both_colours`, `raster::tests::default_colour_spaces::forms_inherit_the_default_colour_spaces_they_do_not_override`, `raster::tests::default_colour_spaces::unusable_defaults_leave_device_colour_alone`; (39) `raster::tests::knockout_groups_replace_what_their_elements_cover`; (40) `raster::tests::visibility_expressions_decide_what_a_membership_dictionary_shows`; (41) `raster::tests::a_multiply_highlight_inside_a_transparency_group_keeps_the_text`; (42) `raster::tests::text_rendering_modes_other_than_invisible_draw_type3_glyphs_as_filled`; (43) `raster::tests::lab_honours_the_space_white_point`, `raster::tests::indexed_palettes_over_lab_keep_their_colours` |; (46) `raster::tests::stencil_masks_are_drawn_at_the_fill_opacity_once`; (47) `raster::tests::a_five_point_polyline_is_not_painted_as_its_bounding_box`, `raster::tests::a_path_that_visits_every_corner_out_of_order_is_not_a_rectangle`, `raster::tests::a_rectangle_is_still_painted_as_a_rectangle`; (48) `raster::tests::icc_profile_cache::icc_profiles_with_identical_dictionaries_draw_their_own_colours`, `raster::tests::icc_profile_cache::icc_profiles_used_again_are_read_once` +| hayro-interpret | 0.7.0 | Apache-2.0 OR MIT | (1) select `/AP /N` appearance states via `/AS` (checkboxes, radios, stateful icons) and honour the NoView flag; (2) Type3 Unicode fallback to `/Encoding` glyph names, then printable ASCII; (3) `Type3Glyph::advance_width`; (4) `InterpreterSettings::ocg_overrides` for viewer layer toggles; (5) soft masks whose group lacks `/CS` were dropped (Chrome gradient text); (6) paint nesting cap `MAX_PAINT_NESTING` for tiling patterns and Type 3 glyphs (self-referencing content overflowed the stack); (7) CID width ranges in `/W` and `/W2` clamped to `MAX_CID` (a `0 4294967295 w` range hung); (8) `InterpreterSettings::hide_comments` skips markup annotations but keeps widgets and links (View ▸ Hide all comments); (9) `image_size_ok`: images and masks over 2^28 pixels are not decoded (a fuzzed `/W 4294967295` stencil mask allocated 4 GiB); (10) `MAX_NESTED_PAINTS`: nested paints (forms, Type 3 glyphs and tiling patterns inside one another) share a per-page budget of 50,000 (a glyph showing eight glyphs of its own font, sixteen deep, hung); (11) depends on skrifa 0.47 instead of 0.42 (upstream read-fonts fixed a Type 1 tokenizer that recursed through runs of numbers: a fuzzed font hung or overflowed the stack), with the one API change (`Charset::glyph_id` returns `Option`); (12) annotation appearances whose `/BBox` or `/Matrix` collapses to a line or a point are skipped (mapping them onto `/Rect` divided by zero and handed the device infinite or NaN matrices); (13) a Highlight annotation without a usable normal appearance is drawn from its `/QuadPoints`, `/C` and `/CA` with Multiply blending, as pdfcraft-annot draws highlights (it was not drawn at all); (14) simple-font `/Widths` (`read_widths`): entries only for codes 0–255 (one entry per code below `/FirstChar` was allocated: `4294967295` asked for 34 GB), an entry that isn't a number leaves its code missing instead of ending the array, and a `/LastChar` below `/FirstChar` gives no widths instead of discarding the font for a standard one; (15) the appearance box is mapped onto `/Rect` corner to corner: the translation now accounts for the scale (an appearance whose box didn't start at the origin and differed in size from `/Rect` was drawn shifted); (16) a clip set by `W`/`W*` takes effect after whichever path-painting operator ends the path (`S`, `f`, `B`, … as well as `n`; ISO 32000-2 §8.5.4): it was applied only on `n`, so `re W* S` clipped nothing and the clip stayed pending for a later, unrelated `n`; (17) `InterpreterSettings::cancelled`: once set, interpretation stops before the next content operator (a render pool sets it for every worker when dropped, after an edit or on closing, and for one worker when the watchdog gives up on its render, so renders whose answers nobody can receive stop instead of running to the end); (18) `MAX_PAGE_CONTENT`: the content a page holds at once (its contents, and the forms, tiling patterns and Type 3 glyph procedures being painted) stays within 512 MiB (`decode_content`; once it runs out, `InterpreterWarning::ContentTruncated` is reported (once per page by pdfcraft-render) so a truncated page is not a clean render; each level gives its bytes back when it finishes): forms nest fifty deep and each level kept its content while the next ran, so a form painting itself around a stream inflating to 256 MiB would have held over 12 GB; and Type 0 sampled functions take at most `MAX_SAMPLE_ENTRIES` (2^21) table points and `MAX_SAMPLE_VALUES` (2^23) samples, read no further than `/Size` asks (256 MiB of 1-bit samples asked for over 8 GB); (19) mesh shadings (types 4–7) are sampled only inside the area being drawn (`ShadingPattern::encode_within`, which the vendored hayro renderer calls with that area plus a pixel), within 8 visits per pixel of it plus 4 per triangle (`TriangleSampler`), and patches are triangulated one at a time: points far off the page were sampled over their whole bounding box, up to 65536 × 65536 entries (a fuzzed 147-byte patch stream; a synthetic page passed 6 GB in six seconds), ten thousand overlapping triangles ran for minutes, and every patch became 722 triangles before any limit; meshes hold at most `MAX_MESH_PATCHES` (2^16) patches and `MAX_MESH_TRIANGLES` (2^20) triangles, a lattice needs two to 2^19 vertices per row (`/VerticesPerRow 0` pushed empty rows for ever; one huge row read the whole stream), and a page samples at most `MAX_PAGE_MESH_TRIANGLES` (2^25) mesh triangles in all (a mesh painted a thousand times was triangulated a thousand times); each cut logs a warning; (20) PDF function construction is limited to 64 levels and 10,000 total nodes with cycle detection; calculator parsing and evaluation share procedure depth/work budgets, integer division/bitshift/rotation/index are panic-free, and fixed-size stack errors propagate; (21) a Type 0 font drawn with a substitute that isn't CID-keyed (it isn't embedded, or its program doesn't load) draws `.notdef` for a CID of an Adobe character collection (Japan1, GB1, CNS1, Korea1) that it can't find by Unicode: hayro took the CID for a glyph index of the substitute, which numbers its glyphs its own way, so the character was drawn as an unrelated one (𠮷 as "Ｑ" in BIZ UDPGothic); (22) a Type 0 font drawn with a substitute finds a character given with a variation selector (`/ToUnicode`, or Adobe-Japan1-UCS2, which maps the JIS X 0208 forms of the kanji JIS X 0213:2004 redrew that way: CID 1247 is 噂 U+E0100) through the substitute's variation sequences, else as the character alone: such a CID found no glyph and was drawn as an unrelated one; (23) backport of hayro 0.8's `gradient` module: `EncodedShadingPattern::as_svg_gradient` turns an axial or radial shading into stops of an sRGB gradient (within 0.01 per channel, a stop at every stitching-function bound via `Function::stitching_bounds`) when a native gradient draws it exactly (extended past the box drawn in; a radial one with no background colour); (24) `DecodedImageCache`: decoded image pixels shared across renders through the interpreter cache within a 64 MiB budget (an image-heavy page decoded its thousands of images again per tile and per scale), keyed by every decode input (`DecodedImageKey`: the stream's place and dictionary bytes, its object, mask/stencil/raster output, interpolation, decode target size, preview or final); images under a transfer function and inline images are not cached, an image over the budget is never kept, and a poisoned lock decodes uncached; (25) `Device::actual_text` hands the device a marked content sequence's `/ActualText` (inline or named in `/Properties`), so text extraction keeps the Thai, Indic and Arabic clusters Chrome marks this way (their shaped tone marks and vowels have no Unicode of their own); (26) non-embedded simple fonts drawn with a standard-font stand-in build the Latin Extended-A letters the stand-in lacks (`/Gbreve`, `/Idotaccent`, `/Scedilla`, `/aogonek`, `/ccaron`…) from the base letter and accent it has (`StandardKind::composite_glyph`, `ACCENTED_LETTERS`): PDFium's Foxit stand-ins hold only the Standard/WinAnsi/Mac Roman glyphs, so iText letters with Times New Roman unembedded showed blanks for Ğ, İ, Ş, ğ and ş; (27) an embedded TrueType font without a base encoding reads its codes as WinAnsi for text extraction (`EmbeddedKind::code_to_unicode_name`), matching pdftotext's output, instead of Standard: OCR text layers of scans write WinAnsi codes with no `/Encoding`, and Standard has no ü, ö or ß and reads é as Ø (glyph selection unchanged); (28) an image whose pixels hold at most 16 bits of samples (an 8-bit Separation or DeviceN image, a 1-, 2- or 4-bit gray or indexed one) is converted once per possible pixel value through a lookup table instead of once per pixel (`rgb_via_lookup`): a spot-colour tint transform is often a PostScript calculator program, run for every pixel, so a fillable character sheet's 3304 × 4105 DeviceN background took four seconds and 430 MB to draw a page; each entry is converted on its own (`convert_each`), so a tint transform giving the wrong number of components for a value the image never uses cannot shift the colours of the values it does; (29) `Device::begin_annotations`/`end_annotations` bracket the page's annotations, so a device can tell comment paints from document paints (reading colours recolour the document, never a comment, #1548); (30) the Annex D.2 alias codes read as the ink they draw: WinAnsiEncoding 0xA0 and 0xAD, and MacRomanEncoding 0xCA, carry the glyph names `space` and `hyphen` -- the names those tables also give 0x20 and 0x2D -- not `nbspace` and `sfthyphen`, whose glyph list values U+00A0 and U+00AD are invisible formatting characters. Extraction hid drawn ink behind them, so search and redaction matched neither (`XiaoBiaoSong.pdf` in the pdf.js corpus drew 104 soft hyphens; across the 989 files the soft hyphens we extract fall from 117 to 13 and the non-breaking spaces from 30 to 26, what remains coming from `/ToUnicode` maps, which say so explicitly). The base table only, in the text direction only: a `/Differences` name keeps its glyph list value and glyph lookup is untouched, both as pdftotext does; (31) a Type 0 sampled function keeps its samples in one flat array indexed by strides (`Type0::entry`) instead of a hash map keyed by each grid point: the map was built again, one hashed insert per sample, every time a shading using the function was painted (a 2,097,152-sample ramp painted 30 times took 38 s; `bug1721218` in the pdf.js corpus spent most of a 1.5 s page on it); the interpolation is unchanged, so results are bit for bit the same, a sample missing from a short stream still gives no value, and the maximum sample value is computed without overflowing at 32 bits per sample; (32) the decoded-image cache (24) keys a decode by the size the image is drawn at only when a JPX stream decodes it or its soft mask or mask (`decode_reads_target`): JPX is the only filter that decodes a reduced resolution for a size, so a JPEG or Flate scan was decoded again at every zoom step and kept once per zoom level; one decode now serves every size; `InterpreterCache::decoded_image_decodes` counts the cache's decodes; (33) Coons and tensor-product patches copy their vertices' colours less: each point of a patch's 20 × 20 triangle grid is interpolated once (`generate_patch_triangles`; each cell interpolated its four corners itself, 1,444 interpolations a patch for 400 points), a cell's second triangle takes the vertices its first copied, and the mesh sampler moves a patch's triangles into device space instead of copying each vertex (`TriangleSampler::sample_owned`); the same values, 10–16% faster on mesh-gradient pages; (34) `color::tint_transform_runs`: how many colours the calling thread has run through a Separation or DeviceN tint transform (counted once per batch in `convert_f32`), so a test counts patch (28)'s once-per-sample-value runs instead of timing them (its 10 s deadline failed on a busy FreeBSD VM in CI); (35) `EncodedShadingPattern::extent`: where a mesh shading (types 4–7) can paint at all, the pixels its triangles touched when it was encoded (`None` for the types that paint everywhere, and when the shading has a `/Background`), so the renderer samples a texture there alone and skips a mesh with nothing on the canvas; (36) a mesh shading's pixel whose corner no triangle holds strictly (on an edge two triangles share, or on the mesh's side) takes the triangle that covers its centre, edges included (`Triangle::covers`): testing only the corner left a white seam along every shared edge, and the colour is still taken at the corner; (38) the `/DefaultGray`, `/DefaultRGB` and `/DefaultCMYK` colour spaces of a resource dictionary (ISO 32000-2 §8.6.5.6) replace the device space content selects: with `g`/`G`, `rg`/`RG`, `k`/`K` and `cs`/`CS` (by name, or through a `/ColorSpace` resource that is a device space), and as the `/ColorSpace` of an image XObject, an inline image or a shading painted by `sh` or as a shading pattern (`DefaultColorSpaces`, looked up once per content window in the resources and those they inherit, so a form XObject without a default of its own draws in its page's, as Poppler and MuPDF draw it); a default that is a Lab, Indexed, Pattern or device space, has another number of components or is no colour space is ignored and device colour is drawn as before; device spaces inside other spaces (an Indexed base, a Separation or DeviceN alternate, a Pattern's underlying space), shading dictionaries, masks and transparency group spaces are unchanged, and the decoded-image cache (24) keys a decode by the default it was drawn in. hayro drew device colour as it was, so pages that calibrate it this way (common in PDF/A) came out in the wrong colours; (39) knockout groups (#2971): `Device::set_knockout` right after a `/K true` group is pushed; `Device::knockout_image` for an unmasked image before its group is pushed, when `Device::in_knockout_group`: the device clears its placement rectangle only if it will draw the image (on the canvas, not a preview's tiny image, and it decodes), decoded at its draw size so the draw reuses that decode; and, when `Device::knockout_erase_wanted`, a nested group's content drawn once between `begin_knockout_erase` and `end_knockout_erase` before the group itself, which then always gets a layer of its own; (40) an optional content membership dictionary's visibility expression (`/VE`: `/And`, `/Or`, `/Not`, nested up to 32 deep) decides its visibility ahead of `/OCGs` and `/P` (ISO 32000-2 §8.11.2.2); it was ignored, so content under it stayed visible with its layers hidden; a malformed expression, including one naming an object the document doesn't list in `/OCProperties /OCGs`, falls back to `/OCGs` and `/P` (#2986); (41) a transparency group that is neither isolated nor knockout, entered at full opacity with no soft mask and Normal blending, is drawn straight onto the page instead of into a layer of its own (§11.4.7: its elements composite with the backdrop; the layer made a `/BM /Multiply` highlight appearance an opaque block, #1968); (42) text in a Type 3 font is drawn filled whatever the text rendering mode, except mode 3 (invisible): only that mode has any effect on Type 3 fonts (ISO 32000-1 §9.3.6). Stroking ran the glyph procedure a second time, so `2 Tr` made bitmap fonts look bold, `1 Tr` painted an uncoloured glyph in the stroking colour, and `7 Tr` hid it; (43) Lab colour spaces honour their own `/WhitePoint`: L\*a\*b\* goes to XYZ relative to that white, is Bradford-adapted to D65 and converted to sRGB (ISO 32000-2 §8.6.5.4) instead of through `LAB.icc` as if every white were D50 (a D65 blue came out 45 levels too red; the profile is removed); components normalised to [0, 1] (`manual_scale`, an `/Indexed` palette over Lab) map onto L\* 0..100 and the a\*/b\* `/Range`; (44) a Type 0 font whose `/ToUnicode` is the name `/Identity-H` or `/Identity-V` (not a CMap stream; WSCAD writes it) reads each code as its Unicode value, as Acrobat shows it (`Type0Font::identity_to_unicode`); the identity CMap the name resolved to maps codes to CIDs only, so such pages had no text to select or find; (46) an image XObject's group applies the fill opacity, so the fill alpha is reset to 1 before a stencil mask takes the fill colour as its paint, and a pattern paint drops the opacity it captured at `scn` (`Pattern::without_opacity`) (it was applied twice: a mask at `ca 0.5` was drawn at a quarter, #2975); (47) `path_as_rect`: a five-element path is a rectangle only when its four points are distinct, each step from one to the next runs along a single axis, and the last returns to the first. The length test admits one MoveTo and four LineTo as well as one MoveTo, three LineTo and a ClosePath, and asking only that every corner of the bounding box was visited says nothing about the order or the steps between, so a five-point polyline whose ends reached the corners was painted as that box -- a fourth side the path does not have -- and a clip of that shape was discarded; the coordinates are compared exactly, since the matrix is applied after this check and a `cm` of 10,000 turns `is_nearly_equal`'s 256th of a unit into tens of device units (#3327); (48) an `/ICCBased` colour space is cached by what its profile is read from (`IccProfileKey`): the stream object it names, looked up without reading the stream, and then the stream's dictionary and its own bytes (decrypted, not decoded), so a byte-for-byte copy of a profile in another object is read once, and a profile stream written directly in its array, which reports the object holding the array as every other stream written there does, is told apart by its bytes. hayro keyed it by the stream dictionary's bytes alone, so two different profiles with the same `/N` and `/Length` shared whichever was read first, and colours in the second were drawn through the first. `InterpreterCache::icc_profile_reads` counts the profiles read | (1) `raster::tests::widget_appearance_states`; (2, 3) `xtask text-oracle` on corpus/pdfjs issue918; (4) `raster::tests::layer_override_hides_content`; (5) `raster::tests::soft_mask_without_group_colour_space_is_applied`; (6) `raster::tests::self_referencing_tiling_pattern_terminates`, `raster::tests::self_referencing_type3_glyph_terminates`; (7) `raster::tests::huge_cid_width_ranges_terminate`; (8) `raster::tests::hiding_comments_keeps_fields`; (9) `raster::tests::absurd_mask_and_fax_sizes_are_refused_before_decoding`; (10) `raster::tests::type3_glyphs_that_fan_out_into_themselves_finish`; (11) `raster::tests::type1_font_programs_with_long_runs_of_numbers_load_quickly`; (12) `raster::tests::degenerate_annotation_appearances_are_skipped`; (13) `raster::tests::highlights_without_an_appearance_are_drawn`; (14) `raster::tests::malformed_simple_font_widths_keep_later_widths_and_the_font`, `raster::tests::huge_simple_font_first_char_terminates`; (15) `raster::tests::annotation_appearance_box_maps_onto_rect`; (16) `raster::tests::clipping_path_applies_after_any_painting_operator`; (17) `raster::tests::dropping_the_pool_cancels_its_renders`, `raster::tests::the_watchdog_stops_the_render_it_gives_up_on`; (18) `raster::tests::page_contents_and_nested_forms_share_one_budget`, `raster::tests::content_painted_again_and_again_keeps_drawing`, `raster::tests::sampled_functions_with_billions_of_samples_are_refused`; (19) `raster::tests::mesh_shadings_far_off_the_page_render`, `raster::tests::many_overlapping_mesh_triangles_finish`, `raster::tests::mesh_shadings_with_many_patches_or_no_rows_finish`; (20) `raster::tests::function_limits_calculator_arithmetic_and_depth`, `raster::tests::function_limits_public_construction_work_and_depth`, `raster::tests::function_limits_stitching_cycles_are_refused_and_shared_children_work`, `raster::tests::function_limits_invalid_calculator_keeps_the_page_renderable`; (21) `raster::tests::cids_a_substitute_lacks_are_not_drawn_as_other_glyphs`; (22) `raster::tests::variation_sequences_draw_their_kanji_in_a_substitute`; (23) `raster::tests::axial_and_radial_shadings_draw_as_native_gradients`; (24) `raster::tests::tiles_match_full_render`, `raster::tests::skipping_paths_off_the_canvas_changes_no_pixel` (both decode through the cache), `raster::tests::decoded_images_with_identical_dictionaries_keep_their_own_pixels`, `raster::tests::decoded_images_under_a_transfer_function_are_not_shared`; (25) `text::tests::actual_text_replaces_the_glyphs_it_marks_in_every_writing_direction`, `text::tests::actual_text_follows_the_pdf_rules_for_marked_content`; (26) `raster::tests::accented_letters_the_standard_stand_ins_lack_are_built_from_their_parts`; (27) `text::tests::truetype_without_an_encoding_reads_its_codes_as_win_ansi`; (28) `raster::tests::spot_colour_images_run_their_tint_transform_once_per_sample_value`, `raster::tests::images_converted_through_a_table_keep_every_samples_colour`, `raster::tests::a_malformed_tint_for_an_unused_sample_keeps_the_used_colours`; (29) `raster::tests::reading_colors_keep_highlight_annotations_readable`; (30) `text::tests::the_annex_d2_alias_codes_read_as_an_ordinary_hyphen_and_space`; (31) `raster::tests::shadings_with_large_sampled_functions_paint_quickly`, `raster::tests::sampled_functions_interpolate_their_grid_as_before`; (32) `raster::tests::decoded_images_serve_every_size_they_are_drawn_at`; (33) `raster::tests::patch_meshes_keep_their_corner_colours`; (34) `raster::tests::spot_colour_images_run_their_tint_transform_once_per_sample_value`; (35) `mesh_extent_tests::{a_mesh_paints_its_triangles_alone, a_mesh_keeps_the_graphics_state_alpha, a_mesh_off_the_canvas_paints_nothing}` (`cargo test -p hayro`); (36) `raster::tests::mesh_triangles_sharing_an_edge_leave_no_seam`; (38) `raster::tests::default_colour_spaces::device_rgb_operators_draw_in_default_rgb`, `raster::tests::default_colour_spaces::device_cmyk_operators_draw_in_default_cmyk`, `raster::tests::default_colour_spaces::device_gray_operators_draw_in_default_gray`, `raster::tests::default_colour_spaces::device_images_draw_in_the_default_colour_spaces`, `raster::tests::default_colour_spaces::device_shadings_draw_in_the_default_colour_spaces`, `raster::tests::default_colour_spaces::an_image_drawn_with_and_without_a_default_keeps_both_colours`, `raster::tests::default_colour_spaces::forms_inherit_the_default_colour_spaces_they_do_not_override`, `raster::tests::default_colour_spaces::unusable_defaults_leave_device_colour_alone`; (39) `raster::tests::knockout_groups_replace_what_their_elements_cover`; (40) `raster::tests::visibility_expressions_decide_what_a_membership_dictionary_shows`; (41) `raster::tests::a_multiply_highlight_inside_a_transparency_group_keeps_the_text`; (42) `raster::tests::text_rendering_modes_other_than_invisible_draw_type3_glyphs_as_filled`; (43) `raster::tests::lab_honours_the_space_white_point`, `raster::tests::indexed_palettes_over_lab_keep_their_colours` |; (44) `text::tests::identity_to_unicode_names_give_text`; (46) `raster::tests::stencil_masks_are_drawn_at_the_fill_opacity_once`; (47) `raster::tests::a_five_point_polyline_is_not_painted_as_its_bounding_box`, `raster::tests::a_path_that_visits_every_corner_out_of_order_is_not_a_rectangle`, `raster::tests::a_rectangle_is_still_painted_as_a_rectangle`; (48) `raster::tests::icc_profile_cache::icc_profiles_with_identical_dictionaries_draw_their_own_colours`, `raster::tests::icc_profile_cache::icc_profiles_used_again_are_read_once` | | hayro | 0.7.1 | Apache-2.0 OR MIT | (1) `RenderSettings::{x_offset, y_offset}` to render one tile of a large page; (2) images over `MAX_IMAGE_PIXELS` (2^28) are skipped (a fuzzed `/W 4294967295` hung in resampling); (3) `tiling_cell_scale` keeps a tiling pattern's cell pixmap within 3000 px a side whatever its `/XStep`/`/YStep` (a fuzzed step far beyond `/BBox` allocated 2 GB); (4) stroke widths clamped to 4 × (canvas width + height) device pixels (a fuzzed `/LW 9223372036854775807` allocated 10 GB in stroke expansion); (5) `usable_dash_pattern`: a stroke whose dash pattern could cut it into more than `MAX_DASHES_PER_STROKE` (250,000) pieces, counting kurbo's restart of the pattern at every subpath and `ClosePath`, is drawn solid (a fuzzed `/D [[11] 0]` on a line from x = 92234775807 allocated over 5 GB, `[0.000001] 0 d` on a 20 pt line over 10 GB); (6) a dash pattern with a negative or non-finite entry, or a period that isn't positive, is drawn solid (`[-1 -1] 0 d` looped for ever in kurbo); (7) `render_size` and `render_into`: render into a buffer the caller allocates (PdfCraft renders into pixel-aligned memory that the GUI takes over as texture data); `render` is `render_into` on a new pixmap; (8) checked image source/target/RGBA, alpha-premultiplication scratch and intermediate sizes before resampling, finite transforms and exact source/mask buffers; sources wider than u16 may shrink before backend limits, with a 2^20 source-axis cap and conservative 128 MiB filter-planning estimate; an over-limit or failed resize keeps the original image resolution and geometry when the source fits the backend, otherwise skips it, with a logged warning; staging, resampling output/scratch, RGBA conversion and padding buffers use fallible reserves; (9) `may_paint_canvas`: a fill or stroke whose control-point box, grown by how far its paint reaches past the path (half the stroke width times the larger of the miter limit and √2) plus 0.5 user units and 2 pixels, misses the canvas is not drawn, so a tile no longer strokes and fills every path of its page (item 5 of #623); `set_skip_offscreen_paths` turns this off for the regression test; (10) backport of hayro 0.8: shadings that `as_svg_gradient` can represent are painted as a `vello_cpu` linear or two-point radial gradient instead of a texture sampled pixel by pixel through the shading function (InDesign fades are luminosity soft masks of one full-page axial `sh`: 40 on a report page took 15 s at 144 dpi, now under 1 s); without upstream's half-pixel shift, which suits vello_cpu 0.3 but moves 0.0.8's gradients off pixel centres; (11) `may_paint_rect`: an image whose stream-size box, mapped to the device, misses the canvas is skipped before decoding (decoded data can only shrink against that box), so a tile of an image-heavy page no longer decodes, resamples and rasterizes every image on it; (12) preview renders (`InterpreterSettings::preview`: backdrops, thumbnails, grid cells, signature-drag backgrounds) skip images under `PREVIEW_MIN_IMAGE_AREA` device pixels, while final renders always draw them; (13) reading colours (#897): `RenderSettings::color_replacement` takes a `ColorReplacement` (text colour, background, `only_black`, `line_art`) that remaps solid text and line-art paint while rendering, glyph outlines counting as text and stencil-mask ink as line art; images, image alpha masks, soft masks and pattern cells keep their colours; `render_pdf` gains a `color_replacement: Option` argument (pass `None` for true colours); the upstream-only `[dev-dependencies.log]` entry is removed from `Cargo.toml` (log is a normal dependency since the logging patch), so `cargo test -p hayro` runs this crate's unit tests; (14) a shrunk, interpolated, axis-aligned image is snapped to whole device pixels: shrunk once to its rounded device rectangle and drawn 1:1 at the rounded corner, instead of a ceil-sized raster resampled again at a fractional scale and offset (`snap_minified_image`; scans looked soft next to Acrobat); (15) a soft mask of another size than its image is filtered when shrunk, whatever its `/Interpolate` (#1649): such a mask carries the shape on its own (Word saves text as a tiny black image with a glyph-bitmap mask), and nearest-neighbour dropped every stroke thinner than the shrink factor; (16) under a soft mask or a blend mode, axial and radial shadings are sampled instead of drawn as native gradients (#2128): vello_cpu 0.0.8 copies an opaque gradient over fully covered tiles without the mask or blend mode, so PowerPoint's faded gradient fills came out solid; (17) the renderer keeps annotation paints out of the reading-colours mapping between `begin_annotations` and `end_annotations` (#1548); (18) the page renders as one transparency group; under reading colours its background is painted inside it as the initial backdrop, so a Multiply highlight blends with the dark background instead of staying an opaque block, and otherwise the background stays outside, so a blend mode over an empty part of the page keeps its colour instead of blending with the white paper (#2970); (19) `set_paint` samples a mesh shading's texture over its triangles alone (`EncodedShadingPattern::extent`, padded rather than repeated) and skips a mesh with none on the canvas: a BricsCAD plan's 310 one-triangle `sh` each covered the whole tile, 2 s per 1024 px tile at any zoom (the watchdog gave the page up on a busy machine), now about 70 ms; (20) a tiling pattern whose step is too large for a cell pixmap, where what is painted (a fill, or a stroke with its width) can only show the first cell (PowerPoint's `/XStep 2147480100` for a picture fill drawn once), has its `/BBox` drawn at full scale with a transparent border and padded instead of tiled (`shows_one_cell`; a cell whose scale isn't finite and positive, from a `/BBox` too thin to divide by, keeps the old path: `usable_cell_scale`); shrinking the cell to fit the 3000 px limit of (3) left the picture under a pixel, so network diagrams lost their pictures (#3349); (22) knockout groups: while one is innermost, fills, strokes, glyph outlines and shadings are drawn with `Compose::Copy` (weighted by coverage), so each element replaces what earlier elements of the group drew instead of compositing over it; an image or nested group, drawn into a layer of its own, first has what is under its shape cleared (`knockout_image` for an unmasked image's rectangle, only when `draw_image` will draw it, decoded at the size it is drawn; a knockout erase pass that draws a nested group's content with every paint clearing instead of painting, a shading fill's clear clipped to the shading's `/BBox` as its paint is) and then composites over it (#2971) | (1) `raster::tests::tiles_match_full_render`; (2) `raster::tests::absurd_image_dimensions_are_skipped`; (3) `raster::tests::tiling_cells_stay_small_whatever_the_step`; (4) `raster::tests::absurd_predictor_columns_and_line_widths_render`; (5) `raster::tests::dash_patterns_that_cut_a_stroke_into_billions_of_pieces_render`; (6) `raster::tests::invalid_dash_patterns_terminate_and_are_drawn_solid`; (7) `raster::tests::pixels_are_a_plain_renders_bytes_in_a_buffer_a_gui_can_take_over`; (8) `raster::tests::image_resampling_rejects_anisotropic_target_growth`, `raster::tests::image_resampling_bounds_intermediate_before_planning`, `raster::tests::image_resampling_rejects_invalid_sources_and_scales`, `raster::tests::image_resampling_keeps_valid_sizes_and_exact_limits`, `raster::tests::resampling_fallback_preserves_original_geometry_and_pixels`, `raster::tests::image_resampling_wide_sources_shrink_before_backend_side_limits`, `raster::tests::image_resampling_wide_transparent_sources_still_shrink`, `raster::tests::image_resampling_padded_backend_dimensions_stay_checked`, `raster::tests::image_resampling_mismatched_alpha_mask_keeps_pixels_and_placement`; (9) `raster::tests::skipping_paths_off_the_canvas_changes_no_pixel`; (10) `raster::tests::axial_and_radial_shadings_draw_as_native_gradients`; (11) `raster::tests::skipping_paths_off_the_canvas_changes_no_pixel`; (12) `raster::tests::preview_renders_skip_tiny_images_but_finals_draw_everything`; (13) `color_replacement_tests::{black_maps_to_text_colour, white_maps_to_background, midgrey_interpolates, alpha_copied_verbatim, only_black_threshold, line_art_false_skips_paths_but_not_text}`, `color_replacement_render_tests::{color_replacement_maps_text_and_line_art, color_replacement_maps_stencil_mask_ink}`; (14) `raster::tests::interpolated_minified_image_edges_snap_to_whole_pixels` (`cargo test -p hayro`); (15) `raster::tests::shrunk_soft_mask_larger_than_its_image_keeps_thin_strokes`; (16) `raster::tests::soft_mask_applies_to_a_shading_pattern_fill`; (17), (18) `raster::tests::reading_colors_keep_highlight_annotations_readable`; (19) `mesh_extent_tests::{a_mesh_paints_its_triangles_alone, a_mesh_keeps_the_graphics_state_alpha, a_mesh_off_the_canvas_paints_nothing}` (`cargo test -p hayro`); (20) `raster::tests::a_tiling_pattern_drawn_once_shows_its_cell`, `raster::tests::a_large_step_still_tiles_where_the_next_cell_shows`, `raster::tests::a_tiling_cell_too_thin_to_scale_keeps_the_old_path` |; (22) `raster::tests::knockout_groups_replace_what_their_elements_cover` | hayro | 0.7.1 | Apache-2.0 OR MIT | (1) `RenderSettings::{x_offset, y_offset}` to render one tile of a large page; (2) images over `MAX_IMAGE_PIXELS` (2^28) are skipped (a fuzzed `/W 4294967295` hung in resampling); (3) `tiling_cell_scale` keeps a tiling pattern's cell pixmap within 3000 px a side whatever its `/XStep`/`/YStep` (a fuzzed step far beyond `/BBox` allocated 2 GB); (4) stroke widths clamped to 4 × (canvas width + height) device pixels (a fuzzed `/LW 9223372036854775807` allocated 10 GB in stroke expansion); (5) `usable_dash_pattern`: a stroke whose dash pattern could cut it into more than `MAX_DASHES_PER_STROKE` (250,000) pieces, counting kurbo's restart of the pattern at every subpath and `ClosePath`, is drawn solid (a fuzzed `/D [[11] 0]` on a line from x = 92234775807 allocated over 5 GB, `[0.000001] 0 d` on a 20 pt line over 10 GB); (6) a dash pattern with a negative or non-finite entry, or a period that isn't positive, is drawn solid (`[-1 -1] 0 d` looped for ever in kurbo); (7) `render_size` and `render_into`: render into a buffer the caller allocates (PdfCraft renders into pixel-aligned memory that the GUI takes over as texture data); `render` is `render_into` on a new pixmap; (8) checked image source/target/RGBA, alpha-premultiplication scratch and intermediate sizes before resampling, finite transforms and exact source/mask buffers; sources wider than u16 may shrink before backend limits, with a 2^20 source-axis cap and conservative 128 MiB filter-planning estimate; an over-limit or failed resize keeps the original image resolution and geometry when the source fits the backend, otherwise skips it, with a logged warning; staging, resampling output/scratch, RGBA conversion and padding buffers use fallible reserves; (9) `may_paint_canvas`: a fill or stroke whose control-point box, grown by how far its paint reaches past the path (half the stroke width times the larger of the miter limit and √2) plus 0.5 user units and 2 pixels, misses the canvas is not drawn, so a tile no longer strokes and fills every path of its page (item 5 of #623); `set_skip_offscreen_paths` turns this off for the regression test; (10) `/Interpolate` governs the pre-resize too (#624): a minified image that does not request interpolation is pre-resized with nearest-neighbour, planned without the Catmull-Rom filter limits (`image_resampling_size` takes the flag); a Type 3 glyph image is sampled bicubically only when interpolated; a minified image with a matching soft mask is nearest-sampled straight into premultiplied RGBA; (11) `may_paint_rect`: an image whose stream-size box, mapped to the device, misses the canvas is skipped before decoding (decoded data can only shrink against that box), so a tile of an image-heavy page no longer decodes, resamples and rasterizes every image on it; (12) preview renders (`InterpreterSettings::preview`: backdrops, thumbnails, grid cells, signature-drag backgrounds) skip images under `PREVIEW_MIN_IMAGE_AREA` device pixels, while final renders always draw them; (13) reading colours (#897): `RenderSettings::color_replacement` takes a `ColorReplacement` (text colour, background, `only_black`, `line_art`) that remaps solid text and line-art paint while rendering, glyph outlines counting as text and stencil-mask ink as line art; images, image alpha masks, soft masks and pattern cells keep their colours; `render_pdf` gains a `color_replacement: Option` argument (pass `None` for true colours); the upstream-only `[dev-dependencies.log]` entry is removed from `Cargo.toml` (log is a normal dependency since the logging patch), so `cargo test -p hayro` runs this crate's unit tests; (14) a shrunk, interpolated, axis-aligned image is snapped to whole device pixels: shrunk once to its rounded device rectangle and drawn 1:1 at the rounded corner, instead of a ceil-sized raster resampled again at a fractional scale and offset (`snap_minified_image`; scans looked soft next to Acrobat); (15) a soft mask of another size than its image is filtered when shrunk, whatever its `/Interpolate` (#1649): such a mask carries the shape on its own (Word saves text as a tiny black image with a glyph-bitmap mask), and nearest-neighbour dropped every stroke thinner than the shrink factor; (16) under a soft mask or a blend mode, axial and radial shadings are sampled instead of drawn as native gradients (#2128): vello_cpu 0.0.8 copies an opaque gradient over fully covered tiles without the mask or blend mode, so PowerPoint's faded gradient fills came out solid; (17) the renderer keeps annotation paints out of the reading-colours mapping between `begin_annotations` and `end_annotations` (#1548); (18) the page renders as one transparency group; under reading colours its background is painted inside it as the initial backdrop, so a Multiply highlight blends with the dark background instead of staying an opaque block, and otherwise the background stays outside, so a blend mode over an empty part of the page keeps its colour instead of blending with the white paper (#2970); (19) `set_paint` samples a mesh shading's texture over its triangles alone (`EncodedShadingPattern::extent`, padded rather than repeated) and skips a mesh with none on the canvas: a BricsCAD plan's 310 one-triangle `sh` each covered the whole tile, 2 s per 1024 px tile at any zoom (the watchdog gave the page up on a busy machine), now about 70 ms; (20) a tiling pattern whose step is too large for a cell pixmap, where what is painted (a fill, or a stroke with its width) can only show the first cell (PowerPoint's `/XStep 2147480100` for a picture fill drawn once), has its `/BBox` drawn at full scale with a transparent border and padded instead of tiled (`shows_one_cell`; a cell whose scale isn't finite and positive, from a `/BBox` too thin to divide by, keeps the old path: `usable_cell_scale`); shrinking the cell to fit the 3000 px limit of (3) left the picture under a pixel, so network diagrams lost their pictures (#3349); (22) knockout groups: while one is innermost, fills, strokes, glyph outlines and shadings are drawn with `Compose::Copy` (weighted by coverage), so each element replaces what earlier elements of the group drew instead of compositing over it; an image or nested group, drawn into a layer of its own, first has what is under its shape cleared (`knockout_image` for an unmasked image's rectangle, only when `draw_image` will draw it, decoded at the size it is drawn; a knockout erase pass that draws a nested group's content with every paint clearing instead of painting, a shading fill's clear clipped to the shading's `/BBox` as its paint is) and then composites over it (#2971) | (1) `raster::tests::tiles_match_full_render`; (2) `raster::tests::absurd_image_dimensions_are_skipped`; (3) `raster::tests::tiling_cells_stay_small_whatever_the_step`; (4) `raster::tests::absurd_predictor_columns_and_line_widths_render`; (5) `raster::tests::dash_patterns_that_cut_a_stroke_into_billions_of_pieces_render`; (6) `raster::tests::invalid_dash_patterns_terminate_and_are_drawn_solid`; (7) `raster::tests::pixels_are_a_plain_renders_bytes_in_a_buffer_a_gui_can_take_over`; (8) `raster::tests::image_resampling_rejects_anisotropic_target_growth`, `raster::tests::image_resampling_bounds_intermediate_before_planning`, `raster::tests::image_resampling_rejects_invalid_sources_and_scales`, `raster::tests::image_resampling_keeps_valid_sizes_and_exact_limits`, `raster::tests::resampling_fallback_preserves_original_geometry_and_pixels`, `raster::tests::image_resampling_wide_sources_shrink_before_backend_side_limits`, `raster::tests::image_resampling_wide_transparent_sources_still_shrink`, `raster::tests::image_resampling_padded_backend_dimensions_stay_checked`, `raster::tests::image_resampling_mismatched_alpha_mask_keeps_pixels_and_placement`; (9) `raster::tests::skipping_paths_off_the_canvas_changes_no_pixel`; (10) `raster::tests::non_interpolated_images_are_not_smoothed_when_minified`, `raster::tests::non_interpolated_gray_images_are_not_smoothed_when_minified`, `raster::tests::non_integer_minification_samples_the_same_columns`, `raster::tests::non_integer_minification_samples_colour_and_mask_from_the_same_columns`, `raster::tests::non_interpolated_alpha_images_keep_their_colours_and_mask_when_minified`, `raster::tests::non_interpolated_type3_glyph_images_are_not_smoothed_when_minified`, `raster::tests::image_resampling_nearest_accepts_wide_sources_the_filter_refuses`, `raster::tests::wide_non_interpolated_sources_draw_after_shrinking`; (11) `raster::tests::skipping_paths_off_the_canvas_changes_no_pixel`; (12) `raster::tests::preview_renders_skip_tiny_images_but_finals_draw_everything`; (13) `color_replacement_tests::{black_maps_to_text_colour, white_maps_to_background, midgrey_interpolates, alpha_copied_verbatim, only_black_threshold, line_art_false_skips_paths_but_not_text}`, `color_replacement_render_tests::{color_replacement_maps_text_and_line_art, color_replacement_maps_stencil_mask_ink}`; (14) `raster::tests::interpolated_minified_image_edges_snap_to_whole_pixels` (`cargo test -p hayro`); (15) `raster::tests::shrunk_soft_mask_larger_than_its_image_keeps_thin_strokes`; (16) `raster::tests::soft_mask_applies_to_a_shading_pattern_fill`; (17), (18) `raster::tests::reading_colors_keep_highlight_annotations_readable`; (19) `mesh_extent_tests::{a_mesh_paints_its_triangles_alone, a_mesh_keeps_the_graphics_state_alpha, a_mesh_off_the_canvas_paints_nothing}` (`cargo test -p hayro`); (20) `raster::tests::a_tiling_pattern_drawn_once_shows_its_cell`, `raster::tests::a_large_step_still_tiles_where_the_next_cell_shows`, `raster::tests::a_tiling_cell_too_thin_to_scale_keeps_the_old_path` |; (22) `raster::tests::knockout_groups_replace_what_their_elements_cover` @@ -63,6 +63,13 @@ Covered by `filter::jpx::tests::jpx_palette_is_applied_without_a_color_space`, `filter::jpx::tests::palette_colours_are_eight_bit_and_other_samples_keep_their_precision` and `raster::tests::a_jp2_palette_keeps_its_colours_whatever_the_index_precision`. +Glyph names outside the Adobe Glyph List (#3169, #3170): hayro-interpret's `glyph_name_to_text` +maps a name by the glyph list specification's full mapping (drop from the first `.`, split on `_`, +each part through the list, `uniXXXX…` or `uXXXX`; a `uni` part with any group that isn't a +character maps to nothing), so `f_i`, `f_l` and `hyphen.case` read as text. A name that only +numbers the glyph (`C39`, `c251`) still reads as nothing: the code's StandardEncoding character +would be a guess, wrong wherever the font's /Encoding puts the glyph elsewhere (review on #3239). Covered by `text::tests::glyph_names_outside_the_glyph_list_still_read_as_text`. + Text extraction: hayro-interpret exposes `OutlineGlyph::is_vertical` from a Type0 font’s CMap writing mode. PdfCraft follows negative glyph-space y for WMode 1 so upright vertical columns can be searched (#126). Lab Bradford follow-up to #1237: Lab adapts XYZ to D65 with white-point ratios @@ -71,3 +78,14 @@ black-point normalization is unchanged. `lab_bradford_tests` checks independent D50/equal-energy neutral and chromatic oracles, D65 identity, and direct/Indexed consistency (including an asymmetric chromatic Range), and zero/overflowing cone-response fallback. + +## Clipping-only text reporting (#3109 / #3116) + +`hayro-interpret/src/interpret/text.rs` reports mode 7 glyphs after installing their +clip, and `hayro-interpret/src/types.rs` distinguishes `GlyphDrawMode::Clip` from +invisible OCR text. `hayro/src/renderer.rs` does not paint either mode directly. +This preserves clipping-only titles when `TextFilter.images` excludes invisible OCR. +Coverage: `text::tests::clipping_mode_filter_keeps_titles_and_excludes_ocr` (plain +and ActualText filtered extraction), and +`text::tests::clipping_mode_text_is_extracted_and_searchable_like_other_rendering_modes` +(extraction, search, glyph geometry, clip-through pixels and no-paint pixels). diff --git a/vendor/hayro-interpret/src/font/cid.rs b/vendor/hayro-interpret/src/font/cid.rs index b99dcbc11..145135925 100644 --- a/vendor/hayro-interpret/src/font/cid.rs +++ b/vendor/hayro-interpret/src/font/cid.rs @@ -45,9 +45,16 @@ pub(crate) struct Type0Font { /// PdfCraft patch: whether the CIDs belong to an Adobe character collection (Japan1, GB1, /// CNS1, Korea1), whose glyph numbering no substitute other than a CID-keyed CFF font shares. adobe_collection: bool, + /// PdfCraft patch: `/ToUnicode /Identity-H` (a name, not a CMap stream), which CAD exporters + /// such as WSCAD write for fonts whose two-byte codes are UTF-16 code units. Acrobat shows + /// such text as the code's Unicode value; the identity CMap the name resolves to maps codes + /// to CIDs only and gave no text at all. + identity_to_unicode: bool, } impl Type0Font { + /// Reads a Type 0 font from its dictionary: the descendant CID font, the encoding CMap and + /// the code-to-Unicode map (`/ToUnicode`, the `/Identity-H` name, or the UCS2 CMap). pub(crate) fn new( dict: &Dict<'_>, font_resolver: &FontResolverFn, @@ -117,6 +124,9 @@ impl Type0Font { let cache_key = dict.cache_key(); let mut to_unicode = read_to_unicode(dict, cmap_resolver); + let identity_to_unicode = dict + .get::>(TO_UNICODE) + .is_some_and(|n| matches!(n.as_ref(), b"Identity-H" | b"Identity-V")); let mut to_unicode_is_cid_indexed = false; // If there is no ToUnicode map, try to get the UCS2 CMap. @@ -162,6 +172,7 @@ impl Type0Font { fallback, to_unicode_is_cid_indexed, adobe_collection, + identity_to_unicode, }) } @@ -430,7 +441,16 @@ impl Type0Font { } } + /// The text a character code stands for, or `None` when the font gives none (or, for an + /// identity `/ToUnicode` name, when the code is a control or a lone surrogate). pub(crate) fn char_code_to_unicode(&self, code: u32) -> Option { + if self.identity_to_unicode { + // Controls and lone surrogates are not text. The same rule builds the editing + // map, `identity_to_unicode` in crates/fonts/src/pdf.rs: keep both alike. + return char::from_u32(code) + .filter(|c| !c.is_control()) + .map(BfString::Char); + } if let Some(to_unicode) = &self.to_unicode { let key = if self.to_unicode_is_cid_indexed { self.code_to_cid(code).unwrap_or(0) diff --git a/vendor/hayro-interpret/src/font/mod.rs b/vendor/hayro-interpret/src/font/mod.rs index 7c8c8d086..103ee3961 100644 --- a/vendor/hayro-interpret/src/font/mod.rs +++ b/vendor/hayro-interpret/src/font/mod.rs @@ -669,6 +669,49 @@ pub(crate) fn glyph_name_to_unicode(name: &str) -> Option { }) } +/// PdfCraft patch: the text of a glyph name `glyph_name_to_unicode` doesn't know, by the +/// Adobe Glyph List Specification's full mapping: drop everything from the first `.` +/// (`hyphen.case`, `a.sc`), split the rest on `_` into components (`f_i`, `f_f_l`), and map each +/// one through the glyph list, as `uniXXXX[XXXX…]` (BMP values, four hex digits each) or as +/// `uXXXX` to `uXXXXXX`. A component it can't map, or a `uni` component with any group that +/// isn't a character, adds nothing; `None` when nothing is left. +pub(crate) fn glyph_name_to_text(name: &str) -> Option { + let hex = |h: &str| u32::from_str_radix(h, 16).ok().and_then(char::from_u32); + // The specification's digits are uppercase only (`uni20ac` maps to nothing, review on + // #3239). + let upper_hex = |d: &str| d.bytes().all(|b| matches!(b, b'0'..=b'9' | b'A'..=b'F')); + let base = name.split('.').next().unwrap_or_default(); + let mut text = String::new(); + for part in base.split('_') { + if let Some(s) = glyph_names::get(part) { + text.push_str(s); + } else if let Some(digits) = part + .strip_prefix("uni") + .filter(|d| !d.is_empty() && d.len() % 4 == 0 && upper_hex(d)) + { + // All ASCII, so every fourth byte is a char boundary. A group that isn't a + // character (a surrogate) makes the whole component map to nothing. + let chars: Option = (0..digits.len()) + .step_by(4) + .map(|at| digits.get(at..at + 4).and_then(hex)) + .collect(); + text.extend(chars); + } else if let Some(c) = part + .strip_prefix('u') + .filter(|d| (4..=6).contains(&d.len()) && upper_hex(d)) + .and_then(hex) + { + text.push(c); + } + } + let mut chars = text.chars(); + match (chars.next(), chars.next()) { + (None, _) => None, + (Some(c), None) => Some(BfString::Char(c)), + _ => Some(BfString::String(text)), + } +} + pub(crate) fn unicode_from_name(name: &str) -> Option { let convert = |input: &str| u32::from_str_radix(input, 16).ok().and_then(char::from_u32); diff --git a/vendor/hayro-interpret/src/font/standard_font.rs b/vendor/hayro-interpret/src/font/standard_font.rs index 139454a0b..1f386a2ce 100644 --- a/vendor/hayro-interpret/src/font/standard_font.rs +++ b/vendor/hayro-interpret/src/font/standard_font.rs @@ -397,7 +397,7 @@ impl StandardKind { }) } - fn code_to_ps_name(&self, code: u8) -> Option<&str> { + pub(crate) fn code_to_ps_name(&self, code: u8) -> Option<&str> { let bf = self.base_font; self.encodings diff --git a/vendor/hayro-interpret/src/font/true_type.rs b/vendor/hayro-interpret/src/font/true_type.rs index b36266985..c65043996 100644 --- a/vendor/hayro-interpret/src/font/true_type.rs +++ b/vendor/hayro-interpret/src/font/true_type.rs @@ -2,7 +2,7 @@ use crate::font::blob::{CffFontBlob, OpenTypeFontBlob}; use crate::font::generated::{glyph_names, mac_os_roman, mac_roman, standard, win_ansi}; use crate::font::standard_font::StandardKind; use crate::font::{ - Encoding, FallbackFontQuery, FontFlags, glyph_name_to_unicode, read_to_unicode, + Encoding, FallbackFontQuery, FontFlags, glyph_name_to_text, glyph_name_to_unicode, read_to_unicode, strip_subset_prefix, unicode_from_name, }; use crate::util::OptionLog; @@ -190,10 +190,11 @@ impl TrueTypeFont { } match &self.kind { - Kind::Embedded(e) => e - .code_to_unicode_name(code as u8) - .and_then(glyph_name_to_unicode) - .map(BfString::Char), + // PdfCraft patch (#3169): a name outside the glyph list reads by the glyph list + // specification's full mapping (`f_i`, `hyphen.case`). + Kind::Embedded(e) => e.code_to_unicode_name(code as u8).and_then(|name| { + glyph_name_to_unicode(name).map(BfString::Char).or_else(|| glyph_name_to_text(name)) + }), Kind::Standard(s) => s.char_code_to_unicode(code as u8).map(BfString::Char), } diff --git a/vendor/hayro-interpret/src/font/type1.rs b/vendor/hayro-interpret/src/font/type1.rs index 02f76bfb8..4ef5f953f 100644 --- a/vendor/hayro-interpret/src/font/type1.rs +++ b/vendor/hayro-interpret/src/font/type1.rs @@ -2,7 +2,8 @@ use crate::font::blob::{CffFontBlob, Type1FontBlob}; use crate::font::standard_font::{StandardFont, StandardKind, select_standard_font}; use crate::font::true_type::{Width, read_encoding, read_widths}; use crate::font::{ - Encoding, FallbackFontQuery, glyph_name_to_unicode, normalized_glyph_name, read_to_unicode, + Encoding, FallbackFontQuery, glyph_name_to_text, glyph_name_to_unicode, normalized_glyph_name, + read_to_unicode, }; use crate::{CMapResolverFn, CacheKey, FontResolverFn}; use hayro_cmap::{BfString, CMap}; @@ -117,11 +118,25 @@ impl Type1Font { } let code = char_code as u8; - match &self.1 { - Kind::Standard(s) => s.char_code_to_unicode(code).map(BfString::Char), - Kind::Cff(c) => c.char_code_to_unicode(code).map(BfString::Char), - Kind::Type1(t) => t.char_code_to_unicode(code).map(BfString::Char), + let found = match &self.1 { + Kind::Standard(s) => s.char_code_to_unicode(code), + Kind::Cff(c) => c.char_code_to_unicode(code), + Kind::Type1(t) => t.char_code_to_unicode(code), + }; + if let Some(c) = found { + return Some(BfString::Char(c)); } + // PdfCraft patch (#3169, #3170): a name outside the glyph list still reads by the + // glyph list specification's full mapping (`f_i`, `hyphen.case`). A name that only + // numbers the glyph (`C39`, `c251`) says nothing about the character and reads as + // nothing: the code's StandardEncoding character would be a guess, wrong wherever the + // font's /Encoding puts the glyph elsewhere (review on #3239). + let name = match &self.1 { + Kind::Standard(s) => s.code_to_ps_name(code), + Kind::Cff(c) => c.code_to_ps_name(code), + Kind::Type1(t) => t.code_to_ps_name(code), + }?; + glyph_name_to_text(name) } } diff --git a/vendor/hayro-interpret/src/interpret/text.rs b/vendor/hayro-interpret/src/interpret/text.rs index ddec0400e..cd5e147f9 100644 --- a/vendor/hayro-interpret/src/interpret/text.rs +++ b/vendor/hayro-interpret/src/interpret/text.rs @@ -126,6 +126,16 @@ pub(crate) fn show_glyph<'a>( } TextRenderingMode::Clip => { clip_glyph(ctx, glyph, glyph_transform); + // PdfCraft patch: report clipping text separately from invisible OCR text. + // Extraction filters must retain text that is filled through its + // clipping path without rendering devices drawing it directly. + device.draw_glyph( + glyph, + ctx.get().ctm, + glyph_transform, + &get_paint(ctx, false), + &GlyphDrawMode::Clip, + ); } TextRenderingMode::FillAndClip => { clip_glyph(ctx, glyph, glyph_transform); diff --git a/vendor/hayro-interpret/src/types.rs b/vendor/hayro-interpret/src/types.rs index 64d2b849b..685ed3e99 100644 --- a/vendor/hayro-interpret/src/types.rs +++ b/vendor/hayro-interpret/src/types.rs @@ -284,6 +284,8 @@ pub enum GlyphDrawMode { Fill, /// Draw using a stroke. Stroke(StrokeProps), + /// PdfCraft patch: text that shapes a clip; report it without painting a glyph. + Clip, /// Invisible text (for text extraction but not visual rendering). Invisible, } diff --git a/vendor/hayro/src/renderer.rs b/vendor/hayro/src/renderer.rs index 7f6eb88bd..8909cb62a 100644 --- a/vendor/hayro/src/renderer.rs +++ b/vendor/hayro/src/renderer.rs @@ -1913,8 +1913,8 @@ impl<'a> Device<'a> for Renderer { GlyphDrawMode::Stroke(s) => { Self::stroke_glyph(self, glyph, transform, glyph_transform, paint, s); } - GlyphDrawMode::Invisible => { - // Don't render invisible text for visual output + GlyphDrawMode::Invisible | GlyphDrawMode::Clip => { + // PdfCraft patch: neither mode paints; clipping was installed by the interpreter. } } }