Merge r2 track t07: text-extract-select

# Conflicts:
#	crates/edit/src/tests.rs
#	crates/render/src/raster.rs
#	vendor/README.md
This commit is contained in:
aiongg
2026-10-22 02:20:03 +00:00
23 changed files with 740 additions and 36 deletions
+2
View File
@@ -189,6 +189,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
- **2026-10-20 (M6, calculation order page):** Prepare a form ▸ Calculation order, a page of its own, lists the calculated fields in the order they calculate with their formulas and values, moved with Up/Down arrows (as Acrobat's Set Field Calculation Order), Alt+↑/↓ or by dragging. A card says what is wrong with the order and holds Fix order; a loop has a card of its own that opens one of its formulas. On the page, each calculated field shows its step, and arrows show what the field in focus reads and what reads it. Estimates unchanged.
- **2026-10-20 (M6, calculation order):** The order a form's fields calculate in (`/CO`) is read, checked, fixed and set headlessly with `form_calculation_order`: it says which fields read one calculated later (they show stale values), which read each other in a loop, and which the PDF's order leaves out; Fix order moves only what must move, in one undo step, and the form recalculates. Editing a formula no longer moves its field to the end of the order, and a field the order lists twice calculates once. Estimates unchanged.
- **2026-10-20 (M2, #3574):** Forms without Resources use page Properties for ActualText and layer visibility. Synthetic tests cover boundaries, nested/shared forms, and save/reopen. Estimates unchanged.
- **2026-10-20 (M2, PR #3348 review):** Tracking samples retain short repeated punctuation (up to four glyphs), keeping tracked ellipses intact and searchable while longer TOC leaders still preserve title word spaces. Synthetic regressions cover isolated and surrounded ellipses. Estimates unchanged.
- **2026-10-19 (M6, renames keep formulas):** Renaming a field renames it in every calculation that reads it, in the same undo step: `AFSimple_Calculate` field lists, simplified field notation (the comment and the JavaScript after it) and `getField` calls naming it in one string literal; the rest of each script is kept as written. A sum used to drop the renamed field, notation read it as 0 and a script kept a stale value. Field Properties says how many formulas changed and `form_set_props` lists them as `formulas_updated`. Names built at run time and other events' scripts are not rewritten. Estimates unchanged.
- **2026-10-19 (M7, #3591):** A background added to a page that already has one is painted over the old one instead of going under it and staying hidden. Behind the content, backgrounds come first, in the order added, and then behind-content watermarks, so a watermark added after a background no longer disappears. Synthetic regressions in `crates/edit` check the order. Estimates unchanged.
@@ -210,6 +211,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
- **2026-10-18 (M5, PR #1971 review):** Nested FDF annotation values that identify a source `/Page` or `/Pages` dictionary become null at every copy depth. A synthetic three-page PDF saved and reopened before export verifies that destination actions cannot pull source page content into the FDF, while unrelated nested metadata and shared references remain intact. FDF import semantics, copied-graph limits and estimates are unchanged.
- **2026-10-15 (M5, PR #1971 refresh):** Nested FDF annotation values retain shared or cyclic copied graphs. Discovery borrows direct values and records annotation positions with lazy array owners; retained entries are budgeted before cloning. Synthetic stress fixtures include repeated aliases and saved-PDF cache eviction. Limits cover the copied graph; source parsing and page discovery retain their existing limits, with no total source-memory bound. FDF import semantics and estimates are unchanged.
- **2026-10-13 (M5, nested FDF values):** Comment export copies and renumbers nested indirect values, retaining geometry, colours, borders and shared graphs. Graph/data limits return an error before the destination is written; synthetic saved-PDF and automation exchange regressions. Estimates unchanged.
- **2026-10-18 (M2, #3086):** Text extraction estimates letter tracking without repeated punctuation gaps so dotted TOC leaders do not swallow word spaces. Synthetic PDF regressions cover leader lengths, phrase search and selection; strongly tracked text stays intact. Estimates unchanged.
- **2026-10-18 (M6, #2892):** Push-button normal labels are editable per widget through Options and form_set_props, independently of field names; rendering, undo/redo and incremental/full save/reopen are covered. Estimates unchanged.
+6
View File
@@ -153,3 +153,9 @@ sync_github_name = true
real_name = "Eric Kaden"
display_name = "Eric Kaden"
sync_github_name = false
[people.MatrixMagician]
sync_github_name = true
[people.Lcoding15]
real_name = "Livio Oswald"
+3 -1
View File
@@ -263,7 +263,9 @@ pub(crate) fn mcid_texts(doc: &Document, page: &Dict) -> HashMap<i64, String> {
tm = tlm;
}
let pieces: Vec<&Object> = match op.op.as_slice() {
b"TJ" => op.operands.first().and_then(Object::as_array).map(|a| a.iter().collect()).unwrap_or_default(),
// A surplus array (`[(A)] [(B)] TJ`) is malformed, but the operator still takes the
// one immediately before it, as the renderer and every other reader here do (#3569).
b"TJ" => op.operands.last().and_then(Object::as_array).map(|a| a.iter().collect()).unwrap_or_default(),
_ => op.operands.last().into_iter().collect(),
};
for piece in pieces {
+10
View File
@@ -926,6 +926,16 @@ fn an_untagged_document_has_no_headings() {
assert!(headings(&pdf(&objs, "")).is_empty());
}
/// `[(FIRSTARR)] [(LASTARR)] TJ` is malformed (a surplus array operand), but the operator still
/// draws only the one immediately before it, as the renderer and every other reader agree (#3569)
/// -- not the first, which is never drawn at all.
#[test]
fn a_heading_with_a_tj_surplus_array_reads_the_one_the_page_draws() {
let doc = heading_page(&["BT /F1 20 Tf 72 700 Td [(FIRSTARR)] [(LASTARR)] TJ ET".into()]);
let titles: Vec<String> = headings(&doc).into_iter().map(|h| h.title).collect();
assert_eq!(titles, ["LASTARR"]);
}
/// The text of every `"…"` literal with a space in it, outside comments and byte strings.
fn sentences(source: &str) -> Vec<String> {
let b = source.as_bytes();
+35
View File
@@ -5966,3 +5966,38 @@ fn equal_size_rows_need_the_tighter_tolerance_even_without_overlap() {
let blocks = text::text_blocks(&doc, 0).unwrap();
assert_eq!(blocks.len(), 2, "two separate rows stay two paragraphs, not one: {blocks:?}");
}
/// A `TJ` with a surplus array operand (`[(FIRSTARR)] [(LASTARR)] TJ`) is malformed, but the
/// operator still takes the one immediately before it, as the renderer, MuPDF and pdftotext do
/// (#3569) -- not the first, which this reader took before, showing a word the page never draws.
#[test]
fn a_tj_with_a_surplus_array_reads_the_last_one() {
let doc = text_page("BT /F1 24 Tf 20 100 Td [(FIRSTARR)] [(LASTARR)] TJ ET");
let lines = text::text_lines(&doc, 0).unwrap();
assert_eq!(lines.len(), 1);
assert_eq!(lines[0].text, "LASTARR", "the operand immediately before TJ, as the page draws: {lines:?}");
}
/// A producer wrote `0.0001 Tc` on one word of a line (Word through macOS Quartz). That spacing
/// can't be seen, so the word stays on its line; spacing that can be seen still keeps runs apart.
#[test]
fn a_word_with_invisible_extra_spacing_stays_on_its_line() {
let page = |spacing: &str| {
text_page(&format!(
"BT /F1 12 Tf 1 0 0 1 72 700 Tm (for company ) Tj ET \
BT /F1 12 Tf 1 0 0 1 141.36 700 Tm (A) Tj ET \
BT {spacing} Tc /F1 12 Tf 1 0 0 1 149.364 700 Tm (EROSOL) Tj ET"
))
};
let blocks = text::text_blocks(&page("0.0001"), 0).unwrap();
assert_eq!(blocks.iter().map(|b| b.text.as_str()).collect::<Vec<_>>(), ["for company AEROSOL"]);
let mut doc = page("0.0001");
text::rewrite_block(&mut doc, 0, 0, Some("for company AEROSOL LLC"), &BlockStyle::default()).unwrap();
let lines = text::text_lines(&reopen(&doc), 0).unwrap();
assert_eq!(lines.iter().map(|l| l.text.as_str()).collect::<Vec<_>>(), ["for company AEROSOL LLC"]);
assert!((lines[0].rect[0] - 72.0).abs() < 0.01, "{:?}", lines[0].rect);
let blocks = text::text_blocks(&page("0.5"), 0).unwrap();
assert_eq!(blocks.iter().map(|b| b.text.as_str()).collect::<Vec<_>>(), ["for company A", "EROSOL"]);
}
+12 -3
View File
@@ -1085,7 +1085,11 @@ fn interpret(
let Some((name, m)) = ts.font.clone() else { continue };
// Pieces: strings, and TJ number adjustments.
let pieces: Vec<Object> = match op.op.as_slice() {
b"TJ" => op.operands.first().and_then(Object::as_array).cloned().unwrap_or_default(),
// A surplus array (`[(A)] [(B)] TJ`) is malformed, but the operator still
// takes the one immediately before it, as the renderer and every other
// reader here do (#3569) -- not the first, which a reader counting operands
// from the front of a surplus list would otherwise take.
b"TJ" => op.operands.last().and_then(Object::as_array).cloned().unwrap_or_default(),
_ => op.operands.last().cloned().into_iter().collect(),
};
// Tc is already included in the pen advance; TJ may encode tracking instead.
@@ -1384,13 +1388,18 @@ pub fn text_lines(doc: &Document, page: usize) -> Result<Vec<TextLine>, EditErro
/// Joining independent text objects is safe only when they use the same text and graphics
/// transforms and spacing. In particular, a nearby rotated label isn't part of this line.
fn compatible_word(line: &TextLine, shown: &Shown) -> bool {
let same = |a: &[f64], b: &[f64]| a.iter().zip(b).all(|(a, b)| (a - b).abs() < 1e-6);
let within = |a: &[f64], b: &[f64], tolerance: f64| a.iter().zip(b).all(|(a, b)| (a - b).abs() < tolerance);
let same = |a: &[f64], b: &[f64]| within(a, b, 1e-6);
let (a, b) = (&line.origin.state, TextState::of(&shown.state));
// Spacing that differs by less than a thousandth of an em, the unit glyph widths are given in,
// can't be seen. Some producers still write it (`0.0001 Tc` on one word of a line).
let em = shown.state.size.abs() / 1000.0;
line.scope == shown.scope
&& line.paint == shown.paint
&& same(&line.origin.tm[..4], &shown.tm.0[..4])
&& same(&line.origin.ctm, &shown.state.ctm.0)
&& same(&[a.char_spacing, a.word_spacing, a.scale, a.rise], &[b.char_spacing, b.word_spacing, b.scale, b.rise])
&& within(&[a.char_spacing, a.word_spacing], &[b.char_spacing, b.word_spacing], em.max(1e-6))
&& same(&[a.scale, a.rise], &[b.scale, b.rise])
&& a.fill.len() == b.fill.len()
&& a.fill.iter().zip(&b.fill).all(|(a, b)| a.op == b.op && a.operands == b.operands)
}
+71 -5
View File
@@ -1154,7 +1154,8 @@ pub fn glyph_unicode(name: &str) -> Option<char> {
chars.next().filter(|_| chars.len() == 0)
}
/// Code → Unicode for a font: its ToUnicode CMap, else (simple fonts) its encoding.
/// Code → Unicode for a font: its ToUnicode CMap (for a composite font, also the name
/// `/Identity-H` or `/Identity-V`, read as code = Unicode), else (simple fonts) its encoding.
fn unicode_map(doc: &Document, font: &Dict, composite: bool) -> Unicode {
let mut out = Unicode::default();
if !composite {
@@ -1217,14 +1218,44 @@ fn unicode_map(doc: &Document, font: &Dict, composite: bool) -> Unicode {
}
}
}
if let Some(Object::Stream(s)) = font.get(b"ToUnicode").map(|t| doc.resolve(t)).as_deref()
&& let Ok(data) = s.decoded()
{
parse_to_unicode(&data, &mut out);
match font.get(b"ToUnicode").map(|t| doc.resolve(t)).as_deref() {
Some(Object::Stream(s)) => {
if let Ok(data) = s.decoded() {
parse_to_unicode(&data, &mut out);
}
}
// `/ToUnicode /Identity-H` is not a stream, as ISO 32000-2 §9.10.3 requires, but CAD
// exporters (WSCAD among them) write it for composite fonts whose two-byte codes are the
// UTF-16 code units. Acrobat shows them as code = Unicode, and so does the renderer's
// text layer; without it the text could be seen but not edited or found. The same rule
// (code = Unicode, no controls or lone surrogates) is in the vendored renderer,
// `Type0Font::char_code_to_unicode` in vendor/hayro-interpret/src/font/cid.rs: keep both alike.
Some(Object::Name(n)) if composite && matches!(n.as_slice(), b"Identity-H" | b"Identity-V") => {
parse_to_unicode(&identity_to_unicode(), &mut out);
}
_ => {}
}
out
}
/// A ToUnicode CMap mapping each two-byte code to the same UTF-16 code unit, leaving out the
/// surrogates (which are not characters on their own) and the controls (C0, DEL and C1, as the
/// renderer's text layer does). No range crosses a last-byte boundary (§9.10.3).
fn identity_to_unicode() -> Vec<u8> {
let mut ranges = vec![(0x20u32, 0x7Eu32), (0xA0, 0xFF)];
ranges.extend((0x01u32..=0xFF).filter(|h| !(0xD8..=0xDF).contains(h)).map(|h| (h << 8, (h << 8) | 0xFF)));
let mut cmap = String::from("1 begincodespacerange <0000> <FFFF> endcodespacerange\n");
// At most 100 entries per section (§9.7.5.2 limits for CMap files).
for chunk in ranges.chunks(100) {
cmap.push_str(&format!("{} beginbfrange\n", chunk.len()));
for (lo, hi) in chunk {
cmap.push_str(&format!("<{lo:04X}> <{hi:04X}> <{lo:04X}>\n"));
}
cmap.push_str("endbfrange\n");
}
cmap.into_bytes()
}
#[cfg(test)]
mod tests {
use pdfcraft_cos::{Dict, Document, Object, Stream};
@@ -1531,6 +1562,41 @@ mod tests {
assert_eq!(m.decode(&[0, 1, 0, 7, 0, 2]), "M á");
}
/// WSCAD writes `/ToUnicode /Identity-H` (a name, not a CMap stream) on composite fonts whose
/// codes are UTF-16 code units: the codes read as the same Unicode, as in Acrobat. Controls
/// and lone surrogates map to nothing, and a simple font's name is not a ToUnicode map.
#[test]
fn identity_to_unicode_names_read_codes_as_unicode() {
let doc = Document::new_empty();
for name in ["Identity-H", "Identity-V"] {
let f = font(
&mut Document::new_empty(),
vec![
("Subtype", Object::name("Type0")),
("BaseFont", Object::name("RobotoMono-Regular")),
("Encoding", Object::name("Identity-H")),
("DescendantFonts", Object::Array(vec![Object::Dict(Dict::new())])),
("ToUnicode", Object::name(name)),
],
);
let m = Metrics::from_dict(&doc, &f);
assert_eq!(m.decode(&[0, b'Z', 0, 0xFC, 0, b'r', 0x20, 0xAC, 0x4E, 0x2D]), "Zür€中", "{name}");
assert_eq!(m.text_of(0x0001), None);
assert_eq!(m.text_of(0x007F), None, "DEL is a control");
assert_eq!(m.text_of(0x0085), None, "a C1 control");
assert_eq!(m.text_of(0x00A0).as_deref(), Some("\u{A0}"), "the range resumes after C1");
assert_eq!(m.text_of(0x007E).as_deref(), Some("~"));
assert_eq!(m.text_of(0xD800), None);
assert_eq!(m.text_of(0xFFFD).as_deref(), Some("\u{FFFD}"));
}
let simple = font(
&mut Document::new_empty(),
vec![("Subtype", Object::name("TrueType")), ("BaseFont", Object::name("Arial")), ("ToUnicode", Object::name("Identity-H"))],
);
let m = Metrics::from_dict(&doc, &simple);
assert_eq!(m.text_of(0x41).as_deref(), Some("A"), "the encoding still applies");
}
/// Chrome's subsets leave the glyphs of `/DW` width out of `/W` (#1443): "s" (CID 0x56) is
/// still in the font, but nothing is when the font gives no default width to point at.
#[test]
+4 -1
View File
@@ -900,7 +900,10 @@ pub(crate) fn process(doc: &mut Document, scope: &mut Scope<'_>, streams: &[Vec<
tm = tlm;
}
let items: Vec<Object> = match o {
b"TJ" => op.operands.first().and_then(Object::as_array).cloned().unwrap_or_default(),
// A surplus array (`[(A)] [(B)] TJ`) is malformed, but the operator still
// takes the one immediately before it, as the renderer and every other
// reader here do (#3569), so a redaction search matches what is drawn.
b"TJ" => op.operands.last().and_then(Object::as_array).cloned().unwrap_or_default(),
_ => op.operands.last().cloned().into_iter().collect(),
};
// Vertical-writing text is laid out horizontally by `show`, so its glyphs are
+10
View File
@@ -2186,3 +2186,13 @@ fn unused_descriptions_in_shared_or_inherited_resources_are_retired() {
}
mod property_retirement;
/// `[(FIRSTARR)] [(LASTARR)] TJ` is malformed (a surplus array operand), but the operator still
/// draws only the one immediately before it (LASTARR, 7 glyphs, 35 pt wide from x = 10) -- not
/// the first (FIRSTARR, 8 glyphs), which is never drawn at all, as the renderer agrees (#3569).
#[test]
fn a_tj_surplus_arrays_undrawn_glyphs_are_not_seen_under_a_mark() {
let mut doc = one_page(b"BT /F1 10 Tf 10 100 Td [(FIRSTARR)] [(LASTARR)] TJ ET", "", vec![]);
assert_eq!(under(&mut doc, 0, &[[45.0, 95.0, 50.0, 110.0]]), 0, "nothing is drawn past LASTARR's 35pt width");
assert_eq!(under(&mut doc, 0, &[[10.0, 95.0, 45.0, 110.0]]), 7, "LASTARR's 7 glyphs, the one the page draws");
}
+32
View File
@@ -7966,4 +7966,36 @@ trailer << /Root 1 0 R >>
// #1201: /DefaultGray, /DefaultRGB and /DefaultCMYK (vendored hayro-interpret patch 38).
mod default_colour_spaces;
/// #3109 / #3116: clipping-only text (`7 Tr`) installs a clip for subsequent painting
/// but draws no pixels itself.
#[test]
fn test_clip_only_text_raster_preserves_clipping_without_painting() {
let pdf_bytes = |content: &str| {
Arc::new(
format!(
"%PDF-1.7\n1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj\n2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >> endobj\n3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 200 100] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >> endobj\n4 0 obj << /Length {} >> stream\n{content}\nendstream endobj\n5 0 obj << /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >> endobj\ntrailer << /Root 1 0 R >>\n%%EOF",
content.len()
)
.into_bytes(),
)
};
let render = |content: &str| {
let mut r = PageRenderer::new(pdf_bytes(content), RenderConfig::default());
let p = r.render(RenderRequest { page: 0, kind: RequestKind::Pixels, tile: None, scale: 1.0, tag: 0, ..Default::default() });
assert!(p.error.is_none(), "{content}: {:?}", p.error);
p
};
let white = [255, 255, 255, 255];
// The text alone paints nothing.
let p = render("BT /F1 40 Tf 7 Tr 20 60 Td (Hi) Tj ET");
assert!(p.rgba.as_chunks::<4>().0.iter().all(|c| *c == white), "7 Tr alone leaves the page blank");
// A fill afterwards shows only through the text-shaped clip.
let p = render("q BT /F1 40 Tf 7 Tr 20 60 Td (Hi) Tj ET 0 0 1 rg 0 0 200 100 re f Q");
let pixels = p.rgba.as_chunks::<4>().0;
assert!(pixels.contains(&[0, 0, 255, 255]), "fill shows through the 7 Tr clip");
assert!(pixels.contains(&white), "the clip is text-shaped, not the whole page");
}
}
+421 -8
View File
@@ -38,6 +38,22 @@ impl TextFilter {
mod search;
/// Whether `(x, y)` lies in or on the convex quadrilateral `quad`, whichever way it winds. A
/// quad with a NaN corner, or no area, holds nothing.
fn quad_contains(quad: &[[f32; 2]; 4], x: f32, y: f32) -> bool {
let (mut positive, mut negative) = (false, false);
for (k, a) in quad.iter().enumerate() {
let b = quad[(k + 1) % 4];
let cross = (b[0] - a[0]) * (y - a[1]) - (b[1] - a[1]) * (x - a[0]);
positive |= cross > 0.0;
negative |= cross < 0.0;
if cross.is_nan() {
return false;
}
}
positive != negative
}
/// One glyph on the page.
#[derive(Clone, Debug, PartialEq)]
pub struct TextGlyph {
@@ -160,7 +176,26 @@ impl PageText {
}
/// Index of the glyph nearest to a view-space point (for selection anchors).
///
/// A glyph whose drawn em box (`quad`) holds the point wins. Skewed or turned text has bounding
/// boxes wider than the glyphs, so the box of the letter before overlaps this one and would
/// otherwise be found first (#3693). Failing that, the nearest bounding box wins, and a tie
/// goes to the glyph whose centre is closer.
pub fn nearest(&self, x: f32, y: f32) -> Option<usize> {
let centre_distance = |g: &TextGlyph| {
let (cx, cy) = (g.quad.iter().map(|q| q[0]).sum::<f32>() / 4.0, g.quad.iter().map(|q| q[1]).sum::<f32>() / 4.0);
(cx - x).powi(2) + (cy - y).powi(2)
};
let inside = self
.glyphs
.iter()
.enumerate()
.filter(|(_, g)| quad_contains(&g.quad, x, y))
.min_by(|a, b| centre_distance(a.1).total_cmp(&centre_distance(b.1)))
.map(|(i, _)| i);
if inside.is_some() {
return inside;
}
self.glyphs
.iter()
.enumerate()
@@ -180,10 +215,10 @@ impl PageText {
0.0
};
// Prefer the same line: vertical distance weighs more.
(i, dx + dy * 3.0)
(i, dx + dy * 3.0, centre_distance(g))
})
.min_by(|a, b| a.1.total_cmp(&b.1))
.map(|(i, _)| i)
.min_by(|a, b| a.1.total_cmp(&b.1).then(a.2.total_cmp(&b.2)))
.map(|(i, _, _)| i)
}
/// The word around glyph `i` as its first and last glyph (double-click): it stops at word
@@ -549,6 +584,13 @@ impl TextDevice {
}
}
/// Whether `PDFCRAFT_TEXT_DEBUG` is set. `draw_glyph` runs once per glyph and `getenv` scans the
/// whole environment under a lock, so read it once.
fn text_debug() -> bool {
static DEBUG: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*DEBUG.get_or_init(|| std::env::var_os("PDFCRAFT_TEXT_DEBUG").is_some())
}
impl<'a> Device<'a> for TextDevice {
fn set_soft_mask(&mut self, _: Option<SoftMask<'a>>) {}
fn set_blend_mode(&mut self, _: BlendMode) {}
@@ -567,7 +609,7 @@ impl<'a> Device<'a> for TextDevice {
Some(hayro::hayro_interpret::hayro_cmap::BfString::String(s)) => s,
None if replaced => String::new(),
None => {
if std::env::var_os("PDFCRAFT_TEXT_DEBUG").is_some() {
if text_debug() {
eprintln!(
"NOUNICODE {} {:?}",
if matches!(glyph, Glyph::Type3(_)) { "t3" } else { "ol" },
@@ -583,7 +625,7 @@ impl<'a> Device<'a> for TextDevice {
// Glyph space → view space. Glyph space uses 1000 units per em for outline glyphs; the em
// box spans descender (−200) to ascender (800), the advance gives the width.
let t = transform * glyph_transform;
if std::env::var_os("PDFCRAFT_TEXT_DEBUG").is_some() {
if text_debug() {
let kind = if matches!(glyph, Glyph::Type3(_)) { "t3" } else { "ol" };
eprintln!("{kind} {text:?} transform={:?} glyph_transform={:?}", transform.as_coeffs(), glyph_transform.as_coeffs());
}
@@ -1420,13 +1462,15 @@ pub fn layout(glyphs: Vec<TextGlyph>) -> PageText {
fn arrange(glyphs: &[TextGlyph]) -> (Vec<usize>, Vec<u32>, Vec<bool>) {
// Drop "fake bold" duplicates: the same character redrawn at (almost) the same place. A redraw
// also overlaps most of its own width: a narrow letter in condensed text (80 Tz and below) sits
// closer than a fifth of its height to its double, as the second "l" of "Alle" (#1467).
// closer than a fifth of its height to its double, as the second "l" of "Alle" (#1467), and
// OCR layers squeeze words to 20 Tz and below (#2457). Only a zero-width mark, which has no
// width to overlap, is matched by its height alone.
let mut keep = vec![true; glyphs.len()];
for i in 1..glyphs.len() {
let g = &glyphs[i];
let h = (g.rect[3] - g.rect[1]).max(0.1);
let w = g.rect[2] - g.rect[0];
let dx = if w > 0.05 * h { (h * 0.2).min(w * 0.5) } else { h * 0.2 };
let dx = if w > 0.0 { (h * 0.2).min(w * 0.5) } else { h * 0.2 };
for j in (i.saturating_sub(4)..i).rev() {
let p = &glyphs[j];
if keep[j] && p.text == g.text && (p.rect[0] - g.rect[0]).abs() < dx && (p.rect[1] - g.rect[1]).abs() < h * 0.2 {
@@ -1599,8 +1643,25 @@ fn arrange(glyphs: &[TextGlyph]) -> (Vec<usize>, Vec<u32>, Vec<bool>) {
place_glyphs(&glyphs, &height, &segs[si], &mut sc);
let h = segs[si].h;
// Word gaps relative to this line's typical letter gap (handles tracking).
// Long runs of repeated punctuation belong to leaders, not letter tracking: a TOC leader
// must not raise the threshold above the word gaps in its title (#3086).
sc.gaps.clear();
sc.gaps.extend(sc.placed.windows(2).map(|w| glyphs[w[1]].rect[0] - glyphs[w[0]].rect[2]));
sc.gaps.extend(sc.placed.windows(2).enumerate().filter_map(|(w, pair)| {
let &[p, c] = pair else { return None };
let (p, c) = (glyphs.get(p)?, glyphs.get(c)?);
if p.text == c.text && p.text.chars().all(|ch| !ch.is_alphanumeric() && !ch.is_whitespace()) {
// Keep short tracked punctuation, including an ellipsis with a final period.
// Treat five or more matching glyphs as a leader. Three neighbours on each
// side suffice to classify this pair, bounding work even on long runs.
let same = |&&i: &&usize| glyphs.get(i).is_some_and(|g| g.text == p.text);
let before = sc.placed.get(..w).unwrap_or_default().iter().rev().take(3).take_while(same).count();
let after = sc.placed.get(w.saturating_add(2)..).unwrap_or_default().iter().take(3).take_while(same).count();
if before + after >= 3 {
return None;
}
}
Some(c.rect[0] - p.rect[2])
}));
sc.gaps.sort_by(f32::total_cmp);
let typical = sc.gaps.get(sc.gaps.len() / 3).copied().unwrap_or(0.0).max(0.0);
let threshold = (typical + h * WORD_GAP).max(h * WORD_GAP);
@@ -1703,6 +1764,37 @@ mod tests {
}
}
/// WSCAD writes `/ToUnicode /Identity-H`, a name rather than a CMap stream, on Type 0 fonts
/// whose two-byte codes are UTF-16 code units. The codes read as that Unicode text (as in
/// Acrobat), so the page's text can be selected and found; before, it had none. Controls (here a
/// DEL and a C1 control at the line ends) are not text.
#[test]
fn identity_to_unicode_names_give_text() {
use lopdf::{Document, Object, Stream, dictionary};
for name in ["Identity-H", "Identity-V"] {
let mut doc = Document::with_version("1.4");
let cid = doc.add_object(dictionary! {
"Type" => "Font", "Subtype" => "CIDFontType2", "BaseFont" => "RobotoMono-Regular",
"CIDSystemInfo" => dictionary! { "Registry" => Object::string_literal("Adobe"), "Ordering" => Object::string_literal("Identity"), "Supplement" => 0 },
"DW" => 600, "CIDToGIDMap" => "Identity"
});
let font = doc.add_object(dictionary! { "Type" => "Font", "Subtype" => "Type0", "BaseFont" => "RobotoMono-Regular", "Encoding" => "Identity-H", "DescendantFonts" => vec![Object::Reference(cid)], "ToUnicode" => name });
let operators = "BT /F1 12 Tf 1 0 0 1 20 250 Tm <004400650063006B0062006C006100740074007F> Tj 1 0 0 1 20 200 Tm <005A00FC0072006900630068002020AC0085> Tj ET";
let content = doc.add_object(Stream::new(dictionary! {}, operators.as_bytes().to_vec()));
let pages = doc.new_object_id();
let page = doc.add_object(dictionary! { "Type" => "Page", "Parent" => pages, "MediaBox" => vec![Object::Integer(0), Object::Integer(0), Object::Integer(300), Object::Integer(300)], "Resources" => dictionary! {"Font" => dictionary! {"F1" => font}}, "Contents" => content });
doc.objects.insert(pages, dictionary! {"Type" => "Pages", "Kids" => vec![Object::Reference(page)], "Count" => 1}.into());
let catalog = doc.add_object(dictionary! {"Type" => "Catalog", "Pages" => pages});
doc.trailer.set("Root", catalog);
let mut bytes = Vec::new();
doc.save_to(&mut bytes).unwrap();
let pdf = Pdf::new(std::sync::Arc::new(bytes)).unwrap();
let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap();
assert_eq!(text.plain_text(), "Deckblatt\nZürich €", "{name}");
assert_eq!(text.find("zürich"), vec![9..15], "{name}: glyphs 9 to 14");
}
}
/// One page per text matrix, each showing `codes` in Helvetica whose `/ToUnicode` maps the
/// codes in `unicode` (a `bfchar` body) to several characters.
fn ligature_pages(unicode: &str, codes: &str, matrices: &[&str]) -> Pdf {
@@ -1869,6 +1961,84 @@ mod tests {
}
}
/// Issue #3109: text drawn in clipping mode (`7 Tr`) sets the clip for a subsequent fill,
/// shading or image without painting the glyphs directly, and must still reach `TextDevice`
/// with the same boxes and search ranges as modes 0–6.
#[test]
fn clipping_mode_text_is_extracted_and_searchable_like_other_rendering_modes() {
let fill_through = " 0 0 1 rg 0 0 300 300 re f";
let mut pages: Vec<String> =
(0..=7).map(|mode| format!("q BT /F1 24 Tf {mode} Tr 72 150 Td () Tj [(Hello) -200 (World)] TJ ET{fill_through} Q")).collect();
// Page 8: /ActualText wrapping 7 Tr clip-only text.
pages.push(format!("q /Span <</ActualText (Hello World)>> BDC BT /F1 24 Tf 7 Tr 72 150 Td (Glyphs) Tj ET EMC{fill_through} Q"));
// Page 9: 0 Tr underpainting followed by 7 Tr at the same position (outline/gradient overlay) deduplicates.
pages.push(format!("BT /F1 24 Tf 0 Tr 72 150 Td (Hello World) Tj ET q BT /F1 24 Tf 7 Tr 72 150 Td (Hello World) Tj ET{fill_through} Q"));
// Page 10: 90° turned 7 Tr text.
pages.push(format!("q BT /F1 24 Tf 7 Tr 0 1 -1 0 150 100 Tm (Hello World) Tj ET{fill_through} Q"));
// Page 11: 7 Tr without any fill-through (visually blank, still extractable).
pages.push("BT /F1 24 Tf 7 Tr 72 150 Td (Hello World) Tj ET".into());
let refs: Vec<&str> = pages.iter().map(String::as_str).collect();
let pdf = marked_pages(&refs);
let control = marked_text(&pdf, 0);
assert_eq!(control.plain_text(), "Hello World");
let control_rects = control.line_rects(0..5);
assert_eq!(control_rects.len(), 1);
for mode in 0..=7 {
let text = marked_text(&pdf, mode);
assert_eq!(text.plain_text(), "Hello World", "mode {mode} Tr");
assert_eq!(text.find("hello"), vec![0..5], "mode {mode} Tr");
assert_eq!(text.find_opts("Hello", true, true), vec![0..5], "mode {mode} Tr");
assert_eq!(
text.find_with(|chars| chars.starts_with(&['H', 'e', 'l', 'l', 'o']).then_some(0..5).into_iter().collect()),
vec![0..5],
"mode {mode} Tr"
);
assert_eq!(text.line_rects(0..5), control_rects, "mode {mode} Tr");
assert_eq!(text.glyphs, control.glyphs, "mode {mode} Tr");
}
let actual = marked_text(&pdf, 8);
assert_eq!(actual.plain_text(), "Hello World");
assert_eq!(actual.find("Hello"), vec![0..1]);
let deduped = marked_text(&pdf, 9);
assert_eq!(deduped.plain_text(), "Hello World");
let turned = marked_text(&pdf, 10);
assert_eq!(turned.plain_text(), "Hello World");
assert_eq!(turned.find("World"), vec![6..11]);
let unfilled = marked_text(&pdf, 11);
assert_eq!(unfilled.plain_text(), "Hello World");
// Visual check: page 7 (`7 Tr` + blue fill-through) paints blue only inside the clipped
// glyph outlines, whereas page 11 (`7 Tr` alone) paints no ink at all.
let mut renderer = crate::PageRenderer::new(std::sync::Arc::new(pdf.data().as_ref().to_vec()), crate::RenderConfig::default());
let filled_px = renderer.render(crate::RenderRequest { page: 7, kind: crate::RequestKind::Pixels, scale: 1.0, ..Default::default() });
assert!(filled_px.error.is_none());
let blue_pixels = filled_px.rgba.as_chunks::<4>().0.iter().filter(|p| p[0] < 32 && p[1] < 32 && p[2] > 200).count();
let white_pixels = filled_px.rgba.as_chunks::<4>().0.iter().filter(|p| p[..3] == [255, 255, 255]).count();
assert!(blue_pixels > 50 && white_pixels > 80_000, "blue={blue_pixels}, white={white_pixels}");
let unfilled_px = renderer.render(crate::RenderRequest { page: 11, kind: crate::RequestKind::Pixels, scale: 1.0, ..Default::default() });
assert!(unfilled_px.error.is_none());
assert!(unfilled_px.rgba.as_chunks::<4>().0.iter().all(|p| p[..3] == [255, 255, 255]));
}
#[test]
fn clipping_mode_filter_keeps_titles_and_excludes_ocr() {
let pdf = marked_pages(&[
"q BT /F1 24 Tf 7 Tr 30 200 Td (Clipped heading) Tj ET 0 0 1 rg 0 0 300 300 re f Q BT /F1 12 Tf 3 Tr 30 100 Td (Hidden OCR) Tj ET",
"q /Span <</ActualText (Accessible heading)>> BDC BT /F1 24 Tf 7 Tr 30 200 Td (Clipped heading) Tj ET EMC 0 0 1 rg 0 0 300 300 re f Q",
]);
assert!(marked_text(&pdf, 0).plain_text().contains("Hidden OCR"));
for (page, expected) in [(0, "Clipped heading"), (1, "Accessible heading")] {
let text = extract_page_filtered(&pdf, page, &InterpreterSettings::default(), TextFilter { images: true, ..Default::default() }).unwrap();
assert_eq!(text.plain_text(), expected);
assert!(!text.find("heading").is_empty());
}
}
/// One page per `(encoding, content)`, in Helvetica (`/F1`).
fn encoded_pages(pages: &[(lopdf::Object, &[u8])]) -> Pdf {
use lopdf::{Document, Object, Stream, dictionary};
@@ -1923,6 +2093,64 @@ mod tests {
assert_eq!(texts(2), ["a", "\u{ad}", "b", "\u{a0}", "c"], "a named glyph keeps its glyph list value");
}
/// #3169, #3170: glyph names outside the Adobe Glyph List still read as text. Ligature
/// (`f_i`) and suffixed (`hyphen.case`) names go by the glyph list specification's full
/// mapping. A name that only numbers the glyph (`c251` small caps, `C39`, `C174`, `C39.sc`)
/// reads as nothing, not as a guess from the code (review on #3239: with `/C1` at 233,
/// StandardEncoding's 233 would read `Ø` for a glyph drawn as `é`), and so do a ZapfDingbats
/// name (`a12`) and a name with no meaning left (`foo`).
#[test]
fn glyph_names_outside_the_glyph_list_still_read_as_text() {
use lopdf::{Object, dictionary};
let name = |n: &str| Object::Name(n.as_bytes().to_vec());
let differences = Object::Dictionary(dictionary! {
"Type" => "Encoding",
"Differences" => vec![
Object::Integer(30), name("f_i"), name("f_l"), Object::Integer(2), name("hyphen.case"),
Object::Integer(39), name("C39"), Object::Integer(97), name("c251"), Object::Integer(174), name("C174"),
Object::Integer(88), name("a12"), name("foo"), name("uni00660069"),
Object::Integer(61), name("C39.sc"), Object::Integer(233), name("C1"),
],
});
// One glyph per line: the test font has none of these glyphs, so they have no width, and
// on one line a later `fi` would sit on the first and be dropped as its fake-bold copy.
let mut content = b"BT /F1 20 Tf 20 290 Td".to_vec();
for code in *b"\x1e\x1f\x02'a\xaeXYZ=\xe9" {
content.extend_from_slice(b" (");
content.push(code);
content.extend_from_slice(b") Tj 0 -25 Td");
}
content.extend_from_slice(b" ET");
let pdf = encoded_pages(&[(differences, &content[..])]);
let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap().plain_text();
let lines: Vec<&str> = text.lines().map(str::trim).filter(|l| !l.is_empty()).collect();
assert_eq!(lines, ["fi", "fl", "-", "fi"], "{text:?}");
}
/// Review on #3239: the glyph list specification's `uni` and `u` forms take uppercase hex
/// digits only, and map a name with lowercase ones to nothing ("uni20ac" in its examples).
/// Names upstream's own lookup can't read (several groups, a suffix) show it: lowercase reads
/// as nothing, uppercase as its characters. A bare `uni20ac` or `u1040c` is still read by
/// upstream's lenient `unicode_from_name`, outside this mapping.
#[test]
fn glyph_names_with_lowercase_hex_read_as_nothing() {
use lopdf::{Object, dictionary};
let name = |n: &str| Object::Name(n.as_bytes().to_vec());
let differences = Object::Dictionary(dictionary! {
"Type" => "Encoding",
"Differences" => vec![Object::Integer(65), name("uni20ac0308"), name("uni20AC0308"), name("u1040c.alt"), name("u1040C.alt"), name("f_uni20ac"), name("f_uni20AC")],
});
let pdf = encoded_pages(&[(
differences,
&b"BT /F1 20 Tf 20 180 Td (A) Tj 0 -25 Td (B) Tj 0 -25 Td (C) Tj 0 -25 Td (D) Tj 0 -25 Td (E) Tj 0 -25 Td (F) Tj ET"[..],
)]);
// One glyph per line, so each name's text stays its own (and a name read as nothing
// leaves no line).
let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap().plain_text();
let lines: Vec<&str> = text.lines().map(str::trim).filter(|l| !l.is_empty()).collect();
assert_eq!(lines, ["\u{20ac}\u{308}", "\u{1040c}", "f", "f\u{20ac}"], "{text:?}");
}
#[test]
fn narrow_ligatures_keep_every_letter() {
// Split evenly, this "ffi" put its two "f"s so close that the second was dropped as a
@@ -2052,6 +2280,62 @@ mod tests {
TextGlyph { text: t.into(), rect, quad: [[x0, y0], [x1, y0], [x1, rect[3]], [x0, rect[3]]], direction: [1.0, 0.0] }
}
/// A glyph whose em box is `w` × `h` at `origin`, advancing along `dir` with its up edge along
/// `up` (a unit step of each), as a sheared or turned text matrix draws it.
fn tilted(t: &str, origin: [f32; 2], dir: [f32; 2], up: [f32; 2], w: f32, h: f32) -> TextGlyph {
let p = |a: f32, b: f32| [origin[0] + dir[0] * a + up[0] * b, origin[1] + dir[1] * a + up[1] * b];
let quad = [p(0.0, 0.0), p(w, 0.0), p(w, h), p(0.0, h)];
let xs = quad.map(|q| q[0]);
let ys = quad.map(|q| q[1]);
let rect = [
xs.iter().copied().fold(f32::MAX, f32::min),
ys.iter().copied().fold(f32::MAX, f32::min),
xs.iter().copied().fold(f32::MIN, f32::max),
ys.iter().copied().fold(f32::MIN, f32::max),
];
TextGlyph { text: t.into(), rect, quad, direction: dir }
}
/// #3693: pointing at the middle of a glyph in sheared or turned text finds that glyph, not
/// the one before it whose bounding box overlaps it.
#[test]
fn nearest_finds_the_glyph_under_the_pointer_in_sheared_and_turned_text() {
let s = std::f32::consts::FRAC_1_SQRT_2;
// (writing direction, up edge) in view space (y down): upright, sheared 0.35, turned 45°.
for (dir, up) in [([1.0, 0.0], [0.0, -1.0]), ([1.0, 0.0], [0.35, -1.0]), ([s, -s], [-s, -s])] {
let mut advance = 0.0;
let glyphs: Vec<TextGlyph> = "film in the list of tiny jilt flit"
.chars()
.map(|c| {
let w = if "filjt ".contains(c) { 3.0 } else { 6.0 };
let g = tilted(&c.to_string(), [50.0 + dir[0] * advance, 300.0 + dir[1] * advance], dir, up, w, 14.0);
advance += w;
g
})
.collect();
let t = layout(glyphs.clone());
assert_eq!(t.glyphs.len(), glyphs.len(), "layout kept every glyph");
for (i, g) in glyphs.iter().enumerate() {
let cx = g.quad.iter().map(|q| q[0]).sum::<f32>() / 4.0;
let cy = g.quad.iter().map(|q| q[1]).sum::<f32>() / 4.0;
assert_eq!(t.nearest(cx, cy), Some(i), "dir {dir:?} up {up:?}: glyph {i} ({})", g.text);
}
// Past either end of the line, the end glyph is still the nearest.
let (first, last) = (&glyphs[0], &glyphs[glyphs.len() - 1]);
assert_eq!(t.nearest(first.quad[0][0] - dir[0] * 8.0, first.quad[0][1] - dir[1] * 8.0), Some(0));
assert_eq!(t.nearest(last.quad[1][0] + dir[0] * 8.0, last.quad[1][1] + dir[1] * 8.0), Some(glyphs.len() - 1));
}
}
#[test]
fn a_degenerate_or_nan_quad_holds_no_point() {
assert!(!quad_contains(&[[1.0, 1.0]; 4], 1.0, 1.0));
assert!(!quad_contains(&[[0.0, 0.0], [1.0, 0.0], [f32::NAN, 1.0], [0.0, 1.0]], 0.5, 0.5));
assert!(quad_contains(&[[0.0, 0.0], [4.0, 0.0], [4.0, 4.0], [0.0, 4.0]], 4.0, 2.0));
// Wound the other way.
assert!(quad_contains(&[[0.0, 0.0], [0.0, 4.0], [4.0, 4.0], [4.0, 0.0]], 2.0, 2.0));
}
#[test]
fn lines_words_and_search() {
// "Hello world" on line 1, "Next" on line 2.
@@ -2175,6 +2459,49 @@ mod tests {
}
}
/// #2457: OCR text layers squeeze each word into its box down to 20 Tz and below, where a
/// narrow letter is less than a twentieth of its height wide. Doubled letters, and letters
/// repeated two or three places on, are still letters there, not redraws.
#[test]
fn ocr_condensed_double_letters_are_kept() {
// Helvetica advances, in thousandths of an em.
let adv = |c: char| match c {
'M' => 833.0,
'K' | 'B' => 667.0,
'W' => 944.0,
'I' | 't' | 'f' => 278.0,
'i' | 'l' => 222.0,
'm' => 833.0,
'r' => 333.0,
's' => 500.0,
_ => 556.0,
};
for scale in [0.2, 0.1, 0.05] {
for word in ["Hallo", "Mitte", "Wille", "Kommission", "Illustration", "still", "Brunnen", "Kaffee"] {
let mut v = Vec::new();
let mut x = 10.0;
for c in word.chars() {
let w = adv(c) / 100.0 * scale;
v.push(g(&c.to_string(), x, 10.0, x + w));
x += w;
}
assert_eq!(layout(v.clone()).plain_text(), word, "at {scale} horizontal scaling");
// Each letter drawn twice at the same place is still one letter.
let bold: Vec<TextGlyph> = v.iter().flat_map(|c| [c.clone(), c.clone()]).collect();
assert_eq!(layout(bold).plain_text(), word, "fake bold at {scale} horizontal scaling");
// And redrawn a quarter of its own width to the right (the match reaches half of it).
let shifted: Vec<TextGlyph> = v
.iter()
.flat_map(|c| {
let d = (c.rect[2] - c.rect[0]) * 0.25;
[c.clone(), g(&c.text, c.rect[0] + d, 10.0, c.rect[2] + d)]
})
.collect();
assert_eq!(layout(shifted).plain_text(), word, "shifted fake bold at {scale} horizontal scaling");
}
}
}
#[test]
fn two_columns_read_left_then_right_even_if_drawn_interleaved() {
let mut v = Vec::new();
@@ -2195,6 +2522,92 @@ mod tests {
assert_eq!(layout(v).plain_text(), "CHAPTER FOUR");
}
#[test]
fn toc_leaders_do_not_hide_word_gaps() {
use lopdf::{Document, Object, Stream, dictionary};
for dots in [0, 5, 10, 20, 40, 60] {
for (words, expected) in [
("(Was)-333(ist)-333(OTRS?)", "Was ist OTRS?"),
("(Das)-333(OTRS)-333(Web-Interface)", "Das OTRS Web-Interface"),
("(Neuen)-333(Zugang)-333(beantragen)", "Neuen Zugang beantragen"),
] {
// The issue's LaTeX TJ spacing, without embedding the corpus PDF or a font asset.
let mut doc = Document::with_version("1.7");
let font = doc.add_object(dictionary! { "Type" => "Font", "Subtype" => "Type1", "BaseFont" => "Times-Roman" });
let operators = format!("BT /F1 10 Tf 10 100 Td [{words}-441{}-1663(7)] TJ ET", "-500(.)".repeat(dots));
let content = doc.add_object(Stream::new(dictionary! {}, operators.into_bytes()));
let pages = doc.new_object_id();
let page = doc.add_object(dictionary! {
"Type" => "Page", "Parent" => pages,
"MediaBox" => vec![0.into(), 0.into(), 800.into(), Object::Integer(200)],
"Resources" => dictionary! { "Font" => dictionary! { "F1" => font } }, "Contents" => content
});
doc.objects.insert(pages, dictionary! { "Type" => "Pages", "Kids" => vec![Object::Reference(page)], "Count" => 1 }.into());
let catalog = doc.add_object(dictionary! { "Type" => "Catalog", "Pages" => pages });
doc.trailer.set("Root", catalog);
let mut bytes = Vec::new();
doc.save_to(&mut bytes).unwrap();
let pdf = Pdf::new(std::sync::Arc::new(bytes)).unwrap();
let text = extract_page(&pdf, 0, &InterpreterSettings::default()).unwrap();
assert_eq!(text.plain_text(), format!("{expected} {}7", ". ".repeat(dots)), "{dots} leader dots");
let hits = text.find_opts(expected, true, true);
assert_eq!(hits.len(), 1, "{expected:?}, {dots} leader dots");
assert_eq!(text.text_of(hits[0].clone()), expected);
assert_eq!(text.line_rects(hits[0].clone()).len(), 1);
assert_eq!(text.find(&expected.to_lowercase()), hits);
assert!(text.find(&expected.replace(' ', "")).is_empty());
}
}
}
#[test]
fn short_tracked_ellipses_remain_searchable() {
for dots in [3, 4] {
let mut v = Vec::new();
for i in 0..dots {
let x = 10.0 + i as f32 * 4.0;
v.push(g(".", x, 10.0, x + 2.0));
}
let ellipsis = ".".repeat(dots);
let text = layout(v.clone());
assert_eq!(text.plain_text(), ellipsis);
assert_eq!(text.find(&ellipsis), vec![0..dots]);
v.insert(0, g("A", 0.0, 10.0, 6.0));
let x = 12.0 + dots as f32 * 4.0;
v.push(g("B", x, 10.0, x + 6.0));
let text = layout(v);
let expected = format!("A {ellipsis} B");
assert_eq!(text.plain_text(), expected);
assert_eq!(text.find(&expected), vec![0..dots + 2]);
}
}
#[test]
fn leaders_preserve_tracking_in_letters_numbers_and_punctuation() {
for advance in [9.0, 12.0] {
for leader in [".", "-", "…"] {
let mut v = Vec::new();
word(&mut v, "CHAPTER-1", 10.0, 10.0, advance);
let x = 10.0 + 9.0 * advance + 5.0;
word(&mut v, "1.4", x, 10.0, advance);
let mut x = x + 3.0 * advance + 5.0;
for _ in 0..60 {
v.push(g(leader, x, 10.0, x + 2.5));
x += 11.0;
}
let text = layout(v);
assert_eq!(text.plain_text(), format!("CHAPTER-1 1.4 {}", vec![leader; 60].join(" ")));
assert_eq!(text.find("chapter-1 1.4"), vec![0..12]);
}
}
// Decimal punctuation and repeated letters/digits still use their original tracking estimate.
for value in ["1.4", "1111", "llll"] {
let mut v = Vec::new();
word(&mut v, value, 10.0, 10.0, 12.0);
assert_eq!(layout(v).plain_text(), value);
}
}
#[test]
fn rtl_runs_come_out_in_logical_order_and_cjk_has_no_spaces() {
let mut v = Vec::new();
+2
View File
@@ -1,5 +1,6 @@
# Where PdfCraft falls short of Acrobat Pro
> **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-19 · **Change:** minor (gap 2: composite fonts with `/ToUnicode /Identity-H` read as Unicode, hayro-interpret patch 44; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
> **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (gap 12: reading order is scored, not just word F1) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
> **Last reviewed:** 2026-10-14 · **Last updated:** 2026-10-14 · **Change:** minor (M1: raw-deflate fallback preserves decoding-limit errors; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
> **Last reviewed:** 2026-10-12 · **Last updated:** 2026-10-13 · **Change:** minor (gap 14: Vietnamese catalog shipped; Hindi/Indonesian/Korean and Arabic RTL remain) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
@@ -382,6 +383,7 @@ update the numbers.
| Date | Change | Summary |
|---|---|---|
| 2026-10-19 | minor | Gap 2: a Type 0 font whose `/ToUnicode` is the name `/Identity-H` or `/Identity-V` (WSCAD) reads its codes as Unicode, so such pages have text to select, find and edit (hayro-interpret patch 44, `view.render-cid-fonts`). Estimates unchanged. |
| 2026-10-15 | minor | Gap 12: the text oracle scores reading order as well as word F1, which is a bag of words and cannot see order; the first measurement separated 18 improvements from 6 regressions in a change F1 called unchanged on 26 of 29 files. Estimates unchanged. |
| 2026-10-14 | minor | M1: raw-deflate fallback preserves decoding-limit errors in strict and tolerant decoding; synthetic filter and COS stream regressions. Estimates unchanged. |
| 2026-10-15 | patch | Gap 24: record localization and installers as partial (unusable Chinese, Telugu and Arabic; unsigned Linux packages) |
+5
View File
@@ -5,7 +5,11 @@
> **Last reviewed:** 2026-10-20 | **Last updated:** 2026-10-20 | **Change:** minor (M2, #3574: named form properties retain page origin and own dictionary boundaries; estimates unchanged)
> **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** minor (M3, #3541/#3620: comment hover follows individual marked lines and original annotation paint order; estimates unchanged)
> **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** patch (PR #3348 review: retain short tracked punctuation in gap estimates; estimates unchanged)
> **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** minor (M5, PR #1971: bounded nested FDF export excludes source page/page-tree dictionaries; refreshed against upstream 4db6ac4a; estimates unchanged)
> **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** patch (M2 #3086: dotted TOC leaders no longer hide word spaces; estimates unchanged)
> **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M14/#2383 GPU-limit candidate; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
> **Last reviewed:** 2026-10-20 - **Last updated:** 2026-10-20 - **Change:** minor (M6, #3502: form detection uses the existing content-stream boundary policy; estimates unchanged)
> **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M7: the Edit text caret is as tall as the row it is on; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
@@ -123,6 +127,7 @@ total matches the parity estimate.
| M0 | Skeleton: workspace, xtask gates, CI | 15–30 | 98% | 1–2 | GitHub workflow, `deny.toml` (licence audit of every dependency), parity checklist (826 features, `xtask parity`) done. Missing: remaining crate stubs, testkit/oracle crates |
| M1 | COS: filters, crypt, parser, xref, writer | 120–200 | 80% | 20–40 | Done:<br>- filters and crypt: every standard-security revision R2–R6 (RC4, AES-128/256), SASLprep, permissions, creating encryption;<br>- cos: parse and repair (including invalid xref object-number ranges), decoded-stream limits including unfiltered data, decrypt on load, re-encrypt on save, incremental and full writing.<br>Corpus: open/edit/save passes on 958 files, and all 7 password-protected files open.<br>Full saves now pack objects into compressed object streams. An unchanged incremental save of a classic table writes a zero-length cross-reference subsection instead of a section with no subsections (#1339). A full rewrite keeps unknown trailer entries, including objects reachable only from them (#1338). Fuzzing runs nightly (`xtask fuzz`). Raw-deflate fallback preserves decoding-limit errors, including through tolerant COS stream decoding. Missing: ≥ 250 tests, own image codecs |
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. Forms without Resources use page Properties for ActualText and layer visibility; own dictionaries stop fallback. Oversized missing inline-image samples, including header and terminator overhead, no longer pin the content window (#3107 / PR #3438); normal large streamed images retain deterministic sample boundaries. Radial luminosity fades have synthetic coverage for the existing #2132 mask guard, including per-pixel references, antialiased clips, previews, scales and tiles; no additional production fix was needed. |
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. Dotted TOC leaders no longer inflate letter tracking or hide word spaces (#3086); phrase search and selection are covered. Short tracked ellipses retain their gap samples and remain searchable. |
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. The renderer keeps its one-point crop-size floor in physical points after UserUnit scaling, with regression coverage for crop origins, rotations and f32 reciprocal rounding. Existing unit policy and nearly-zero fallback are retained. |
| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings |
| M3 | Viewer app (native + web) | 80–150 | 90% | 10–20 | Acrobat-style shell, find, select, panels, tiles, web build, UI control channel for agents (opt-in). #3808: the Pages inspector offers a 50–150% thumbnail slider and buttons, also available through `ui.set`. Missing: 60 fps test on a 500-page document, snapshot tests of every panel<br>Done since: fit visible, document title in the window; Linux middle-button scrolling; shared three-state theme preferences in the toolbar, View menu and control channel; numeric Linux portal events and idle wakeups completing #881, with native appearance inheritance retained (#446); late browser startup downloads preserve the current document and Save target (#171). Selected-text Copy in Edit and the canvas context menu is covered by focused regressions and native clipboard checks (#1225); document copy restrictions apply to the floating action too. #1364 candidate: absolute local GoToR links ask before target access; headless callers explicitly opt in. Native-opening candidate: chosen tools survive preparation and password retries; startup intake waits for pending preparation. Page-label navigation displays logical labels, retains the current page on unchanged submission and refreshes focused same-document replacements; clean file-watcher reload preserves the chosen scrolling mode, preventing a Single Page reader from jumping to page 1 during replacement. Synthetic control regressions cover these behaviors. #3149: page-list changes discard render demand based on old geometry; actual raster dimensions are checked against the GPU limit before upload. Synthetic crash and continued-rendering regressions cover the replacement path. Save-prompt shortcuts yield to dialogs above them; stacked link-consent UI regression covers Escape and Return. |
+4 -4
View File
@@ -782,8 +782,8 @@ milestone = "M2"
status = "partial"
commands = []
tools = []
evidence = ["crates/render/src/raster.rs::huge_cid_width_ranges_terminate", "crates/render/src/raster.rs::non_embedded_japanese_cid_fonts_draw_with_a_craft_fonts_face", "crates/render/src/raster.rs::mincho_substitute_covers_half_width_katakana"]
notes = "CID-to-Unicode tables are M2.4; a non-embedded Adobe-Japan1 font is drawn in a craft-fonts Japanese face (Mincho or Gothic by name; BIZ UDMincho for Mincho, which covers half-width katakana) when built with craft-fonts; non-embedded GB1, CNS1 and Korea1 fonts still draw nothing (no allowed face)"
evidence = ["crates/render/src/raster.rs::huge_cid_width_ranges_terminate", "crates/render/src/raster.rs::non_embedded_japanese_cid_fonts_draw_with_a_craft_fonts_face", "crates/render/src/raster.rs::mincho_substitute_covers_half_width_katakana", "crates/render/src/text.rs::identity_to_unicode_names_give_text", "crates/fonts/src/pdf.rs::identity_to_unicode_names_read_codes_as_unicode"]
notes = "CID-to-Unicode tables are M2.4; a non-embedded Adobe-Japan1 font is drawn in a craft-fonts Japanese face (Mincho or Gothic by name; BIZ UDMincho for Mincho, which covers half-width katakana) when built with craft-fonts; non-embedded GB1, CNS1 and Korea1 fonts still draw nothing (no allowed face); a `/ToUnicode` given as the name `/Identity-H` or `/Identity-V` (WSCAD) reads codes as Unicode for selection, search and editing, without control characters"
[[feature]]
id = "view.render-image-codecs"
@@ -1394,8 +1394,8 @@ milestone = "M3"
status = "shipped"
commands = ["edit.find"]
tools = ["text_find"]
evidence = ["crates/ui-egui/tests/ui.rs::find_counts_matches_across_pages", "crates/ui-egui/tests/ui.rs::find_reports_no_matches", "crates/automation/tests/automation.rs::open_inspect_render_and_find", "crates/render/src/text.rs::identity_v_columns_are_searchable_in_reading_order"]
notes = "Horizontal and WMode 1 vertical text are grouped independently, including a horizontal heading above right-to-left vertical columns. Complex interleaved flows still use a geometric reading-order heuristic."
evidence = ["crates/ui-egui/tests/ui.rs::find_counts_matches_across_pages", "crates/ui-egui/tests/ui.rs::find_reports_no_matches", "crates/automation/tests/automation.rs::open_inspect_render_and_find", "crates/render/src/text.rs::identity_v_columns_are_searchable_in_reading_order", "crates/render/src/text.rs::toc_leaders_do_not_hide_word_gaps", "crates/render/src/text.rs::short_tracked_ellipses_remain_searchable"]
notes = "Horizontal and WMode 1 vertical text are grouped independently, including a horizontal heading above right-to-left vertical columns. Dotted TOC leaders preserve word spaces for phrase search (#3086), while short tracked ellipses remain intact and searchable; sparse lines and condensed OCR gaps below the existing height-based threshold remain limitations. Complex interleaved flows still use a geometric reading-order heuristic."
[[feature]]
id = "view.find-next-previous"
+19 -1
View File
File diff suppressed because one or more lines are too long
+20
View File
@@ -45,9 +45,16 @@ pub(crate) struct Type0Font {
/// PdfCraft patch: whether the CIDs belong to an Adobe character collection (Japan1, GB1,
/// CNS1, Korea1), whose glyph numbering no substitute other than a CID-keyed CFF font shares.
adobe_collection: bool,
/// PdfCraft patch: `/ToUnicode /Identity-H` (a name, not a CMap stream), which CAD exporters
/// such as WSCAD write for fonts whose two-byte codes are UTF-16 code units. Acrobat shows
/// such text as the code's Unicode value; the identity CMap the name resolves to maps codes
/// to CIDs only and gave no text at all.
identity_to_unicode: bool,
}
impl Type0Font {
/// Reads a Type 0 font from its dictionary: the descendant CID font, the encoding CMap and
/// the code-to-Unicode map (`/ToUnicode`, the `/Identity-H` name, or the UCS2 CMap).
pub(crate) fn new(
dict: &Dict<'_>,
font_resolver: &FontResolverFn,
@@ -117,6 +124,9 @@ impl Type0Font {
let cache_key = dict.cache_key();
let mut to_unicode = read_to_unicode(dict, cmap_resolver);
let identity_to_unicode = dict
.get::<Name<'_>>(TO_UNICODE)
.is_some_and(|n| matches!(n.as_ref(), b"Identity-H" | b"Identity-V"));
let mut to_unicode_is_cid_indexed = false;
// If there is no ToUnicode map, try to get the UCS2 CMap.
@@ -162,6 +172,7 @@ impl Type0Font {
fallback,
to_unicode_is_cid_indexed,
adobe_collection,
identity_to_unicode,
})
}
@@ -430,7 +441,16 @@ impl Type0Font {
}
}
/// The text a character code stands for, or `None` when the font gives none (or, for an
/// identity `/ToUnicode` name, when the code is a control or a lone surrogate).
pub(crate) fn char_code_to_unicode(&self, code: u32) -> Option<BfString> {
if self.identity_to_unicode {
// Controls and lone surrogates are not text. The same rule builds the editing
// map, `identity_to_unicode` in crates/fonts/src/pdf.rs: keep both alike.
return char::from_u32(code)
.filter(|c| !c.is_control())
.map(BfString::Char);
}
if let Some(to_unicode) = &self.to_unicode {
let key = if self.to_unicode_is_cid_indexed {
self.code_to_cid(code).unwrap_or(0)
+43
View File
@@ -669,6 +669,49 @@ pub(crate) fn glyph_name_to_unicode(name: &str) -> Option<char> {
})
}
/// PdfCraft patch: the text of a glyph name `glyph_name_to_unicode` doesn't know, by the
/// Adobe Glyph List Specification's full mapping: drop everything from the first `.`
/// (`hyphen.case`, `a.sc`), split the rest on `_` into components (`f_i`, `f_f_l`), and map each
/// one through the glyph list, as `uniXXXX[XXXX…]` (BMP values, four hex digits each) or as
/// `uXXXX` to `uXXXXXX`. A component it can't map, or a `uni` component with any group that
/// isn't a character, adds nothing; `None` when nothing is left.
pub(crate) fn glyph_name_to_text(name: &str) -> Option<BfString> {
let hex = |h: &str| u32::from_str_radix(h, 16).ok().and_then(char::from_u32);
// The specification's digits are uppercase only (`uni20ac` maps to nothing, review on
// #3239).
let upper_hex = |d: &str| d.bytes().all(|b| matches!(b, b'0'..=b'9' | b'A'..=b'F'));
let base = name.split('.').next().unwrap_or_default();
let mut text = String::new();
for part in base.split('_') {
if let Some(s) = glyph_names::get(part) {
text.push_str(s);
} else if let Some(digits) = part
.strip_prefix("uni")
.filter(|d| !d.is_empty() && d.len() % 4 == 0 && upper_hex(d))
{
// All ASCII, so every fourth byte is a char boundary. A group that isn't a
// character (a surrogate) makes the whole component map to nothing.
let chars: Option<String> = (0..digits.len())
.step_by(4)
.map(|at| digits.get(at..at + 4).and_then(hex))
.collect();
text.extend(chars);
} else if let Some(c) = part
.strip_prefix('u')
.filter(|d| (4..=6).contains(&d.len()) && upper_hex(d))
.and_then(hex)
{
text.push(c);
}
}
let mut chars = text.chars();
match (chars.next(), chars.next()) {
(None, _) => None,
(Some(c), None) => Some(BfString::Char(c)),
_ => Some(BfString::String(text)),
}
}
pub(crate) fn unicode_from_name(name: &str) -> Option<char> {
let convert = |input: &str| u32::from_str_radix(input, 16).ok().and_then(char::from_u32);
+1 -1
View File
@@ -397,7 +397,7 @@ impl StandardKind {
})
}
fn code_to_ps_name(&self, code: u8) -> Option<&str> {
pub(crate) fn code_to_ps_name(&self, code: u8) -> Option<&str> {
let bf = self.base_font;
self.encodings
+6 -5
View File
@@ -2,7 +2,7 @@ use crate::font::blob::{CffFontBlob, OpenTypeFontBlob};
use crate::font::generated::{glyph_names, mac_os_roman, mac_roman, standard, win_ansi};
use crate::font::standard_font::StandardKind;
use crate::font::{
Encoding, FallbackFontQuery, FontFlags, glyph_name_to_unicode, read_to_unicode,
Encoding, FallbackFontQuery, FontFlags, glyph_name_to_text, glyph_name_to_unicode, read_to_unicode,
strip_subset_prefix, unicode_from_name,
};
use crate::util::OptionLog;
@@ -190,10 +190,11 @@ impl TrueTypeFont {
}
match &self.kind {
Kind::Embedded(e) => e
.code_to_unicode_name(code as u8)
.and_then(glyph_name_to_unicode)
.map(BfString::Char),
// PdfCraft patch (#3169): a name outside the glyph list reads by the glyph list
// specification's full mapping (`f_i`, `hyphen.case`).
Kind::Embedded(e) => e.code_to_unicode_name(code as u8).and_then(|name| {
glyph_name_to_unicode(name).map(BfString::Char).or_else(|| glyph_name_to_text(name))
}),
Kind::Standard(s) => s.char_code_to_unicode(code as u8).map(BfString::Char),
}
+20 -5
View File
@@ -2,7 +2,8 @@ use crate::font::blob::{CffFontBlob, Type1FontBlob};
use crate::font::standard_font::{StandardFont, StandardKind, select_standard_font};
use crate::font::true_type::{Width, read_encoding, read_widths};
use crate::font::{
Encoding, FallbackFontQuery, glyph_name_to_unicode, normalized_glyph_name, read_to_unicode,
Encoding, FallbackFontQuery, glyph_name_to_text, glyph_name_to_unicode, normalized_glyph_name,
read_to_unicode,
};
use crate::{CMapResolverFn, CacheKey, FontResolverFn};
use hayro_cmap::{BfString, CMap};
@@ -117,11 +118,25 @@ impl Type1Font {
}
let code = char_code as u8;
match &self.1 {
Kind::Standard(s) => s.char_code_to_unicode(code).map(BfString::Char),
Kind::Cff(c) => c.char_code_to_unicode(code).map(BfString::Char),
Kind::Type1(t) => t.char_code_to_unicode(code).map(BfString::Char),
let found = match &self.1 {
Kind::Standard(s) => s.char_code_to_unicode(code),
Kind::Cff(c) => c.char_code_to_unicode(code),
Kind::Type1(t) => t.char_code_to_unicode(code),
};
if let Some(c) = found {
return Some(BfString::Char(c));
}
// PdfCraft patch (#3169, #3170): a name outside the glyph list still reads by the
// glyph list specification's full mapping (`f_i`, `hyphen.case`). A name that only
// numbers the glyph (`C39`, `c251`) says nothing about the character and reads as
// nothing: the code's StandardEncoding character would be a guess, wrong wherever the
// font's /Encoding puts the glyph elsewhere (review on #3239).
let name = match &self.1 {
Kind::Standard(s) => s.code_to_ps_name(code),
Kind::Cff(c) => c.code_to_ps_name(code),
Kind::Type1(t) => t.code_to_ps_name(code),
}?;
glyph_name_to_text(name)
}
}
+10
View File
@@ -126,6 +126,16 @@ pub(crate) fn show_glyph<'a>(
}
TextRenderingMode::Clip => {
clip_glyph(ctx, glyph, glyph_transform);
// PdfCraft patch: report clipping text separately from invisible OCR text.
// Extraction filters must retain text that is filled through its
// clipping path without rendering devices drawing it directly.
device.draw_glyph(
glyph,
ctx.get().ctm,
glyph_transform,
&get_paint(ctx, false),
&GlyphDrawMode::Clip,
);
}
TextRenderingMode::FillAndClip => {
clip_glyph(ctx, glyph, glyph_transform);
+2
View File
@@ -284,6 +284,8 @@ pub enum GlyphDrawMode {
Fill,
/// Draw using a stroke.
Stroke(StrokeProps),
/// PdfCraft patch: text that shapes a clip; report it without painting a glyph.
Clip,
/// Invisible text (for text extraction but not visual rendering).
Invisible,
}
+2 -2
View File
@@ -1913,8 +1913,8 @@ impl<'a> Device<'a> for Renderer {
GlyphDrawMode::Stroke(s) => {
Self::stroke_glyph(self, glyph, transform, glyph_transform, paint, s);
}
GlyphDrawMode::Invisible => {
// Don't render invisible text for visual output
GlyphDrawMode::Invisible | GlyphDrawMode::Clip => {
// PdfCraft patch: neither mode paints; clipping was installed by the interpreter.
}
}
}