fix(fonts): read the Annex D.2 alias codes as the ink they draw

#2305 fixed WinAnsi 0xAD in `crates/fonts`, and closed with two loose ends it named: the
user-visible extraction path goes through hayro, not that table, so it still reported the
invisible character; and the same Annex D.2 note makes 0xA0 the `space` glyph. This finishes
both, and makes the two tables in the product agree about the same byte.

WinAnsiEncoding gives codes 0xA0 and 0xAD the glyph names `space` and `hyphen` -- the names it
also gives 0x20 and 0x2D -- and MacRomanEncoding gives 0xCA `space`. The tables called them
`nbspace` and `sfthyphen`, whose Adobe Glyph List values U+00A0 and U+00AD are invisible
formatting characters, so extraction hid ink the reader can see: "well-known" drawn with the
0xAD byte matched no search and no redaction pattern, and a non-breaking space did not match the
space a reader typed.

Evidence. Two sources transcribe the table the same way (an Annex D implementation's docs, and
the WinAnsi.enc encoding vector R ships, which gives `/space` at octal 240 and `/hyphen` at
255), and a black-box check settles the rest: on a synthetic fixture pdftotext reads the
base-encoded codes as `a-b a b`, and keeps U+00AD for a code an explicit `/Differences` names
`sfthyphen`. We now do both. Over the 989-file pdf.js corpus the soft hyphens we extract fall
from 117 to 13 and the non-breaking spaces from 30 to 26; everything left comes from `/ToUnicode`
maps, which say what they mean and are not touched.

Scope. The base table only, in the text direction only.

- A `/Differences` entry naming `sfthyphen` or `nbspace` is a deliberate statement about one
  code and keeps its glyph list value, as pdftotext does.
- Glyph lookup is untouched, so a font program's own `sfthyphen` program is still found first.
  In hayro that is why the alias sits in the Unicode path rather than in the generated encoding
  table: `code_to_ps_name` is shared with lookup.
- Writing is unaffected: `encode` takes the lowest code showing a character, so ' ' and '-'
  still write 0x20 and 0x2D, and a caller holding a literal U+00A0 still writes 0xA0.

Tests. `fonts` pins both tables and the whole decode/encode path including the `/Differences`
case; `render` pins the extraction a reader actually gets, over WinAnsi, MacRoman and
`/Differences`. Each was watched failing with each cell and each match arm reverted in turn, so
no assertion is passing by accident. fonts 70, render 296+4, redact 49, a11y 43 and edit 156 are
green, with fmt, clippy, `xtask layers` (37 crates) and `xtask assets` clean.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tanish Manku
2026-10-15 12:52:53 -07:00
co-authored by Claude Opus 5
parent dc41044ba7
commit 13e3154ab3
10 changed files with 173 additions and 20 deletions
+9 -6
View File
@@ -5,11 +5,13 @@ pub(crate) static WIN_ANSI: [u32; 256] = [
44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 0, 8364, 0, 8218, 402, 8222, 8230, 8224, 8225, 710, 8240, 352, 8249, 338,
0, 381, 0, 0, 8216, 8217, 8220, 8221, 8226, 8211, 8212, 732, 8482, 353, 8250, 339, 0, 382, 376, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169,
// 0xAD is `hyphen`, not U+00AD: WinAnsiEncoding's table (Annex D.2) gives codes 0x2D and 0xAD
// the same glyph name, so the byte a Windows codepage calls a soft hyphen draws, and extracts
// as, an ordinary hyphen. Emitting U+00AD instead hides the character from search and from
// redaction's pattern matching, because a soft hyphen is invisible and matches nothing.
// 0xA0 is `space` and 0xAD is `hyphen`, not U+00A0 and U+00AD: WinAnsiEncoding's table
// (Annex D.2) gives those codes the same glyph names as 0x20 and 0x2D, so the bytes a Windows
// codepage calls a non-breaking space and a soft hyphen draw, and extract as, an ordinary
// space and hyphen. Emitting the formatting characters instead hides them from search and
// from redaction's pattern matching: a soft hyphen is invisible and matches nothing, and a
// non-breaking space does not match the space a reader typed.
0, 381, 0, 0, 8216, 8217, 8220, 8221, 8226, 8211, 8212, 732, 8482, 353, 8250, 339, 0, 382, 376, 32, 161, 162, 163, 164, 165, 166, 167, 168, 169,
170, 171, 172, 45, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198,
199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227,
228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255,
@@ -21,7 +23,8 @@ pub(crate) static MAC_ROMAN: [u32; 256] = [
113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 0, 196, 197, 199, 201, 209, 214, 220, 225, 224, 226, 228, 227, 229, 231,
233, 232, 234, 235, 237, 236, 238, 239, 241, 243, 242, 244, 246, 245, 250, 249, 251, 252, 8224, 176, 162, 163, 167, 8226, 182, 223, 174, 169,
8482, 180, 168, 8800, 198, 216, 8734, 177, 8804, 8805, 165, 181, 8706, 8721, 8719, 960, 8747, 170, 186, 937, 230, 248, 191, 161, 172, 8730, 402,
8776, 8710, 171, 187, 8230, 160, 192, 195, 213, 338, 339, 8211, 8212, 8220, 8221, 8216, 8217, 247, 9674, 255, 376, 8260, 8364, 8249, 8250, 64257,
// 0xCA is `space`, the same glyph name Annex D.2 gives 0x20, for the reason WinAnsi 0xA0 is.
8776, 8710, 171, 187, 8230, 32, 192, 195, 213, 338, 339, 8211, 8212, 8220, 8221, 8216, 8217, 247, 9674, 255, 376, 8260, 8364, 8249, 8250, 64257,
64258, 8225, 183, 8218, 8222, 8240, 194, 202, 193, 203, 200, 205, 206, 207, 204, 211, 212, 63743, 210, 218, 219, 217, 305, 710, 732, 175, 728,
729, 730, 184, 733, 731, 711,
];
+27 -13
View File
@@ -305,31 +305,45 @@ mod tests {
assert_eq!(literal(b"a(b)\\c"), b"(a\\(b\\)\\\\c)");
}
/// WinAnsiEncoding's 0xAD is the `hyphen` glyph, not U+00AD.
/// WinAnsiEncoding's 0xAD is the `hyphen` glyph and its 0xA0 is `space`, not U+00AD and
/// U+00A0; MacRomanEncoding's 0xCA is `space` too.
///
/// Annex D.2 gives codes 0x2D and 0xAD the same glyph name, so the byte a Windows codepage
/// calls a soft hyphen draws an ordinary hyphen and must extract as one. Reading it as U+00AD
/// hid the character: a soft hyphen is invisible, so a search for "well-known" missed text
/// that drew exactly that, and `redact`'s pattern matching missed it too. One corpus fixture
/// alone extracted 104 of them.
/// Annex D.2 gives each of those codes the same glyph name as the ordinary one (0x2D, 0x20),
/// so the bytes a Windows codepage calls a soft hyphen and a non-breaking space draw ordinary
/// ink and must extract as it. Reading them as the formatting characters hid them: a soft
/// hyphen is invisible, so a search for "well-known" missed text that drew exactly that, and
/// `redact`'s pattern matching missed it too; a non-breaking space does not match the space a
/// reader types. Over the 989-file pdf.js corpus this takes the soft hyphens we extract from
/// 117 to 13 (104 of them in `XiaoBiaoSong.pdf` alone) and the non-breaking spaces from 30 to
/// 26; what remains comes from `/ToUnicode` maps, which say so explicitly and are not touched.
/// pdftotext reads both codes as the ordinary character.
#[test]
fn win_ansi_soft_hyphen_byte_reads_as_an_ordinary_hyphen() {
fn the_alias_codes_read_as_the_glyph_they_draw() {
assert_eq!(encodings::WIN_ANSI.get(0xAD), Some(&u32::from('-')));
assert_eq!(encodings::WIN_ANSI.get(0x2D), Some(&u32::from('-')));
// Writing is unaffected: '-' takes the canonical low code, and a caller holding a literal
// U+00AD still writes the 0xAD byte, which is the one that draws a hyphen.
assert_eq!(encodings::WIN_ANSI.get(0xA0), Some(&u32::from(' ')));
assert_eq!(encodings::WIN_ANSI.get(0x20), Some(&u32::from(' ')));
assert_eq!(encodings::MAC_ROMAN.get(0xCA), Some(&u32::from(' ')));
assert_eq!(encodings::MAC_ROMAN.get(0x20), Some(&u32::from(' ')));
// MacRoman has no soft hyphen; 0xAD there is `notequal`, and keeps it.
assert_eq!(encodings::MAC_ROMAN.get(0xAD), Some(&0x2260));
// Writing is unaffected: '-' and ' ' take the canonical low codes, and a caller holding a
// literal U+00AD or U+00A0 still writes the high byte, which is the one that draws them.
assert_eq!(win_ansi_byte('-'), Some(0x2D));
assert_eq!(win_ansi_byte(' '), Some(0x20));
assert_eq!(win_ansi_byte('\u{00ad}'), Some(0xAD));
assert_eq!(win_ansi("a-b"), b"a-b");
assert_eq!(win_ansi_byte('\u{00a0}'), Some(0xA0));
assert_eq!(win_ansi("a-b c"), b"a-b c");
}
#[test]
fn win_ansi_covers_the_0x80_to_0x9f_glyphs() {
// Every character in the WinAnsiEncoding table encodes to its code and back (#347), except
// where two codes share one glyph: 0xAD is `hyphen`, the same glyph as 0x2D, so it decodes
// to '-' and '-' encodes back to the canonical 0x2D. Round-tripping the *code* is what
// where two codes share one glyph: 0xAD is `hyphen`, the same glyph as 0x2D, and 0xA0 is
// `space`, the same glyph as 0x20, so each decodes to the ordinary character and that
// character encodes back to the canonical low code. Round-tripping the *code* is what
// cannot hold there; round-tripping the character still does.
const ALIASES: &[u8] = &[0xAD];
const ALIASES: &[u8] = &[0xA0, 0xAD];
for (code, &u) in encodings::WIN_ANSI.iter().enumerate() {
let Some(c) = char::from_u32(u).filter(|_| u != 0) else { continue };
if ALIASES.contains(&(code as u8)) {
+26
View File
@@ -1863,6 +1863,32 @@ mod encoding_tests {
}
}
/// The Annex D.2 alias codes read, through the whole decode path, as the ink they draw.
///
/// This is the map `redact` matches its patterns on (`unicode_map`), so a code that decodes
/// to an invisible formatting character is a silent miss, not a cosmetic one: "well-known"
/// written with the 0xAD byte matched no search and no redaction pattern.
#[test]
fn the_alias_codes_decode_as_ordinary_ink_and_encode_back_to_the_low_code() {
let (doc, d) = simple("Helvetica", None, Some(Object::name("WinAnsiEncoding")));
let m = Metrics::from_dict(&doc, &d);
assert_eq!(m.decode(b"well\xadknown"), "well-known");
assert_eq!(m.decode(b"a\xa0b"), "a b");
assert_eq!(m.encode("well-known a b").as_deref(), Some(&b"well-known a b"[..]), "the low code is the one written");
let (doc, d) = simple("Helvetica", None, Some(Object::name("MacRomanEncoding")));
let m = Metrics::from_dict(&doc, &d);
assert_eq!(m.decode(b"a\xcab"), "a b");
assert_eq!(m.decode(b"a\xadb"), "a\u{2260}b", "MacRoman 0xAD is `notequal`, not a hyphen");
assert_eq!(m.encode("a b").as_deref(), Some(&b"a b"[..]));
// An explicit /Differences name still says what it says: this is a property of those
// codes in those tables, not of the glyph names.
let (doc, d) = simple("Helvetica", None, Some(differences(vec![Object::Int(0x41), Object::name("sfthyphen"), Object::name("nbspace")])));
let m = Metrics::from_dict(&doc, &d);
assert_eq!(m.decode(b"AB"), "\u{ad}\u{a0}");
}
/// Pins a deliberate restraint, so it is not "fixed" into a regression: a great many ordinary
/// Latin TrueType subsets set the `/FontDescriptor` symbolic flag and still rely on a Latin
/// encoding. Suppressing the implicit table on the flag would lose real text on a large share
+54
View File
@@ -1740,6 +1740,60 @@ mod tests {
}
}
/// One page per `(encoding, content)`, in Helvetica (`/F1`).
fn encoded_pages(pages: &[(lopdf::Object, &[u8])]) -> Pdf {
use lopdf::{Document, Object, Stream, dictionary};
let mut doc = Document::with_version("1.7");
let pages_id = doc.new_object_id();
let mut kids = Vec::new();
for (encoding, content) in pages {
let font =
doc.add_object(dictionary! { "Type" => "Font", "Subtype" => "Type1", "BaseFont" => "Helvetica", "Encoding" => encoding.clone() });
let content = doc.add_object(Stream::new(dictionary! {}, content.to_vec()));
let page = doc.add_object(dictionary! { "Type" => "Page", "Parent" => pages_id, "MediaBox" => vec![Object::Integer(0), Object::Integer(0), Object::Integer(300), Object::Integer(300)], "Resources" => dictionary! {"Font" => dictionary! {"F1" => font}}, "Contents" => content });
kids.push(Object::Reference(page));
}
let count = kids.len() as i64;
doc.objects.insert(pages_id, dictionary! {"Type" => "Pages", "Kids" => kids, "Count" => count}.into());
let catalog = doc.add_object(dictionary! {"Type" => "Catalog", "Pages" => pages_id});
doc.trailer.set("Root", catalog);
let mut bytes = Vec::new();
doc.save_to(&mut bytes).unwrap();
Pdf::new(std::sync::Arc::new(bytes)).unwrap()
}
/// The Annex D.2 alias codes extract as the ink they draw, not as formatting characters.
///
/// WinAnsiEncoding gives 0xAD and 0xA0 the glyph names `hyphen` and `space`, the names it
/// also gives 0x2D and 0x20, and MacRomanEncoding gives 0xCA `space`. Reporting the Adobe
/// Glyph List values U+00AD and U+00A0 for them hid visible text: a soft hyphen is invisible,
/// so "well-known" written that way matched no search and no redaction pattern.
/// `XiaoBiaoSong.pdf` in the pdf.js corpus drew 104 of them.
///
/// A `/Differences` entry naming `sfthyphen` or `nbspace` is a deliberate statement about one
/// code and keeps its glyph list value. pdftotext draws the same line: on this fixture it
/// reads the base-encoded codes as "a-b a b" and the named ones as U+00AD.
#[test]
fn the_annex_d2_alias_codes_read_as_an_ordinary_hyphen_and_space() {
use lopdf::{Object, dictionary};
let name = |n: &str| Object::Name(n.as_bytes().to_vec());
let differences = Object::Dictionary(dictionary! {
"Type" => "Encoding",
"BaseEncoding" => "WinAnsiEncoding",
"Differences" => vec![Object::Integer(88), name("sfthyphen"), Object::Integer(89), name("nbspace")],
});
let pdf = encoded_pages(&[
(name("WinAnsiEncoding"), &b"BT /F1 20 Tf 1 0 0 1 20 150 Tm (a\xadb\xa0c) Tj ET"[..]),
(name("MacRomanEncoding"), &b"BT /F1 20 Tf 1 0 0 1 20 150 Tm (a\xcab) Tj ET"[..]),
(differences, &b"BT /F1 20 Tf 1 0 0 1 20 150 Tm (aXbYc) Tj ET"[..]),
]);
let texts =
|page| extract_page(&pdf, page, &InterpreterSettings::default()).unwrap().glyphs.iter().map(|g| g.text.clone()).collect::<Vec<_>>();
assert_eq!(texts(0), ["a", "-", "b", " ", "c"], "WinAnsi 0xAD and 0xA0");
assert_eq!(texts(1), ["a", " ", "b"], "MacRoman 0xCA");
assert_eq!(texts(2), ["a", "\u{ad}", "b", "\u{a0}", "c"], "a named glyph keeps its glyph list value");
}
#[test]
fn narrow_ligatures_keep_every_letter() {
// Split evenly, this "ffi" put its two "f"s so close that the second was dropped as a
+1 -1
View File
File diff suppressed because one or more lines are too long
+23
View File
@@ -444,6 +444,29 @@ impl Encoding {
Self::BuiltIn => None,
}
}
/// PdfCraft patch: the glyph name an Annex D.2 *alias code* reads as when the glyph is taken
/// as text rather than looked up in a font program; `None` for every other code.
///
/// Annex D.2 gives WinAnsiEncoding codes 0xA0 and 0xAD the glyph names `space` and `hyphen`,
/// the same names it gives 0x20 and 0x2D, and MacRomanEncoding 0xCA `space`. The tables above
/// call them `nbspace` and `sfthyphen`, whose Adobe Glyph List values U+00A0 and U+00AD are
/// *formatting* characters, so extraction hid ink the reader can see: a soft hyphen is
/// invisible, and text that drew one matched no search and no redaction pattern -- one corpus
/// file alone produced 104 of them -- while a non-breaking space did not match the space a
/// reader typed. pdftotext reads both codes as the ordinary character.
///
/// Only the base table is aliased, in the text direction only. A `/Differences` entry naming
/// `sfthyphen` or `nbspace` is a deliberate statement and keeps its glyph list value, which
/// is what pdftotext does too; and glyph *lookup* is untouched, so a font program's own
/// `sfthyphen` program is still found first.
fn text_alias(&self, code: u8) -> Option<&'static str> {
match (self, code) {
(Self::WinAnsi, 0xA0) | (Self::MacRoman, 0xCA) => Some("space"),
(Self::WinAnsi, 0xAD) => Some("hyphen"),
_ => None,
}
}
}
/// The font stretch.
+8
View File
@@ -546,6 +546,14 @@ impl StandardKind {
}
pub(crate) fn char_code_to_unicode(&self, code: u8) -> Option<char> {
// PdfCraft patch: an Annex D.2 alias code reads as the ordinary character; see
// `Encoding::text_alias`. `code_to_ps_name` is shared with glyph lookup and stays as it
// is, so the alias is applied here, after `/Differences` has had its say.
if !self.encodings.contains_key(&code)
&& let Some(name) = self.encoding.text_alias(code)
{
return glyph_name_to_unicode(name);
}
self.code_to_ps_name(code).and_then(glyph_name_to_unicode)
}
+6
View File
@@ -286,7 +286,13 @@ impl EmbeddedKind {
self.differences
.get(&code)
.map(|s| s.as_str())
// PdfCraft patch: an Annex D.2 alias code reads as the ordinary character; see
// `Encoding::text_alias`.
.or_else(|| self.encoding.text_alias(code))
.or_else(|| self.encoding.map_code(code))
// The last resort reads the code as WinAnsi whatever the font said, so it takes the
// same aliasing.
.or_else(|| Encoding::WinAnsi.text_alias(code))
.or_else(|| win_ansi::get(code))
}
+16
View File
@@ -239,6 +239,14 @@ impl Type1Kind {
}
fn char_code_to_unicode(&self, code: u8) -> Option<char> {
// PdfCraft patch: an Annex D.2 alias code reads as the ordinary character; see
// `Encoding::text_alias`. `code_to_ps_name` is shared with glyph lookup and stays as it
// is, so the alias is applied here, after `/Differences` has had its say.
if !self.encodings.contains_key(&code)
&& let Some(name) = self.encoding.text_alias(code)
{
return glyph_name_to_unicode(name);
}
self.code_to_ps_name(code).and_then(glyph_name_to_unicode)
}
}
@@ -336,6 +344,14 @@ impl CffKind {
}
fn char_code_to_unicode(&self, code: u8) -> Option<char> {
// PdfCraft patch: an Annex D.2 alias code reads as the ordinary character; see
// `Encoding::text_alias`. `code_to_ps_name` is shared with glyph lookup and stays as it
// is, so the alias is applied here, after `/Differences` has had its say.
if !self.encodings.contains_key(&code)
&& let Some(name) = self.encoding.text_alias(code)
{
return glyph_name_to_unicode(name);
}
self.code_to_ps_name(code).and_then(glyph_name_to_unicode)
}
}
+3
View File
@@ -108,6 +108,9 @@ impl<'a> Type3<'a> {
.encodings
.get(&code)
.map(|s| s.as_str())
// PdfCraft patch: an Annex D.2 alias code reads as the ordinary character;
// see `Encoding::text_alias`.
.or_else(|| self.encoding.text_alias(code))
.or_else(|| self.encoding.map_code(code));
name.and_then(crate::font::glyph_name_to_unicode)
.or_else(|| (0x20..0x7f).contains(&code).then_some(code as char))