fix(render): draw missing spacing glyphs in Japanese CID substitutes as spaces (#3601)

UniJIS-UCS2-HW-H maps code 0x0020 to CID 231 / U+2002. The OpenType
substitute for a non-embedded Japanese CID font lacks that glyph, so
.notdef boxes were drawn. Type0Font::map_via_unicode now falls back to
the substitute's ordinary-space glyph for explicit spacing characters
its charmap lacks. Native and variation-selector glyphs keep priority;
PDF advances and Unicode text are unchanged.

Fixes #3601.

Co-Authored-By: Claude <noreply@anthropic.com>

ROADMAP.md

@@ -2,6 +2,7 @@
 
 **Stage: alpha** · next: beta, ~25% (ready for real work 50% → 75%) and ~400–700 h away
 
+> **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** minor (M2, #3601: missing spacing glyphs in non-embedded Japanese CID substitutes draw as blanks while PDF widths and Unicode stay unchanged; estimates unchanged)
 > **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** minor (M5, PR #1971: bounded nested FDF export excludes source page/page-tree dictionaries; refreshed against upstream 4db6ac4a; estimates unchanged)
 > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M7: the Edit text caret is as tall as the row it is on; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
 > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M6: CSV tab-containing headers; estimates unchanged)
@@ -172,6 +173,8 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
 
 ## Progress log
 
+- **2026-10-20 (M2, #3601):** A missing spacing glyph in an OpenType substitute uses its ordinary-space glyph. Synthetic Japanese CID renderer tests check blank spaces, `/W` and `/DW` placement, and unchanged Unicode extraction. Font selection and embedded fonts are unchanged. Estimates unchanged.
+
 - **2026-10-19 (M14, #3373):** The main menu and its long submenus scroll in short windows, keeping their commands reachable. Estimates unchanged.
 
 - **2026-10-19 (M3, #3387):** Save-prompt shortcuts yield to later consent and utility dialogs; a synthetic UI regression checks that an edited document stays open. Estimates unchanged.

crates/render/src/raster.rs

@@ -6701,6 +6701,61 @@ trailer << /Root 1 0 R >>
         }
     }
 
+    // #3601: UniJIS-UCS2-HW-H maps code 0x0020 to CID 231 / U+2002. The PDF's
+    // explicit 500-unit width must win over the substitute's ordinary-space width.
+    fn japanese_spacing_pdf(content: &str, widths: &str) -> Vec<u8> {
+        format!(
+            "%PDF-1.7
+1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj
+2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >> endobj
+3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 180 60] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >> endobj
+4 0 obj << /Length {} >> stream
+{content}
+endstream endobj
+5 0 obj << /Type /Font /Subtype /Type0 /BaseFont /MS-Mincho /Encoding /UniJIS-UCS2-HW-H /DescendantFonts [6 0 R] >> endobj
+6 0 obj << /Type /Font /Subtype /CIDFontType2 /BaseFont /MS-Mincho /CIDSystemInfo << /Registry (Adobe) /Ordering (Japan1) /Supplement 4 >>
+  /FontDescriptor 7 0 R /DW 1000 {widths} >> endobj
+7 0 obj << /Type /FontDescriptor /FontName /MS-Mincho /Flags 6 /FontBBox [0 -141 1000 859] /ItalicAngle 0 /Ascent 859 /Descent -141 /CapHeight 700 /StemV 80 >> endobj
+trailer << /Root 1 0 R >>
+%%EOF",
+            content.len()
+        )
+        .into_bytes()
+    }
+
+    #[test]
+    fn japanese_substitute_missing_spaces_draw_no_ink() {
+        if pdfcraft_fonts::document_japanese_font().is_none() {
+            eprintln!("built without craft-fonts (CRAFT_FONTS_DIR unset): no Japanese face to check");
+            return;
+        }
+        let out = render_first_page(japanese_spacing_pdf("BT /F1 20 Tf 10 30 Td <002000200020> Tj ET", "/W [231 [500]]"));
+        assert!(out.rgba.as_chunks::<4>().0.iter().all(|pixel| *pixel == [255; 4]), "spacing glyphs must leave the page white");
+    }
+
+    #[test]
+    fn japanese_substitute_spaces_keep_pdf_widths_and_unicode() {
+        if pdfcraft_fonts::document_japanese_font().is_none() {
+            eprintln!("built without craft-fonts (CRAFT_FONTS_DIR unset): no Japanese face to check");
+            return;
+        }
+        for (widths, second_x) in [("/W [231 [500]]", 50), ("", 70)] {
+            let pdf = japanese_spacing_pdf("BT /F1 20 Tf 10 30 Td <65E500200020672C> Tj ET", widths);
+            let out = render_first_page(pdf.clone());
+            // 日 advances 20 pt; each space advances 10 pt (/W) or 20 pt (/DW).
+            // A second independent text matrix pins 本 at the expected original PDF position.
+            let reference = format!("BT /F1 20 Tf 1 0 0 1 10 30 Tm <65E5> Tj 1 0 0 1 {second_x} 30 Tm <672C> Tj ET");
+            let expected = render_first_page(japanese_spacing_pdf(&reference, widths));
+            assert!(expected.rgba.as_chunks::<4>().0.iter().any(|pixel| pixel[0] < 128), "the Japanese control must draw ink");
+            assert!(out.rgba.as_ref() == expected.rgba.as_ref(), "spaces must be blank without changing {widths:?} advances");
+            let mut renderer = PageRenderer::new(Arc::new(pdf), RenderConfig::default());
+            let text = renderer.render(RenderRequest { page: 0, kind: RequestKind::Text, ..Default::default() });
+            assert!(text.error.is_none(), "{:?}", text.error);
+            let glyphs: String = text.text.unwrap().glyphs.iter().map(|glyph| glyph.text.as_str()).collect();
+            assert_eq!(glyphs, "日\u{2002}\u{2002}本", "glyph substitution must not rewrite Unicode");
+        }
+    }
+
     /// Half-width katakana are common in Japanese documents set in a non-embedded Mincho font
     /// such as HeiseiMin-W3. The Mincho substitute must have those glyphs: one that lacks them
     /// (Shippori Mincho) draws them as .notdef.

docs/roadmap.md

@@ -1,5 +1,6 @@
 # PdfCraft roadmap: milestones and what's next
 
+> **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** minor (M2, #3601: missing spacing glyphs in non-embedded Japanese CID substitutes draw as blanks while PDF widths and Unicode stay unchanged; estimates unchanged)
 > **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** minor (M5, PR #1971: bounded nested FDF export excludes source page/page-tree dictionaries; refreshed against upstream 4db6ac4a; estimates unchanged)
 > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M14/#2383 GPU-limit candidate; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
 > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M7: the Edit text caret is as tall as the row it is on; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
@@ -111,7 +112,7 @@ total matches the parity estimate.
 |---|---|---|---|---|---|
 | M0 | Skeleton: workspace, xtask gates, CI | 15–30 | 98% | 1–2 | GitHub workflow, `deny.toml` (licence audit of every dependency), parity checklist (826 features, `xtask parity`) done. Missing: remaining crate stubs, testkit/oracle crates |
 | M1 | COS: filters, crypt, parser, xref, writer | 120–200 | 80% | 20–40 | Done:<br>- filters and crypt: every standard-security revision R2–R6 (RC4, AES-128/256), SASLprep, permissions, creating encryption;<br>- cos: parse and repair (including invalid xref object-number ranges), decoded-stream limits including unfiltered data, decrypt on load, re-encrypt on save, incremental and full writing.<br>Corpus: open/edit/save passes on 958 files, and all 7 password-protected files open.<br>Full saves now pack objects into compressed object streams. An unchanged incremental save of a classic table writes a zero-length cross-reference subsection instead of a section with no subsections (#1339). A full rewrite keeps unknown trailer entries, including objects reachable only from them (#1338). Fuzzing runs nightly (`xtask fuzz`). Raw-deflate fallback preserves decoding-limit errors, including through tolerant COS stream decoding. Missing: ≥ 250 tests, own image codecs |
-| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. |
+| M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. Missing spacing glyphs in non-embedded Japanese CID substitutes render blank without changing PDF advances or Unicode (#3601). |
 | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. The renderer keeps its one-point crop-size floor in physical points after UserUnit scaling, with regression coverage for crop origins, rotations and f32 reciprocal rounding. Existing unit policy and nearly-zero fallback are retained. |
 | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings |
 | M3 | Viewer app (native + web) | 80–150 | 90% | 10–20 | Acrobat-style shell, find, select, panels, tiles, web build, UI control channel for agents (opt-in). Missing: 60 fps test on a 500-page document, snapshot tests of every panel<br>Done since: fit visible, document title in the window; Linux middle-button scrolling; shared three-state theme preferences in the toolbar, View menu and control channel; numeric Linux portal events and idle wakeups completing #881, with native appearance inheritance retained (#446); late browser startup downloads preserve the current document and Save target (#171). Selected-text Copy in Edit and the canvas context menu is covered by focused regressions and native clipboard checks (#1225); document copy restrictions apply to the floating action too. #1364 candidate: absolute local GoToR links ask before target access; headless callers explicitly opt in. Native-opening candidate: chosen tools survive preparation and password retries; startup intake waits for pending preparation. Page-label navigation displays logical labels, retains the current page on unchanged submission and refreshes focused same-document replacements; clean file-watcher reload preserves the chosen scrolling mode, preventing a Single Page reader from jumping to page 1 during replacement. Synthetic control regressions cover these behaviors. #3149: page-list changes discard render demand based on old geometry; actual raster dimensions are checked against the GPU limit before upload. Synthetic crash and continued-rendering regressions cover the replacement path. Save-prompt shortcuts yield to dialogs above them; stacked link-consent UI regression covers Escape and Return. |
@@ -196,6 +197,8 @@ M2 (renderer), M7 (content editing), M10 (OCR and Office), M11 (Preflight and st
 
 ## Revision history
 
+**2026-10-20 · minor:** M2/#3601: substitute-font spacing aliases preserve blank ink, original `/W` and `/DW` advances, and Unicode extraction. Synthetic renderer regressions cover the reported Japanese CID fixture. Estimates unchanged.
+
 **2026-10-15 · minor:** M14/#2383 candidate: GPU device requests clamp unsupported limits while retaining the adapter texture extent. Synthetic V3D/downlevel regressions and Windows GPU-required tests pass; Raspberry Pi hardware remains unverified. Estimates unchanged.
 
 | Date | Change | Summary |

vendor/README.md

@@ -51,3 +51,12 @@ black-point normalization is unchanged. `lab_bradford_tests` checks independent
 D50/equal-energy neutral and chromatic oracles, D65 identity, and direct/Indexed
 consistency (including an asymmetric chromatic Range), and zero/overflowing
 cone-response fallback.
+
+Missing spacing glyphs (#3601): in `hayro-interpret` `Type0Font::map_via_unicode`,
+an OpenType substitute uses its ordinary-space glyph only when its own charmap
+lacks a spacing character (including Adobe-Japan1 U+2002). Native glyphs and
+variation forms retain priority; embedded fonts, PDF advances and Unicode decoding
+are unchanged. Renderer regressions `japanese_substitute_missing_spaces_draw_no_ink`
+and `japanese_substitute_spaces_keep_pdf_widths_and_unicode` use the synthetic
+MS-Mincho / UniJIS-UCS2-HW-H fixture with craft-fonts, including `/W` and `/DW`
+placement controls.

vendor/hayro-interpret/src/font/cid.rs

@@ -264,7 +264,13 @@ impl Type0Font {
                 {
                     return Some(glyph);
                 }
-                charmap.map(character)
+                // PdfCraft patch: a substitute may lack a spacing character (#3601; e.g.
+                // Adobe-Japan1's U+2002). Draw its ordinary space instead of .notdef;
+                // the PDF's widths and Unicode mapping still determine advance and text.
+                charmap.map(character).or_else(|| match character {
+                    '\u{00a0}' | '\u{2000}'..='\u{200a}' | '\u{202f}' | '\u{205f}' | '\u{3000}' => charmap.map(' '),
+                    _ => None,
+                })
             }
             FontType::Cff(c) => {
                 // Map codepoint to glyph name via AFL, and then look it up.