Files
wordcraft/crates/docbin/src/notes.rs
T
663904a2e2 Import Word 97-2003 .doc files (#28) (#145)
* D0+D1: open Word 97-2003 .doc — container, FIB, piece table text (#28)

New wordcraft-docbin crate (L2) reads the OLE2 container via the cfb
crate (MIT) and parses the FIB and piece table from the [MS-DOC]
specification: text pieces decode as UTF-16LE or Windows-1252 (table
moved to wordcraft-doc::encoding), subdocument CP ranges, and Word's
special characters map to the model's (tabs, breaks, hyphens). .doc/.dot
registered in OPEN_EXTS and the I/O dispatch. Encrypted and Word 6/95
files give clear errors; every length is checked and proptest
truncation/flips never panic.

* D2: .doc formatting — stylesheet, fonts, CHPX/PAPX FKPs, sprms (#28)

Sprm decoder with operand-size rules (incl. sprmTDefTable/sprmPChgTabs
exceptions and the Prm0 isprm table), bin tables + ChpxFkp/PapxFkp page
lookup, STSH styles (StdfBase layout validated against real Word files:
bchUpe == cbStd, sti 0 = Normal) and the SttbfFfn font table in both
extended and self-delimiting flavours. The CP walk now splits runs on
formatting changes and resolves paragraph styles: bold/italic/strike/
underline/caps/hidden, colours (Ico palette + COLORREF), half-point
sizes, super/subscript, fonts, highlights; alignment, indents, spacing,
keep options, outline levels. Unknown sprms are skipped leniently;
an env-gated corpus test opens real files (styles + text asserted).

* D3: .doc structure — sections, headers/footers, tables, lists (#28)

- Sections: PlcfSed/Sepx with 16-bit measurement sprms (sprmSXaPage & co),
  explicit sprmSBOrientation with a width/height fallback; even/odd/first
  header and footer stories from Plcfhdd become document parts.
- Tables: sprmTDefTable grids (operand length fixed to cb+1 per the spec),
  horizontal merges (sprmTMerge), vertical merges (sprmTVertMerge), cell
  shading (sprmTSetShd / sprmTDefTableShd80), cell widths, valign, row
  height/header/cant-split, and table borders (sprmTTableBorders80 Brc80s).
- Lists: PlfLst (LSTF + appended LVLs) and PlfLfo (LFO + LFOLVL overrides)
  resolved into the numbering model; sprmPIlvl/sprmPIlfo mark paragraphs.
- Combined fixture (intro + table + list + 2 sections + header) round-trips
  through Document JSON; hostile proptests now mutate the structured fixture.

* D4: .doc notes, fields, bookmarks and inline pictures (#28)

- Footnotes/endnotes: PlcffndRef/PlcfEndRef reference CPs paired with the
  PlcffndTxt/PlcfEndTxt story ranges; each story walks into a Footnote or
  Endnote part and the 0x02 anchor becomes a NoteRef object.
- Fields: 0x13/0x14/0x15 state machine in the walker — instruction text is
  hidden, the cached result kept; HYPERLINK fields keep their result as
  linked runs, everything else becomes one Field object.
- Bookmarks: SttbfBkmk names + Plcfbkf/Plcfbkl CPs become BookmarkStart/
  BookmarkEnd objects at their anchors.
- Pictures: sprmCPicLocation points into the Data stream; the PICF header
  gives the display size (dxaGoal/dyaGoal scaled by mx/my) and the payload
  is format-sniffed (PNG/JPEG/GIF/BMP; metafiles are dropped with a warn).
- Hostile proptests now truncate/flip a fixture with all of the above.

* D5: .doc import hardening, caps and roadmap (#28)

- Cap inline pictures at 5 000 per document.
- Engine test: .doc/.dot bytes dispatch through the Word 97-2003 reader
  (garbage must error, never panic or open empty).
- ROADMAP: Word 97-2003 import row (60% — text, formatting, styles,
  sections, headers/footers, tables, lists, notes, fields, bookmarks,
  raster pictures).

Real-corpus validation stays opt-in: WORDCRAFT_DOCBIN_FIXTURES=<dir>
cargo test -p wordcraft-docbin corpus_files_open.

* Corpus fixes: compressed-piece FC space, PAPX query point, none-level labels (#28)

Validated against two real files (kept locally under plan/word/fixtures,
gitignored): a 1 MB Lorem-ipsum sample and a 2003 styles/lists/table
sample.

- FKP bin tables and pages address the WordDocument stream in true byte
  offsets; a compressed piece stores its base fc doubled (byte = fc/2),
  so fc_of_cp now unwinds the doubling. Before, every PAPX/CHPX lookup in
  compressed (cp1252) files landed on the wrong entry — tables collapsed
  to one row, lists and heading styles were lost.
- The paragraph-properties query now uses the paragraph mark's own FC
  (inside the paragraph) instead of the FC after it, which picked up the
  next paragraph's FKP entry.
- Fixture papx entries re-keyed to paragraph-start CPs, the real rgfc
  convention.
- Counters::next_label returns nothing for NumFormat::None levels (Word's
  numFmt none shows no label; also fixes a latent docx-reader case).
- Symbol/Wingdings bullet chars map to Unicode glyphs (• ▪ ○ ●).

With this, both corpus files render headings, styles, nested and numbered
lists, the tables and the PAGE field correctly; metafile-wrapped images
and OLE objects stay dropped by policy (logged).

* docbin: bound the text walk by real text; dedupe and cap pictures (#28)

Hostile CP numbers no longer drive the work:
- header/footnote/endnote story ranges from the Table stream are clamped to their
  subdocument (ccpHdd/ccpFtn/ccpEdn) with checked base additions, and every walk stops at
  the last piece's CP; a story spanning [0, 4e9) used to push billions of U+FFFD;
- characters are found by binary search over the pieces (PieceTable::char_at), and an
  unreadable stretch (gap, truncated or out-of-stream piece) is skipped in one step rather
  than one U+FFFD per CP; the old per-character scan was O(chars x pieces);
- the piece CPs are made non-decreasing at parse, so inverted pieces are empty and binary
  search stays sound; surrogate pairs now decode;
- note references are looked up by binary search; the style's CharProps is borrowed, not
  cloned, per character;
- pictures are interned by fc_pic (5000 anchors on one picture share one media entry and
  one copy) and the bytes copied per document are capped at 256 MB.

Each fix has a crafted-bytes regression test; against the previous code the story,
piece-claim, last-piece and many-pieces tests hang (killed at 60 s) and the shared
picture test copies the picture once per anchor (80 GB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docbin: checked offset arithmetic in lists, sections and table rows (#28)

- list.rs: PlfLst/PlfLfo/LVL offsets (at + 2 + cLst*28, at + 4 + lfoMac*16, off += len,
  ...) saturate, which matters on 32-bit targets (wasm32) where fc + n can overflow usize;
  the LFO override path no longer indexes abstracts directly.
- sections.rs: the Sepx range (fc_sepx + 2 + cb) is checked; Plcfhdd is read only as far
  as the stories a capped section count can use (it used to reserve lcb/4 entries).
- table.rs: sprmTDyaRowHeight = i16::MIN negated in i16 (overflow panic in debug builds);
  now -(v as f32).

Regression tests: the row height test failed before (overflow panic with debug assertions, a 0 pt height without); the list test pins the saturating
offsets (it only overflowed on 32-bit targets).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Format the merged OPEN_EXTS list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:36:38 -04:00

110 lines
4.2 KiB
Rust

//! Footnotes, endnotes and bookmarks: `PlcffndRef`/`PlcffndTxt`, `PlcfendRef`/`PlcfendTxt`,
//! and `SttbfBkmk` + `Plcfbkf`/`Plcfbkl` ([MS-DOC] §2.9).
/// Cap on notes and bookmarks per document.
const MAX: usize = 10_000;
/// Parse the note reference/story PLCs. `ref_pair`/`txt_pair` are the FIB pair indices
/// (`(2, 3)` for footnotes, `(46, 47)` for endnotes); story ranges are relative to the
/// note subdocument.
pub(crate) fn parse_notes(table: &[u8], fib: &crate::fib::Fib, ref_pair: usize, txt_pair: usize) -> Vec<(u32, (u32, u32))> {
let refs = fib.pair(ref_pair).and_then(|(fc, lcb)| plc_of(table, fc, lcb, 2)).unwrap_or_default();
let txts = fib.pair(txt_pair).and_then(|(fc, lcb)| plc_of(table, fc, lcb, 0)).unwrap_or_default();
if txts.len() < 2 {
return Vec::new();
}
let mut out = Vec::new();
// The reference PLC's aFtnIdx order matches the story order; pair them by index.
for (i, &cp) in refs.iter().enumerate() {
if let (Some(a), Some(b)) = (txts.get(i), txts.get(i + 1))
&& b > a
{
out.push((cp, (*a, *b)));
}
}
out
}
/// The CPs of a PLC (the trailing element is a limit and is kept so callers can pair
/// range ends) plus the data-element size check. Returns the CPs only.
fn plc_of(table: &[u8], fc: u32, lcb: u32, data_size: usize) -> Option<Vec<u32>> {
if lcb == 0 {
return None;
}
let plc = table.get(fc as usize..(fc as usize).saturating_add(lcb as usize))?;
if plc.len() < 4 {
return None;
}
// aCP has one more element than aData: n = (len - data_size) / (4 + data_size).
let n = (plc.len() - data_size) / (4 + data_size);
if n == 0 || n > MAX {
return None;
}
let mut cps = Vec::with_capacity(n.min(MAX));
for i in 0..=n.min(MAX) {
match plc.get(i * 4..i * 4 + 4) {
Some(b) => cps.push(u32::from_le_bytes([b[0], b[1], b[2], b[3]])),
None => break,
}
}
Some(cps)
}
/// A bookmark boundary to insert while walking.
pub(crate) enum Mark {
Start(String),
End(String),
}
/// Parse `SttbfBkmk` + `Plcfbkf` + `Plcfbkl` into (cp, event) pairs sorted for a sequential
/// walk. Hidden bookmarks (leading `_`) are kept: the engine can decide.
pub(crate) fn parse_bookmarks(table: &[u8], fib: &crate::fib::Fib) -> Vec<(u32, Mark)> {
let names = fib.pair(crate::fib::pair::STTBF_BKMK).and_then(|(fc, lcb)| sttb_strings(table, fc, lcb)).unwrap_or_default();
let starts = fib.pair(crate::fib::pair::PLCF_BKF).and_then(|(fc, lcb)| plc_of(table, fc, lcb, 4)).unwrap_or_default();
let ends = fib.pair(crate::fib::pair::PLCF_BKL).and_then(|(fc, lcb)| plc_of(table, fc, lcb, 2)).unwrap_or_default();
let n = starts.len().saturating_sub(1).min(ends.len().saturating_sub(1)).min(names.len()).min(MAX);
let mut out = Vec::with_capacity(n * 2);
for i in 0..n {
let (Some(&a), Some(&b)) = (starts.get(i), ends.get(i)) else { break };
if b < a {
continue;
}
if let Some(name) = names.get(i) {
if name.is_empty() {
continue;
}
out.push((a, Mark::Start(name.clone())));
out.push((b, Mark::End(name.clone())));
}
}
out.sort_by_key(|(cp, m)| (*cp, matches!(m, Mark::End(_))));
out
}
/// Strings of an extended STTB (fExtend = 0xFFFF, 2-byte chars, no extra data).
fn sttb_strings(table: &[u8], fc: u32, lcb: u32) -> Option<Vec<String>> {
let b = table.get(fc as usize..(fc as usize).saturating_add(lcb as usize))?;
if b.len() < 6 {
return None;
}
let c_data = u16::from_le_bytes([b[2], b[3]]) as usize;
if c_data > MAX {
return None;
}
let mut at = 6usize;
let mut out = Vec::with_capacity(c_data);
for _ in 0..c_data {
let cch = match b.get(at..at + 2) {
Some(x) => u16::from_le_bytes([x[0], x[1]]) as usize,
None => break,
};
at += 2;
let cch = cch.min(40);
let chars = b.get(at..at + cch * 2)?;
at += cch * 2;
let s: String = chars.as_chunks::<2>().0.iter().map(|c| char::from_u32(u16::from_le_bytes(*c) as u32).unwrap_or('\u{FFFD}')).collect();
out.push(s);
}
Some(out)
}