Files
wordcraft/crates/docbin/src/fkp.rs
T
663904a2e2 Import Word 97-2003 .doc files (#28) (#145)
* D0+D1: open Word 97-2003 .doc — container, FIB, piece table text (#28)

New wordcraft-docbin crate (L2) reads the OLE2 container via the cfb
crate (MIT) and parses the FIB and piece table from the [MS-DOC]
specification: text pieces decode as UTF-16LE or Windows-1252 (table
moved to wordcraft-doc::encoding), subdocument CP ranges, and Word's
special characters map to the model's (tabs, breaks, hyphens). .doc/.dot
registered in OPEN_EXTS and the I/O dispatch. Encrypted and Word 6/95
files give clear errors; every length is checked and proptest
truncation/flips never panic.

* D2: .doc formatting — stylesheet, fonts, CHPX/PAPX FKPs, sprms (#28)

Sprm decoder with operand-size rules (incl. sprmTDefTable/sprmPChgTabs
exceptions and the Prm0 isprm table), bin tables + ChpxFkp/PapxFkp page
lookup, STSH styles (StdfBase layout validated against real Word files:
bchUpe == cbStd, sti 0 = Normal) and the SttbfFfn font table in both
extended and self-delimiting flavours. The CP walk now splits runs on
formatting changes and resolves paragraph styles: bold/italic/strike/
underline/caps/hidden, colours (Ico palette + COLORREF), half-point
sizes, super/subscript, fonts, highlights; alignment, indents, spacing,
keep options, outline levels. Unknown sprms are skipped leniently;
an env-gated corpus test opens real files (styles + text asserted).

* D3: .doc structure — sections, headers/footers, tables, lists (#28)

- Sections: PlcfSed/Sepx with 16-bit measurement sprms (sprmSXaPage & co),
  explicit sprmSBOrientation with a width/height fallback; even/odd/first
  header and footer stories from Plcfhdd become document parts.
- Tables: sprmTDefTable grids (operand length fixed to cb+1 per the spec),
  horizontal merges (sprmTMerge), vertical merges (sprmTVertMerge), cell
  shading (sprmTSetShd / sprmTDefTableShd80), cell widths, valign, row
  height/header/cant-split, and table borders (sprmTTableBorders80 Brc80s).
- Lists: PlfLst (LSTF + appended LVLs) and PlfLfo (LFO + LFOLVL overrides)
  resolved into the numbering model; sprmPIlvl/sprmPIlfo mark paragraphs.
- Combined fixture (intro + table + list + 2 sections + header) round-trips
  through Document JSON; hostile proptests now mutate the structured fixture.

* D4: .doc notes, fields, bookmarks and inline pictures (#28)

- Footnotes/endnotes: PlcffndRef/PlcfEndRef reference CPs paired with the
  PlcffndTxt/PlcfEndTxt story ranges; each story walks into a Footnote or
  Endnote part and the 0x02 anchor becomes a NoteRef object.
- Fields: 0x13/0x14/0x15 state machine in the walker — instruction text is
  hidden, the cached result kept; HYPERLINK fields keep their result as
  linked runs, everything else becomes one Field object.
- Bookmarks: SttbfBkmk names + Plcfbkf/Plcfbkl CPs become BookmarkStart/
  BookmarkEnd objects at their anchors.
- Pictures: sprmCPicLocation points into the Data stream; the PICF header
  gives the display size (dxaGoal/dyaGoal scaled by mx/my) and the payload
  is format-sniffed (PNG/JPEG/GIF/BMP; metafiles are dropped with a warn).
- Hostile proptests now truncate/flip a fixture with all of the above.

* D5: .doc import hardening, caps and roadmap (#28)

- Cap inline pictures at 5 000 per document.
- Engine test: .doc/.dot bytes dispatch through the Word 97-2003 reader
  (garbage must error, never panic or open empty).
- ROADMAP: Word 97-2003 import row (60% — text, formatting, styles,
  sections, headers/footers, tables, lists, notes, fields, bookmarks,
  raster pictures).

Real-corpus validation stays opt-in: WORDCRAFT_DOCBIN_FIXTURES=<dir>
cargo test -p wordcraft-docbin corpus_files_open.

* Corpus fixes: compressed-piece FC space, PAPX query point, none-level labels (#28)

Validated against two real files (kept locally under plan/word/fixtures,
gitignored): a 1 MB Lorem-ipsum sample and a 2003 styles/lists/table
sample.

- FKP bin tables and pages address the WordDocument stream in true byte
  offsets; a compressed piece stores its base fc doubled (byte = fc/2),
  so fc_of_cp now unwinds the doubling. Before, every PAPX/CHPX lookup in
  compressed (cp1252) files landed on the wrong entry — tables collapsed
  to one row, lists and heading styles were lost.
- The paragraph-properties query now uses the paragraph mark's own FC
  (inside the paragraph) instead of the FC after it, which picked up the
  next paragraph's FKP entry.
- Fixture papx entries re-keyed to paragraph-start CPs, the real rgfc
  convention.
- Counters::next_label returns nothing for NumFormat::None levels (Word's
  numFmt none shows no label; also fixes a latent docx-reader case).
- Symbol/Wingdings bullet chars map to Unicode glyphs (• ▪ ○ ●).

With this, both corpus files render headings, styles, nested and numbered
lists, the tables and the PAGE field correctly; metafile-wrapped images
and OLE objects stay dropped by policy (logged).

* docbin: bound the text walk by real text; dedupe and cap pictures (#28)

Hostile CP numbers no longer drive the work:
- header/footnote/endnote story ranges from the Table stream are clamped to their
  subdocument (ccpHdd/ccpFtn/ccpEdn) with checked base additions, and every walk stops at
  the last piece's CP; a story spanning [0, 4e9) used to push billions of U+FFFD;
- characters are found by binary search over the pieces (PieceTable::char_at), and an
  unreadable stretch (gap, truncated or out-of-stream piece) is skipped in one step rather
  than one U+FFFD per CP; the old per-character scan was O(chars x pieces);
- the piece CPs are made non-decreasing at parse, so inverted pieces are empty and binary
  search stays sound; surrogate pairs now decode;
- note references are looked up by binary search; the style's CharProps is borrowed, not
  cloned, per character;
- pictures are interned by fc_pic (5000 anchors on one picture share one media entry and
  one copy) and the bytes copied per document are capped at 256 MB.

Each fix has a crafted-bytes regression test; against the previous code the story,
piece-claim, last-piece and many-pieces tests hang (killed at 60 s) and the shared
picture test copies the picture once per anchor (80 GB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docbin: checked offset arithmetic in lists, sections and table rows (#28)

- list.rs: PlfLst/PlfLfo/LVL offsets (at + 2 + cLst*28, at + 4 + lfoMac*16, off += len,
  ...) saturate, which matters on 32-bit targets (wasm32) where fc + n can overflow usize;
  the LFO override path no longer indexes abstracts directly.
- sections.rs: the Sepx range (fc_sepx + 2 + cb) is checked; Plcfhdd is read only as far
  as the stories a capped section count can use (it used to reserve lcb/4 entries).
- table.rs: sprmTDyaRowHeight = i16::MIN negated in i16 (overflow panic in debug builds);
  now -(v as f32).

Regression tests: the row height test failed before (overflow panic with debug assertions, a 0 pt height without); the list test pins the saturating
offsets (it only overflowed on 32-bit targets).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Format the merged OPEN_EXTS list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:36:38 -04:00

109 lines
5.3 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Character and paragraph formatting lookup: the bin tables (`PlcBteChpx`/`PlcBtePapx`) in
//! the Table stream point at 512-byte FKP pages in the WordDocument stream, which map FC
//! ranges to CHPX (grpprls) and PAPX (istd + grpprl) ([MS-DOC] §2.4.6.1/2.4.6.2).
use crate::DocbinError;
/// Largest bin table (entries) we accept.
const MAX_BINS: usize = 100_000;
/// A parsed `PlcBte*`: FC breakpoints and the FKP page each range lives in.
pub(crate) struct Bins {
fcs: Vec<u32>,
/// Page number (× 512 = byte offset in the WordDocument stream).
pns: Vec<u32>,
}
impl Bins {
/// Parse the Plc at `table[fc..fc+lcb]` (aFc[n+1] u32s + aPn[n] u32s).
pub(crate) fn parse(table: &[u8], fc: u32, lcb: u32) -> Result<Bins, DocbinError> {
let fc = fc as usize;
let lcb = lcb as usize;
if lcb < 4 || !(lcb - 4).is_multiple_of(8) {
return Ok(Bins { fcs: Vec::new(), pns: Vec::new() });
}
let end = fc.checked_add(lcb).ok_or_else(|| DocbinError::Malformed("bin table overflow".into()))?;
let plc = table.get(fc..end).ok_or_else(|| DocbinError::Malformed("bin table outside the Table stream".into()))?;
let n = (lcb - 4) / 8;
if n > MAX_BINS {
return Err(DocbinError::Limit(format!("bin table has {n} entries")));
}
let at = |i: usize| -> Option<u32> { plc.get(i * 4..i * 4 + 4).map(|b| u32::from_le_bytes([b[0], b[1], b[2], b[3]])) };
let mut fcs = Vec::with_capacity(n + 1);
for i in 0..=n {
fcs.push(at(i).ok_or_else(|| DocbinError::Malformed("truncated bin aFc".into()))?);
}
let mut pns = Vec::with_capacity(n);
for i in 0..n {
// PnBte* keeps the page number in its low bits; the rest must be zero.
pns.push(at(n + 1 + i).ok_or_else(|| DocbinError::Malformed("truncated bin aPn".into()))? & 0x003F_FFFF);
}
Ok(Bins { fcs, pns })
}
/// The page containing `fc`, as a slice of the WordDocument stream.
fn page<'w>(&self, word: &'w [u8], fc: u32) -> Option<&'w [u8]> {
let i = self.fcs.partition_point(|&f| f <= fc).checked_sub(1)?;
let pn = *self.pns.get(i)?;
let at = pn.checked_mul(512)? as usize;
word.get(at..at.checked_add(512)?)
}
/// The grpprl of the Chpx covering `fc` ([MS-DOC] ChpxFkp: rgfc u32s, rgb offsets ×2,
/// crun in the last byte).
pub(crate) fn chpx<'w>(&self, word: &'w [u8], fc: u32) -> &'w [u8] {
let Some(page) = self.page(word, fc) else { return &[] };
let Some(&crun) = page.get(511) else { return &[] };
let crun = crun.min(0x65) as usize;
let rgfc = |i: usize| -> Option<u32> { page.get(i * 4..i * 4 + 4).map(|b| u32::from_le_bytes([b[0], b[1], b[2], b[3]])) };
// rgb array sits right after rgfc (crun+1 u32s); each byte is a half-offset into the
// region between rgb's end and crun.
let rgb_at = (crun + 1) * 4;
let j = (0..crun).rev().find(|&j| rgfc(j).is_some_and(|f| f <= fc));
let Some(j) = j else { return &[] };
let Some(&off) = page.get(rgb_at + j) else { return &[] };
if off == 0 {
return &[];
}
let at = off as usize * 2;
let Some(&cb) = page.get(at) else { return &[] };
page.get(at + 1..at + 1 + cb as usize).unwrap_or(&[])
}
/// The (istd, grpprl) of the Papx of the paragraph whose mark is just before `fc`
/// ([MS-DOC] PapxFkp: rgfc u32s — each the FC *following* a paragraph mark —, rgbx BX
/// entries (1 offset byte + 12 PHE bytes), crun in the last byte; the PAPX itself is a
/// PapxInFkp with the cb/cb' size quirk). Query with the FC of the character after the mark.
pub(crate) fn papx<'w>(&self, word: &'w [u8], fc: u32) -> (u16, &'w [u8]) {
let Some(page) = self.page(word, fc) else { return (0, &[]) };
let Some(&crun) = page.get(511) else { return (0, &[]) };
// BX is 13 bytes; cap crun at what fits before byte 511.
let max_run = (511usize.saturating_sub((crun as usize + 1) * 4)) / 13;
let crun = (crun as usize).min(max_run);
let rgfc = |i: usize| -> Option<u32> { page.get(i * 4..i * 4 + 4).map(|b| u32::from_le_bytes([b[0], b[1], b[2], b[3]])) };
let rgbx_at = (crun + 1) * 4;
let j = (0..crun).rev().find(|&j| rgfc(j).is_some_and(|f| f <= fc));
let Some(j) = j else { return (0, &[]) };
let Some(&off) = page.get(rgbx_at + j * 13) else { return (0, &[]) };
if off == 0 {
return (0, &[]);
}
let at = off as usize * 2;
let u16at = |o: usize| -> Option<u16> { page.get(o..o + 2).map(|b| u16::from_le_bytes([b[0], b[1]])) };
let Some(cb) = page.get(at).copied() else { return (0, &[]) };
// PapxInFkp: if cb is nonzero the GrpPrlAndIstd (istd u16 + grpprl) is 2×cb-1 bytes;
// if cb is zero, the next byte cb' (1 byte) gives 2×cb' bytes of it.
let (data_at, data_len) = if cb != 0 {
(at + 1, 2 * cb as usize - 1)
} else {
let cb2 = page.get(at + 1).copied().unwrap_or(0) as usize;
(at + 2, 2 * cb2)
};
if data_len < 2 {
return (0, &[]);
}
let istd = u16at(data_at).unwrap_or(0);
(istd, page.get(data_at + 2..data_at + data_len).unwrap_or(&[]))
}
}