mirror of
https://github.com/storytold/wordcraft.git
synced 2026-10-29 12:57:09 -06:00
df9c29f6733d632b3c8cd024315a2febb2fbdd36
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
29320804f2 |
RTF import: Mac OS Roman and DOS code pages (\mac, \pc, \pca) (#1653) (#1661)
* RTF import: Mac OS Roman text (`\mac`, `\fcharset77`) (#1653) A Macintosh RTF document (`{\rtf1\mac …}`) had its `\'xx` bytes read as Windows-1252: LibreOffice's fdo85889-mac.rtf showed "ñòó" instead of "ÒÚÛ". Mac OS Roman (code page 10000, with the euro sign at 0xDB) joins the single-byte code pages, and the reader uses it for `\mac` documents, fonts with the Mac charset (`\fcharset77`) and `\cpg` / `\ansicpg` 10000. Tests: the code page table (accents, euro, quotes) and two RTF cases, the issue's file and a Mac-charset font in a Windows document. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * RTF import: DOS code pages (`\pc`, `\pca`, OEM `\cpg`) (#1653) With the OEM tables from #1146 now on main, the RTF reader decodes DOS text too: `\pc` documents as code page 437, `\pca` as 850, a font's `\cpg` 437/737/850/852/857/866 and `\fcharset254` (PC 437). `encoding::code_page_byte` decodes a byte of document text in any code page RTF names (Windows, OEM or Mac OS Roman; ASCII below 0x80 stays ASCII), and `is_byte_code_page` says which those are. `SINGLE_BYTE_CODE_PAGES` stays Windows-only, so `text.typeCode`'s `ansi` pages are unchanged. A `\cpg` / `\ansicpg` number out of the u16 range is now ignored; before, `pv as u16` truncated it to another code page (65973 read as 437). Tests: document text in DOS, Mac and Windows pages; `\pc`, `\pca` with a `\cpg437` font, and an out-of-range `\ansicpg`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
086d3d0384 |
Alt+numpad character codes (#961) (#1146)
* #961: Alt+numpad decimal character codes (ANSI, OEM 437/850, Unicode) Hold Alt, type a decimal code, release Alt: the character is typed at the caret through the new text.typeCode command. A leading 0 means Windows-1252, no leading 0 the OEM code page (437, or 850 for Western European locales), above 255 the Unicode character. Not on macOS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * #961: Alt codes follow the system code pages; ignore auto-repeat, drop on blur text.typeCode takes optional ansi (874, 1250-1258) and oem (437, 737, 850, 852, 857, 866) code pages; OEM 852/866/737/857 tables added from the public mapping files. On Windows the shell passes the system locale's pages and, for codes up to 255, lets the character Windows composes replace ours (above 255 ours stays and Windows' byte-reduced text isn't typed). Auto-repeat of a held digit adds nothing, and losing focus drops a half-typed code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Dmitrii Khlapov <19991457+dkhlapov@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
e989fc2b3d |
Open RTF in its code page, and Word/RTF files whose extension is wrong (#896)
RTF import read every 8-bit byte as Windows-1252, so Cyrillic (and any other non-Western) RTF came out as mojibake. Bytes now decode in the current font's code page (\fcharset, \cpg), else the default font's, else the document's (\ansicpg), for the single-byte Windows code pages 874 and 1250-1258. A .doc, .docx or .rtf whose content is another of these formats now opens by its signature, as Word does: RTF saved as .doc (hh.ru résumés and accounting exports are commonly saved that way), a .docx renamed .doc, a Word 97-2003 file renamed .docx. Co-authored-by: Магда М.Г. <mail@mgmagda.ru> Co-authored-by: Brandon Thomas <echelon@gmail.com> |
||
|
|
663904a2e2 |
Import Word 97-2003 .doc files (#28) (#145)
* D0+D1: open Word 97-2003 .doc — container, FIB, piece table text (#28) New wordcraft-docbin crate (L2) reads the OLE2 container via the cfb crate (MIT) and parses the FIB and piece table from the [MS-DOC] specification: text pieces decode as UTF-16LE or Windows-1252 (table moved to wordcraft-doc::encoding), subdocument CP ranges, and Word's special characters map to the model's (tabs, breaks, hyphens). .doc/.dot registered in OPEN_EXTS and the I/O dispatch. Encrypted and Word 6/95 files give clear errors; every length is checked and proptest truncation/flips never panic. * D2: .doc formatting — stylesheet, fonts, CHPX/PAPX FKPs, sprms (#28) Sprm decoder with operand-size rules (incl. sprmTDefTable/sprmPChgTabs exceptions and the Prm0 isprm table), bin tables + ChpxFkp/PapxFkp page lookup, STSH styles (StdfBase layout validated against real Word files: bchUpe == cbStd, sti 0 = Normal) and the SttbfFfn font table in both extended and self-delimiting flavours. The CP walk now splits runs on formatting changes and resolves paragraph styles: bold/italic/strike/ underline/caps/hidden, colours (Ico palette + COLORREF), half-point sizes, super/subscript, fonts, highlights; alignment, indents, spacing, keep options, outline levels. Unknown sprms are skipped leniently; an env-gated corpus test opens real files (styles + text asserted). * D3: .doc structure — sections, headers/footers, tables, lists (#28) - Sections: PlcfSed/Sepx with 16-bit measurement sprms (sprmSXaPage & co), explicit sprmSBOrientation with a width/height fallback; even/odd/first header and footer stories from Plcfhdd become document parts. - Tables: sprmTDefTable grids (operand length fixed to cb+1 per the spec), horizontal merges (sprmTMerge), vertical merges (sprmTVertMerge), cell shading (sprmTSetShd / sprmTDefTableShd80), cell widths, valign, row height/header/cant-split, and table borders (sprmTTableBorders80 Brc80s). - Lists: PlfLst (LSTF + appended LVLs) and PlfLfo (LFO + LFOLVL overrides) resolved into the numbering model; sprmPIlvl/sprmPIlfo mark paragraphs. - Combined fixture (intro + table + list + 2 sections + header) round-trips through Document JSON; hostile proptests now mutate the structured fixture. * D4: .doc notes, fields, bookmarks and inline pictures (#28) - Footnotes/endnotes: PlcffndRef/PlcfEndRef reference CPs paired with the PlcffndTxt/PlcfEndTxt story ranges; each story walks into a Footnote or Endnote part and the 0x02 anchor becomes a NoteRef object. - Fields: 0x13/0x14/0x15 state machine in the walker — instruction text is hidden, the cached result kept; HYPERLINK fields keep their result as linked runs, everything else becomes one Field object. - Bookmarks: SttbfBkmk names + Plcfbkf/Plcfbkl CPs become BookmarkStart/ BookmarkEnd objects at their anchors. - Pictures: sprmCPicLocation points into the Data stream; the PICF header gives the display size (dxaGoal/dyaGoal scaled by mx/my) and the payload is format-sniffed (PNG/JPEG/GIF/BMP; metafiles are dropped with a warn). - Hostile proptests now truncate/flip a fixture with all of the above. * D5: .doc import hardening, caps and roadmap (#28) - Cap inline pictures at 5 000 per document. - Engine test: .doc/.dot bytes dispatch through the Word 97-2003 reader (garbage must error, never panic or open empty). - ROADMAP: Word 97-2003 import row (60% — text, formatting, styles, sections, headers/footers, tables, lists, notes, fields, bookmarks, raster pictures). Real-corpus validation stays opt-in: WORDCRAFT_DOCBIN_FIXTURES=<dir> cargo test -p wordcraft-docbin corpus_files_open. * Corpus fixes: compressed-piece FC space, PAPX query point, none-level labels (#28) Validated against two real files (kept locally under plan/word/fixtures, gitignored): a 1 MB Lorem-ipsum sample and a 2003 styles/lists/table sample. - FKP bin tables and pages address the WordDocument stream in true byte offsets; a compressed piece stores its base fc doubled (byte = fc/2), so fc_of_cp now unwinds the doubling. Before, every PAPX/CHPX lookup in compressed (cp1252) files landed on the wrong entry — tables collapsed to one row, lists and heading styles were lost. - The paragraph-properties query now uses the paragraph mark's own FC (inside the paragraph) instead of the FC after it, which picked up the next paragraph's FKP entry. - Fixture papx entries re-keyed to paragraph-start CPs, the real rgfc convention. - Counters::next_label returns nothing for NumFormat::None levels (Word's numFmt none shows no label; also fixes a latent docx-reader case). - Symbol/Wingdings bullet chars map to Unicode glyphs (• ▪ ○ ●). With this, both corpus files render headings, styles, nested and numbered lists, the tables and the PAGE field correctly; metafile-wrapped images and OLE objects stay dropped by policy (logged). * docbin: bound the text walk by real text; dedupe and cap pictures (#28) Hostile CP numbers no longer drive the work: - header/footnote/endnote story ranges from the Table stream are clamped to their subdocument (ccpHdd/ccpFtn/ccpEdn) with checked base additions, and every walk stops at the last piece's CP; a story spanning [0, 4e9) used to push billions of U+FFFD; - characters are found by binary search over the pieces (PieceTable::char_at), and an unreadable stretch (gap, truncated or out-of-stream piece) is skipped in one step rather than one U+FFFD per CP; the old per-character scan was O(chars x pieces); - the piece CPs are made non-decreasing at parse, so inverted pieces are empty and binary search stays sound; surrogate pairs now decode; - note references are looked up by binary search; the style's CharProps is borrowed, not cloned, per character; - pictures are interned by fc_pic (5000 anchors on one picture share one media entry and one copy) and the bytes copied per document are capped at 256 MB. Each fix has a crafted-bytes regression test; against the previous code the story, piece-claim, last-piece and many-pieces tests hang (killed at 60 s) and the shared picture test copies the picture once per anchor (80 GB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docbin: checked offset arithmetic in lists, sections and table rows (#28) - list.rs: PlfLst/PlfLfo/LVL offsets (at + 2 + cLst*28, at + 4 + lfoMac*16, off += len, ...) saturate, which matters on 32-bit targets (wasm32) where fc + n can overflow usize; the LFO override path no longer indexes abstracts directly. - sections.rs: the Sepx range (fc_sepx + 2 + cb) is checked; Plcfhdd is read only as far as the stories a capped section count can use (it used to reserve lcb/4 entries). - table.rs: sprmTDyaRowHeight = i16::MIN negated in i16 (overflow panic in debug builds); now -(v as f32). Regression tests: the row height test failed before (overflow panic with debug assertions, a 0 pt height without); the list test pins the saturating offsets (it only overflowed on 32-bit targets). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Format the merged OPEN_EXTS list Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Brandon Thomas <echelon@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |