mirror of
https://github.com/storytold/wordcraft.git
synced 2026-11-01 06:16:58 -06:00
midasdf
2d645dba03
`\ansicpg932`/`936`/`949`/`950`, a font's `\fcharset128`/`129`/`134`/`136`, `\cpg932` and the OEM charset 255 were read one byte at a time, so Japanese, Korean and Chinese text came out as mojibake. The reader now holds the 8-bit text of a multibyte code page and decodes the run with encoding_rs (SHIFT_JIS, GBK, EUC_KR/UHC, BIG5): a lead byte and its trail byte decode together, escaped (`\'hh`) or raw. A code page switch (`\f`, `\fcharset`, `\cpg`, `\ansicpg`), a group end and the end of the document flush what is pending; invalid sequences read as U+FFFD, as Word writes them, and never lose the rest of the text. The pending buffer is capped at 4096 bytes. Single-byte code pages, `\uN?` escapes and their `\ucN` skips, the font table and every existing test are unchanged. Tests cover Shift-JIS (escaped, raw and through a `\fcharset128` font), CP949/CP936/CP950, a mixed Japanese/ASCII/ Cyrillic document whose code-page switch must flush, `\uN?` alongside multibyte text, an invalid sequence and a Windows-1252 regression.