Jan Hallvard LarsenandClaude Opus 5.5 c63ca54227 Import: RTF and Tagged Text keep characters past U+FFFF ...

Placing an RTF or Tagged Text file dropped every character outside the
Basic Multilingual Plane: emoji, mathematical letters and the rarer CJK
ideographs such as 𠮷 (U+20BB7, used in Japanese names). The text
around them came in, so the loss was silent. It also broke our own round
trip: Export Text writes these characters correctly and Place lost them.

Both formats carry such a character as a UTF-16 surrogate pair, two
units in a row: \u-10179?\u-8704? in RTF, <0xD83D><0xDE00> in Tagged
Text. Both readers passed each unit to char::from_u32 on its own. Half a
pair is not a character, so each half was dropped.

A small shared Utf16Units joins the units: a first half waits for the
second, and the pair becomes one character. Half a pair with no partner
is still left out, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-08 16:33:44 +01:00

..

src

Import: RTF and Tagged Text keep characters past U+FFFF

2026-10-08 16:33:44 +01:00

Cargo.toml

H: place text files — Word (.docx), RTF, plain text; autoflow

2026-10-02 08:55:28 -04:00