mirror of
https://github.com/storytold/vectorcraft.git
synced 2026-11-01 13:47:21 -06:00
Place: Word .doc text with a broken surrogate pair is refused, and fields without a result warn
A 16-bit piece holding half of a UTF-16 surrogate pair was placed with U+FFFD in its place; the document is now refused. A pair split between two pieces is decoded as one character. A field with no result (no separator, never ended, or nested past 64) is left out with its own warning, and the warning about fields placed as their results is given only for fields that have one.
This commit is contained in:
dotCooCoo
@@ -10,8 +10,9 @@
|
||||
//! and ones last saved with fast save (`fComplex`) are refused.
|
||||
//! - Text: the main document's `ccpText` characters, from the pieces the Clx's PlcPcd lists
|
||||
//! (§2.9.38, §2.8.35) in their order, each in 16-bit Unicode or in compressed 8-bit text with the
|
||||
//! mapping §2.9.73 gives. A paragraph mark (U+000D) ends a paragraph, and so does a page or
|
||||
//! section break (U+000C) at the end of a paragraph's properties (PapxFkp); a line break
|
||||
//! mapping §2.9.73 gives; 16-bit text with half of a surrogate pair is refused. A paragraph mark
|
||||
//! (U+000D) ends a paragraph, and so does a page or section break (U+000C) at the end of a
|
||||
//! paragraph's properties (PapxFkp); a line break
|
||||
//! (U+000B) starts a new paragraph in the same formatting. A table's cells (cell marks, U+0007)
|
||||
//! are separated by tabs and each row (a table terminating mark) is a paragraph. Tabs are tabs,
|
||||
//! and U+001E and U+001F are the nonbreaking and optional hyphens. Of a field (U+0013, U+0014 and
|
||||
@@ -34,11 +35,11 @@
|
||||
//! Hidden text, deleted revisions, field instructions, pictures and embedded objects, note and
|
||||
//! comment references and the other stories (footnotes, endnotes, headers, footers, comments and
|
||||
//! text boxes) are left out. The reply's warnings name each kind of content that was left out or
|
||||
//! changed, once: pictures and objects, fields, the other stories, tracked changes and tables. A
|
||||
//! stream larger than [`MAX_STREAM`] bytes, a main text longer than [`text::MAX_TEXT`] characters,
|
||||
//! more than [`MAX_PIECES`] pieces or [`MAX_PAGES`] property pages, and placed type of more than
|
||||
//! [`MAX_PARAGRAPHS`] paragraphs or [`MAX_RUNS`] runs are refused, and so is a structure that lies
|
||||
//! outside its stream.
|
||||
//! changed, once: pictures and objects, fields, fields left out (no result, or nested too deep), the
|
||||
//! other stories, tracked changes and tables. A stream larger than [`MAX_STREAM`] bytes, a main
|
||||
//! text longer than [`text::MAX_TEXT`] characters, more than [`MAX_PIECES`] pieces or
|
||||
//! [`MAX_PAGES`] property pages, and placed type of more than [`MAX_PARAGRAPHS`] paragraphs or
|
||||
//! [`MAX_RUNS`] runs are refused, and so is a structure that lies outside its stream.
|
||||
|
||||
use std::io::{Cursor, Read};
|
||||
|
||||
@@ -287,27 +288,31 @@ fn main_text(wd: &[u8], pieces: &[Piece], ccp: u32, cmd: &str) -> Result<Vec<Cha
|
||||
return Err(damaged(cmd, "its piece table doesn't cover its text"));
|
||||
}
|
||||
let outside = || damaged(cmd, "a piece of its text lies outside its WordDocument stream");
|
||||
let mut out = Vec::with_capacity(ccp as usize);
|
||||
// Each character position's UTF-16 code unit, file position and byte count, in character
|
||||
// order; a surrogate pair may be split between two pieces.
|
||||
let mut units: Vec<(u16, u32, u8)> = Vec::with_capacity(ccp as usize);
|
||||
for p in pieces.iter().take_while(|p| p.cp < ccp) {
|
||||
let n = (p.end.min(ccp) - p.cp) as usize;
|
||||
if p.compressed {
|
||||
let b = bytes_at(wd, p.fc as usize, n).ok_or_else(outside)?;
|
||||
out.extend(b.iter().zip(p.fc..).map(|(&c, fc)| (compressed(c), fc, 1)));
|
||||
// Every character compressed text maps to is in the Basic Multilingual Plane.
|
||||
units.extend(b.iter().zip(p.fc..).map(|(&c, fc)| (u16::try_from(u32::from(compressed(c))).unwrap_or(0xFFFD), fc, 1)));
|
||||
continue;
|
||||
}
|
||||
let b = bytes_at(wd, p.fc as usize, n.checked_mul(2).ok_or_else(outside)?).ok_or_else(outside)?;
|
||||
let units: Vec<(u16, u32)> = b.as_chunks::<2>().0.iter().zip((p.fc..).step_by(2)).map(|(&u, fc)| (u16::from_le_bytes(u), fc)).collect();
|
||||
let mut i = 0;
|
||||
while let Some(&(u, fc)) = units.get(i) {
|
||||
let pair = units.get(i + 1).map(|&(low, _)| low).filter(|low| (0xD800..0xDC00).contains(&u) && (0xDC00..0xE000).contains(low));
|
||||
let c = match pair {
|
||||
Some(low) => char::decode_utf16([u, low]).next().and_then(|c| c.ok()),
|
||||
None => char::from_u32(u32::from(u)),
|
||||
};
|
||||
let width = if pair.is_some() { 4 } else { 2 };
|
||||
out.push((c.unwrap_or(char::REPLACEMENT_CHARACTER), fc, width));
|
||||
i += usize::from(width / 2);
|
||||
}
|
||||
units.extend(b.as_chunks::<2>().0.iter().zip((p.fc..).step_by(2)).map(|(&u, fc)| (u16::from_le_bytes(u), fc, 2)));
|
||||
}
|
||||
let mut out = Vec::with_capacity(units.len());
|
||||
let mut i = 0;
|
||||
while let Some(&(u, fc, width)) = units.get(i) {
|
||||
let low = units.get(i + 1).map(|&(low, _, w)| (low, w)).filter(|(low, _)| (0xD800..0xDC00).contains(&u) && (0xDC00..0xE000).contains(low));
|
||||
let c = match low {
|
||||
Some((low, _)) => char::decode_utf16([u, low]).next().and_then(|c| c.ok()),
|
||||
None => char::from_u32(u32::from(u)),
|
||||
};
|
||||
let c = c.ok_or_else(|| damaged(cmd, "its text holds half of a UTF-16 surrogate pair"))?;
|
||||
out.push((c, fc, low.map_or(width, |(_, w)| width + w)));
|
||||
i += if low.is_some() { 2 } else { 1 };
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
@@ -802,7 +807,10 @@ impl Para {
|
||||
#[derive(Clone, Copy, Default)]
|
||||
struct Seen {
|
||||
pictures: bool,
|
||||
/// Fields with a result.
|
||||
fields: bool,
|
||||
/// Fields with no result, or nested past [`MAX_FIELDS`].
|
||||
unread_fields: bool,
|
||||
revisions: bool,
|
||||
tables: bool,
|
||||
}
|
||||
@@ -931,10 +939,10 @@ impl<'a> Reader<'a> {
|
||||
}
|
||||
match c {
|
||||
'\u{13}' => {
|
||||
self.seen.fields = true;
|
||||
if self.fields.len() < MAX_FIELDS && self.field_overflow == 0 {
|
||||
self.fields.push(true);
|
||||
} else {
|
||||
self.seen.unread_fields = true;
|
||||
self.field_overflow += 1;
|
||||
}
|
||||
return;
|
||||
@@ -942,14 +950,16 @@ impl<'a> Reader<'a> {
|
||||
'\u{14}' if self.field_overflow == 0 => {
|
||||
if let Some(instructions) = self.fields.last_mut() {
|
||||
*instructions = false;
|
||||
self.seen.fields = true;
|
||||
}
|
||||
return;
|
||||
}
|
||||
'\u{15}' => {
|
||||
if self.field_overflow > 0 {
|
||||
self.field_overflow -= 1;
|
||||
} else {
|
||||
self.fields.pop();
|
||||
} else if self.fields.pop() == Some(true) {
|
||||
// A field that ends without a separator has no result.
|
||||
self.seen.unread_fields = true;
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1052,6 +1062,10 @@ impl<'a> Reader<'a> {
|
||||
if self.seen.fields {
|
||||
out.push("the Word document's fields were placed as the text of their results".to_string());
|
||||
}
|
||||
// A field still in its instructions when the text ends has no result either.
|
||||
if self.seen.unread_fields || self.fields.contains(&true) {
|
||||
out.push("the Word document's fields without a result that could be read were left out".to_string());
|
||||
}
|
||||
if self.seen.revisions {
|
||||
out.push("the Word document's tracked changes were accepted: deleted text was left out and inserted text kept".to_string());
|
||||
}
|
||||
|
||||
@@ -547,7 +547,14 @@ impl Word97 {
|
||||
c => u8::try_from(u32::from(c)).unwrap(),
|
||||
}));
|
||||
} else {
|
||||
wd.extend(text.encode_utf16().flat_map(u16::to_le_bytes));
|
||||
// U+FFFF and U+FFFE stand for the high surrogate 0xD800 and the low surrogate 0xDE00
|
||||
// on their own.
|
||||
let unit = |u| match u {
|
||||
0xFFFF => 0xD800,
|
||||
0xFFFE => 0xDE00,
|
||||
u => u,
|
||||
};
|
||||
wd.extend(text.encode_utf16().map(unit).flat_map(u16::to_le_bytes));
|
||||
}
|
||||
}
|
||||
// Each character's file position and size, in character order.
|
||||
@@ -814,9 +821,23 @@ fn malformed_doc_fails_cleanly() {
|
||||
assert!(e.contains("piece table is damaged"), "{e}");
|
||||
let e = place(&mut s, "a.doc", &Word97 { pieces: vec![("\r\r", true)], ..Word97::default() }.build()).unwrap_err();
|
||||
assert!(e.contains("no text to place"), "{e}");
|
||||
// 16-bit text with half of a surrogate pair is refused rather than placed with a stand-in.
|
||||
let e = place(&mut s, "a.doc", &Word97 { pieces: vec![("bad \u{FFFF} text\r", false)], ..Word97::default() }.build()).unwrap_err();
|
||||
assert!(e.contains("half of a UTF-16 surrogate pair"), "{e}");
|
||||
// A surrogate pair split between two pieces is one character.
|
||||
let r = place(&mut s, "a.doc", &Word97 { pieces: vec![("x\u{FFFF}", false), ("\u{FFFE}y\r", false)], ..Word97::default() }.build()).unwrap();
|
||||
assert_eq!(placed(&s, &r).0, "x\u{10200}y");
|
||||
// A field that ends without a result is left out, with its own warning.
|
||||
let r = place(&mut s, "a.doc", &Word97 { pieces: vec![("a\u{13} PAGE \u{15}b\r", true)], ..Word97::default() }.build()).unwrap();
|
||||
assert_eq!(placed(&s, &r).0, "ab");
|
||||
assert_eq!(r["warnings"], json!(["the Word document's fields without a result that could be read were left out"]));
|
||||
// So is a field that never ends, with the text after it.
|
||||
let r = place(&mut s, "a.doc", &Word97 { pieces: vec![("a\rb\u{13} PAGE c\r", true)], ..Word97::default() }.build()).unwrap();
|
||||
assert_eq!(placed(&s, &r).0, "a\nb");
|
||||
assert_eq!(r["warnings"], json!(["the Word document's fields without a result that could be read were left out"]));
|
||||
let r = place(&mut s, "a.doc", &words().build()).unwrap();
|
||||
assert_eq!(placed(&s, &r).0, "words");
|
||||
assert_eq!(s.doc().unwrap().history.undo.len(), 2, "failed places leave no undo step");
|
||||
assert_eq!(s.doc().unwrap().history.undo.len(), 5, "failed places leave no undo step");
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
+5
-4
@@ -3838,10 +3838,11 @@ font, the paragraph style and the character style (and the styles each is based
|
||||
text is read; table cells are separated by tabs and each row is a paragraph, and a field places as its result. Hidden
|
||||
and deleted text, field instructions, pictures and embedded objects, headers, footers, notes, comments and text boxes
|
||||
are left out, and `warnings` names each kind of content left out or changed once (pictures and objects, fields,
|
||||
tracked changes, tables, and the other stories the document has). Documents saved by Word 6.0 or Word 95, encrypted or
|
||||
password-protected documents and documents last saved with fast save are refused, and so are a stream larger than
|
||||
64 MiB, a document with more than 4 MiB of text, 100,000 pieces, 16,384 pages of properties or 100,000 paragraphs or
|
||||
runs, and one whose structures lie outside their streams.
|
||||
tracked changes, tables, and the other stories the document has); a field with no result is left out with its own
|
||||
warning. Documents saved by Word 6.0 or Word 95, encrypted or password-protected documents and documents last saved with
|
||||
fast save are refused, and so are a stream larger than 64 MiB, a document with more than 4 MiB of text, 100,000 pieces,
|
||||
16,384 pages of properties or 100,000 paragraphs or runs, one whose 16-bit text holds half of a surrogate pair, and one
|
||||
whose structures lie outside their streams.
|
||||
`file.place` answers `format: "doc"`.
|
||||
|
||||
## PDF presets
|
||||
|
||||
@@ -1093,7 +1093,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
|
||||
- File › Place reads RTF files as area type with their formatting: each run's font, size, bold and italic, underline, strikethrough, caps, superscript or subscript, baseline shift and color, and each paragraph's alignment, indents, space before and after and direction. Table cells are separated by tabs; hidden text, pictures, headers, footers and notes are left out (`file.place`, `crates/engine/src/tests/textimport.rs`).
|
||||
- File › Place reads Word `.docx` documents as area type with the same formatting, taken from the document defaults, the paragraph and character styles (their toggle properties reversing each other) and direct formatting, the theme's fonts included. Table rows are paragraphs with their cells separated by tabs; hidden and deleted text, field instructions, pictures, headers, footers and notes are left out (`file.place`, `crates/engine/src/tests/textimport.rs`).
|
||||
- A Word paragraph justified with a low, medium or high kashida length places justified with Insert Kashidas set to Short, Medium or Long (`file.place`, `crates/engine/src/tests/textimport.rs`).
|
||||
- File › Place reads Word 97-2003 `.doc` documents as area type with the same formatting, read from the main text's pieces, the style sheet's paragraph and character styles and direct formatting. Table rows are paragraphs with their cells separated by tabs and fields place as their results; hidden and deleted text, pictures, headers, footers, notes, comments and text boxes are left out, and `warnings` names each kind left out or changed. Encrypted documents, documents last saved with fast save and Word 6.0 and Word 95 documents are refused (`file.place`, `crates/engine/src/tests/textimport.rs`).
|
||||
- File › Place reads Word 97-2003 `.doc` documents as area type with the same formatting, read from the main text's pieces, the style sheet's paragraph and character styles and direct formatting. Table rows are paragraphs with their cells separated by tabs and fields place as their results; fields without a result, hidden and deleted text, pictures, headers, footers, notes, comments and text boxes are left out, and `warnings` names each kind left out or changed. Encrypted documents, documents last saved with fast save and Word 6.0 and Word 95 documents are refused (`file.place`, `crates/engine/src/tests/textimport.rs`).
|
||||
- On the desktop, File › Place reads and decodes files off the UI thread, one at a time, and places each once it is read, with "Placing" in the status bar and the busy spinner until then: the Place dialog's OK, a file dropped on the canvas, a click or drag of the place cursor, and the OK of the PDF, DXF and text import options. The files and thumbnails the place cursor is loaded with are read the same way, and the Place dialog's lines about its files are read on a thread of their own. Files are placed in the order they were started; a file is not placed while a dialog is open or a drag is under way, and a file whose document was closed meanwhile is not placed. `file.place` and `file.place.queue` over MCP and scripts reply as before.
|
||||
- The pixels of a PNG, JPEG, GIF or WebP file that Place decodes to check them are kept in the image store the renderers share, so the canvas's first frame with the placed image draws them without decoding the file again on the UI thread: File › Place, a file dropped on the canvas, a click or drag of the place cursor, `file.place`, and Paste of such an image. A CMYK JPEG is still decoded when it is first drawn (`crates/ui-egui/src/placing.rs`, `crates/render/src/tests.rs`).
|
||||
- A photo whose EXIF orientation turns or flips it (a camera held upright or upside down) is placed and opened upright, as it was taken; the image is stored as it is seen (#4284).
|
||||
|
||||
Reference in New Issue
Block a user