diff --git a/Cargo.lock b/Cargo.lock index 7f9a0d3e5..cd0ff683a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5193,6 +5193,7 @@ version = "0.9.0" name = "wordcraft-doc" version = "0.9.0" dependencies = [ + "base64", "proptest", "serde", "serde_json", diff --git a/ROADMAP.md b/ROADMAP.md index 9cb35f9e9..a8b33d87c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,7 +2,7 @@ **Stage: alpha** · next: beta, ~15 points (60% → 75% ready for real work) and ~150–230 h away -> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-21 · **Change:** trivial (personal templates in File › New and Save as Word Template, #2852; previously trivial: progress log: legacy menu keys, #1697; previously trivial: progress log: shape gradients and transparency, #1500; previously trivial: progress log: Focus fills the screen, and Full Screen View shows the pages alone, #1678; previously trivial: progress log: text above a float's anchor wraps around it, #2048; previously trivial: progress log: File › Recover follows the app theme, #3243; previously trivial: progress log: caret inside ligatures, #1974; previously trivial: text typed into a content control takes the control's formatting; localized placeholder styles recognised; previously trivial: forms protection keeps its content controls: no inserting, Properties or Remove Content Control; previously trivial: long navigation headings wrap onto further lines within the pane, #3015; previously trivial: clearing a no-match font query keeps the recent heading visible, #2068; previously trivial: a word after a slash wraps whole, like Word, #2963; previously trivial: progress log: a header-cell picture watermark sits behind the text, #1775; previously trivial: symbols keep their font when the text font changes, #2036; previously trivial: opening a DOCX no longer copies its pictures; previously trivial: Add Font imports every selected font file, #1789); previously trivial (RTF export writes sections (gap 19); previously trivial: ODT export writes sections (gap 19); previously trivial: Read Aloud speaks each sentence in its own language, #1843; previously trivial: hint-reliant CJK outlines; previously trivial: Ctrl+wheel zoom no longer scrolls the document, #1743; previously trivial: progress log: Turkish case mapping in AutoCorrect, #1923; previously trivial: a mouse drag on a table leaves the view where it is, #1738; previously trivial: progress log: Hebrew field results and smart quotes, #1670; previously trivial: Backspace removes a selected table, rows or columns, #1844; previously trivial: progress log: mixed Hebrew/Latin text keeps its order in Word, #1543; previously trivial: progress log: web typing in non-Latin text no longer repeats, #1480; previously trivial: progress log: font family caches bounded and math font lookups follow font changes, #1354; previously trivial: `-->` AutoCorrect replacement, #2223; previously trivial: numbered headings, #1793; previously trivial: Navigation Find exposes Results, #2096; previously trivial: progress log: Columns on selected text, #2024; previously trivial: progress log: legacy outline, shadow, emboss and imprint text effects drawn, #2534; previously trivial: progress log: Indonesian proofing, #498; previously trivial: progress log: DOCX glossary documents (building blocks); previously trivial: progress log: Read Mode keeps the keyboard and shows whole, unobscured columns, #1772); previously trivial: progress log: font list entries of tall-ascent fonts no longer cut off, #2570); previously trivial: progress log: Split Cells and Convert Text to Table dialogs follow the Word check, #2592; previously trivial: a new source's Tag is made like Word's, three letters of the author and two digits of the year with a number added when taken, and an edit keeps the Tag, #2094; previously trivial: Web Layout fills the window width, #1876; previously trivial: progress log: Different First Page headers reachable from page 1, #2826; previously trivial: Language dialog keeps inherited proofing language and no-proof choices (#2717); previously trivial: progress log: web preferences kept between visits, #2132; previously trivial: progress log: undrawable private-use list bullets show the level's default bullet, #1683; previously trivial: progress log: `w:eastAsianLayout` kept on save; previously trivial: table style editor formats every part, #714); previously trivial: VML freeforms read and kept on save; previously trivial: progress log: consistent preferred cell widths; previously trivial: multi-paragraph field instructions no longer add blank lines; previously trivial: page-break-only list paragraphs hide their labels; previously trivial: progress log: grouped `flowChartOr` shapes draw their circle and crossing spokes; previously trivial: progress log: copies keep their formatting in other programs, #1523; previously trivial: progress log: privacy options on save, #1062; previously trivial: progress log: equation pictures in theme colours on Windows, #1002; previously trivial: symbol fonts, #776; previously trivial: the save prompt answers from the keyboard, #861; previously trivial: table column and row markers on the rulers, #700; previously trivial: table borders and the resize handle drag on the page, #217; previously trivial: formatting revisions landed, #41; previously trivial: Draw tab ink, #307, and Draw Table and Eraser, #303, landed: catalog 402/434; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; restructured to the craftrules progress-docs standard) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4 +> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-21 · **Change:** trivial (progress log: recovery copies and .json files store pictures as base64, #586 follow-up; previously trivial: personal templates in File › New and Save as Word Template, #2852; previously trivial: progress log: legacy menu keys, #1697; previously trivial: progress log: shape gradients and transparency, #1500; previously trivial: progress log: Focus fills the screen, and Full Screen View shows the pages alone, #1678; previously trivial: progress log: text above a float's anchor wraps around it, #2048; previously trivial: progress log: File › Recover follows the app theme, #3243; previously trivial: progress log: caret inside ligatures, #1974; previously trivial: text typed into a content control takes the control's formatting; localized placeholder styles recognised; previously trivial: forms protection keeps its content controls: no inserting, Properties or Remove Content Control; previously trivial: long navigation headings wrap onto further lines within the pane, #3015; previously trivial: clearing a no-match font query keeps the recent heading visible, #2068; previously trivial: a word after a slash wraps whole, like Word, #2963; previously trivial: progress log: a header-cell picture watermark sits behind the text, #1775; previously trivial: symbols keep their font when the text font changes, #2036; previously trivial: opening a DOCX no longer copies its pictures; previously trivial: Add Font imports every selected font file, #1789); previously trivial (RTF export writes sections (gap 19); previously trivial: ODT export writes sections (gap 19); previously trivial: Read Aloud speaks each sentence in its own language, #1843; previously trivial: hint-reliant CJK outlines; previously trivial: Ctrl+wheel zoom no longer scrolls the document, #1743; previously trivial: progress log: Turkish case mapping in AutoCorrect, #1923; previously trivial: a mouse drag on a table leaves the view where it is, #1738; previously trivial: progress log: Hebrew field results and smart quotes, #1670; previously trivial: Backspace removes a selected table, rows or columns, #1844; previously trivial: progress log: mixed Hebrew/Latin text keeps its order in Word, #1543; previously trivial: progress log: web typing in non-Latin text no longer repeats, #1480; previously trivial: progress log: font family caches bounded and math font lookups follow font changes, #1354; previously trivial: `-->` AutoCorrect replacement, #2223; previously trivial: numbered headings, #1793; previously trivial: Navigation Find exposes Results, #2096; previously trivial: progress log: Columns on selected text, #2024; previously trivial: progress log: legacy outline, shadow, emboss and imprint text effects drawn, #2534; previously trivial: progress log: Indonesian proofing, #498; previously trivial: progress log: DOCX glossary documents (building blocks); previously trivial: progress log: Read Mode keeps the keyboard and shows whole, unobscured columns, #1772); previously trivial: progress log: font list entries of tall-ascent fonts no longer cut off, #2570); previously trivial: progress log: Split Cells and Convert Text to Table dialogs follow the Word check, #2592; previously trivial: a new source's Tag is made like Word's, three letters of the author and two digits of the year with a number added when taken, and an edit keeps the Tag, #2094; previously trivial: Web Layout fills the window width, #1876; previously trivial: progress log: Different First Page headers reachable from page 1, #2826; previously trivial: Language dialog keeps inherited proofing language and no-proof choices (#2717); previously trivial: progress log: web preferences kept between visits, #2132; previously trivial: progress log: undrawable private-use list bullets show the level's default bullet, #1683; previously trivial: progress log: `w:eastAsianLayout` kept on save; previously trivial: table style editor formats every part, #714); previously trivial: VML freeforms read and kept on save; previously trivial: progress log: consistent preferred cell widths; previously trivial: multi-paragraph field instructions no longer add blank lines; previously trivial: page-break-only list paragraphs hide their labels; previously trivial: progress log: grouped `flowChartOr` shapes draw their circle and crossing spokes; previously trivial: progress log: copies keep their formatting in other programs, #1523; previously trivial: progress log: privacy options on save, #1062; previously trivial: progress log: equation pictures in theme colours on Windows, #1002; previously trivial: symbol fonts, #776; previously trivial: the save prompt answers from the keyboard, #861; previously trivial: table column and row markers on the rulers, #700; previously trivial: table borders and the resize handle drag on the page, #217; previously trivial: formatting revisions landed, #41; previously trivial: Draw tab ink, #307, and Draw Table and Eraser, #303, landed: catalog 402/434; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; restructured to the craftrules progress-docs standard) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4 > **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-18 · **Change:** trivial (English dialects in proofing, #1281; previously trivial: triple borders draw three lines, #1740; previously trivial: progress log: double-clicking a later page's footer no longer scrolls up, #2103; previously trivial: long navigation headings wrap onto further lines within the pane, #3015; previously trivial: clearing a no-match font query keeps the recent heading visible, #2068; previously trivial: a word after a slash wraps whole, like Word, #2963; previously trivial: RTF export writes sections (gap 19); previously trivial: ODT export writes sections (gap 19); previously trivial: progress log: pasted blockquotes are indented paragraphs, #1270; previously trivial: progress log: print-area strings translated in every catalog; previously trivial: progress log: comments readable in the dark theme, #2045; previously trivial: progress log: ODT sections, #1379; previously trivial: progress log: table deletes, Change Case and Unlink Fields respect Track Changes and locked content controls, #1329; previously trivial: progress log: rounded rectangle corners, #1476; previously trivial: progress log: Save As type list, #1399; previously trivial: progress log: privacy options on save, #1062; previously trivial: progress log: equation pictures in theme colours on Windows, #1002; previously trivial: symbol fonts, #776; previously trivial: the save prompt answers from the keyboard, #861; previously trivial: table column and row markers on the rulers, #700; previously trivial: table borders and the resize handle drag on the page, #217; previously trivial: formatting revisions landed, #41; previously trivial: Draw tab ink, #307, and Draw Table and Eraser, #303, landed: catalog 402/434; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; restructured to the craftrules progress-docs standard) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4 > **Printing update:** 2026-10-12 · direct desktop printing through craft-print; details below. @@ -242,6 +242,7 @@ Imported Word 97–2003 `.doc`/`.dot` files now ask for a supported DOCX target | 2026-10-16 | Text with the legacy Outline, Shadow, Emboss or Engrave font effects (`w:outline`, `w:shadow`, `w:emboss`, `w:imprint`) is drawn with them on screen and in PDFs, no longer as plain text: hollow outlined letters, or a grey copy behind the letters; the PDF keeps the text (#2534) | | 2026-10-15 | Table Layout › Cell Margins opens an editable dialog initialized from the table defaults; confirming an unchanged dialog preserves imported margins, and changing one side retains the others (#1960) | | 2026-10-15 | Word's native ink inside groups and drawing canvases survives DOCX saves (gap 2): the ink member keeps its InkML part, relationship, content type and picture fallback, stays in its place among the members, moves with the group and is drawn from its fallback picture; deleting the group drops its part | +| 2026-10-15 | Crash recovery copies and native .json files store pictures and kept parts as base64 text instead of arrays of numbers (#586 follow-up, #518): with a 27 MiB picture a recovery copy is 36 MiB instead of 96 MiB and a .json file 36 MiB instead of 286 MiB, written and read about five times faster. Copies and files written before still open | | 2026-10-14 | Consistent preferred cell widths override a stale table grid when both totals agree, for fixed and AutoFit tables. Conflicting widths, spans and percentages remain unchanged. | | 2026-10-14 | Legacy VML freeforms (`v:shape` with its own `path`), such as the rule lines in many older headers, are read as freeform shapes: they show on screen and in PDFs and are saved back as DrawingML custom geometry instead of vanishing | | 2026-10-14 | Field instructions spanning paragraphs no longer add blank lines during pagination; empty result paragraphs still take a line. | @@ -600,6 +601,7 @@ Footer page-table height and body wrapping are corrected. Other pagination gaps | 2026-10-17 | minor | Progress log: Scale to paper size on File › Print, #2108 | | 2026-10-17 | trivial | Non-overlapping body floating tables clear header tables; overlapping tables keep their requested position | | 2026-10-15 | trivial | Progress log: caret inside ligatures (#1974) | +| 2026-10-15 | trivial | Progress log: recovery copies and .json files store pictures as base64 (#586 follow-up) | | 2026-10-14 | trivial | Progress log: row table border exceptions | | 2026-10-12 | minor | Czech proofing review: short-word splits rank after deletion, NOSUGGEST excludes suggestions while accepting typed forms, affix comments apply only to headers, ratios keep spaces and repeated-word messages use Czech quotes; synthetic engine end-to-end and latency checks | | 2026-10-12 | minor | Czech spelling from the installed Hunspell dictionary, runtime-table suggestions and generic checks; Czech interface writes cs-CZ documents; no Czech hyphenation patterns bundled; estimates unchanged | diff --git a/crates/doc/Cargo.toml b/crates/doc/Cargo.toml index f0111eb18..9ab62d34d 100644 --- a/crates/doc/Cargo.toml +++ b/crates/doc/Cargo.toml @@ -10,6 +10,7 @@ workspace = true [dependencies] wordcraft-geom = { workspace = true } +base64 = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } thiserror = { workspace = true } diff --git a/crates/doc/src/byte_store.rs b/crates/doc/src/byte_store.rs new file mode 100644 index 000000000..75fae6453 --- /dev/null +++ b/crates/doc/src/byte_store.rs @@ -0,0 +1,130 @@ +//! How [`crate::Document::media`] and [`crate::Document::passthrough`] are written in the native +//! JSON form (`.json` files and crash-recovery copies): each part's bytes as one base64 string. +//! serde's default for bytes, an array of numbers, took about 3.6 times their size compact and +//! over six times pretty-printed (a 27 MB picture made a 99 MB recovery copy, #586). +//! +//! Reading takes either form, so `.json` files and recovery copies written before still open. +use std::collections::BTreeMap; +use std::fmt; +use std::sync::Arc; + +use base64::Engine as _; +use base64::display::Base64Display; +use base64::engine::general_purpose::STANDARD; +use serde::de::{self, Deserializer, MapAccess, SeqAccess, Visitor}; +use serde::ser::{Serialize, Serializer}; + +type Store = BTreeMap>>; + +/// One part's bytes, encoded as it is written out: no second copy of a large picture is made. +struct Encoded<'a>(&'a [u8]); + +impl Serialize for Encoded<'_> { + fn serialize(&self, s: S) -> Result { + s.collect_str(&Base64Display::new(self.0, &STANDARD)) + } +} + +pub(crate) fn serialize(store: &Store, s: S) -> Result { + s.collect_map(store.iter().map(|(name, bytes)| (name, Encoded(bytes)))) +} + +pub(crate) fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result { + d.deserialize_map(StoreVisitor) +} + +struct StoreVisitor; + +impl<'de> Visitor<'de> for StoreVisitor { + type Value = Store; + + fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str("a map of part names to base64 text or byte arrays") + } + + fn visit_map>(self, mut map: A) -> Result { + let mut store = Store::new(); + while let Some((name, Bytes(bytes))) = map.next_entry::()? { + store.insert(name, Arc::new(bytes)); + } + Ok(store) + } +} + +/// A part's bytes as either form. +struct Bytes(Vec); + +impl<'de> de::Deserialize<'de> for Bytes { + fn deserialize>(d: D) -> Result { + d.deserialize_any(BytesVisitor).map(Bytes) + } +} + +struct BytesVisitor; + +impl<'de> Visitor<'de> for BytesVisitor { + type Value = Vec; + + fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str("base64 text or an array of bytes") + } + + fn visit_str(self, text: &str) -> Result, E> { + STANDARD.decode(text).map_err(|e| E::custom(format!("bytes are not valid base64: {e}"))) + } + + fn visit_bytes(self, bytes: &[u8]) -> Result, E> { + Ok(bytes.to_vec()) + } + + /// The form written before: one number per byte. + fn visit_seq>(self, mut seq: A) -> Result, A::Error> { + // The length hint comes from the file: trust it only so far. + let mut out = Vec::with_capacity(seq.size_hint().unwrap_or(0).min(1 << 20)); + while let Some(b) = seq.next_element::()? { + out.push(b); + } + Ok(out) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[derive(serde::Serialize, serde::Deserialize, PartialEq, Debug, Default)] + struct Parts { + #[serde(default, with = "super")] + media: Store, + } + + fn parts(entries: &[(&str, &[u8])]) -> Parts { + Parts { media: entries.iter().map(|(n, b)| ((*n).to_string(), Arc::new(b.to_vec()))).collect() } + } + + #[test] + fn bytes_are_written_as_base64_and_read_back() { + let all: Vec = (0..=255).collect(); + for n in [0, 1, 2, 3, 4, 255, 256] { + let p = parts(&[("a.png", &all[..n.min(all.len())])]); + let text = serde_json::to_string(&p).unwrap(); + assert_eq!(serde_json::from_str::(&text).unwrap(), p, "{n}"); + } + assert_eq!(serde_json::to_string(&parts(&[("m", b"Man"), ("ma", b"Ma")])).unwrap(), r#"{"media":{"m":"TWFu","ma":"TWE="}}"#); + } + + #[test] + fn number_arrays_written_before_still_read() { + let old = r#"{"media":{"a.png":[0,1,127,128,254,255],"empty":[]}}"#; + assert_eq!(serde_json::from_str::(old).unwrap(), parts(&[("a.png", &[0, 1, 127, 128, 254, 255]), ("empty", &[])])); + } + + #[test] + fn hostile_values_are_errors_not_panics() { + for bad in [r#""T""#, r#""TWFu!""#, r#""-_""#, r#""TQ=x""#, "[256]", "[-1]", "[1.5]", "true", "{}", r#"["a"]"#] { + let text = format!(r#"{{"media":{{"a.png":{bad}}}}}"#); + assert!(serde_json::from_str::(&text).is_err(), "{bad}"); + } + assert!(serde_json::from_str::(r#"{"media":"x"}"#).is_err()); + } +} diff --git a/crates/doc/src/lib.rs b/crates/doc/src/lib.rs index 04421a5d2..eb0cea8a0 100644 --- a/crates/doc/src/lib.rs +++ b/crates/doc/src/lib.rs @@ -11,6 +11,7 @@ #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] pub mod bidi; +mod byte_store; pub mod chart; mod cjk_numbers; pub mod cluster; @@ -723,11 +724,11 @@ pub struct Document { /// The document's own endnote separators and continuation notice. #[serde(skip_serializing_if = "NoteSeparators::is_empty")] pub endnote_separators: NoteSeparators, - /// Embedded media (images) by key. - #[serde(skip_serializing_if = "BTreeMap::is_empty")] + /// Embedded media (images) by key. Native JSON writes each as base64 (`byte_store.rs`). + #[serde(skip_serializing_if = "BTreeMap::is_empty", with = "byte_store")] pub media: BTreeMap>>, /// Source-format parts we keep verbatim for round-trip (e.g. custom XML), by part name. - #[serde(skip_serializing_if = "BTreeMap::is_empty")] + #[serde(skip_serializing_if = "BTreeMap::is_empty", with = "byte_store")] pub passthrough: BTreeMap>>, /// The document's own building blocks (its glossary document), in file order. #[serde(skip_serializing_if = "Vec::is_empty")] diff --git a/crates/engine/src/recovery/desktop.rs b/crates/engine/src/recovery/desktop.rs index 676a6ed0c..ad38ed0da 100644 --- a/crates/engine/src/recovery/desktop.rs +++ b/crates/engine/src/recovery/desktop.rs @@ -706,6 +706,26 @@ mod tests { assert_eq!(Store::files(&t.0.join(id)).unwrap().len(), 1); } + /// Copies are written with picture bytes in base64 (#586 follow-up); those written before kept + /// them as arrays of numbers and are still listed and restored, bytes intact. + #[test] + fn copies_with_byte_arrays_written_before_still_restore() { + let t = Temp::new(); + let id = new_id().unwrap(); + let mut s = snapshot(&id, "Synthetic draft with a picture"); + s.generation = 1; + s.document.media.insert("media/image1.png".into(), Arc::new(vec![0, 1, 127, 128, 254, 255])); + let mut value = serde_json::to_value(&s).unwrap(); + assert_eq!(value["document"]["media"]["media/image1.png"], "AAF/gP7/"); + value["document"]["media"]["media/image1.png"] = json!([0, 1, 127, 128, 254, 255]); + private_dir(&t.0.join(&id)).unwrap(); + fs::write(t.0.join(&id).join("snapshot-00000000000000000001.json"), serde_json::to_vec(&value).unwrap()).unwrap(); + let mut store = Store::new(t.0.clone()).unwrap(); + store.scan().unwrap(); + assert_eq!(store.inventory().len(), 1); + assert_eq!(store.load(&id).unwrap().document, s.document); + } + #[test] fn failed_pruning_cannot_push_a_record_beyond_the_discovery_limit() { let t = Temp::new(); diff --git a/crates/engine/tests/recovery_format.rs b/crates/engine/tests/recovery_format.rs index 38f372ac7..f889d6a83 100644 --- a/crates/engine/tests/recovery_format.rs +++ b/crates/engine/tests/recovery_format.rs @@ -67,3 +67,88 @@ fn existing_native_json_without_byte_stores_still_opens() { let doc = wordcraft_engine::io::open_bytes("old.json", &serde_json::to_vec(&value).unwrap()).unwrap(); assert_eq!(doc.plain_text(wordcraft_doc::StoryRef::Body), "Old native document"); } + +/// Picture and part bytes are one base64 string each in native JSON (`.json` files and recovery +/// copies, #586). An array of numbers took about 3.6 times their size, and over ten times in a +/// pretty-printed `.json` file. +#[test] +fn native_json_stores_bytes_as_base64() { + let mut doc = Document::from_text("Synthetic"); + let compact = |d: &Document| serde_json::to_vec(d).unwrap().len(); + let pretty = |d: &Document| wordcraft_engine::io::save_bytes("doc.json", d).unwrap(); + let (without, without_pretty) = (compact(&doc), pretty(&doc).len()); + let bytes: Vec = (0..300_000u32).map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8).collect(); + doc.media.insert("big.png".into(), Arc::new(bytes.clone())); + doc.passthrough.insert("customXml/item1.xml".into(), Arc::new(b"".to_vec())); + let saved = pretty(&doc); + // Base64 is 4/3 of the bytes; allow a little for names and punctuation. + for (with, before) in [(compact(&doc), without), (saved.len(), without_pretty)] { + assert!(with - before < bytes.len() * 14 / 10, "{} bytes for {}", with - before, bytes.len()); + } + let value: serde_json::Value = serde_json::from_slice(&saved).unwrap(); + assert_eq!(value["passthrough"]["customXml/item1.xml"], "PHN5bnRoZXRpYy8+"); + let reopened = wordcraft_engine::io::open_bytes("doc.json", &saved).unwrap(); + assert_eq!((reopened.media, reopened.passthrough), (doc.media, doc.passthrough)); +} + +/// `.json` files and recovery copies written before kept bytes as arrays of numbers: they still +/// open, with the same bytes. +#[test] +fn native_json_with_byte_arrays_still_opens() { + let doc = complete_document(); + let mut value = serde_json::to_value(&doc).unwrap(); + for (store, parts) in [("media", &doc.media), ("passthrough", &doc.passthrough)] { + for (name, bytes) in parts { + value[store][name] = serde_json::json!(bytes.as_slice()); + } + } + assert!(value["media"]["synthetic.png"].is_array()); + let reopened = wordcraft_engine::io::open_bytes("old.json", &serde_json::to_vec_pretty(&value).unwrap()).unwrap(); + assert_eq!(reopened, doc); +} + +/// Bytes that are neither base64 nor an array of bytes make the file fail to open, never panic. +#[test] +fn native_json_with_bad_bytes_is_an_error() { + let saved = wordcraft_engine::io::save_bytes("doc.json", &complete_document()).unwrap(); + for bad in [serde_json::json!("not base64!"), serde_json::json!([1, 2, 300]), serde_json::json!({"a": 1}), serde_json::json!(7)] { + let mut value: serde_json::Value = serde_json::from_slice(&saved).unwrap(); + value["media"]["synthetic.png"] = bad.clone(); + assert!(wordcraft_engine::io::open_bytes("doc.json", &serde_json::to_vec(&value).unwrap()).is_err(), "{bad}"); + } +} + +/// Measures native JSON with a large picture: the compact form a recovery copy wraps, and a +/// pretty-printed `.json` file. Run on demand: +/// `cargo test --release -p wordcraft-engine --test recovery_format -- --ignored --nocapture`. +#[test] +#[ignore = "measurement"] +fn native_json_size_with_a_large_picture() { + let mut doc = Document::from_text("Synthetic document with one large picture."); + // Incompressible bytes, as a photograph's are. + let mut x = 0x2545_f491_4f6c_dd1du64; + let noise: Vec = (0..27u32 << 20) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect(); + doc.media.insert("media/image1.png".into(), Arc::new(noise)); + let mib = |n: usize| n as f64 / f64::from(1u32 << 20); + let t0 = std::time::Instant::now(); + let compact = serde_json::to_vec(&doc).unwrap(); + let written = t0.elapsed(); + let t0 = std::time::Instant::now(); + let back: Document = serde_json::from_slice(&compact).unwrap(); + println!("27 MiB picture, compact (recovery copy): {:.1} MiB, written in {written:?}, read in {:?}", mib(compact.len()), t0.elapsed()); + assert_eq!(back.media, doc.media); + let t0 = std::time::Instant::now(); + let pretty = wordcraft_engine::io::save_bytes("doc.json", &doc).unwrap(); + let written = t0.elapsed(); + let t0 = std::time::Instant::now(); + let back = wordcraft_engine::io::open_bytes("doc.json", &pretty).unwrap(); + println!("27 MiB picture, .json file (pretty): {:.1} MiB, written in {written:?}, read in {:?}", mib(pretty.len()), t0.elapsed()); + assert_eq!(back.media, doc.media); +} diff --git a/docs/file-format-parity.md b/docs/file-format-parity.md index 0b12d8b2e..1e06bcdc8 100644 --- a/docs/file-format-parity.md +++ b/docs/file-format-parity.md @@ -1,6 +1,6 @@ # File-format parity with Microsoft Word -> **Last reviewed:** 2026-10-11 · **Last updated:** 2026-10-19 · **Change:** trivial (ODT headers and footers read from each master page, #863; previously trivial: shape gradients and transparency, #1500; previously trivial: RTF import compares heading and code formatting with the file's stylesheet; previously trivial: text typed into a content control takes the control's formatting; localized placeholder styles recognised; previously trivial: forms protection keeps its content controls: no inserting, Properties or Remove Content Control; previously trivial: Word XML and web pages saved as `.doc` open by what they hold, #2853; previously trivial: a VML shape in a header cell is always placed on the page, user-checked in Word for Mac, #1775; previously trivial: `w:sym` characters keep their symbol font when the text's font changes and are written back as `w:sym`, #2036; previously trivial: RTF export writes sections (gap 19); previously trivial: ODT export writes sections (gap 19); previously trivial: two lines in one and horizontal in vertical (`w:eastAsianLayout`) kept on save; previously trivial: PDF outline targets follow leading page breaks; previously trivial: positional tabs `w:ptab` read and kept on save, #2367; previously trivial: DOCX: mixed right-to-left runs split and marked `w:rtl` with their language, #1543; previously trivial: `w:doNotTrackMoves` round-trips and turns move tracking off; previously trivial: legacy outline, shadow, emboss and imprint text effects drawn, #2534; previously trivial: DOCX glossary documents (building blocks) read, kept and written; previously trivial: cached field results keep their visible run formatting; previously trivial: cached REF results retain hyperlink destinations; previously trivial: CLI conversions can explicitly refresh fields; previously trivial: a copied equation also goes in the clipboard's MathML formats, #1297; previously trivial: a pasted HTML `
` is an indented paragraph, not the Quote style, #1270); previously trivial: every table style part round-trips, #714; previously trivial: VML freeforms read and kept on save; previously trivial: multi-paragraph field instructions no longer add blank lines; previously trivial: `flowChartOr` draws its circle and cross spokes, as in Word; previously trivial: RTF: Mac OS Roman and DOS text, #1653; previously trivial: VML text boxes keep their internal margins, fit to their text and vertical alignment; hidden drawings not shown; previously trivial: PDF export draws horizontal and vertical lines inside groups, #1474; previously trivial: symbol-font text exported as Unicode, #776; previously trivial: text-box margins, Resize shape to fit text and freeforms in groups, #743; previously trivial: formatted paste from the system clipboard's HTML and RTF, #687; previously trivial: independent complex-script formatting inheritance and round-trip regression tests; previously trivial: import/export fixes for TXT, RTF, ODT, HTML and LaTeX, #433–#440; previously major: first version; every format Word reads or writes, measured from the readers and writers on origin/main) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4, plus Word for Windows' extra formats +> **Last reviewed:** 2026-10-11 · **Last updated:** 2026-10-19 · **Change:** trivial (native .json stores pictures and kept parts as base64, #586 follow-up; previously trivial: ODT headers and footers read from each master page, #863; previously trivial: shape gradients and transparency, #1500; previously trivial: RTF import compares heading and code formatting with the file's stylesheet; previously trivial: text typed into a content control takes the control's formatting; localized placeholder styles recognised; previously trivial: forms protection keeps its content controls: no inserting, Properties or Remove Content Control; previously trivial: Word XML and web pages saved as `.doc` open by what they hold, #2853; previously trivial: a VML shape in a header cell is always placed on the page, user-checked in Word for Mac, #1775; previously trivial: `w:sym` characters keep their symbol font when the text's font changes and are written back as `w:sym`, #2036; previously trivial: RTF export writes sections (gap 19); previously trivial: ODT export writes sections (gap 19); previously trivial: two lines in one and horizontal in vertical (`w:eastAsianLayout`) kept on save; previously trivial: PDF outline targets follow leading page breaks; previously trivial: positional tabs `w:ptab` read and kept on save, #2367; previously trivial: DOCX: mixed right-to-left runs split and marked `w:rtl` with their language, #1543; previously trivial: `w:doNotTrackMoves` round-trips and turns move tracking off; previously trivial: legacy outline, shadow, emboss and imprint text effects drawn, #2534; previously trivial: DOCX glossary documents (building blocks) read, kept and written; previously trivial: cached field results keep their visible run formatting; previously trivial: cached REF results retain hyperlink destinations; previously trivial: CLI conversions can explicitly refresh fields; previously trivial: a copied equation also goes in the clipboard's MathML formats, #1297; previously trivial: a pasted HTML `
` is an indented paragraph, not the Quote style, #1270); previously trivial: every table style part round-trips, #714; previously trivial: VML freeforms read and kept on save; previously trivial: multi-paragraph field instructions no longer add blank lines; previously trivial: `flowChartOr` draws its circle and cross spokes, as in Word; previously trivial: RTF: Mac OS Roman and DOS text, #1653; previously trivial: VML text boxes keep their internal margins, fit to their text and vertical alignment; hidden drawings not shown; previously trivial: PDF export draws horizontal and vertical lines inside groups, #1474; previously trivial: symbol-font text exported as Unicode, #776; previously trivial: text-box margins, Resize shape to fit text and freeforms in groups, #743; previously trivial: formatted paste from the system clipboard's HTML and RTF, #687; previously trivial: independent complex-script formatting inheritance and round-trip regression tests; previously trivial: import/export fixes for TXT, RTF, ODT, HTML and LaTeX, #433–#440; previously major: first version; every format Word reads or writes, measured from the readers and writers on origin/main) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4, plus Word for Windows' extra formats Every format Word opens or saves, what WordCraft does with it, and how it's tested. Fidelity is the share of a typical real-world file's content and formatting that survives (read: shows correctly; @@ -267,6 +267,7 @@ Imported text fields use the first nonempty cached result run's formatting on th | 2026-10-16 | trivial | DOCX: `wp:anchor/@layoutInCell="0"` and VML `o:allowincell="f"` are read and written back (`layoutInCell="0"`), so a header-cell picture watermark is placed on the page (#1775) | | 2026-10-16 | trivial | Equations from Word files no longer lose content: body-level `m:oMathPara`, `mc:AlternateContent Requires="m"`, hyperlinks/simple fields/`mc:AlternateContent` inside an equation, `w:sym` and Symbol-font text in math (#1273) | | 2026-10-13 | trivial | Recipient lists: cell text capped at 64 MiB a sheet (shared-string amplification), an XML spreadsheet named .xls explained (#1287) | +| 2026-10-15 | trivial | Native .json writes pictures and kept parts as base64 text instead of arrays of numbers (a 27 MiB picture: 36 MiB instead of 286 MiB); files with the old arrays still open (#586 follow-up) | | 2026-10-13 | trivial | A cell's explicit top vertical alignment (`w:vAlign w:val="top"`) is kept on read and write and beats a table style that centres or bottom-aligns (#1288) | | 2026-10-14 | trivial | Row table border exceptions survive DOCX saves | | 2026-10-12 | trivial | Fields keep `w:dirty` through DOCX round trips — on buffered fields and range-field begin markers alike — and computing a result (Update Field, Update Table) clears it like Word's F9, so a generator's placeholder TOC still refreshes when Word opens the file until WordCraft replaces the placeholder (#877) | diff --git a/docs/recovery.md b/docs/recovery.md index 00ce84506..45a05479f 100644 --- a/docs/recovery.md +++ b/docs/recovery.md @@ -32,7 +32,9 @@ be lost if a crash happens before the next snapshot or while its write is still Version 1 snapshots wrap the native JSON document with an id, document name, timestamp and monotonic generation. Native JSON now includes media bytes, opaque preserved parts, and -chart/SmartArt/OLE source markup and relationships. Older JSON without these fields still opens. +chart/SmartArt/OLE source markup and relationships. Media and preserved parts are written as one +base64 string each. Older JSON without these fields, or with their bytes as arrays of numbers (as +written before base64), still opens. Paragraph cache revisions are transient and regenerated on load; undo history and selection are not part of the document snapshot. Recovery preserves the current in-memory model. It cannot recover content already lost during import or make later export more faithful than the exporter. @@ -50,8 +52,9 @@ Windows is therefore unverified. Limits are 256 MiB of encoded JSON per snapshot, 2 GiB of snapshot files per profile, 128 records and 16 generations per record during discovery. Ordinarily only two valid generations remain. Copies are retained until saved or explicitly discarded; there is no automatic age-based deletion. -Over-limit writes report an error and retain the previous valid copy. JSON stores binary parts -as byte arrays, so it can be substantially larger than DOCX. This first version has a fixed +Over-limit writes report an error and retain the previous valid copy. Base64 takes 4/3 of a +part's size (a 27 MiB picture makes a 36 MiB copy), and DOCX compresses, so a copy can still be +larger than the DOCX it came from. This first version has a fixed interval, English fallback for new labels, and no cloud, version-history or browser recovery. ## Commands and verification