Merge PR #2688: M6: distinguish CSV header tabs from TSV delimiters

ROADMAP.md: next's file; the PR's progress line and revision row added once; status-line edit dropped.
docs/roadmap.md: next's file; the PR's revision row and M6 milestone sentence added once; status-line edit dropped.
crates/xfdf/README.md: all three paragraphs kept (#1917, #2686, this PR).
crates/xfdf/src/tests.rs: #2686's test and this PR's two tests all kept.
parity/acrobat-features.toml: form.data-xml-csv-txt keeps every evidence entry and note.
This commit is contained in:
aiongg
2026-10-16 14:03:59 +00:00
6 changed files with 86 additions and 8 deletions
+2
View File
@@ -157,6 +157,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
| 8 | 1.0 polish: performance, right-to-left, missing languages, keyboard-only (M14) | 55–95 |
## Progress log
- **2026-10-15 (M6, CSV tab-containing headers):** CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV delimiters without changing field identities. Synthetic export/import and incremental/full save regressions cover header tabs, quotes and line breaks, plus ordinary TSV controls. Formula-cell policy is unchanged. Estimates unchanged.
- **2026-10-15 (M6, AcroForm XML text):** Plain AcroForm XML imports retain all direct text around comments and processing instructions, including CDATA and character references. Synthetic regressions verify exact values and unknown catalog data after incremental and full save/reopen. Nested-element and XFA interpretation are unchanged. Estimates unchanged.
- **2026-10-13 (M6, data exchange fidelity):** XML and XFDF exports preserve tabs and line breaks in field names and comment metadata using character references. Saved-PDF regressions cover distinct names that XML would normalize to the same attribute value, restored field values and replacement of the original comment identities. Estimates unchanged.
- **2026-10-15 (M2, gap 12):** `cargo xtask text-oracle` scores reading order as well as word F1: the longest common subsequence of the two word sequences over the longer one. F1 is a bag of words and cannot see order at all -- measuring #2431, a change that moves text and nothing else, it called 26 of the 29 files whose output changed identical, where the order score separated 18 improvements from 6 regressions. Reports, does not gate; a disagreement with Poppler, not a verdict. Estimates unchanged.
@@ -497,6 +498,7 @@ Newest first: the date, what moved, and the new overall estimate.
| Date | Change | Summary |
|---|---|---|
| 2026-10-15 | minor | M6: CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV delimiters without changing field identities. Estimates unchanged. |
| 2026-10-15 | minor | M6: Plain AcroForm XML imports retain all direct text around comments and processing instructions, including CDATA and character references. Estimates unchanged. |
| 2026-10-13 | minor | M6: XML/XFDF attribute whitespace survives form and comment export/import; estimates unchanged |
| 2026-10-13 | minor | M9: separate page validation overlays preserve signature artwork, and snapshot tabs reuse instead of multiplying; baked captions remain unchanged (#79 open); estimates unchanged |
+4
View File
@@ -24,6 +24,10 @@ character references, so XML attribute normalization cannot change their identit
Plain AcroForm XML imports retain all direct text around comments and processing instructions,
including CDATA and character references. Nested-element and XFA interpretation are unchanged.
CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV
delimiters without changing field identities. Automatic detection of externally authored, ambiguous
tables remains heuristic. Formula-cell policy and tab-delimited export are unchanged.
Import replaces comments with the same `/NM` (and drops the replaced comment's pop-up), draws
appearances PdfCraft can draw, and sets values through the form's own checks (formats,
validation, recalculation); read-only fields take imported values, as in Acrobat. XML is read
+28 -3
View File
@@ -615,7 +615,7 @@ pub fn export_data(doc: &Document, format: Format) -> String {
out
}
Format::Csv => {
let q = |s: &str| if s.contains([',', '"', '\n', '\r']) { format!("\"{}\"", s.replace('"', "\"\"")) } else { s.to_string() };
let q = |s: &str| if s.contains([',', '"', '\n', '\r', '\t']) { format!("\"{}\"", s.replace('"', "\"\"")) } else { s.to_string() };
format!(
"{}\n{}\n",
fields.iter().map(|f| q(&f.0)).collect::<Vec<_>>().join(","),
@@ -1044,7 +1044,7 @@ pub fn merge_csv(files: &[Vec<(String, Vec<String>)>]) -> String {
}
}
}
let q = |s: &str| if s.contains([',', '"', '\n', '\r']) { format!("\"{}\"", s.replace('"', "\"\"")) } else { s.to_string() };
let q = |s: &str| if s.contains([',', '"', '\n', '\r', '\t']) { format!("\"{}\"", s.replace('"', "\"\"")) } else { s.to_string() };
let mut out = columns.iter().map(|c| q(c)).collect::<Vec<_>>().join(",");
out.push('\n');
for f in files {
@@ -1104,6 +1104,31 @@ fn import_fdf(doc: &mut Document, bytes: &[u8]) -> Result<Report, DataError> {
Ok(report)
}
// Inspect the first logical record: tabs within quoted CSV cells are data. A quote
// in the middle of an unquoted TSV name stays literal, as in the TSV reader.
fn has_tab_delimiter(text: &str) -> bool {
let mut quoted = false;
let mut cell_start = true;
let mut chars = text.chars().peekable();
while let Some(c) = chars.next() {
match c {
'"' if quoted && chars.peek() == Some(&'"') => {
chars.next();
}
'"' if quoted || cell_start => {
quoted = !quoted;
cell_start = false;
}
'\t' if !quoted => return true,
'\r' | '\n' if !quoted => return false,
',' if !quoted => cell_start = true,
_ => cell_start = false,
}
}
// An unclosed quote can be a literal TSV field-name character.
quoted && text.lines().next().is_some_and(|line| line.contains('\t'))
}
/// Form data from XML, CSV or tab-delimited text.
fn import_table(doc: &mut Document, text: &str) -> Result<Report, DataError> {
let mut report = Report::default();
@@ -1121,7 +1146,7 @@ fn import_table(doc: &mut Document, text: &str) -> Result<Report, DataError> {
(name, vec![c.children().filter(|node| node.is_text()).filter_map(|node| node.text()).collect()])
})
.collect()
} else if t.lines().next().is_some_and(|l| l.contains('\t')) {
} else if has_tab_delimiter(t) {
tab_delimited(t)?
} else {
// CSV with quotes: a quoted value may hold commas, doubled quotes and line breaks.
+46
View File
@@ -1144,3 +1144,49 @@ fn xml_form_import_preserves_text_around_comments_and_instructions() {
assert_eq!(fields(&original).into_iter().find(|f| f.name == "Name").unwrap().value, vec!["replace me"]);
}
}
#[test]
fn csv_headers_with_tabs_survive_export_import_and_save() {
for name in ["First\tName", "\tName", "Name\t", "Name\t\"quoted\"", "Name\nwith\ta tab"] {
let mut source = one_field();
add_field(&mut source, 0, [50.0, 650.0, 250.0, 670.0], &NewField::Text { multiline: false }, Some("Other")).unwrap();
set_value(&mut source, "Other", &FieldValue::Text("other field".into())).unwrap();
let field = fields(&source).into_iter().find(|f| f.name == "Name").unwrap().obj;
source.update_dict(field, |dict| dict.set(b"T".to_vec(), PdfString::text(name))).unwrap();
set_value(&mut source, name, &FieldValue::Text("original, \"value\"".into())).unwrap();
let bytes = write_full(&source, &SaveOptions::default()).unwrap();
let source = Document::open(Arc::new(bytes.clone())).unwrap();
let row = data_values(&bytes).unwrap();
for csv in [export_data(&source, Format::Csv), merge_csv(&[row])] {
let mut doc = Document::open(Arc::new(bytes.clone())).unwrap();
set_value(&mut doc, name, &FieldValue::Text("replacement".into())).unwrap();
set_value(&mut doc, "Other", &FieldValue::Text("other replacement".into())).unwrap();
let report = import(&mut doc, csv.as_bytes()).unwrap();
assert_eq!(report.fields, 2);
assert!(report.rejected.is_empty(), "{:?}", report.rejected);
for saved in [write_full(&doc, &SaveOptions::default()).unwrap(), write_incremental(&doc, &SaveOptions::default()).unwrap()] {
let reopened = Document::open(Arc::new(saved)).unwrap();
assert_eq!(fields(&reopened).into_iter().find(|f| f.name == name).unwrap().value, vec!["original, \"value\""], "{csv:?}");
assert_eq!(values(&reopened), values(&source), "{csv:?}");
}
}
}
}
#[test]
fn tabular_import_keeps_tsv_quotes_literal_and_csv_value_tabs() {
for name in ["Name", "Na\"me", "\"Name\"", "\"Name"] {
let mut doc = one_field();
add_field(&mut doc, 0, [50.0, 650.0, 250.0, 670.0], &NewField::Text { multiline: false }, Some("Other")).unwrap();
let field = fields(&doc).into_iter().find(|f| f.name == "Name").unwrap().obj;
doc.update_dict(field, |dict| dict.set(b"T".to_vec(), PdfString::text(name))).unwrap();
let text = format!("{name}\tOther\r\nAda\tother field\r\n");
let report = import(&mut doc, text.as_bytes()).unwrap();
assert_eq!(report.fields, 2, "{text:?}");
assert_eq!(fields(&doc).into_iter().find(|f| f.name == name).unwrap().value, vec!["Ada"]);
}
let mut doc = one_field();
add_field(&mut doc, 0, [50.0, 650.0, 250.0, 670.0], &NewField::Text { multiline: false }, Some("Other")).unwrap();
import(&mut doc, b"Name,Other\r\n\"Ada\tLovelace\",other field\r\n").unwrap();
assert_eq!(fields(&doc).into_iter().find(|f| f.name == "Name").unwrap().value, vec!["Ada\tLovelace"]);
}
+2 -1
View File
@@ -117,7 +117,7 @@ total matches the parity estimate.
| M4 | Engine, history, save, organize | 100–180 | 95% | 8–15 | Done:<br>- command registry (menus, shortcuts and palette all use it);<br>- undo/redo; incremental, atomic and encrypted saves;<br>- autosave and crash recovery;<br>- organize, combine, extract, split and insert-from-file, with identical fonts and images stored once;<br>- CLI `edit/combine/extract/split`.<br>Done since: Combine files in its own tab (sortable, resizable columns; folders; early warnings; unlocking protected files), bookmark editing, page labels (Number pages), CLI `run`, Set Page Boxes, Crop tool, Duplicate pages, Ctrl+A / Cmd+A to select all pages in Organize, Replace pages, bookmarks from tagged headings. Missing: page transitions, recovery on the web M4: Insert-from-file preserves source bookmarks; integration tests retained with current main. |
| M5 | Comments (all annotation types, XFDF) | 120–200 | 88% | 12–25 | Done: notes, highlight/underline/strikeout/squiggly, text boxes, ink, lines, arrows, rectangles, ovals, stamps (dynamic, Sign Here, Standard Business), with appearance streams; replies, status, checkmarks, locking, move/resize/restyle/delete; properties dialog; panel filter and sort; hide all; summaries (comments only); XFDF/FDF import and export (partial imports preserve existing reply and review-status links); flatten; Fill & Sign with adjustable text/date size; agent tools.<br>Done since: stamps placed on a rotated page read upright (#1398); polygons, connected lines, clouds, callouts, inserted text, colour and checkmark filters; image signature pointer previews, selection handles and live image movement/resizing with proportional corners and independent edge handles. Missing: custom stamps, replace-text proposals, summary layouts with the page |
| M5 | Comments (all annotation types, XFDF) | 120–200 | 88% | 12–25 | Done: notes, highlight/underline/strikeout/squiggly, text boxes, ink, lines, arrows, rectangles, ovals, stamps (dynamic, Sign Here, Standard Business), with appearance streams; replies, status, checkmarks, locking, move/resize/restyle/delete; properties dialog; panel filter and sort; hide all; summaries (comments only); XFDF/FDF import and export (partial imports preserve existing reply and review-status links); flatten; Fill & Sign with adjustable text/date size; agent tools.<br>Done since: polygons, connected lines, clouds, callouts, inserted text, colour and checkmark filters; image signature pointer previews, selection handles and live image movement/resizing with proportional corners and independent edge handles. A refused restyle or resize of a text box another program drew leaves that comment unchanged (#1327). Make Current Properties Default on an arrow updates arrows, not plain lines (#1302). Flattening keeps a visible comment it cannot draw (#1317). Missing: custom stamps, replace-text proposals, summary layouts with the page |
| M6 | Forms + JavaScript | 160–320 | 50% | 40–75 | Done: filling all field types with regenerated appearances; Prepare a form (every field type, multi-select/shared properties, move/resize/delete, Field Properties General/Appearance/Position/Options/Format/Validate/Calculate); Acrobat's AF format/keystroke/validate/calculate functions and simplified field notation run natively in Acrobat's event order; tab order; push-button actions; flatten; agent tools. Form data exchange (FDF, XFDF, XML, CSV, text) done; XFDF values retain text across XML comments and processing instructions. XML/XFDF exports preserve tabs and line breaks in names and comment metadata. Plain AcroForm XML imports retain all direct text around comments and processing instructions, including CDATA and character references.<br>Done since: JavaScript engine (boa, sandboxed) running custom keystroke/validate/calculate/format and button scripts, document JavaScripts, console. Missing: Actions tab, wider object model (annotations, layers, dialogs), XFA<br>Done since: Multi-select choice fields write ascending `/I` option indices while preserving `/V` selection order; synthetic forms/engine regressions cover reverse selection and save/reopen. CSV export preserves carriage returns in names and values. |
| M6 | Forms + JavaScript | 160–320 | 50% | 40–75 | Done: filling all field types with regenerated appearances; Prepare a form (every field type, multi-select/shared properties, move/resize/delete, Field Properties General/Appearance/Position/Options/Format/Validate/Calculate); Acrobat's AF format/keystroke/validate/calculate functions and simplified field notation run natively in Acrobat's event order; tab order; push-button actions; flatten; agent tools. Form data exchange (FDF, XFDF, XML, CSV, text) done; XFDF values retain text across XML comments and processing instructions. XML/XFDF exports preserve tabs and line breaks in names and comment metadata. Plain AcroForm XML imports retain all direct text around comments and processing instructions, including CDATA and character references. CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV delimiters without changing field identities.<br>Done since: JavaScript engine (boa, sandboxed) running custom keystroke/validate/calculate/format and button scripts, document JavaScripts, console. Missing: Actions tab, wider object model (annotations, layers, dialogs), XFA<br>Done since: Multi-select choice fields write ascending `/I` option indices while preserving `/V` selection order; synthetic forms/engine regressions cover reverse selection and save/reopen. CSV export preserves carriage returns in names and values. |
| M7 | Content editing (text, images, header/footer, watermark) | 250–500 | 20% | 100–180 | Done: header & footer, watermarks, backgrounds, Bates; added text and images that stay editable (move, resize, format, rotate, flip, crop, replace); links (Link tool, Link Properties, create from URLs, remove all). Done since: Edit text preserves Arabic/Hebrew `/ActualText` typing order, with font-independent regressions; single-glyph RTL ligature spans (including lam-alef) join their surrounding word. Done since (#1116): editable text regions grouped across compatible words and subset fonts, one rich-text editor over a background with the source text removed, embedded sfnt faces reused as editor faces, and visible (clipped) bounds for cropped-image selection and resize. Exclusive crops now resize across nested states and content streams, with shared clips fixed.<br>Done since: #594 ordinary paragraph drags preserve overlapping source text; UI/engine regressions, native save/reopen, independent-reader checks and full CI passed in the Linux test environment. #1200 candidate: local GoToR links open in desktop tabs and through root-confined automation; appearance-only edits preserve existing actions. Named destinations remain unapplied and `/NewWindow` is a tab hint. Missing: editing existing text and images in place (the longest pole), image/PDF watermarks |
| M7 | Content editing (text, images, header/footer, watermark) | 250–500 | 20% | 100–180 | Done: header & footer, watermarks, backgrounds, Bates; added text and images that stay editable (move, resize, format, rotate, flip, crop, replace); links (Link tool, Link Properties, create from URLs, remove all). Done since: Edit text preserves Arabic/Hebrew `/ActualText` typing order, with font-independent regressions. Done since (#1116): editable text regions grouped across compatible words and subset fonts, one rich-text editor over a background with the source text removed, embedded sfnt faces reused as editor faces, and visible (clipped) bounds for cropped-image selection and resize. Exclusive crops now resize across nested states and content streams, with shared clips fixed.<br>Done since: #594 ordinary paragraph drags preserve overlapping source text; UI/engine regressions, native save/reopen, independent-reader checks and full CI passed in the Linux test environment. #1200 candidate: local GoToR links open in desktop tabs and through root-confined automation; appearance-only edits preserve existing actions. Named destinations remain unapplied and `/NewWindow` is a tab hint. #1316: replacing a line or paragraph keeps a neighbouring line whose glyph box overlaps when the baselines differ; a second copy on the same baseline is still replaced. Missing: editing existing text and images in place (the longest pole), image/PDF watermarks |
| M7 | Content editing (text, images, header/footer, watermark) | 250–500 | 20% | 100–180 | Done: header & footer, watermarks, backgrounds, Bates; added text and images that stay editable (move, resize, format, rotate, flip, crop, replace); links (Link tool, Link Properties, create from URLs, remove all). Missing: editing existing text and images in place (the longest pole), watermark print/screen visibility and repeat |
@@ -179,6 +179,7 @@ M2 (renderer), M7 (content editing), M10 (OCR and Office), M11 (Preflight and st
| Date | Change | Summary |
|---|---|---|
| 2026-10-15 | minor | M6: CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV delimiters without changing field identities. Estimates unchanged. |
| 2026-10-15 | minor | M6: Plain AcroForm XML imports retain all direct text around comments and processing instructions, including CDATA and character references. Estimates unchanged. |
| 2026-10-13 | minor | M6: XML/XFDF attribute whitespace survives form and comment export/import; estimates unchanged |
| 2026-10-15 | minor | M9 #1895: generic appearance rewriting deferred; separate validation overlays preserve original artwork, exports and printing. Baked captions remain unchanged (#79 open). Estimates unchanged. |
+4 -4
View File
@@ -4704,8 +4704,8 @@ milestone = "M6"
status = "shipped"
commands = ["form.export_data", "form.import_data"]
tools = ["doc_export_data", "doc_import_data"]
evidence = ["crates/xfdf/src/tests.rs::form_data_as_xml_csv_and_text", "crates/automation/tests/automation.rs::comments_and_form_data_travel_as_xfdf_fdf_and_text", "crates/xfdf/src/tests.rs::xml_exports_preserve_whitespace_in_field_names", "crates/xfdf/src/tests.rs::xml_form_import_preserves_text_around_comments_and_instructions"]
notes = "Acrobat's XML (xfdf:original), CSV, tab-delimited; XML/XFDF exports retain tabs and line breaks in field identities through character references; plain AcroForm XML imports retain all direct text around comments and processing instructions, including CDATA and character references. Nested-element and XFA interpretation remain outside this flat exchange format."
evidence = ["crates/xfdf/src/tests.rs::form_data_as_xml_csv_and_text", "crates/automation/tests/automation.rs::comments_and_form_data_travel_as_xfdf_fdf_and_text", "crates/xfdf/src/tests.rs::xml_exports_preserve_whitespace_in_field_names", "crates/xfdf/src/tests.rs::xml_form_import_preserves_text_around_comments_and_instructions", "crates/xfdf/src/tests.rs::csv_headers_with_tabs_survive_export_import_and_save", "crates/xfdf/src/tests.rs::tabular_import_keeps_tsv_quotes_literal_and_csv_value_tabs"]
notes = "Acrobat's XML (xfdf:original), CSV, tab-delimited; XML/XFDF exports retain tabs and line breaks in field identities through character references; plain AcroForm XML imports retain all direct text around comments and processing instructions, including CDATA and character references. Nested-element and XFA interpretation remain outside this flat exchange format. CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV delimiters without changing field identities. Ambiguous externally authored tables remain heuristic; TSV export and formula-cell policy are unchanged."
[[feature]]
id = "form.merge-to-spreadsheet"
@@ -4716,8 +4716,8 @@ milestone = "M6"
status = "shipped"
commands = ["form.merge_data"]
tools = ["form_merge_data"]
evidence = ["crates/xfdf/src/tests.rs::data_files_merge_into_a_spreadsheet", "crates/automation/tests/automation.rs::merging_form_data_into_a_spreadsheet", "crates/ui-egui/tests/javascript.rs::merge_data_files_into_a_spreadsheet"]
notes = "FDF, XFDF or filled-in PDF forms \u2192 CSV, a column per field and a row per file."
evidence = ["crates/xfdf/src/tests.rs::data_files_merge_into_a_spreadsheet", "crates/automation/tests/automation.rs::merging_form_data_into_a_spreadsheet", "crates/ui-egui/tests/javascript.rs::merge_data_files_into_a_spreadsheet", "crates/xfdf/src/tests.rs::csv_headers_with_tabs_survive_export_import_and_save"]
notes = "FDF, XFDF or filled-in PDF forms \u2192 CSV, a column per field and a row per file. CSV header tabs, quotes and line breaks retain field identities through import and save/reopen; ambiguous external table detection remains heuristic."
[[feature]]
id = "form.flatten-fields"