Variables: Import Data Sets… reads tab-separated files and files without a name column

dataset.import read only commas between cells and refused a file whose
first header cell wasn't `name`. A value that didn't match its
variable's kind, and a repeated data set name, were reported with the
row's record number, which is not the file line after a blank line or a
quoted line break. A data set name over 255 characters gave no line.

- When the header row, read with tabs between cells, has a tab and no
  comma outside quotes, tabs separate the cells. A quoted cell may then
  hold tabs and commas.
- Without a `name` column every header cell is a variable, and the rows
  take the first free names Data Set 1, Data Set 2 and so on, as Capture
  Data Set and blank name cells do. A first header cell `name`, in any
  case, still holds the data set names.
- Value errors, a repeated data set name and a name over 255 characters
  give the file line the row starts on.
- The Variables panel's Import Data Sets… dialog also lists .tsv files.
- import_fuzz.rs imports random text, random bytes and mutated exports,
  comma- and tab-separated, into a document with bound text, visibility
  and graph variables, selects up to three of the imported data sets in
  turn, and exports every data set.

Docs: docs/mcp.md, docs/target-app-parity.md.
This commit is contained in:
dotCooCoo
2026-10-17 13:49:49 +07:00
committed by bflatastic
parent a285efda46
commit 5bdd9d539f
6 changed files with 219 additions and 35 deletions
+84 -31
View File
@@ -5,8 +5,9 @@
//! variable, or a graph root to a `graph` variable (CSV data), put a row of values in
//! each dataset, and `dataset.select` swaps the whole document to that row in one undo
//! step. `dataset.import` / `dataset.export` move those rows as CSV (header: `name`
//! then variable names). Agents reach every command through `run_command`, like all
//! engine commands.
//! then variable names); import also reads tab-separated files and files without the
//! `name` column. Agents reach every command through `run_command`, like all engine
//! commands.
use std::collections::{BTreeMap, BTreeSet};
@@ -535,16 +536,61 @@ fn datasets_csv_bytes(p: &Value, cmd: &str) -> Result<Vec<u8>> {
Ok(bytes)
}
/// Parse a datasets CSV into rows of cells (no header interpretation yet): commas between cells,
/// `"` quotes a cell (doubled inside), which may then hold commas and line breaks, as
/// [`csv_escape`] writes them. An unquoted cell is trimmed. A quoted cell keeps what its quotes
/// hold as written, spaces included, and drops white space after the closing quote at the end
/// of the cell. Blank lines are skipped.
fn parse_datasets_csv(text: &str, cmd: &str) -> Result<Vec<Vec<String>>> {
/// A row of a datasets CSV: the file line it starts on, and its cells.
type CsvRow = (usize, Vec<String>);
/// Whether a datasets CSV separates its cells with tabs: its header row (the first line that
/// isn't blank), read with tabs between cells, has a tab outside quotes and no comma outside
/// quotes.
fn tab_separated(text: &str) -> bool {
// `start`: nothing but `\r` read yet in this cell. `blank`: no cell on this line holds
// anything but white space yet.
let (mut in_quotes, mut start, mut blank, mut tab) = (false, true, true, false);
let mut chars = text.chars().peekable();
while let Some(c) = chars.next() {
match c {
'"' if in_quotes => {
if chars.peek() == Some(&'"') {
chars.next();
blank = false;
} else {
in_quotes = false;
}
}
'"' if start => in_quotes = true,
c if in_quotes => blank &= c.is_whitespace(),
',' => return false,
'\t' => {
tab = true;
start = true;
continue;
}
'\r' => continue,
'\n' if blank => {
tab = false;
start = true;
continue;
}
'\n' => return tab,
c => blank &= c.is_whitespace(),
}
start = false;
}
tab
}
/// Parse a datasets CSV into rows of cells (no header interpretation yet). Cells are separated
/// by commas, or by tabs when [`tab_separated`] says so. `"` quotes a cell (doubled inside),
/// which may then hold separators and line breaks, as [`csv_escape`] writes them. An unquoted
/// cell is trimmed. A quoted cell keeps what its quotes hold as written, spaces included, and
/// drops white space after the closing quote at the end of the cell. Blank lines are skipped.
fn parse_datasets_csv(text: &str, cmd: &str) -> Result<Vec<CsvRow>> {
let text = text.strip_prefix('\u{feff}').unwrap_or(text);
let sep = if tab_separated(text) { '\t' } else { ',' };
let (mut rows, mut row, mut cur) = (Vec::new(), Vec::<String>::new(), String::new());
// `closed`: where the closing quote of a quoted cell falls in `cur`.
let (mut in_quotes, mut closed, mut line) = (false, None, 1usize);
// `closed`: where the closing quote of a quoted cell falls in `cur`. `start`: the line the
// current row starts on.
let (mut in_quotes, mut closed, mut line, mut start) = (false, None, 1usize, 1usize);
let mut chars = text.chars().peekable();
loop {
let c = chars.next();
@@ -559,12 +605,12 @@ fn parse_datasets_csv(text: &str, cmd: &str) -> Result<Vec<Vec<String>>> {
}
}
Some('"') if cur.is_empty() => in_quotes = true,
Some(',') if !in_quotes => row.push(end_csv_cell(&mut cur, &mut closed)),
Some(ch) if ch == sep && !in_quotes => row.push(end_csv_cell(&mut cur, &mut closed)),
Some('\r') if !in_quotes => {}
Some('\n') | None if !in_quotes => {
row.push(end_csv_cell(&mut cur, &mut closed));
if row.iter().any(|c| !c.trim().is_empty()) {
rows.push(std::mem::take(&mut row));
rows.push((start, std::mem::take(&mut row)));
// +1 for the header.
if rows.len() > MAX_CSV_ROWS + 1 {
return Err(bad(cmd, format!("more than {MAX_CSV_ROWS} data rows")));
@@ -575,6 +621,7 @@ fn parse_datasets_csv(text: &str, cmd: &str) -> Result<Vec<Vec<String>>> {
return Ok(rows);
}
line += 1;
start = line;
}
None => return Err(bad(cmd, format!("line {line}: unclosed quote"))),
Some(c) => {
@@ -659,27 +706,27 @@ fn datasets_to_csv(vars: &Variables) -> String {
out
}
/// Import datasets from CSV: first column `name`, remaining headers are variable names.
/// Existing rows of the same name are replaced; new names are appended. One undo step.
/// Import datasets from CSV. A first header cell `name` makes the first column the dataset
/// names; the other headers are variable names. Without it, every header is a variable name
/// and the rows take the first free names "Data Set 1", "Data Set 2" and so on. Existing rows
/// of the same name are replaced; new names are appended. One undo step.
fn dataset_import(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "dataset.import";
let bytes = datasets_csv_bytes(p, C)?;
let text = std::str::from_utf8(&bytes).map_err(|_| bad(C, "CSV is not UTF-8"))?;
let rows = parse_datasets_csv(text, C)?;
let Some(header) = rows.first() else {
let Some((_, header)) = rows.first() else {
return Err(bad(C, "CSV is empty"));
};
if header.first().map(|h| h.trim().eq_ignore_ascii_case("name")) != Some(true) {
return Err(bad(C, "first header cell must be `name`"));
}
let name_column = header.first().is_some_and(|h| h.trim().eq_ignore_ascii_case("name"));
let (kinds, existing_names) = {
let st = s.doc()?;
let kinds: BTreeMap<String, VariableKind> = st.doc.variables.variables.iter().map(|v| (v.name.clone(), v.kind)).collect();
let existing: Vec<String> = st.doc.variables.datasets.iter().map(|d| d.name.clone()).collect();
let existing: BTreeSet<String> = st.doc.variables.datasets.iter().map(|d| d.name.clone()).collect();
(kinds, existing)
};
let mut columns: Vec<(usize, String, VariableKind)> = Vec::new();
for (i, h) in header.iter().enumerate().skip(1) {
for (i, h) in header.iter().enumerate().skip(usize::from(name_column)) {
let name = h.trim();
if name.is_empty() {
return Err(bad(C, format!("empty header in column {}", i + 1)));
@@ -691,19 +738,25 @@ fn dataset_import(s: &mut Session, p: &Value) -> Result<Value> {
return Err(bad(C, "CSV has a header but no data rows"));
}
let mut parsed: Vec<DataSet> = Vec::new();
for (r_i, row) in rows.iter().enumerate().skip(1) {
let line = r_i + 1;
let raw_name = row.first().map(|c| c.trim()).unwrap_or("");
// The names of the rows read so far, and the last number given to a row without a name.
let (mut seen, mut number) = (BTreeSet::new(), 0usize);
for &(line, ref row) in rows.iter().skip(1) {
let raw_name = if name_column { row.first().map_or("", |c| c.trim()) } else { "" };
let name = if raw_name.is_empty() {
// Same numbering as Capture Data Set when a row leaves the name blank.
(1..)
.map(|n| format!("Data Set {n}"))
.find(|n| parsed.iter().all(|d| d.name != *n) && !existing_names.iter().any(|e| e == n))
.unwrap_or_else(|| format!("Data Set {line}"))
// Same numbering as Capture Data Set when a row leaves the name blank or the file has
// no name column. The search starts after the last number given.
let (n, name) = (number + 1..)
.map(|n| (n, format!("Data Set {n}")))
.find(|(_, n)| !seen.contains(n) && !existing_names.contains(n))
.unwrap_or_else(|| (line, format!("Data Set {line}")));
number = n;
name
} else if raw_name.chars().count() > MAX_NAME {
return Err(bad(C, format!("line {line}: `name` is longer than {MAX_NAME} characters")));
} else {
named(Some(raw_name), "name", C)?
raw_name.to_string()
};
if parsed.iter().any(|d| d.name == name) {
if !seen.insert(name.clone()) {
return Err(bad(C, format!("line {line}: dataset `{name}` appears twice")));
}
let mut values = BTreeMap::new();
@@ -937,7 +990,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Import Data Sets…",
["Window", "Variables"],
None,
"{path? | data? | dataBase64? | csv?} load datasets from CSV (header: name, then variable names; one row per dataset). Rows whose name already exists are replaced → {imported, updated, names}",
"{path? | data? | dataBase64? | csv?} load datasets from CSV, comma- or tab-separated (header: name, then variable names; one row per dataset; without the name column every header is a variable and the rows take the first free names Data Set 1, 2…). Rows whose name already exists are replaced → {imported, updated, names}",
has_doc,
dataset_import
),
+65 -1
View File
@@ -948,6 +948,70 @@ fn datasets_csv_keeps_spaces_in_quoted_cells() {
assert_eq!(value("C"), text_value(" quoted "));
}
/// A header row with a tab and no comma outside quotes makes tabs the separator; a quoted cell
/// may then hold tabs and commas. A header row with a comma outside quotes keeps commas, even
/// when a cell holds a tab. A tab-separated header with an unclosed quote fails with the reason.
#[test]
fn datasets_csv_reads_tab_separated_files() {
let mut s = session();
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
s.execute("variable.define", &json!({"name": "Show", "kind": "visibility"})).unwrap();
s.execute("variable.define", &json!({"name": "Last, First", "kind": "text"})).unwrap();
let tsv = "name\tTitle\tShow\r\nA\tTwo words\ttrue\r\nB\t\"a, b\tc\"\tfalse\r\n";
let imp = s.execute("dataset.import", &json!({"csv": tsv})).unwrap();
assert_eq!(imp["names"], json!(["A", "B"]));
s.execute("dataset.import", &json!({"csv": "name,Title\nC,one\ttwo"})).unwrap();
s.execute("dataset.import", &json!({"csv": "name\t\"Last, First\"\nD\tSmith, Jo"})).unwrap();
let value = |name: &str, var: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get(var)).cloned();
let text_value = |v: &str| Some(vectorcraft_doc::DataValue::Text(v.into()));
assert_eq!(value("A", "Title"), text_value("Two words"));
assert_eq!(value("A", "Show"), Some(vectorcraft_doc::DataValue::Visible(true)));
assert_eq!(value("B", "Title"), text_value("a, b\tc"));
assert_eq!(value("B", "Show"), Some(vectorcraft_doc::DataValue::Visible(false)));
assert_eq!(value("C", "Title"), text_value("one\ttwo"));
assert_eq!(value("D", "Last, First"), text_value("Smith, Jo"));
let unclosed = s.execute("dataset.import", &json!({"csv": "name\tTitle\t\"Show\nA\tx\ttrue"})).unwrap_err().to_string();
assert!(unclosed.contains("unclosed quote"), "{unclosed}");
}
/// Without a `name` column every header cell is a variable, and the rows take the first free
/// names Data Set 1, 2 and so on. A first header cell `name`, in any case, still holds the names.
#[test]
fn datasets_csv_without_a_name_column_numbers_the_rows() {
let mut s = session();
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
s.execute("dataset.new", &json!({"name": "Data Set 2", "values": {"Title": "kept"}})).unwrap();
let imp = s.execute("dataset.import", &json!({"csv": "Title\nx\ny\n"})).unwrap();
assert_eq!((imp["imported"].as_u64(), imp["updated"].as_u64()), (Some(2), Some(0)));
assert_eq!(imp["names"], json!(["Data Set 1", "Data Set 3"]));
let value = |s: &Session, name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("Title")).cloned();
let text_value = |v: &str| Some(vectorcraft_doc::DataValue::Text(v.into()));
assert_eq!(value(&s, "Data Set 1"), text_value("x"));
assert_eq!(value(&s, "Data Set 2"), text_value("kept"));
assert_eq!(value(&s, "Data Set 3"), text_value("y"));
let named = s.execute("dataset.import", &json!({"csv": "NAME,Title\nData Set 1,z"})).unwrap();
assert_eq!((named["imported"].as_u64(), named["updated"].as_u64()), (Some(0), Some(1)));
assert_eq!(value(&s, "Data Set 1"), text_value("z"));
}
/// An error about a row's values or name gives the file line the row starts on, counting blank
/// lines and line breaks inside quoted cells.
#[test]
fn datasets_csv_row_errors_name_the_file_line() {
let mut s = session();
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
s.execute("variable.define", &json!({"name": "Show", "kind": "visibility"})).unwrap();
let error = |s: &mut Session, csv: &str| s.execute("dataset.import", &json!({"csv": csv})).unwrap_err().to_string();
let blank_line = error(&mut s, "name,Show\n\nX,maybe");
assert!(blank_line.contains("line 3:"), "{blank_line}");
let line_break = error(&mut s, "name,Title,Show\nA,\"one\ntwo\",true\nB,x,maybe");
assert!(line_break.contains("line 4:"), "{line_break}");
let twice = error(&mut s, "name,Title\nA,x\n\nA,y");
assert!(twice.contains("line 4:"), "{twice}");
let long = error(&mut s, &format!("name,Title\n\n{},x", "n".repeat(256)));
assert!(long.contains("line 3:") && long.contains("longer than 255"), "{long}");
}
/// Import / export datasets as CSV: header `name` then variables; round-trip keeps values.
#[test]
fn datasets_csv_round_trips_and_rejects_bad_input() {
@@ -989,7 +1053,7 @@ fn datasets_csv_round_trips_and_rejects_bad_input() {
assert_eq!((again["imported"].as_u64(), again["updated"].as_u64()), (Some(0), Some(2)));
assert!(s.execute("dataset.import", &json!({"csv": "name,Ghost\nX,y"})).is_err());
assert!(s.execute("dataset.import", &json!({"csv": "name,Show\nX,maybe"})).is_err());
assert!(s.execute("dataset.import", &json!({"csv": "Title,Name\nX,y"})).is_err(), "first column must be name");
assert!(s.execute("dataset.import", &json!({"csv": "Title,Name\nX,y"})).is_err(), "no variable named Title");
let out = dir.file("sets.csv");
s.execute("dataset.export", &json!({"path": out})).unwrap();
+63
View File
@@ -1157,6 +1157,69 @@ proptest! {
}
}
// ---------- data set CSV files ----------
/// A document with a text, a visibility and a graph variable, each bound to an object, and two
/// data sets.
fn merge_session() -> vectorcraft_engine::Session {
let mut s = vectorcraft_engine::Session::new();
s.execute("file.new", &json!({"width": 200, "height": 200})).unwrap();
let id = |r: Value| r["id"].as_u64().unwrap();
let t = id(s.execute("text.create", &json!({"x": 10, "y": 10, "text": "One"})).unwrap());
let r = id(s.execute("shape.rectangle", &json!({"x": 10, "y": 40, "width": 50, "height": 40})).unwrap());
let graph = json!({"type": "column", "x": 80, "y": 10, "width": 100, "height": 80, "csv": ",Sales\nA,1\nB,2"});
let g = id(s.execute("graph.create", &graph).unwrap());
for (name, kind, object) in [("Title", "text", t), ("Show", "visibility", r), ("Chart", "graph", g)] {
s.execute("variable.define", &json!({"name": name, "kind": kind})).unwrap();
s.execute("variable.bind", &json!({"variable": name, "ids": [object]})).unwrap();
}
s.execute("dataset.new", &json!({"name": "A", "values": {"Title": "Two, \"quoted\"\nlines", "Show": true, "Chart": ",Q\nX,3"}})).unwrap();
s.execute("dataset.new", &json!({"name": "B", "values": {"Title": " padded ", "Show": false}})).unwrap();
s
}
/// [`merge_session`]'s data sets exported as CSV, made once.
fn saved_datasets() -> String {
static TEXT: std::sync::OnceLock<String> = std::sync::OnceLock::new();
TEXT.get_or_init(|| merge_session().execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string()).clone()
}
/// `dataset.import` with `params` on [`merge_session`]'s document, the first three data sets it
/// read selected in turn and every data set exported again; the document then renders and exports.
fn datasets(what: &str, params: Value) -> Result<(), TestCaseError> {
survive(what, || {
let mut s = merge_session();
if let Ok(r) = no_panic(s.execute("dataset.import", &params)) {
for name in r["names"].as_array().into_iter().flatten().take(3) {
let _ = no_panic(s.execute("dataset.select", &json!({"name": name})));
}
}
let _ = no_panic(s.execute("dataset.export", &json!({})));
Some((*s.doc().ok()?.doc).clone())
})
}
proptest! {
#![proptest_config(config())]
#[test]
fn dataset_csv_garbage_never_panics(s in ".{0,300}", head in prop::sample::select(vec!["", "name,Title,Show,Chart\n", "Title\tShow\tChart\n", "name\tTitle\n", "Show,Chart\n"])) {
datasets("data set CSV garbage", json!({"csv": format!("{head}{s}")}))?;
}
#[test]
fn dataset_csv_bytes_never_panic(bytes in prop::collection::vec(any::<u8>(), 0..300)) {
datasets("data set CSV bytes", json!({"dataBase64": vectorcraft_format::base64_encode(&bytes)}))?;
}
#[test]
fn mutated_dataset_csvs_never_panic(cut in 0usize..200, edits in prop::collection::vec(arb_edit(), 0..10), tabs in any::<bool>()) {
let text = saved_datasets();
let text = if tabs { text.replace(',', "\t") } else { text };
datasets("mutated data set CSV", json!({"csv": mutate_text(&text, cut, &edits)}))?;
}
}
// ---------- shortcut sets ----------
/// Shortcut set `bytes` read with `shortcutSet.read`, with import matches saved to annotate it: an
+2 -2
View File
@@ -319,9 +319,9 @@ pub fn menu(app: &mut VectorcraftApp, ui: &mut Ui) {
}
}
/// Variables panel: Import Data Sets… — pick a CSV and run `dataset.import`.
/// Variables panel: Import Data Sets… opens a file dialog for a comma- or tab-separated file and runs `dataset.import` on the chosen file.
fn import_datasets(app: &mut VectorcraftApp) {
let pick = crate::FilePick { filters: vec![("CSV", &["csv", "txt"])], ..Default::default() };
let pick = crate::FilePick { filters: vec![("CSV", &["csv", "tsv", "txt"])], ..Default::default() };
let Some(path) = crate::picks::open(app, &pick) else { return };
if let Err(e) = app.run("dataset.import", json!({"path": path})) {
app.status(e);
+4 -1
View File
@@ -1083,7 +1083,10 @@ already walks Image Trace. Short agent entry points:
- Data sets: `dataset.list`, `dataset.new {name, values?}`, `dataset.set {name, values?}`, `dataset.rename`,
`dataset.delete`, `dataset.capture`, `dataset.update`, `dataset.select {name}`, `dataset.next` /
`dataset.prev` (each `dataset.select` / next / prev is one undo step over every bound object), `dataset.import {path|data|dataBase64|csv}` /
`dataset.export {path?}` (CSV: `name`, then one column per variable).
`dataset.export {path?}` (CSV: `name`, then one column per variable). `dataset.import` also reads tab-separated
files, and files whose first header cell isn't `name`: every header cell is then a variable, and the rows take the
first free names `Data Set 1`, `Data Set 2` and so on. An import error about a row's values or name gives the file
line the row starts on.
- Batch: `action.batch {commands, source?: folder|datasets, folder?, destination?: none|saveAndClose|folder,
destinationFolder?, overrideOpen?, overrideSave?, overrideExport?, includeSubdirs?, stopOnError?}` plays an
action on every file in a folder or every data set (the Actions panel menu's Batch…).
+1
View File
@@ -415,6 +415,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
- Variables: `dataset.select`, `dataset.next` and `dataset.prev` name the area type and type on a path whose text they rewrote and that doesn't fit, in the reply's `overflow` key, as `text.setText` does.
- Actions panel (record/playback, persisted), generic parameter dialogs for every "…" command.
- Variables (data merge): text, visibility and image variables bound to objects (an object may carry several), dataset rows applied in one undo step (`variable.*`, `dataset.*`) from a Variables panel and Window › Variables, with dialogs to define, bind and edit, Make Text/Visibility/Image Dynamic, Capture/Update Data Set, CSV import/export (`dataset.import` / `dataset.export`), and a row in italics when an edit leaves the art not matching it; an image dataset value is a file path and apply keeps the object's bounds; bindings let go of deleted art and locked art is left alone; every command, like the rest, is drivable over MCP and the CLI.
- Variables: Import Data Sets… (`dataset.import`) reads tab-separated files as well as comma-separated ones, and files without a `name` column, whose rows take the first free names Data Set 1, Data Set 2 and so on. An import error about a row's values or name gives the file line the row starts on.
- JSON-lines control channel with real egui pointer and keyboard injection.
- Synthetic input validates complete gestures before queue mutation: at most 256 drag steps, 16 clicks and 1,024 pending events; up to 64 control requests per frame, with later requests retained for the next frame (#1709).
- The control server writes each reply through a 64 KiB buffer. On Windows 11 it sends the 17 MB `document.inspect` reply of a document with 50,000 objects on one layer to a client over loopback in 0.09 s instead of 34 s.