mirror of
https://github.com/storytold/gridcraft.git
synced 2026-11-03 00:41:56 -06:00
XLSX: Python cells as _xlfn._xlws.PY() formulas, code in the Python part, cached values; opening limits
A Python cell is written to xlsx as [MS-XLSX] 2.3.10 specifies: its formula is _xlfn._xlws.PY(i,0|1), where i is the zero-based index of the cell's script in the workbook's Python part and 0|1 the return type, 0 for an Excel value and 1 for a Python object. The part (2.1.23) is xl/python.xml, content type application/vnd.ms-excel.python+xml, related from the workbook part with http://schemas.microsoft.com/office/2023/09/relationships/Python; its root python holds pythonScripts, one pythonScript per Python cell in the order the cells are written, with the code in code. The code keeps its xl() references as text, so the formula has no further arguments. The cell keeps its cached value. A spilling result is written like a dynamic array (cm="1", t="array" ref=...) with the spilled values cached, so other applications show the values (and #NAME? if they recalculate). The reader turns a PY( formula with or without the _xlfn. and _xlws. prefixes into a Python cell. An index names a script of the Python part, and each %Px% in the script becomes the text of the formula's argument x (2.3.10.4). The string-literal form, PY("chunk"&"chunk",0|1), still reads, from the formula's own text so line breaks in the code survive. A formula naming a script the part doesn't hold keeps only its cached value, with a warning: no calculation can evaluate it. The reader keeps the cached spilled values as the cell's result, so a workbook from Excel shows its arrays without running anything. Python errors are cached as #VALUE!, as #SPILL! and #CALC! already are. Native JSON keeps Cell.py whole: code, return type, the last result and the exact error. Calc lays out a Python cell's kept result: Python cells are nodes of the dependency graph that read nothing, and a pass takes a dirty Python cell's result instead of evaluating it, so its array spills, is blocked (#SPILL!) and unblocked, and feeds its dependents exactly like a formula's. An array that is malformed or has more than 1,000,000 cells (only a crafted file holds one) shows #SPILL!. A dirty Python cell is pending in the pass, as a dirty formula is, so a formula, a range read or a cell-by-cell range walk that reaches it before its turn lays it out first rather than reading its old value. Opening a file can't make the app loop over a claimed size: the reader keeps a Python cell's spill range only when it has at most 1,000,000 cells (a range claiming the whole sheet is dropped with its values, as a formula's spilled values are), and io::adopt_python rebuilds results for at most 4,000,000 cells over the whole workbook, with a warning for the cells it skips. The Python part is read up to 64 MB and 1,000,000 scripts, a script is kept up to one character past what a cell may hold, and the cells take at most 64 MB of code from the part in all, since several formulas can name one script. io::enforce_python_limits drops the code of a cell whose code doesn't fit in 8,192 characters as PY(...) (the cell keeps its cached value, with a warning) and a kept result that isn't a well-formed array within the cap, each with its spill range and spilled values, so the sheet and a save no longer show the old spill. Formats that keep PY() as an ordinary formula (xls, xlsb, ods) are adopted the same way. Opening a file runs nothing. The command-sequence fuzzer sets Python cells (code with xl() references, quotes, line breaks, and code at and over the size limit), and its xlsx round-trip check allows the Python errors becoming #VALUE!, which is what the file format stores. Tests: the written formula naming its script, the Python part with its content type and relationship, return types, a spilling table saved as a dynamic array, Python errors as #VALUE!, the round trip of values, code, long code with quotes and line breaks and the error, and no Python part without Python cells; a workbook laid out as the format specifies (an index, a parameter named as %P2%) read as Python cells with their spilled values and saved back with its code in a Python part; the string-literal form; a formula naming a missing script, with and without a Python part; 20,000 cells naming one long script against the code budget; the part's reader and writer (escaping, odd %P tokens, a long script, a repeated parameter); a blocker in the spill area giving #SPILL! and clearing it spilling again; the native JSON round trip; a spill range claiming the whole sheet; the workbook-wide budget; hostile JSON (code over the limit, a result claiming far more cells than it holds); a malformed result that never spills; PY() formulas at 8,079 and 8,080 characters of code. Calc: formulas, MAX, SUM, COUNTA and SUMPRODUCT read a dirty Python cell whose array is blocked before its turn, in a full and an incremental recalculation, and see the array once it is unblocked; enforce_python_limits removes the spill of dropped code and of a dropped result, and keeps a sound one.
This commit is contained in:
prxnvt
@@ -119,7 +119,139 @@ fn decrypt_message(name: &str, e: gridcraft_xlsx::DecryptError) -> String {
|
||||
}
|
||||
|
||||
/// Parses bytes that aren't encrypted.
|
||||
/// Parses bytes that aren't encrypted, and adopts the Python cells they hold (see
|
||||
/// [`adopt_python`]) within the limits of [`enforce_python_limits`].
|
||||
fn open_plain(name: &str, bytes: &[u8], csv_as_text: bool) -> Result<(Workbook, Vec<String>)> {
|
||||
let (mut wb, mut warnings) = read_plain(name, bytes, csv_as_text)?;
|
||||
warnings.extend(adopt_python(&mut wb));
|
||||
warnings.extend(enforce_python_limits(&mut wb));
|
||||
Ok((wb, warnings))
|
||||
}
|
||||
|
||||
/// The most result cells [`adopt_python`] rebuilds from a file's cached spills, over all its
|
||||
/// Python cells (each one is also limited to [`gridcraft_model::python::MAX_RESULT_CELLS`]).
|
||||
pub const MAX_ADOPTED_RESULT_CELLS: u64 = 4_000_000;
|
||||
|
||||
/// Python cells from a file: a cell whose formula is `PY("…",0|1)` (a format that kept it as a
|
||||
/// formula) becomes a Python cell, and a Python cell whose spilled values the file cached gets
|
||||
/// them back as its result, so it shows them until it runs again. Results are rebuilt for at most
|
||||
/// [`MAX_ADOPTED_RESULT_CELLS`] cells in all; the rest keep only their own cached value. Returns
|
||||
/// the warnings.
|
||||
pub fn adopt_python(wb: &mut Workbook) -> Vec<String> {
|
||||
let mut budget = MAX_ADOPTED_RESULT_CELLS;
|
||||
let mut skipped = 0usize;
|
||||
for si in 0..wb.sheets.len() {
|
||||
let Some(sh) = wb.sheets.get(si) else { continue };
|
||||
let formulas: Vec<(CellRef, gridcraft_model::PyCode)> = sh
|
||||
.cells
|
||||
.iter()
|
||||
.filter_map(|(c, cell)| cell.formula.as_ref().and_then(|f| gridcraft_model::PyCode::from_formula(&f.text)).map(|p| (c, p)))
|
||||
.collect();
|
||||
let max = gridcraft_model::python::MAX_RESULT_CELLS as u64;
|
||||
let spills: Vec<(CellRef, RangeRef)> = sh
|
||||
.spill_ranges
|
||||
.iter()
|
||||
.filter(|(a, r)| r.count() <= max && sh.cell(**a).and_then(|c| c.py.as_ref()).is_some_and(|p| p.output.is_none()))
|
||||
.map(|(a, r)| (*a, *r))
|
||||
.collect();
|
||||
if formulas.is_empty() && spills.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let Some(sh) = wb.sheet_mut(si) else { continue };
|
||||
for (c, py) in formulas {
|
||||
if let Some(cell) = sh.cells.get_mut(c) {
|
||||
cell.formula = None;
|
||||
cell.py = Some(std::sync::Arc::new(py));
|
||||
}
|
||||
}
|
||||
for (anchor, range) in spills {
|
||||
let Some(left) = budget.checked_sub(range.count()) else {
|
||||
skipped += 1;
|
||||
continue;
|
||||
};
|
||||
budget = left;
|
||||
let data: Vec<gridcraft_core::Value> =
|
||||
range.iter().map(|c| if c == anchor { sh.cell(c).map(|x| x.value.clone()).unwrap_or_default() } else { sh.value(c) }).collect();
|
||||
let Some(array) = gridcraft_core::Array::new(range.height() as usize, range.width() as usize, data) else { continue };
|
||||
if let Some(py) = sh.cells.get_mut(anchor).and_then(|x| x.py.as_mut()) {
|
||||
std::sync::Arc::make_mut(py).output =
|
||||
Some(gridcraft_model::PyOutput { value: array.into(), type_name: "list".into(), preview: String::new(), convertible: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
if skipped == 0 {
|
||||
return Vec::new();
|
||||
}
|
||||
vec![format!(
|
||||
"The file's Python results are larger than {MAX_ADOPTED_RESULT_CELLS} cells in all, so {skipped} Python cell(s) show only their first value until they run again."
|
||||
)]
|
||||
}
|
||||
|
||||
/// A file is somebody else's work: a Python cell whose code doesn't fit in Excel's
|
||||
/// 8,192-character formula keeps only its cached value, and a kept result that is not a
|
||||
/// well-formed array of at most [`gridcraft_model::python::MAX_RESULT_CELLS`] cells is dropped
|
||||
/// (the cell keeps its value). Returns the warnings.
|
||||
pub fn enforce_python_limits(wb: &mut Workbook) -> Vec<String> {
|
||||
let mut long = Vec::new();
|
||||
for si in 0..wb.sheets.len() {
|
||||
let Some(sh) = wb.sheets.get(si) else { continue };
|
||||
let mut over = Vec::new();
|
||||
let mut bad_output = Vec::new();
|
||||
for (c, cell) in sh.cells.iter() {
|
||||
let Some(py) = &cell.py else { continue };
|
||||
if !py.fits() {
|
||||
long.push(format!("{}!{}", sh.name, c.a1()));
|
||||
over.push(c);
|
||||
} else if py.output.as_ref().is_some_and(|o| !sane_python_output(&o.value)) {
|
||||
bad_output.push(c);
|
||||
}
|
||||
}
|
||||
if over.is_empty() && bad_output.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let dropped: Vec<CellRef> = over.iter().chain(&bad_output).copied().collect();
|
||||
let Some(sh) = wb.sheet_mut(si) else { continue };
|
||||
for c in over {
|
||||
if let Some(cell) = sh.cells.get_mut(c) {
|
||||
cell.py = None;
|
||||
}
|
||||
}
|
||||
for c in bad_output {
|
||||
if let Some(py) = sh.cells.get_mut(c).and_then(|x| x.py.as_mut()) {
|
||||
std::sync::Arc::make_mut(py).output = None;
|
||||
}
|
||||
}
|
||||
// The spill of a dropped result (or of code dropped to its value) goes with it, so the
|
||||
// sheet and a save no longer show the old spilled values.
|
||||
for c in dropped {
|
||||
if let Some(r) = sh.spill_ranges.remove(&c) {
|
||||
sh.clear_spill(r);
|
||||
}
|
||||
}
|
||||
}
|
||||
let Some(first) = long.first() else { return vec![] };
|
||||
let more = if long.len() > 1 { format!(" (and {} more)", long.len() - 1) } else { String::new() };
|
||||
vec![format!(
|
||||
"A Python cell's code is longer than Excel's {}-character formula limit, so {first}{more} kept only its last calculated value.",
|
||||
gridcraft_model::python::MAX_FORMULA_CHARS
|
||||
)]
|
||||
}
|
||||
|
||||
/// A Python result a file may hold: a scalar, or an array whose size matches its data and has at
|
||||
/// most [`gridcraft_model::python::MAX_RESULT_CELLS`] cells.
|
||||
fn sane_python_output(v: &gridcraft_core::Value) -> bool {
|
||||
match v {
|
||||
gridcraft_core::Value::Array(a) => {
|
||||
a.rows >= 1
|
||||
&& a.cols >= 1
|
||||
&& a.rows.checked_mul(a.cols).is_some_and(|n| n == a.data.len() && n <= gridcraft_model::python::MAX_RESULT_CELLS)
|
||||
}
|
||||
_ => true,
|
||||
}
|
||||
}
|
||||
|
||||
/// Parses bytes that aren't encrypted, as they are.
|
||||
fn read_plain(name: &str, bytes: &[u8], csv_as_text: bool) -> Result<(Workbook, Vec<String>)> {
|
||||
let kind = FileKind::from_path(name);
|
||||
let sniffed = gridcraft_xlsx::sniff(bytes);
|
||||
if sniffed == gridcraft_xlsx::Format::Xls {
|
||||
|
||||
@@ -5095,6 +5095,7 @@ fn a_script_cannot_switch_windows() {
|
||||
|
||||
mod python_cells;
|
||||
mod python_commands;
|
||||
mod python_files;
|
||||
mod seams;
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -0,0 +1,441 @@
|
||||
//! Python cells in files: native JSON, and xlsx as `_xlfn._xlws.PY(index,0|1)` with the code in
|
||||
//! the Python part ([MS-XLSX] 2.1.23, 2.3.10) and cached values. Nothing here runs Python: results
|
||||
//! are cached ones, as a file would hold them.
|
||||
|
||||
use std::io::{Read, Write};
|
||||
use std::sync::Arc;
|
||||
|
||||
use gridcraft_core::{Array, CellError, CellRef, RangeRef, Value};
|
||||
use gridcraft_model::{Cell, PyCode, PyOutput, Workbook};
|
||||
use serde_json::json;
|
||||
|
||||
use super::{s, v};
|
||||
use crate::{Session, Trust};
|
||||
|
||||
fn py(s: &mut Session, cell: &str, code: &str) {
|
||||
s.execute("python.set", json!({"cell": cell, "code": code})).unwrap();
|
||||
}
|
||||
|
||||
/// Gives a Python cell a result, as running it (or a file's cache) would.
|
||||
fn cache(s: &mut Session, cell: &str, value: Value, type_name: &str) {
|
||||
let d = s.doc_mut().unwrap();
|
||||
let wb = Arc::make_mut(&mut d.wb);
|
||||
let sh = wb.sheet_mut(0).unwrap();
|
||||
let c = sh.cells.get_mut(CellRef::parse(cell).unwrap()).unwrap();
|
||||
let p = Arc::make_mut(c.py.as_mut().unwrap());
|
||||
p.output = Some(PyOutput { value, type_name: type_name.into(), preview: String::new(), convertible: true });
|
||||
}
|
||||
|
||||
fn array(rows: usize, cols: usize, data: Vec<Value>) -> Value {
|
||||
Value::Array(Arc::new(Array::new(rows, cols, data).unwrap()))
|
||||
}
|
||||
|
||||
fn save(s: &mut Session, format: &str) -> Vec<u8> {
|
||||
let r = s.execute("file.saveBytes", json!({"format": format})).unwrap();
|
||||
crate::io::base64_decode(r["base64"].as_str().unwrap()).unwrap()
|
||||
}
|
||||
|
||||
fn open(s: &mut Session, name: &str, bytes: &[u8]) {
|
||||
s.execute("file.open", json!({"name": name, "base64": crate::io::base64_encode(bytes)})).unwrap();
|
||||
}
|
||||
|
||||
fn part(bytes: &[u8], name: &str) -> String {
|
||||
let mut z = zip::ZipArchive::new(std::io::Cursor::new(bytes)).unwrap();
|
||||
let mut xml = String::new();
|
||||
z.by_name(name).unwrap().read_to_string(&mut xml).unwrap();
|
||||
xml
|
||||
}
|
||||
|
||||
fn sheet_xml(bytes: &[u8]) -> String {
|
||||
part(bytes, "xl/worksheets/sheet1.xml")
|
||||
}
|
||||
|
||||
/// The package with `edit` applied to each XML part (by name), and `add` added.
|
||||
fn with_parts(bytes: &[u8], edit: impl Fn(&str, &str) -> String, add: &[(&str, &str)]) -> Vec<u8> {
|
||||
let mut z = zip::ZipArchive::new(std::io::Cursor::new(bytes)).unwrap();
|
||||
let mut out = zip::ZipWriter::new(std::io::Cursor::new(Vec::new()));
|
||||
for i in 0..z.len() {
|
||||
let mut f = z.by_index(i).unwrap();
|
||||
let name = f.name().to_string();
|
||||
let mut data = Vec::new();
|
||||
f.read_to_end(&mut data).unwrap();
|
||||
if let Ok(text) = std::str::from_utf8(&data) {
|
||||
data = edit(&name, text).into_bytes();
|
||||
}
|
||||
out.start_file(name, zip::write::SimpleFileOptions::default()).unwrap();
|
||||
out.write_all(&data).unwrap();
|
||||
}
|
||||
for (name, text) in add {
|
||||
out.start_file(*name, zip::write::SimpleFileOptions::default()).unwrap();
|
||||
out.write_all(text.as_bytes()).unwrap();
|
||||
}
|
||||
out.finish().unwrap().into_inner()
|
||||
}
|
||||
|
||||
/// The package with `edit` applied to the first sheet's XML.
|
||||
fn with_sheet_xml(bytes: &[u8], edit: impl Fn(&str) -> String) -> Vec<u8> {
|
||||
with_parts(bytes, |name, text| if name == "xl/worksheets/sheet1.xml" { edit(text) } else { text.to_string() }, &[])
|
||||
}
|
||||
|
||||
/// `xml` with its `<sheetData>` replaced by `data`.
|
||||
fn replace_sheet_data(xml: &str, data: &str) -> String {
|
||||
match (xml.find("<sheetData"), xml.find("</sheetData>"), xml.find("<sheetData/>")) {
|
||||
(Some(a), Some(b), _) => format!("{}{data}{}", &xml[..a], &xml[b + "</sheetData>".len()..]),
|
||||
(_, _, Some(a)) => format!("{}{data}{}", &xml[..a], &xml[a + "<sheetData/>".len()..]),
|
||||
_ => panic!("no sheetData in {xml}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// An empty workbook's package with `data` as its first sheet's cells and, when `scripts` is
|
||||
/// given, a Python part holding them (the relationship and the content type included), as
|
||||
/// [MS-XLSX] 2.1.23 describes it.
|
||||
fn excel_shaped(data: &str, scripts: Option<&[&str]>) -> Vec<u8> {
|
||||
let mut s = s();
|
||||
let plain = save(&mut s, "xlsx");
|
||||
let Some(scripts) = scripts else { return with_sheet_xml(&plain, |xml| replace_sheet_data(xml, data)) };
|
||||
let codes: String = scripts.iter().map(|c| format!("<pythonScript><code>{c}</code></pythonScript>")).collect();
|
||||
let python = format!(
|
||||
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<python xmlns=\"http://schemas.microsoft.com/office/spreadsheetml/2023/python\"><environmentDefinition id=\"{{00000000-0000-0000-0000-000000000000}}\"/><pythonScripts>{codes}</pythonScripts></python>"
|
||||
);
|
||||
const REL: &str =
|
||||
"<Relationship Id=\"rIdPy\" Type=\"http://schemas.microsoft.com/office/2023/09/relationships/Python\" Target=\"python/python1.xml\"/>";
|
||||
const TYPE: &str = "<Override PartName=\"/xl/python/python1.xml\" ContentType=\"application/vnd.ms-excel.python+xml\"/>";
|
||||
let edit = |name: &str, text: &str| match name {
|
||||
"xl/worksheets/sheet1.xml" => replace_sheet_data(text, data),
|
||||
"xl/_rels/workbook.xml.rels" => text.replace("</Relationships>", &format!("{REL}</Relationships>")),
|
||||
"[Content_Types].xml" => text.replace("</Types>", &format!("{TYPE}</Types>")),
|
||||
_ => text.to_string(),
|
||||
};
|
||||
with_parts(&plain, edit, &[("xl/python/python1.xml", &python)])
|
||||
}
|
||||
|
||||
fn code(s: &Session, cell: &str) -> Option<PyCode> {
|
||||
s.doc().unwrap().wb.active().unwrap().cell(CellRef::parse(cell).unwrap()).and_then(|c| c.py.as_deref()).cloned()
|
||||
}
|
||||
|
||||
/// A workbook with a value, a formula over a Python cell, a spilling table, an object, long code
|
||||
/// with quotes and line breaks, and an error.
|
||||
fn build() -> (Session, String) {
|
||||
let mut s = s();
|
||||
s.execute("cell.set", json!({"cell": "A1", "input": "21"})).unwrap();
|
||||
py(&mut s, "B1", "xl('A1') * 2");
|
||||
cache(&mut s, "B1", Value::Number(42.0), "int");
|
||||
s.execute("cell.set", json!({"cell": "A5", "input": "=B1+1"})).unwrap();
|
||||
py(&mut s, "C1", "[[1, 'two'], [3, \"four\"]]");
|
||||
cache(&mut s, "C1", array(2, 2, vec![1.0.into(), Value::text("two"), 3.0.into(), Value::text("four")]), "list");
|
||||
s.execute("python.set", json!({"cell": "E1", "code": "[1, 2]", "output": "object"})).unwrap();
|
||||
cache(&mut s, "E1", array(2, 1, vec![1.0.into(), 2.0.into()]), "list");
|
||||
let long = format!("# {}\ntext = \"say \\\"hi\\\"\"\nlen(text)", "x".repeat(7_900));
|
||||
assert!(PyCode::new(&long).fits());
|
||||
py(&mut s, "F1", &long);
|
||||
cache(&mut s, "F1", Value::Number(8.0), "int");
|
||||
py(&mut s, "G1", "1/0");
|
||||
{
|
||||
let d = s.doc_mut().unwrap();
|
||||
let sh = Arc::make_mut(&mut d.wb).sheet_mut(0).unwrap();
|
||||
sh.cells.get_mut(CellRef::parse("G1").unwrap()).unwrap().value = Value::Error(CellError::Python);
|
||||
}
|
||||
s.execute("formulas.calculateNow", json!({})).unwrap();
|
||||
assert_eq!(v(&s, "B1"), Value::Number(42.0));
|
||||
assert_eq!(v(&s, "A5"), Value::Number(43.0));
|
||||
assert_eq!(v(&s, "D2"), Value::text("four"));
|
||||
assert_eq!(v(&s, "E1"), Value::text("list"), "a Python object shows its type");
|
||||
assert_eq!(v(&s, "E2"), Value::Empty, "and doesn't spill");
|
||||
(s, long)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn xlsx_stores_the_code_in_the_python_part_with_cached_values() {
|
||||
let (mut s, long) = build();
|
||||
let bytes = save(&mut s, "xlsx");
|
||||
let xml = sheet_xml(&bytes);
|
||||
// The formula names the cell's script by index, in the order the cells are written: B1, C1,
|
||||
// E1, F1, G1.
|
||||
assert!(xml.contains("<f>_xlfn._xlws.PY(0,0)</f><v>42</v>"), "{xml}");
|
||||
assert!(xml.contains(">_xlfn._xlws.PY(2,1)</f>"), "the Python object cell has return type 1: {xml}");
|
||||
assert!(!xml.contains("xl('A1')"), "the code is not in the sheet");
|
||||
let python = part(&bytes, "xl/python.xml");
|
||||
assert!(
|
||||
python.contains("<python xmlns=\"http://schemas.microsoft.com/office/spreadsheetml/2023/python\"><pythonScripts><pythonScript><code>xl('A1') * 2</code></pythonScript>"),
|
||||
"{python}"
|
||||
);
|
||||
assert_eq!(python.matches("<pythonScript>").count(), 5);
|
||||
assert!(
|
||||
part(&bytes, "[Content_Types].xml").contains("<Override PartName=\"/xl/python.xml\" ContentType=\"application/vnd.ms-excel.python+xml\"/>")
|
||||
);
|
||||
assert!(
|
||||
part(&bytes, "xl/_rels/workbook.xml.rels")
|
||||
.contains("Type=\"http://schemas.microsoft.com/office/2023/09/relationships/Python\" Target=\"python.xml\"")
|
||||
);
|
||||
assert!(xml.contains("t=\"array\" ref=\"C1:D2\""), "the spill is saved like a dynamic array's: {xml}");
|
||||
assert!(xml.contains("#VALUE!") && !xml.contains("#PYTHON!"), "Python errors are cached as #VALUE!");
|
||||
|
||||
open(&mut s, "book.xlsx", &bytes);
|
||||
assert_eq!(s.doc().unwrap().trust, Trust::FromFile);
|
||||
// Cached values show, spills included, and the code is back.
|
||||
assert_eq!(v(&s, "B1"), Value::Number(42.0));
|
||||
assert_eq!(v(&s, "A5"), Value::Number(43.0));
|
||||
assert_eq!(
|
||||
(v(&s, "C1"), v(&s, "D1"), v(&s, "C2"), v(&s, "D2")),
|
||||
(Value::Number(1.0), Value::text("two"), Value::Number(3.0), Value::text("four"))
|
||||
);
|
||||
assert_eq!(code(&s, "B1").unwrap().code, "xl('A1') * 2");
|
||||
assert!(code(&s, "E1").unwrap().object);
|
||||
assert_eq!(code(&s, "F1").unwrap().code, long, "long code with quotes and line breaks survives");
|
||||
assert_eq!(v(&s, "G1"), Value::Error(CellError::Value));
|
||||
// A blocker typed into the spill area gives #SPILL!; clearing it spills again, without
|
||||
// running anything.
|
||||
s.execute("cell.set", json!({"cell": "D2", "input": "x"})).unwrap();
|
||||
assert_eq!(v(&s, "C1"), Value::Error(CellError::Spill));
|
||||
s.execute("edit.clearContents", json!({"range": "D2"})).unwrap();
|
||||
assert_eq!(v(&s, "D2"), Value::text("four"));
|
||||
// Saved again, the file holds the same formulas and scripts.
|
||||
let again = save(&mut s, "xlsx");
|
||||
assert!(sheet_xml(&again).contains("<f>_xlfn._xlws.PY(0,0)</f>"));
|
||||
assert_eq!(part(&again, "xl/python.xml"), python);
|
||||
// A workbook without Python cells has no Python part.
|
||||
let mut plain = super::s();
|
||||
assert!(!save(&mut plain, "xlsx").windows(10).any(|w| w == b"python.xml"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn excel_authored_python_cells_are_read_as_python() {
|
||||
// A workbook as [MS-XLSX] 2.3.10 describes it: `_xlfn._xlws.PY(index,0|1,params…)` naming a
|
||||
// script of the Python part, whose code names the parameters as `%Px%`, with a spilled result
|
||||
// cached.
|
||||
let data = "<sheetData><row r=\"1\"><c r=\"A1\" cm=\"1\"><f t=\"array\" ref=\"A1:A2\">_xlfn._xlws.PY(0,0)</f><v>1</v></c>\
|
||||
<c r=\"B1\" t=\"str\"><f>_xlfn._xlws.PY(1,1,C1:C3)</f><v>DataFrame</v></c></row>\
|
||||
<row r=\"2\"><c r=\"A2\"><v>2</v></c></row></sheetData>";
|
||||
let bytes = excel_shaped(data, Some(&["[1, 2]", "df = xl(\"%P2%\", headers=True)\ndf.describe()"]));
|
||||
let mut s = s();
|
||||
open(&mut s, "excel.xlsx", &bytes);
|
||||
let a1 = code(&s, "A1").expect("an Excel Python cell is a Python cell");
|
||||
assert_eq!(a1.code, "[1, 2]");
|
||||
assert!(!a1.object);
|
||||
assert_eq!((v(&s, "A1"), v(&s, "A2")), (Value::Number(1.0), Value::Number(2.0)), "its spilled values are kept");
|
||||
let cell = s.doc().unwrap().wb.active().unwrap().cell(CellRef::new(0, 0)).unwrap().clone();
|
||||
assert!(cell.formula.is_none());
|
||||
let b1 = code(&s, "B1").unwrap();
|
||||
assert_eq!(b1.code, "df = xl(\"C1:C3\", headers=True)\ndf.describe()", "a parameter is put back as its reference");
|
||||
assert!(b1.object);
|
||||
assert_eq!(v(&s, "B1"), Value::text("DataFrame"));
|
||||
// Saved by GridCraft, the code goes back into a Python part, the reference as text.
|
||||
let again = save(&mut s, "xlsx");
|
||||
assert!(part(&again, "xl/python.xml").contains("<code>df = xl(\"C1:C3\", headers=True)\ndf.describe()</code>"));
|
||||
// The string-literal form (other applications, files GridCraft wrote before) still reads.
|
||||
let literal = "<sheetData><row r=\"1\"><c r=\"A1\"><f>_xlfn._xlws.PY(\"1 + 1\",1)</f><v>2</v></c></row></sheetData>";
|
||||
open(&mut s, "literal.xlsx", &excel_shaped(literal, None));
|
||||
assert_eq!(code(&s, "A1").map(|p| (p.code, p.object)), Some(("1 + 1".to_string(), true)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_py_formula_naming_a_missing_script_keeps_only_its_value() {
|
||||
let data = "<sheetData><row r=\"1\"><c r=\"A1\"><f>_xlfn._xlws.PY(0,0)</f><v>5</v></c><c r=\"B1\"><f>_xlfn._xlws.PY(3,0)</f><v>6</v></c></row></sheetData>";
|
||||
for (name, bytes) in [("one.xlsx", excel_shaped(data, Some(&["5"]))), ("none.xlsx", excel_shaped(data, None))] {
|
||||
let (wb, warnings) = crate::io::open_bytes(name, &bytes).unwrap();
|
||||
let sh = wb.sheet(0).unwrap();
|
||||
let b1 = sh.cell(CellRef::parse("B1").unwrap()).unwrap();
|
||||
assert!(b1.py.is_none() && b1.formula.is_none(), "{name}: no code to keep, and no formula #NAME? would replace");
|
||||
assert_eq!(b1.value, Value::Number(6.0), "{name}");
|
||||
assert!(warnings.iter().any(|w| w.contains("Python")), "{name}: {warnings:?}");
|
||||
}
|
||||
let (wb, _) = crate::io::open_bytes("one.xlsx", &excel_shaped(data, Some(&["5"]))).unwrap();
|
||||
assert_eq!(wb.sheet(0).unwrap().cell(CellRef::parse("A1").unwrap()).unwrap().py.as_ref().map(|p| p.code.as_str()), Some("5"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn many_cells_naming_one_long_script_take_from_one_budget() {
|
||||
// 20,000 cells name one script of 8,000 characters: 160 MB of code if each took a copy.
|
||||
let long = "x".repeat(8_000);
|
||||
let mut data = String::from("<sheetData>");
|
||||
for r in 1..=20_000 {
|
||||
data.push_str(&format!("<row r=\"{r}\"><c r=\"A{r}\"><f>_xlfn._xlws.PY(0,0)</f><v>1</v></c></row>"));
|
||||
}
|
||||
data.push_str("</sheetData>");
|
||||
let (wb, warnings) = crate::io::open_bytes("many.xlsx", &excel_shaped(&data, Some(&[&long]))).unwrap();
|
||||
let sh = wb.sheet(0).unwrap();
|
||||
let with_code = sh.cells.iter().filter(|(_, c)| c.py.is_some()).count();
|
||||
assert_eq!(with_code, 64 * 1024 * 1024 / 8_000);
|
||||
assert!(warnings.iter().any(|w| w.contains("Python")), "{warnings:?}");
|
||||
assert_eq!(sh.value(CellRef::parse("A20000").unwrap()), Value::Number(1.0), "the others keep their values");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_json_keeps_python_cells_and_their_results() {
|
||||
let (mut s, long) = build();
|
||||
let bytes = save(&mut s, "json");
|
||||
open(&mut s, "book.json", &bytes);
|
||||
assert_eq!(s.doc().unwrap().trust, Trust::FromFile);
|
||||
assert_eq!(v(&s, "B1"), Value::Number(42.0));
|
||||
assert_eq!(v(&s, "D2"), Value::text("four"));
|
||||
assert_eq!(v(&s, "G1"), Value::Error(CellError::Python), "the native format keeps the exact error");
|
||||
assert_eq!(code(&s, "F1").unwrap().code, long);
|
||||
let e1 = code(&s, "E1").unwrap();
|
||||
assert!(e1.object);
|
||||
assert_eq!(e1.output.unwrap().type_name, "list");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_spill_range_claiming_the_whole_sheet_is_dropped_on_open() {
|
||||
let (mut s, _) = build();
|
||||
let bytes = save(&mut s, "xlsx");
|
||||
let evil = with_sheet_xml(&bytes, |xml| xml.replace("ref=\"C1:D2\"", "ref=\"C1:XFD1048576\""));
|
||||
assert_ne!(evil, bytes);
|
||||
// Walking the claimed range would take hours; the test finishing is the check.
|
||||
open(&mut s, "evil.xlsx", &evil);
|
||||
let sh = s.doc().unwrap().wb.active().unwrap();
|
||||
assert!(sh.spill_ranges.values().all(|r| r.count() <= gridcraft_model::python::MAX_RESULT_CELLS as u64), "{:?}", sh.spill_ranges);
|
||||
// Edits afterwards don't walk the claimed range either.
|
||||
s.execute("cell.set", json!({"cell": "H9", "input": "1"})).unwrap();
|
||||
s.execute("formulas.calculateNow", json!({})).unwrap();
|
||||
assert_eq!(v(&s, "C1"), Value::Number(1.0), "the anchor keeps its cached value");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rebuilt_results_have_a_budget_across_the_workbook() {
|
||||
// Five Python cells each claiming a 1,000 x 1,000 spill: more than the budget in all.
|
||||
let mut wb = Workbook::new();
|
||||
let sh = wb.sheet_mut(0).unwrap();
|
||||
for i in 0..5u32 {
|
||||
let anchor = CellRef::new(0, i * 1_000);
|
||||
sh.set_cell(anchor, Cell { py: Some(Arc::new(PyCode::new("x"))), ..Cell::value(Value::Number(1.0)) });
|
||||
sh.spill_ranges.insert(anchor, RangeRef::new(anchor, CellRef::new(999, i * 1_000 + 999)));
|
||||
}
|
||||
let warnings = crate::io::adopt_python(&mut wb);
|
||||
let sh = wb.sheet(0).unwrap();
|
||||
let rebuilt = sh.cells.iter().filter(|(_, c)| c.py.as_ref().is_some_and(|p| p.output.is_some())).count();
|
||||
assert_eq!(rebuilt as u64, crate::io::MAX_ADOPTED_RESULT_CELLS / 1_000_000);
|
||||
assert!(warnings.iter().any(|w| w.contains("Python")), "{warnings:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_python_content_in_files_is_capped() {
|
||||
let (mut s, _) = build();
|
||||
let bytes = save(&mut s, "json");
|
||||
// Make the Python cells hostile: code over the limit, and a result whose shape claims far
|
||||
// more cells than it holds (the array fields are public, so a crafted file can say anything).
|
||||
let mut wb: Workbook = serde_json::from_slice(&bytes).unwrap();
|
||||
let sh = wb.sheet_mut(0).unwrap();
|
||||
let b1 = sh.cells.get_mut(CellRef::parse("B1").unwrap()).unwrap();
|
||||
Arc::make_mut(b1.py.as_mut().unwrap()).code = "y".repeat(9_000);
|
||||
let c1 = sh.cells.get_mut(CellRef::parse("C1").unwrap()).unwrap();
|
||||
let out = Arc::make_mut(c1.py.as_mut().unwrap()).output.as_mut().unwrap();
|
||||
out.value = Value::Array(Arc::new(Array { rows: 100_000, cols: 100_000, data: vec![Value::Number(1.0)] }));
|
||||
let bytes = serde_json::to_vec(&wb).unwrap();
|
||||
let (wb, warnings) = crate::io::open_bytes("evil.json", &bytes).unwrap();
|
||||
let sh = wb.sheet(0).unwrap();
|
||||
let b1 = sh.cell(CellRef::parse("B1").unwrap()).unwrap();
|
||||
assert!(b1.py.is_none(), "code over the formula limit is dropped");
|
||||
assert_eq!(b1.value, Value::Number(42.0), "the cached value stays");
|
||||
assert!(warnings.iter().any(|w| w.contains("Python cell")), "{warnings:?}");
|
||||
let c1 = sh.cell(CellRef::parse("C1").unwrap()).unwrap();
|
||||
assert!(c1.py.as_ref().unwrap().output.is_none(), "a malformed result is dropped");
|
||||
open(&mut s, "evil.json", &bytes);
|
||||
s.execute("formulas.calculateNow", json!({})).unwrap();
|
||||
assert_eq!(v(&s, "C1"), Value::Number(1.0));
|
||||
|
||||
// PY formulas from other formats become Python cells; non-literal PY calls stay formulas.
|
||||
let mut wb = Workbook::new();
|
||||
let sh = wb.sheet_mut(0).unwrap();
|
||||
sh.set_cell(CellRef::new(0, 0), Cell::formula(gridcraft_model::Formula::new("=PY(\"1+\"&\"1\",1)")));
|
||||
sh.set_cell(CellRef::new(1, 0), Cell::formula(gridcraft_model::Formula::new("=PY(A1,0)")));
|
||||
sh.set_cell(CellRef::new(2, 0), Cell::formula(gridcraft_model::Formula::new("=_xlfn._xlws.PY(\"2\",0)")));
|
||||
crate::io::adopt_python(&mut wb);
|
||||
let sh = wb.sheet(0).unwrap();
|
||||
let a1 = sh.cell(CellRef::new(0, 0)).unwrap();
|
||||
assert_eq!(a1.py.as_ref().map(|p| (p.code.as_str(), p.object)), Some(("1+1", true)));
|
||||
assert!(a1.formula.is_none());
|
||||
assert!(sh.cell(CellRef::new(1, 0)).unwrap().py.is_none());
|
||||
assert_eq!(sh.cell(CellRef::new(2, 0)).unwrap().py.as_ref().map(|p| p.code.as_str()), Some("2"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_malformed_result_never_spills() {
|
||||
let mut s = s();
|
||||
py(&mut s, "A1", "x");
|
||||
let bad = Array { rows: 4_000_000_000, cols: 4_000_000_000, data: vec![] };
|
||||
cache(&mut s, "A1", Value::Array(Arc::new(bad)), "list");
|
||||
s.execute("formulas.calculateNow", json!({})).unwrap();
|
||||
assert_eq!(v(&s, "A1"), Value::Error(CellError::Spill));
|
||||
assert!(s.doc().unwrap().wb.active().unwrap().spill.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn py_formulas_over_the_size_limit_keep_only_their_value() {
|
||||
let lits = |n: usize| {
|
||||
let mut v = vec![format!("\"{}\"", "x".repeat(255)); n / 255];
|
||||
if !n.is_multiple_of(255) {
|
||||
v.push(format!("\"{}\"", "x".repeat(n % 255)));
|
||||
}
|
||||
v.join("&")
|
||||
};
|
||||
let mut wb = Workbook::new();
|
||||
let sh = wb.sheet_mut(0).unwrap();
|
||||
let cell = |text: String| Cell { value: Value::text("cached"), ..Cell::formula(gridcraft_model::Formula::new(&text)) };
|
||||
sh.set_cell(CellRef::new(0, 0), cell(format!("_xlfn._xlws.PY({},0)", lits(8_080))));
|
||||
sh.set_cell(CellRef::new(1, 0), cell(format!("_xlfn._xlws.PY({},0)", lits(8_079))));
|
||||
let xlsx = gridcraft_xlsx::write_xlsx(&wb).unwrap();
|
||||
let (wb, warnings) = crate::io::open_bytes("long.xlsx", &xlsx).unwrap();
|
||||
let sh = wb.sheet(0).unwrap();
|
||||
let over = sh.cell(CellRef::new(0, 0)).unwrap();
|
||||
assert!(over.py.is_none() && over.formula.is_none(), "8,080 characters of code is over the limit");
|
||||
assert_eq!(over.value, Value::text("cached"));
|
||||
let within = sh.cell(CellRef::new(1, 0)).unwrap();
|
||||
assert_eq!(within.py.as_ref().map(|p| p.code.chars().count()), Some(8_079));
|
||||
assert!(warnings.iter().any(|w| w.contains("A1")), "{warnings:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_dropped_python_result_takes_its_spill_with_it() {
|
||||
// A1's code is over the limit and B1's result is malformed; each had spilled down to row 3,
|
||||
// and C1's well-formed result keeps its spill.
|
||||
let mut wb = Workbook::new();
|
||||
let sh = wb.sheet_mut(0).unwrap();
|
||||
let out = |value: Value| Some(PyOutput { value, type_name: "list".into(), preview: String::new(), convertible: true });
|
||||
let col = array(3, 1, vec![Value::Number(7.0), Value::Number(8.0), Value::Number(9.0)]);
|
||||
let bad = Value::Array(Arc::new(Array { rows: 3, cols: 1, data: vec![Value::Number(7.0)] }));
|
||||
for (at, code, value) in [("A1", "y".repeat(9_000), col.clone()), ("B1", "b".into(), bad), ("C1", "c".into(), col)] {
|
||||
let at = CellRef::parse(at).unwrap();
|
||||
let py = PyCode { output: out(value), ..PyCode::new(&code) };
|
||||
sh.set_cell(at, Cell { value: Value::Number(7.0), formula: None, style: Default::default(), py: Some(Arc::new(py)) });
|
||||
let r = RangeRef::new(at, CellRef::new(2, at.col));
|
||||
sh.spill_ranges.insert(at, r);
|
||||
for c in r.iter().skip(1) {
|
||||
sh.spill.insert(c, Value::Number(8.0));
|
||||
}
|
||||
}
|
||||
let warnings = crate::io::enforce_python_limits(&mut wb);
|
||||
assert!(warnings.iter().any(|w| w.contains("A1")), "{warnings:?}");
|
||||
let sh = wb.sheet(0).unwrap();
|
||||
for (anchor, below) in [("A1", "A2"), ("B1", "B3")] {
|
||||
assert!(!sh.spill_ranges.contains_key(&CellRef::parse(anchor).unwrap()), "{anchor}'s spill range stays");
|
||||
assert_eq!(sh.value(CellRef::parse(below).unwrap()), Value::Empty, "{below} still shows {anchor}'s old spill");
|
||||
assert_eq!(sh.value(CellRef::parse(anchor).unwrap()), Value::Number(7.0), "{anchor} keeps its value");
|
||||
}
|
||||
assert_eq!(sh.value(CellRef::parse("C2").unwrap()), Value::Number(8.0), "a sound result keeps its spill");
|
||||
// A save writes no value for the dropped spills.
|
||||
let xml = sheet_xml(&gridcraft_xlsx::write_xlsx(&wb).unwrap());
|
||||
assert!(!xml.contains("r=\"A2\"") && !xml.contains("r=\"B3\""), "{xml}");
|
||||
assert!(xml.contains("r=\"C2\""), "{xml}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_formula_entered_in_a_python_spill_takes_the_area_as_a_formula_would() {
|
||||
// A1's cached array spills over A1:A3; =SEQUENCE(2) entered in A2 is not blocked by that old
|
||||
// spill (the Python cell is laid out again in the same pass), so it spills and A1 is blocked.
|
||||
let mut s = s();
|
||||
py(&mut s, "A1", "[1, 2, 3]");
|
||||
cache(&mut s, "A1", array(3, 1, vec![10.0.into(), 20.0.into(), 30.0.into()]), "list");
|
||||
s.execute("formulas.calculateNow", json!({})).unwrap();
|
||||
assert_eq!(v(&s, "A3"), Value::Number(30.0));
|
||||
s.execute("cell.set", json!({"cell": "A2", "input": "=SEQUENCE(2)"})).unwrap();
|
||||
assert_eq!((v(&s, "A2"), v(&s, "A3")), (Value::Number(1.0), Value::Number(2.0)));
|
||||
assert_eq!(v(&s, "A1"), Value::Error(CellError::Spill));
|
||||
let wb = &s.doc().unwrap().wb;
|
||||
let mut fresh = (**wb).clone();
|
||||
gridcraft_calc::Calc::new().recalc_all(&mut fresh);
|
||||
for c in ["A1", "A2", "A3"] {
|
||||
let c = CellRef::parse(c).unwrap();
|
||||
assert_eq!(wb.active().unwrap().value(c), fresh.active().unwrap().value(c), "{c}");
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user