Compare commits

..
Author SHA1 Message Date
Dylan M. Taylor bc2dabc7ea FORMULATEXT and the in-cell editor keep Hidden formulas hidden (#3298)
* FORMULATEXT and the cell editor keep a hidden formula's source hidden

* address review: recalculate formulas when sheet protection changes

* address review: skip the full recalculation for a no-op unprotect

* address review: recalculate on protect/unprotect without marking the edit structural
2026-11-11 05:48:31 -05:00
Dylan M. Taylor e14f783503 Analysis ToolPak: stop building a report once it outgrows the output limit (#3207)
* Analysis ToolPak: stop building a report once it outgrows the output limit

Report builders filled their whole output map before write_report
checked its size, so Descriptive Statistics grouped by rows over a
tall, nearly empty column allocated about 16 entries for each of up
to a million variables before refusing the report.

Grid now tracks its extent as values are stored and keeps nothing
once the report's rows x columns pass MAX_OUTPUT_CELLS; write_report
refuses such a report. extent() no longer rescans every entry.

* address review: stop the descriptive builder once the grid is too large

* address review: descriptive statistics reads variables one at a time
2026-11-11 05:40:31 -05:00
Dylan M. Taylor 542aa176ce Wildcards: match * and ? patterns in one pass over the text (#3205)
* Wildcards: match * and ? patterns in one pass over the text

After a mismatch the matcher restarted the pattern after the last
star at the next text position, so a long repetitive cell against a
near-matching pattern cost text x pattern comparisons. AutoFilter
custom criteria, Find and Replace all call it once per cell.

wildcard() now runs the pattern as a bit-parallel automaton: one pass
over the text, one step per 64 pattern positions for each character.
Results are unchanged; a test compares it with the old matcher on
every short text and pattern over a small alphabet.

* address review: build wildcard literal masks lazily with a bounded cache

* address review: keep masks of the most frequent wildcard literals past the cache cap
2026-11-11 05:19:24 -05:00
Dylan M. Taylor 67a0b7f325 Security: chart rasterising, spline points and category labels stay bounded (#3204)
* chart: bound scan conversion, spline points and category label bytes

- The rasteriser tested every edge of a path at every sub-sample of every
  covered row. It now keeps an active-edge list (edges sorted by top, joining
  and leaving as the scan passes them), and one rasterize call has a total
  scan-work allowance after which remaining fills are skipped.
- Smoothed lines budgeted interpolated points per run, so gap-separated short
  runs escaped both decimation and the cap. The budget is now shared by all
  runs of a line; once spent, runs are drawn through their data points.
- Category labels copied every text value in full. Labels are clipped to 255
  characters before copying, within an 8 MiB budget per series.

* address review: chart raster and label budget fixes

* address review: bound polyline point count before stroke_paths
2026-11-11 05:07:04 -05:00
Dylan M. Taylor bbe0825404 Security: TEXTJOIN builds its result piece by piece within the text cap (#3202)
* functions: TEXTJOIN builds its result piece by piece within the text cap

TEXTJOIN converted every input and delimiter into an owned String before
joining, so many references to one long shared text copied gigabytes before
the joined length was ever checked. Errors are now found first without
building text (so they still win in argument order), then each piece and
delimiter is converted only as it is appended, stopping at the cap.

* address review: check the cap before copying each text piece
2026-11-11 04:55:21 -05:00
Dylan M. Taylor aa4951fdf4 Security: WORKDAY, PIVOTBY, SORT and the D-functions stop rescanning their input (#3200)
* functions: stop WORKDAY, PIVOTBY, SORT and the D-functions rescanning their input

- WORKDAY/WORKDAY.INTL counted the holidays each step passed by scanning every
  holiday, once per step; a run of consecutive holidays made that quadratic.
  The working-day holidays are kept sorted and counted by binary search.
- PIVOTBY totals rescanned the whole source (rebuilding keys) once per total
  row and column. The grouping pass now also indexes rows by row key and by
  column key, and totals read from those indexes in the same order.
- SORT copied one key per sort_index entry for every row, so a long
  sort_index with repeated columns multiplied the source. Repeats can't break
  a tie their first use left, so only first uses are kept.
- DSUM and the other database functions tested every record against every
  criteria row. A selection whose records x conditions exceed 5e8 is #NUM!,
  and each criteria label's database column is found once, not per cell.

* address review: bounds-check SORT indices before narrowing; empty criteria rows are not charged
2026-11-11 04:36:48 -05:00
Dylan M. Taylor 79fe7b8eb6 Forecast Sheet: count hindcast rows against the work limit (#3199)
* Forecast Sheet: count hindcast rows against the work limit

MAX_WORK bounded history rows x forecast rows, but a Forecast Start
inside the history adds a hindcast row for every history point after
it, each holding the same ETS formulas that refit the model over the
history before the start. Those rows were not counted, so a start in
the middle of 10,000 rows of history built millions of uncharged
model fits per recalculation.

The forecast row cap is now computed from the rows the model is fit
to, after subtracting the hindcast rows; a start so early that the
hindcast alone passes the limit is refused, and the default number of
periods stays within the cap.

* address review: base the in-history forecast cap on the slots the fit range spans

* address review: document the hindcast-aware forecast work limit

* address review: document the fitted-row minimum in the forecast work limit

* address review: say the forecast start row counts toward the hindcast work limit
2026-11-11 04:22:54 -05:00
24 changed files with 785 additions and 205 deletions
+6 -3
View File
@@ -52,13 +52,16 @@ impl<'h> Evaluator<'h> {
"ISFORMULA" | "FORMULATEXT" => {
let Some(Ev::R(a)) = args.first().map(|a| self.eval(a)) else { return err(CellError::NA) };
let Some(a) = a.first() else { return err(CellError::NA) };
let f = self.host.workbook().sheet(a.sheet).and_then(|s| s.cell(a.range.start)).and_then(|c| c.formula.clone());
let wb = self.host.workbook();
let f = wb.sheet(a.sheet).and_then(|s| s.cell(a.range.start)).and_then(|c| c.formula.clone());
// A formula formatted Hidden on a protected sheet stays hidden from FORMULATEXT too.
let hidden = wb.sheet(a.sheet).is_some_and(|s| s.is_protected() && wb.styles.get(s.style_id(a.range.start)).protection.hidden);
if name == "ISFORMULA" {
Ev::V(Value::Bool(f.is_some()))
} else {
match f {
Some(f) => Ev::V(Value::text(format!("={}", f.text))),
None => err(CellError::NA),
Some(f) if !hidden => Ev::V(Value::text(format!("={}", f.text))),
_ => err(CellError::NA),
}
}
}
+84 -7
View File
@@ -7,6 +7,20 @@ use crate::{Prim, Rgba};
/// Largest image we allocate (pixels).
const MAX_PIXELS: u64 = 64 * 1024 * 1024;
const SUB: usize = 4;
/// Scan-conversion work one [`rasterize`] call may do: edge tests (an active edge at one
/// sub-sample) plus covered pixels blended. Edges are only tested on the rows they cross, but a
/// caller can still pass enough geometry to keep the rasteriser busy for minutes on a small
/// image; once the allowance is spent the remaining fills are skipped. 2^28 is about a second,
/// far beyond any chart.
const MAX_SCAN_WORK: u64 = 1 << 28;
/// Most points one filled path may have: more is skipped before any edge is built. Chart
/// renderers cap and decimate their paths far below this; it only bounds direct callers.
const MAX_FILL_POINTS: usize = 1 << 22;
/// Most points one stroked polyline may have: more is skipped before any segment path is built
/// (each segment becomes its own quad). Charts hold at most 100,000 points per series and draw far
/// fewer; it only bounds direct callers.
const MAX_STROKE_POINTS: usize = 1 << 17;
type Path = Vec<Vec<[f32; 2]>>;
@@ -15,6 +29,8 @@ struct Canvas {
h: usize,
buf: Vec<u8>,
acc: Vec<f32>,
/// Scan-conversion work left (see [`MAX_SCAN_WORK`]).
work: u64,
}
impl Canvas {
@@ -41,7 +57,12 @@ impl Canvas {
/// Fills a multi-contour path with the even-odd rule.
fn fill(&mut self, path: &Path, color: Rgba) {
let mut edges: Vec<[f32; 4]> = Vec::new();
// Out of work, or more geometry than the allowance could ever scan: no edge list is built.
if self.work == 0 || path.iter().map(Vec::len).fold(0usize, usize::saturating_add) > MAX_FILL_POINTS {
return;
}
// Each edge as [top y, bottom y, x0, y0, x1, y1].
let mut edges: Vec<[f32; 6]> = Vec::new();
let (mut minx, mut miny, mut maxx, mut maxy) = (f32::INFINITY, f32::INFINITY, f32::NEG_INFINITY, f32::NEG_INFINITY);
for c in path {
if c.len() < 3 || !c.iter().all(|p| p[0].is_finite() && p[1].is_finite()) {
@@ -54,7 +75,7 @@ impl Canvas {
miny = miny.min(p[1]);
maxy = maxy.max(p[1]);
if p[1] != q[1] {
edges.push([p[0], p[1], q[0], q[1]]);
edges.push([p[1].min(q[1]), p[1].max(q[1]), p[0], p[1], q[0], q[1]]);
}
}
}
@@ -72,8 +93,28 @@ impl Canvas {
if self.acc.len() < bw + 1 {
self.acc.resize(bw + 1, 0.0);
}
// Active-edge list: edges join when the scan reaches their top and leave past their
// bottom, so each row tests only the edges crossing it instead of every edge.
edges.sort_by(|a, b| a[0].total_cmp(&b[0]));
let mut pending = edges.iter().peekable();
let mut active: Vec<[f32; 6]> = Vec::new();
let mut xs: Vec<f32> = Vec::new();
for py in y0..y1 {
while let Some(e) = pending.next_if(|e| e[0] < py as f32 + 1.0) {
active.push(*e);
}
active.retain(|e| e[1] > py as f32);
if active.is_empty() {
continue;
}
// The row's edge tests and its `bw` accumulator clears and blends, charged before any of
// them, so rows with no coverage still count.
let cost = (active.len() * SUB) as u64 + bw as u64;
if cost > self.work {
self.work = 0;
return;
}
self.work -= cost;
for v in self.acc.iter_mut().take(bw) {
*v = 0.0;
}
@@ -81,10 +122,9 @@ impl Canvas {
for s in 0..SUB {
let sy = py as f32 + (s as f32 + 0.5) / SUB as f32;
xs.clear();
for e in &edges {
let (ya, yb) = if e[1] < e[3] { (e[1], e[3]) } else { (e[3], e[1]) };
if sy >= ya && sy < yb {
xs.push(e[0] + (sy - e[1]) * (e[2] - e[0]) / (e[3] - e[1]));
for e in &active {
if sy >= e[0] && sy < e[1] {
xs.push(e[2] + (sy - e[3]) * (e[4] - e[2]) / (e[5] - e[3]));
}
}
xs.sort_by(f32::total_cmp);
@@ -154,6 +194,9 @@ fn rounded_rect(x: f32, y: f32, w: f32, h: f32, r: f32) -> Vec<[f32; 2]> {
/// Thick polyline → one quad per segment (each a separate path so overlaps don't cancel).
fn stroke_paths(pts: &[[f32; 2]], width: f32, dash: bool, closed: bool) -> Vec<Path> {
if pts.len() > MAX_STROKE_POINTS {
return Vec::new();
}
let width = width.clamp(1.0, 50.0);
let mut segs: Vec<([f32; 2], [f32; 2])> = pts.windows(2).map(|w| (w[0], w[1])).collect();
if closed && let (Some(a), Some(b)) = (pts.last(), pts.first()) {
@@ -212,7 +255,7 @@ pub fn rasterize(prims: &[Prim], w: u32, h: u32, background: Rgba) -> Vec<u8> {
for _ in 0..w * h {
buf.extend_from_slice(&background);
}
let mut cv = Canvas { w, h, buf, acc: Vec::new() };
let mut cv = Canvas { w, h, buf, acc: Vec::new(), work: MAX_SCAN_WORK };
for p in prims {
match p {
Prim::Rect { x, y, w, h, fill, stroke, radius } => {
@@ -274,6 +317,40 @@ pub fn rasterize(prims: &[Prim], w: u32, h: u32, background: Rgba) -> Vec<u8> {
mod tests {
use super::*;
/// Security: each row tests only the edges crossing it. A polygon of 100,000 tiny edges over
/// 4,000 rows tested every edge at every sub-sample of every row (1.6 billion tests).
#[test]
fn many_short_edges_scan_only_their_rows() {
let mut pts: Vec<[f32; 2]> = (0..100_000).map(|i| [if i % 2 == 0 { 0.0 } else { 0.5 }, i as f32 * 0.04]).collect();
pts.push([2.0, 4_000.0]);
pts.push([2.0, 0.0]);
let img = rasterize(&[Prim::Polygon { pts, fill: [255, 0, 0, 255], stroke: None }], 2, 4_000, [0, 0, 0, 0]);
// The right-hand pixel of every row is inside the polygon.
assert_eq!(img.get(4..8), Some(&[255, 0, 0, 255][..]));
assert_eq!(img.get(4 * (2 * 3_999 + 1)..4 * (2 * 3_999 + 2)), Some(&[255, 0, 0, 255][..]));
}
/// Security: scan conversion stops once the call's work allowance is spent.
#[test]
fn fills_stop_when_the_work_allowance_is_spent() {
let square = vec![vec![[0.0, 0.0], [4.0, 0.0], [4.0, 4.0], [0.0, 4.0]]];
let mut cv = Canvas { w: 4, h: 4, buf: vec![0; 64], acc: Vec::new(), work: 2 * SUB as u64 * 4 + 16 };
cv.fill(&square, [255; 4]);
assert!(cv.buf.iter().all(|&b| b == 255));
cv.buf = vec![0; 64];
cv.fill(&square, [255; 4]);
assert!(cv.buf.iter().all(|&b| b == 0));
}
/// Security: a polyline over the point cap builds no segment paths at all, solid or dashed.
#[test]
fn oversized_polylines_build_no_segment_paths() {
let line = |n: usize| -> Vec<[f32; 2]> { (0..n).map(|i| [i as f32, (i % 2) as f32]).collect() };
assert!(stroke_paths(&line(MAX_STROKE_POINTS + 1), 1.0, false, false).is_empty());
assert!(stroke_paths(&line(MAX_STROKE_POINTS + 1), 1.0, true, false).is_empty());
assert_eq!(stroke_paths(&line(MAX_STROKE_POINTS), 1.0, false, false).len(), MAX_STROKE_POINTS - 1);
}
#[test]
fn dashed_geometry_stays_within_the_dash_budget() {
// Forty very long dashed lines, each allowed 10_000 dashes per segment: before the
+34 -6
View File
@@ -136,7 +136,13 @@ pub(super) fn decimate_area(top: Vec<[f32; 2]>, base: Vec<[f32; 2]>, max: usize)
/// Draws a polyline split at gaps (`None`), decimated when long. With `smooth` (the plot box),
/// each run is interpolated with a Catmull-Rom spline so the line curves through the data points.
///
/// The points of the smoothed runs (data points included) are budgeted across all the runs of the line, not per run:
/// short runs between gaps escape decimation, and 100 sub-segments each would turn a
/// 100,000-point series of three-point runs into millions of points. Past the budget the
/// remaining runs are drawn through their data points only.
pub(super) fn draw_runs(ctx: &mut Ctx, pts: &[Option<[f32; 2]>], color: Rgba, width: f32, smooth: Option<Bx>) {
let mut budget = MAX_LINE_PTS * 2;
let mut run: Vec<[f32; 2]> = Vec::new();
for p in pts.iter().chain(std::iter::once(&None)) {
match p {
@@ -145,7 +151,11 @@ pub(super) fn draw_runs(ctx: &mut Ctx, pts: &[Option<[f32; 2]>], color: Rgba, wi
if run.len() >= 2 {
let r = decimate(std::mem::take(&mut run), MAX_LINE_PTS);
let r = match smooth {
Some(plot) => smooth_run(r, plot),
Some(plot) => {
let r = smooth_run(r, plot, budget);
budget = budget.saturating_sub(r.len());
r
}
None => r,
};
ctx.line(r, color, width);
@@ -158,15 +168,15 @@ pub(super) fn draw_runs(ctx: &mut Ctx, pts: &[Option<[f32; 2]>], color: Rgba, wi
}
/// Catmull-Rom spline through `pts`: every segment gets the same number of sub-segments (up to
/// 100), chosen so the whole curve stays within `MAX_LINE_PTS * 2` points. Endpoints are
/// 100), chosen so the whole curve stays within `cap` points (one per data point when `cap` is
/// smaller than that). Endpoints are
/// duplicated so the curve passes through the first and last points, and the curve is clamped
/// to the `plot` box (a spline overshoots its data near sharp turns).
fn smooth_run(pts: Vec<[f32; 2]>, plot: Bx) -> Vec<[f32; 2]> {
fn smooth_run(pts: Vec<[f32; 2]>, plot: Bx, cap: usize) -> Vec<[f32; 2]> {
let n = pts.len();
if n < 3 {
return pts;
}
let cap = MAX_LINE_PTS * 2;
let steps = (cap / n).clamp(1, 100);
let mut out: Vec<[f32; 2]> = Vec::with_capacity(n.saturating_mul(steps).min(cap) + 1);
let at = |i: isize| -> [f32; 2] {
@@ -219,7 +229,7 @@ mod smooth_tests {
fn long_runs_are_smoothed_uniformly_within_the_cap() {
let plot = Bx { x: 0.0, y: 0.0, w: 3000.0, h: 100.0 };
let pts: Vec<[f32; 2]> = (0..1500).map(|i| [i as f32 * 2.0, if i % 2 == 0 { 20.0 } else { 80.0 }]).collect();
let out = smooth_run(pts, plot);
let out = smooth_run(pts, plot, MAX_LINE_PTS * 2);
assert!(out.len() <= MAX_LINE_PTS * 2 + 1, "{}", out.len());
// Two sub-segments per segment all the way to the end, not 100 each and then a straight tail.
assert_eq!(out.len(), 1499 * 2 + 1);
@@ -231,11 +241,29 @@ mod smooth_tests {
let plot = Bx { x: 0.0, y: 10.0, w: 300.0, h: 100.0 };
// A step right at the plot edges: Catmull-Rom overshoots past both.
let pts = vec![[0.0, 110.0], [100.0, 110.0], [110.0, 10.0], [200.0, 10.0], [300.0, 10.0]];
let out = smooth_run(pts, plot);
let out = smooth_run(pts, plot, MAX_LINE_PTS * 2);
assert!(out.len() > 5);
assert!(out.iter().all(|p| (0.0..=300.0).contains(&p[0]) && (10.0..=110.0).contains(&p[1])));
}
/// Security: smoothing budgets its added points across every run of a line. A series of
/// three-point runs between gaps used to get 100 sub-segments per run, 67 times its data.
#[test]
fn gap_separated_runs_share_one_smoothing_budget() {
let plot = Bx { x: 0.0, y: 0.0, w: 1e6, h: 100.0 };
let pts: Vec<Option<[f32; 2]>> =
(0..100_000).map(|i| if i % 4 == 3 { None } else { Some([i as f32, if i % 2 == 0 { 20.0 } else { 80.0 }]) }).collect();
let m = crate::ApproxMeasure;
let mut ctx = Ctx { out: Vec::new(), m: &m };
draw_runs(&mut ctx, &pts, WHITE, 1.0, Some(plot));
let total: usize = ctx.out.iter().map(|p| if let Prim::Line { pts, .. } = p { pts.len() } else { 0 }).sum();
// 75,000 data points, plus at most the budget added by interpolation.
assert!(total <= 75_000 + MAX_LINE_PTS * 2, "{total}");
assert_eq!(ctx.out.len(), 25_000);
// The first runs are still smoothed.
assert!(matches!(ctx.out.first(), Some(Prim::Line { pts, .. }) if pts.len() > 3));
}
#[test]
fn decimate_area_preserves_paired_vertical_strips() {
let top: Vec<[f32; 2]> = (0..1000).map(|i| [i as f32 * 0.25, ((i * 37) % 100) as f32]).collect();
+103 -16
View File
@@ -184,6 +184,53 @@ fn value_text(v: &Value) -> String {
}
}
/// Characters kept of one category label: an axis label never shows more (labels are fitted to
/// the axis), and copying more only costs memory.
const MAX_LABEL_CHARS: usize = 255;
/// Bytes of category label text kept per series. One long text broadcast by a short formula
/// (`IF(SEQUENCE(100000), long_text)`) is a single shared value, but turning every element into
/// its own label copied it 100,000 times (gigabytes). Labels past the budget are empty.
const MAX_CATEGORY_BYTES: usize = 8 << 20;
/// At most [`MAX_LABEL_CHARS`] characters and `room` bytes of `s`, cut at a character boundary.
fn clip_label(s: &str, room: usize) -> String {
let mut end = 0;
for (n, (i, c)) in s.char_indices().enumerate() {
if n >= MAX_LABEL_CHARS || i + c.len_utf8() > room {
break;
}
end = i + c.len_utf8();
}
s.get(..end).unwrap_or_default().to_string()
}
/// The label of category `i` (value `v`) in at most `room` bytes: formatted with its source
/// cell's number format when it is a number.
fn category_label(wb: &Workbook, src: Option<(&gridcraft_model::Sheet, RangeRef)>, i: usize, v: &Value, room: usize) -> String {
if room == 0 {
return String::new(); // nothing fits: skip the number formatting
}
if let (Some((s, r)), Value::Number(_)) = (src, v) {
let w = r.width().max(1) as usize;
let at = r.start.offset((i / w) as i64, (i % w) as i64);
if let Some(c) = at {
let code = &wb.styles.get(s.style_id(c)).num_fmt.0;
if !code.eq_ignore_ascii_case("general") {
return clip_label(&format_value(v, &NumberFormat::parse_cached(code), wb.date_system).text, room);
}
}
}
// Text is clipped before it is copied: the source may be one long shared string.
let mut v = v;
while let Value::Array(a) = v {
v = a.get(0, 0).unwrap_or(&Value::Empty);
}
match v {
Value::Text(t) => clip_label(t, room),
_ => clip_label(&value_text(v), room),
}
}
/// Category labels, formatted with each source cell's number format (dates, currency…), and
/// the outer levels of a multi-level axis: a category range wider than a column of values (or
/// taller than a row of values) groups its innermost labels, year beside month. Each outer level
@@ -221,29 +268,29 @@ fn category_table(wb: &Workbook, sheet: usize, formula: &str, values: &str, limi
None => eval_list_in(wb, sheet, formula, limit, 0),
};
let src = cells.and_then(|(sh, r)| wb.sheet(sh).map(|s| (s, r)));
// The retained levels are known before any label is charged: a cell on a discarded level
// gets no label and spends none of the budget. `None`: a flat list (every cell is a label).
let layout = src.and_then(|(_, r)| {
let levels = levels?;
let (w, h) = (r.width().max(1) as usize, r.height().max(1) as usize);
(vals.len() == w.saturating_mul(h)).then_some((levels, w, h)) // else truncated or not a plain block
});
let kept =
|i: usize| layout.is_none_or(|(l, w, h)| if l.by_columns { i % w + 1 + l.outer as usize >= w } else { i / w + 1 + l.outer as usize >= h });
let mut room = MAX_CATEGORY_BYTES;
let texts: Vec<String> = vals
.iter()
.enumerate()
.map(|(i, v)| {
if let (Some((s, r)), Value::Number(_)) = (src, v) {
let w = r.width().max(1) as usize;
let at = r.start.offset((i / w) as i64, (i % w) as i64);
if let Some(c) = at {
let code = &wb.styles.get(s.style_id(c)).num_fmt.0;
if !code.eq_ignore_ascii_case("general") {
return format_value(v, &NumberFormat::parse_cached(code), wb.date_system).text;
}
}
if !kept(i) {
return String::new();
}
value_text(v)
let label = category_label(wb, src, i, v, room);
room = room.saturating_sub(label.len());
label
})
.collect();
let Some((_, r)) = src else { return (texts, vec![]) };
let Some(levels) = levels else { return (texts, vec![]) };
let (w, h) = (r.width().max(1) as usize, r.height().max(1) as usize);
if texts.len() != w.saturating_mul(h) {
return (texts, vec![]); // truncated or not a plain block: a flat list
}
let Some((levels, w, h)) = layout else { return (texts, vec![]) };
let at = |row: usize, col: usize| texts.get(row * w + col).cloned().unwrap_or_default();
let column = |c: usize| (0..h).map(|row| at(row, c)).collect::<Vec<String>>();
let row = |rr: usize| (0..w).map(|c| at(rr, c)).collect::<Vec<String>>();
@@ -369,6 +416,46 @@ pub fn resolve(wb: &Workbook, sheet: usize, chart: &Chart) -> ChartData {
ChartData { categories, category_levels, series, palette }
}
#[cfg(test)]
mod label_budget_tests {
use gridcraft_model::Workbook;
/// Security: category labels are clipped before they are copied, with one byte budget per
/// series. A formula broadcasting one long text to 100,000 categories used to copy it
/// 100,000 times (3.3 GB).
#[test]
fn category_labels_share_one_byte_budget() {
let wb = Workbook::new();
let (texts, _) = super::category_table(&wb, 0, r#"=IF(SEQUENCE(100000),REPT("x",32767))"#, "", super::MAX_POINTS);
assert_eq!(texts.len(), 100_000);
assert_eq!(texts.first().map(String::len), Some(super::MAX_LABEL_CHARS));
assert!(texts.iter().map(String::len).sum::<usize>() <= super::MAX_CATEGORY_BYTES);
// Labels are cut at character boundaries.
assert_eq!(super::clip_label("aé", 2), "a");
assert_eq!(super::clip_label("aé", 3), "aé");
}
/// Cells on discarded outer levels (only the innermost labels and three outer levels are
/// kept) spend none of the byte budget.
#[test]
fn discarded_levels_do_not_spend_the_label_budget() {
let mut wb = Workbook::new();
let rows = 8000u32;
if let Some(sh) = wb.sheet_mut(0) {
for r in 0..rows {
for c in 0..5 {
sh.set_value(gridcraft_core::CellRef::new(r, c), gridcraft_core::Value::text("x".repeat(255)));
}
}
}
// Five columns: column A is a fourth outer level and is discarded.
let (inner, outer) = super::category_table(&wb, 0, "=A1:E8000", "", super::MAX_POINTS);
assert_eq!(outer.len(), 3);
assert_eq!(inner.last().map(String::len), Some(super::MAX_LABEL_CHARS));
assert_eq!(outer.first().and_then(|l| l.last()).map(String::len), Some(super::MAX_LABEL_CHARS));
}
}
#[cfg(test)]
mod tests {
use gridcraft_model::Workbook;
+2 -9
View File
@@ -147,10 +147,10 @@ impl RangeRef {
RangeRef::new(CellRef::new(0, 0), CellRef::new(MAX_ROWS - 1, MAX_COLS - 1))
}
pub fn height(&self) -> u32 {
self.end.row.saturating_sub(self.start.row).saturating_add(1)
self.end.row - self.start.row + 1
}
pub fn width(&self) -> u32 {
self.end.col.saturating_sub(self.start.col).saturating_add(1)
self.end.col - self.start.col + 1
}
pub fn count(&self) -> u64 {
self.height() as u64 * self.width() as u64
@@ -321,13 +321,6 @@ pub fn split_areas(reference: &str) -> Vec<&str> {
mod tests {
use super::*;
#[test]
fn hostile_range_dimensions_saturate_instead_of_wrapping() {
let r = RangeRef { start: CellRef::new(0, 0), end: CellRef::new(u32::MAX, u32::MAX) };
assert_eq!((r.height(), r.width()), (u32::MAX, u32::MAX));
assert_eq!(r.count(), u64::from(u32::MAX) * u64::from(u32::MAX));
}
#[test]
fn areas_split_only_at_commas_outside_quoted_sheet_names() {
assert_eq!(split_areas("B2:B4,D1"), ["B2:B4", "D1"]);
+7 -2
View File
@@ -366,6 +366,10 @@ pub struct Ctx<'a> {
pub wb: Workbook,
pub changed: Vec<Key>,
pub structural: bool,
/// Recalculate every formula without the other effects of `structural` (table headers are not
/// resynchronized, array formulas are still checked): for edits that change only what a
/// function reads, like sheet protection.
pub recalc_all: bool,
pub sel: &'a mut Selection,
/// Rows whose automatic height should be recomputed (sheet, row).
pub fit_rows: Vec<(usize, u32)>,
@@ -400,13 +404,14 @@ pub(crate) fn commit<R>(d: &mut DocState, f: impl FnOnce(&mut Ctx) -> Result<R>)
wb: (*d.wb).clone(),
changed: Vec::new(),
structural: false,
recalc_all: false,
sel: &mut sel,
fit_rows: Vec::new(),
protection_checked: false,
edits: Vec::new(),
};
let r = f(&mut ctx)?;
let Ctx { mut wb, mut changed, structural, mut fit_rows, protection_checked, edits, .. } = ctx;
let Ctx { mut wb, mut changed, structural, recalc_all, mut fit_rows, protection_checked, edits, .. } = ctx;
// Equations whose source changed (or that arrived without a layout, as from a paste) are
// laid out here, so painting only ever reads the cache.
wb.refresh_equations();
@@ -435,7 +440,7 @@ pub(crate) fn commit<R>(d: &mut DocState, f: impl FnOnce(&mut Ctx) -> Result<R>)
if let Some(t0) = t0 {
eprintln!("commit: edit {:?}, {} changed", since(t0), changed.len());
}
if structural {
if structural || recalc_all {
d.calc.recalc_all(&mut wb);
} else if !changed.is_empty() {
d.calc.cells_changed(&mut wb, &changed);
+67 -20
View File
@@ -152,26 +152,32 @@ fn input_range(s: &Session, p: &Json, keys: &[&str]) -> Result<(usize, RangeRef)
/// Splits a range into variables (columns, or rows when `by_rows`); with `labels` the first
/// cell of each names it, else `default(i)` does.
fn variables(s: &Session, si: usize, r: RangeRef, by_rows: bool, labels: bool, default: &dyn Fn(usize) -> String) -> Result<Vec<Var>> {
let d = s.doc()?;
let sh = d.wb.sheet(si).ok_or(EngineError::NoDocument)?;
let (outer, inner) = if by_rows { (r.height(), r.width()) } else { (r.width(), r.height()) };
let outer = if by_rows { r.height() } else { r.width() };
let mut out = Vec::with_capacity(outer as usize);
for i in 0..outer {
let cell = |j: u32| if by_rows { CellRef::new(r.start.row + i, r.start.col + j) } else { CellRef::new(r.start.row + j, r.start.col + i) };
let mut vals: Vec<Value> = (0..inner).map(|j| sh.value(cell(j))).collect();
let mut name = default(i as usize);
if labels && !vals.is_empty() {
let first = vals.remove(0);
let t = first.display();
if !t.trim().is_empty() {
name = t;
}
}
out.push(Var { name, vals });
out.push(variable(s, si, r, by_rows, labels, default, i)?);
}
Ok(out)
}
/// Variable `i` of [`variables`], read alone so a caller that may stop early needn't build them all.
fn variable(s: &Session, si: usize, r: RangeRef, by_rows: bool, labels: bool, default: &dyn Fn(usize) -> String, i: u32) -> Result<Var> {
let d = s.doc()?;
let sh = d.wb.sheet(si).ok_or(EngineError::NoDocument)?;
let inner = if by_rows { r.width() } else { r.height() };
let cell = |j: u32| if by_rows { CellRef::new(r.start.row + i, r.start.col + j) } else { CellRef::new(r.start.row + j, r.start.col + i) };
let mut vals: Vec<Value> = (0..inner).map(|j| sh.value(cell(j))).collect();
let mut name = default(i as usize);
if labels && !vals.is_empty() {
let first = vals.remove(0);
let t = first.display();
if !t.trim().is_empty() {
name = t;
}
}
Ok(Var { name, vals })
}
/// The numbers of a variable; blanks are skipped when `skip_blank`, anything else that isn't a
/// number is an error.
fn numbers(v: &[Value], skip_blank: bool) -> Result<Vec<f64>> {
@@ -272,11 +278,25 @@ struct Grid {
cells: BTreeMap<(u32, u32), Value>,
/// Cells shown as percentages.
pct: Vec<(u32, u32)>,
/// Rows and columns spanned by `cells`.
rows: u32,
cols: u32,
/// A cell was dropped because the report outgrew [`MAX_OUTPUT_CELLS`]: the report is refused.
too_large: bool,
}
impl Grid {
/// Stores a value. Past [`MAX_OUTPUT_CELLS`] (the most a report writes) nothing more is kept,
/// so a builder that would produce millions of entries stops allocating them, and
/// [`write_report`] refuses the report.
fn put(&mut self, r: u32, c: u32, v: Value) {
if !v.is_empty() {
let (rows, cols) = (self.rows.max(r.saturating_add(1)), self.cols.max(c.saturating_add(1)));
if self.too_large || u64::from(rows) * u64::from(cols) > MAX_OUTPUT_CELLS {
self.too_large = true;
return;
}
(self.rows, self.cols) = (rows, cols);
self.cells.insert((r, c), v);
}
}
@@ -296,9 +316,7 @@ impl Grid {
}
/// (rows, cols) spanned.
fn extent(&self) -> (u32, u32) {
let rows = self.cells.keys().map(|(r, _)| r + 1).max().unwrap_or(0);
let cols = self.cells.keys().map(|(_, c)| c + 1).max().unwrap_or(0);
(rows, cols)
(self.rows, self.cols)
}
}
@@ -310,7 +328,7 @@ fn idx(i: usize) -> u32 {
fn descriptive(s: &Session, p: &Json) -> Result<Grid> {
let (si, r) = input_range(s, p, &["range", "input"])?;
let vars = variables(s, si, r, by_rows(p), flag(p, "labels"), &|i| format!("Column{}", i + 1))?;
let (by_rows, labels) = (by_rows(p), flag(p, "labels"));
let summary = p.get("summary").map(|_| flag(p, "summary")).unwrap_or(true);
let conf = confidence(p, "confidence")?;
let kl = num(p, "kthLargest").filter(|k| *k >= 1.0).map(|k| k as usize);
@@ -319,7 +337,14 @@ fn descriptive(s: &Session, p: &Json) -> Result<Grid> {
return Err(bad(CMD, "choose at least one of summary, confidence, kthLargest or kthSmallest"));
}
let mut g = Grid::default();
for (i, v) in vars.iter().enumerate() {
// Variables are read one at a time: a million-row range grouped by rows is a million variables.
for i in 0..(if by_rows { r.height() } else { r.width() }) {
// The report is refused once the grid is too large: the rest would only build discarded labels.
if g.too_large {
break;
}
let v = variable(s, si, r, by_rows, labels, &|i| format!("Column{}", i + 1), i)?;
let i = i as usize;
let x = numbers(&v.vals, true)?;
let c = idx(i).saturating_mul(2);
g.txt(0, c, &v.name);
@@ -1749,7 +1774,7 @@ fn sampling(s: &Session, p: &Json, seed: Option<u64>) -> Result<Grid> {
fn write_report(s: &mut Session, p: &Json, tool: &str, g: Grid) -> Result<Json> {
let (rows, cols) = g.extent();
let (rows, cols) = (rows.max(1), cols.max(1));
if u64::from(rows) * u64::from(cols) > MAX_OUTPUT_CELLS {
if g.too_large || u64::from(rows) * u64::from(cols) > MAX_OUTPUT_CELLS {
return Err(bad(CMD, "the report is too large"));
}
let d = s.doc()?;
@@ -1914,6 +1939,28 @@ mod tests {
x
}
#[test]
fn a_report_stops_growing_past_the_output_limit() {
// Entries past the limit are not kept, and the report is refused.
let mut g = super::Grid::default();
g.num(1999, 1999, 1.0);
assert!(!g.too_large);
g.num(0, 2000, 1.0);
assert!(g.too_large);
assert_eq!(g.cells.len(), 1);
assert_eq!(g.extent(), (2000, 2000));
// Row-grouped statistics of a tall, almost empty column: one variable per row would make
// millions of report entries; the report is refused instead of built.
let mut s = s();
s.execute("range.setValues", json!({"range": "A1", "values": [[1]]})).unwrap();
s.execute("range.setValues", json!({"range": "A1000000", "values": [[2]]})).unwrap();
let err = s
.execute("data.analysisToolPak", json!({"tool": "descriptive", "range": "A1:A1000000", "groupedBy": "rows", "output": "C1"}))
.expect_err("report too large");
assert!(err.to_string().contains("too large"), "{err}");
assert_eq!(v(&s, "C1"), Value::Empty);
}
#[test]
fn forecast_standard_errors_match_the_full_window_rescan() {
// #1043: the running sum must give what re-adding every window gave, gaps included.
+1 -2
View File
@@ -184,8 +184,7 @@ pub(crate) fn grow_autofilter(sh: &mut Sheet, at: CellRef) {
pub(crate) fn apply_filter(wb: &mut Workbook, sheet: usize) {
let Some(sh) = wb.sheet(sheet) else { return };
let Some(af) = sh.autofilter.clone() else { return };
// An imported range can name rows past the grid; none of those exist to hide.
let r = RangeRef::new(af.range.start, CellRef::new(af.range.end.row.min(gridcraft_core::MAX_ROWS - 1), af.range.end.col));
let r = af.range;
let mut hide = Vec::new();
// Precompute per-criterion top10 threshold and average once, not per row.
let mut col_stats: std::collections::HashMap<usize, (Option<f64>, f64)> = std::collections::HashMap::new();
-7
View File
@@ -5,9 +5,6 @@ use super::{
// ---------------------------------------------------------------- tools
/// Most cells (rows times compared columns) one Remove Duplicates call reads.
const MAX_COMPARED_CELLS: u64 = 50_000_000;
pub(super) fn remove_duplicates(s: &mut Session, p: &Json) -> Result<Json> {
let (sheet, r) = data_target(s, p, "data.removeDuplicates")?;
let d = s.doc()?;
@@ -27,10 +24,6 @@ pub(super) fn remove_duplicates(s: &mut Session, p: &Json) -> Result<Json> {
if cols.is_empty() || cols.iter().any(|c| !(r.start.col..=r.end.col).contains(c)) {
return Err(bad("data.removeDuplicates", "`columns` must name at least one column of the range"));
}
// Rows are compared one by one, so a sparse sheet's enclosing rectangle is bounded like a list.
if r.count() > MAX_COMPARED_CELLS {
return Err(bad("data.removeDuplicates", "the range is too large"));
}
let start = r.start.row + header as u32;
let mut removed = 0;
edit(s, |cx| {
+1 -4
View File
@@ -337,14 +337,11 @@ pub(super) fn circle_invalid(s: &mut Session, _: &Json) -> Result<Json> {
let sheet = d.wb.active_sheet();
let Some(sh) = d.wb.sheet(sheet) else { return Err(EngineError::NoDocument) };
let mut bad_cells = Vec::new();
// One scan allowance for the whole command: duplicate or overlapping ranges don't multiply it.
let mut budget = 100_000usize;
for dv in &sh.validations {
for r in &dv.ranges {
let r = sh.used_range().and_then(|u| r.intersection(&u));
let Some(r) = r else { continue };
for c in r.iter().take(budget) {
budget -= 1;
for c in r.iter().take(100_000) {
let text = sh.input_text(c);
if !text.is_empty() && check_validation(&d.wb, sheet, c, &text).is_some() {
bad_cells.push(c.a1());
+80 -26
View File
@@ -14,8 +14,13 @@ fn matches(hay: &str, what: &str, case: bool, whole: bool) -> bool {
}
/// `*`/`?` wildcard match with `~` escapes.
///
/// The pattern runs as a bit-parallel automaton over the text: one pass, each character costing
/// one step per 64 pattern positions. Backtracking to the last `*` after a mismatch rescans the
/// text and costs text × pattern comparisons on a long, repetitive cell, for every cell a
/// filter or Find visits.
pub(crate) fn wildcard(text: &str, pat: &str) -> bool {
let t: Vec<char> = text.chars().collect();
// The pattern as (character, escaped) items, runs of `*` folded into one.
let mut p: Vec<(char, bool)> = Vec::new();
let mut it = pat.chars();
while let Some(c) = it.next() {
@@ -23,32 +28,85 @@ pub(crate) fn wildcard(text: &str, pat: &str) -> bool {
if let Some(n) = it.next() {
p.push((n, true));
}
} else {
} else if c != '*' || p.last() != Some(&('*', false)) {
p.push((c, false));
}
}
let (mut ti, mut pi) = (0usize, 0usize);
let (mut star, mut mark) = (None, 0usize);
while ti < t.len() {
if pi < p.len() && (p[pi] == ('?', false) || (p[pi].0 == t[ti] && p[pi] != ('*', false))) {
ti += 1;
pi += 1;
} else if pi < p.len() && p[pi] == ('*', false) {
star = Some(pi);
mark = ti;
pi += 1;
} else if let Some(sp) = star {
pi = sp + 1;
mark += 1;
ti = mark;
} else {
return false;
let words = (p.len() + 1).div_ceil(64);
let bit = |set: &mut [u64], i: usize| set[i / 64] |= 1 << (i % 64);
// Positions holding a `*` and a `?`, and where each literal character sits in the pattern.
let mut stars = vec![0u64; words];
let mut any = vec![0u64; words];
let mut at: std::collections::HashMap<char, Vec<usize>> = std::collections::HashMap::new();
for (i, item) in p.iter().enumerate() {
match item {
('*', false) => bit(&mut stars, i),
('?', false) => bit(&mut any, i),
(c, _) => at.entry(*c).or_default().push(i),
}
}
while pi < p.len() && p[pi] == ('*', false) {
pi += 1;
// A literal's bit mask is built when the text first reads it, and kept for at most
// `MAX_CACHED_MASKS` literals: a long pattern of mostly distinct characters would otherwise
// allocate a mask of the pattern's width for each one up front. When there are more literals
// than that, the ones kept are those at the most pattern positions (the dearest to rebuild);
// each of the rest has at most pattern length / `MAX_CACHED_MASKS` positions, so rebuilding
// its mask in `scratch` costs less than one step of the scan.
const MAX_CACHED_MASKS: usize = 256;
let cacheable: Option<std::collections::HashSet<char>> = (at.len() > MAX_CACHED_MASKS).then(|| {
let mut by_count: Vec<(&char, usize)> = at.iter().map(|(c, v)| (c, v.len())).collect();
by_count.sort_unstable_by_key(|&(_, n)| std::cmp::Reverse(n));
by_count.into_iter().take(MAX_CACHED_MASKS).map(|(c, _)| *c).collect()
});
let mut masks: std::collections::HashMap<char, Vec<u64>> = std::collections::HashMap::new();
let mut scratch = vec![0u64; words];
// A set bit i: the text read so far matches the first i pattern items. A `*` matches nothing
// as well, so a state at a `*` is also at the item after it.
let close = |set: &mut Vec<u64>| {
let mut carry = 0u64;
for w in 0..words {
let s = set[w] & stars[w];
let next = (s << 1) | carry;
carry = s >> 63;
set[w] |= next;
}
};
let mut state = vec![0u64; words];
bit(&mut state, 0);
close(&mut state);
let mut next = vec![0u64; words];
for ch in text.chars() {
let lit: Option<&[u64]> = match at.get(&ch) {
None => None,
Some(positions) if cacheable.as_ref().is_none_or(|k| k.contains(&ch)) => {
let mask = masks.entry(ch).or_insert_with(|| {
let mut m = vec![0u64; words];
positions.iter().for_each(|&i| bit(&mut m, i));
m
});
Some(mask)
}
Some(positions) => {
scratch.fill(0);
positions.iter().for_each(|&i| bit(&mut scratch, i));
Some(&scratch)
}
};
let mut carry = 0u64;
let mut alive = 0u64;
for w in 0..words {
// Items that read `ch` advance; a `*` reads it and stays.
let m = state[w] & (any[w] | lit.map_or(0, |l| l[w]));
next[w] = (m << 1) | carry | (state[w] & stars[w]);
carry = m >> 63;
alive |= next[w];
}
if alive == 0 {
return false;
}
std::mem::swap(&mut state, &mut next);
close(&mut state);
}
pi == p.len()
state[p.len() / 64] & (1 << (p.len() % 64)) != 0
}
pub(super) fn find(s: &mut Session, p: &Json) -> Result<Json> {
@@ -141,11 +199,7 @@ pub(super) fn replace(s: &mut Session, p: &Json) -> Result<Json> {
let mut total = 0usize;
for si in sheets {
let cells: Vec<(CellRef, String)> =
// A hidden formula's source is not searched: a count or error would reveal its text.
cx.wb
.sheet(si)
.map(|sh| sh.cells.iter().filter(|(c, _)| !crate::display::formula_hidden(&cx.wb, sh, *c)).map(|(c, _)| (c, sh.input_text(c))).collect())
.unwrap_or_default();
cx.wb.sheet(si).map(|sh| sh.cells.iter().map(|(c, _)| (c, sh.input_text(c))).collect()).unwrap_or_default();
for (c, text) in cells {
if (!all && c != active) || selection.as_ref().is_some_and(|sel| !sel.contains(c)) {
continue;
+97
View File
@@ -324,6 +324,103 @@ fn wildcards() {
assert!(wildcard("", "*"));
}
/// The backtracking matcher `wildcard` replaced, as the reference for its results.
fn wildcard_backtracking(text: &str, pat: &str) -> bool {
let t: Vec<char> = text.chars().collect();
let mut p: Vec<(char, bool)> = Vec::new();
let mut it = pat.chars();
while let Some(c) = it.next() {
if c == '~' {
if let Some(n) = it.next() {
p.push((n, true));
}
} else {
p.push((c, false));
}
}
let (mut ti, mut pi) = (0usize, 0usize);
let (mut star, mut mark) = (None, 0usize);
while ti < t.len() {
if pi < p.len() && (p[pi] == ('?', false) || (p[pi].0 == t[ti] && p[pi] != ('*', false))) {
ti += 1;
pi += 1;
} else if pi < p.len() && p[pi] == ('*', false) {
star = Some(pi);
mark = ti;
pi += 1;
} else if let Some(sp) = star {
pi = sp + 1;
mark += 1;
ti = mark;
} else {
return false;
}
}
while pi < p.len() && p[pi] == ('*', false) {
pi += 1;
}
pi == p.len()
}
#[test]
fn wildcard_agrees_with_backtracking() {
// Every text up to 5 characters over a small alphabet against every pattern up to 4 items.
let alphabet = ['a', 'b', '*', '?', '~'];
let all = |n: usize, chars: &[char]| -> Vec<String> {
let mut out = vec![String::new()];
let mut level = vec![String::new()];
for _ in 0..n {
level = level.iter().flat_map(|s| chars.iter().map(move |c| format!("{s}{c}"))).collect();
out.extend(level.iter().cloned());
}
out
};
let texts = all(5, &['a', 'b', '*']);
for pat in all(4, &alphabet) {
for text in &texts {
assert_eq!(wildcard(text, &pat), wildcard_backtracking(text, &pat), "{text:?} ~ {pat:?}");
}
}
// Patterns longer than one 64-bit word.
let long = format!("*{}*b?", "a".repeat(70));
for text in [format!("x{}yb1", "a".repeat(70)), format!("{}b", "a".repeat(71)), "a".repeat(200)] {
assert_eq!(wildcard(&text, &long), wildcard_backtracking(&text, &long), "{text}");
}
}
#[test]
fn wildcard_with_many_distinct_literals_agrees_with_backtracking() {
// 600 distinct characters in the pattern: more than the masks kept, so the rest are rebuilt.
let chars: Vec<char> = (0x4E00u32..0x4E00 + 600).filter_map(char::from_u32).collect();
let word: String = chars.iter().collect();
let pat = format!("*{word}?{}", chars[3]);
for text in [format!("x{word}yq{}", chars[3]), word.clone(), format!("{word}{word}"), chars.iter().rev().collect::<String>()] {
assert_eq!(wildcard(&text, &pat), wildcard_backtracking(&text, &pat), "{} chars", text.chars().count());
}
assert!(wildcard(&format!("{word}zz{}", chars[3]).replace("zz", "z"), &pat));
}
#[test]
fn wildcard_keeps_masks_of_frequent_literals_past_the_cache_limit() {
// 300 distinct characters come before the repeated `a`: past the cached masks, `a` (5,000 positions)
// still has to be read from a kept mask, or it is rebuilt for each of the 20,000 `a` in the text.
let rare: String = (0x4E00u32..0x4E00 + 300).filter_map(char::from_u32).collect();
let pat = format!("*{rare}{}b", "a".repeat(5_000));
let text = "a".repeat(20_000);
assert!(!wildcard(&text, &pat));
assert!(wildcard(&format!("{rare}{}b", "a".repeat(5_000)), &pat));
assert!(!wildcard(&format!("{rare}{}", "a".repeat(5_000)), &pat));
}
#[test]
fn wildcard_on_long_repetitive_text_is_linear() {
// The backtracking matcher compared about text × pattern characters here (10^8).
let text = "a".repeat(20_000);
let pat = format!("*{}b", "a".repeat(5_000));
assert!(!wildcard(&text, &pat));
assert!(wildcard(&format!("{text}b"), &pat));
}
#[test]
fn tsv() {
assert_eq!(parse_tsv("a\tb\n1\t2\n"), vec![vec!["a", "b"], vec!["1", "2"]]);
+73 -3
View File
@@ -36,13 +36,28 @@ fn max_periods(history: usize) -> u32 {
u32::try_from(MAX_WORK / h).unwrap_or(u32::MAX).clamp(1, MAX_PERIODS)
}
/// Most forecast rows when the model is fit to the first `fit` of `history` rows, which span
/// `slots` timeline slots once their gaps are filled. With a Forecast Start inside the history,
/// every row after it is forecast too (a hindcast row holds the same ETS formulas, each refitting
/// the model over the `slots` slots), so those rows (counted from the raw rows) count against
/// [`MAX_WORK`] before any forecast row does. `None` when the hindcast alone exceeds it.
fn max_periods_from(history: usize, fit: usize, slots: usize) -> Option<u32> {
if fit >= history {
return Some(max_periods(history));
}
let f = u64::try_from(slots.max(fit)).unwrap_or(u64::MAX).max(1);
let hindcast = u64::try_from(history - fit).unwrap_or(u64::MAX);
let rest = (MAX_WORK / f).checked_sub(hindcast).filter(|r| *r >= 1)?;
Some(u32::try_from(rest).unwrap_or(u32::MAX).min(MAX_PERIODS))
}
pub fn specs() -> Vec<CommandSpec> {
vec![cmd!(
"data.forecastSheet",
"Forecast Sheet",
["Data", "Forecast"],
None,
"{timeline?: \"A2:A25\", values?: \"B2:B25\" (default: the first two columns of `range`, the selection or the current region; a text first cell is the header), end?: timeline value or date text | periods?: n (default: a third of the history, at least 1; ≤10,000 and history × periods ≤ 500,000; history ≤ 10,000 rows), forecastStart?: a timeline value or date text on the history, before its last point (hindcast: the model is fit to the points before it, the forecast columns start there and run over the actuals; default and the last point: forecast after the history), confidenceInterval?: bool (default true), confidence?: 0.95 or 95, seasonality?: \"auto\" | 0 (none) | 2–8,760, fillMissing?: interpolate|zeros, aggregate?: average|count|counta|max|median|min|sum, statistics?: bool, chart?: line|column} → {sheet, table, chart, history, periods}. Inserts a sheet before the active one: timeline, values, Forecast (FORECAST.ETS), Lower/Upper Confidence Bound (FORECAST.ETS.CONFINT), optional FORECAST.ETS.STAT table, and a chart.",
"{timeline?: \"A2:A25\", values?: \"B2:B25\" (default: the first two columns of `range`, the selection or the current region; a text first cell is the header), end?: timeline value or date text | periods?: n (default: a third of the history, at least 1; ≤10,000 and history × periods ≤ 500,000, or with a forecastStart inside the history (hindcast rows are forecast too) (the greater of the fitted rows and fitted slots × (periods + rows from forecastStart on, that row included)) ≤ 500,000; history ≤ 10,000 rows), forecastStart?: a timeline value or date text on the history, before its last point (hindcast: the model is fit to the points before it, the forecast columns start there and run over the actuals; default and the last point: forecast after the history), confidenceInterval?: bool (default true), confidence?: 0.95 or 95, seasonality?: \"auto\" | 0 (none) | 2–8,760, fillMissing?: interpolate|zeros, aggregate?: average|count|counta|max|median|min|sum, statistics?: bool, chart?: line|column} → {sheet, table, chart, history, periods}. Inserts a sheet before the active one: timeline, values, Forecast (FORECAST.ETS), Lower/Upper Confidence Bound (FORECAST.ETS.CONFINT), optional FORECAST.ETS.STAT table, and a chart.",
has_doc,
forecast_sheet
)]
@@ -257,13 +272,23 @@ fn forecast_sheet(s: &mut Session, p: &Json) -> Result<Json> {
}
}
};
let cap = max_periods(rows.len());
// The slots the model is fit over: the fit rows' span of the timeline, gaps included.
let slots = match (fit < rows.len(), times.get(fit.wrapping_sub(1)), times.first()) {
(true, Some(last_fit), Some(first)) => match (timeline.steps_after_last(*last_fit), timeline.steps_after_last(*first)) {
(Some(b), Some(a)) => ((b - a).round() + 1.0).clamp(0.0, 1e9) as usize,
_ => fit,
},
_ => fit,
};
let cap = max_periods_from(rows.len(), fit, slots).ok_or_else(|| {
EngineError::Other("The forecast start is too early for this much history: choose a later forecast start or less history.".into())
})?;
let periods = match end {
Some(end) => steps_to(&timeline, end, cap)?,
None => match number(p, "periods")? {
Some(n) if n >= 1.0 && n.ceil() <= f64::from(cap) => n.ceil() as u32,
Some(_) => return Err(bad(CMD, format!("`periods` is from 1 to {cap} for {} rows of history", rows.len()))),
None => default_periods(rows.len()),
None => default_periods(rows.len()).min(cap),
},
};
let future: Vec<f64> =
@@ -532,6 +557,51 @@ mod tests {
assert_eq!(s.doc().unwrap().wb.sheets.len(), sheets - 1);
}
#[test]
fn hindcast_rows_count_against_the_work_limit() {
use super::{MAX_WORK, max_periods, max_periods_from};
assert_eq!(max_periods_from(1_300, 1_300, 1_300), Some(max_periods(1_300)));
// 2,000 rows fit to the first 1,899: 101 hindcast rows, then 162 forecast rows.
assert_eq!(max_periods_from(2_000, 1_899, 1_899), Some(162));
// Fit to the first 1,000: the 1,000 hindcast rows alone are twice the limit.
assert_eq!(max_periods_from(2_000, 1_000, 1_000), None);
// 800 rows fit to the first 400, which span 571 slots (gaps filled): each formula refits
// 571 slots, so 475 forecast rows after the 400 hindcast rows, not 850.
assert_eq!(max_periods_from(800, 400, 571), Some(475));
assert!((475 + 400) * 571 <= MAX_WORK as usize);
for (h, f) in [(2_000usize, 1_899usize), (10_000, 9_990), (100, 2), (60, 50)] {
if let Some(p) = max_periods_from(h, f, f) {
assert!((u64::from(p) + (h - f) as u64) * f as u64 <= MAX_WORK, "{h} {f}");
}
}
let mut s = Session::new();
s.new_workbook();
let mut rows = vec![json!(["t", "y"])];
rows.extend((1..=2_000).map(|t| json!([t, (t % 7) as f64 + t as f64 * 0.5])));
s.execute("range.setValues", json!({"range": "A1", "values": rows})).unwrap();
let sheets = s.doc().unwrap().wb.sheets.len();
let err = s.execute("data.forecastSheet", json!({"range": "A1:B2001", "forecastStart": 1_001})).expect_err("hindcast too long");
assert!(err.to_string().contains("forecast start is too early"), "{err}");
let err = s.execute("data.forecastSheet", json!({"range": "A1:B2001", "forecastStart": 1_900, "periods": 163})).expect_err("past the cap");
assert!(err.to_string().contains("from 1 to 162"), "{err}");
assert_eq!(s.doc().unwrap().wb.sheets.len(), sheets);
}
#[test]
fn the_hindcast_cap_follows_the_slots_the_fit_spans_not_its_rows() {
// 800 rows, 20% of the slots missing: the first 400 rows span 500 slots, so each ETS
// formula refits 500 slots and the cap is 1,000 - 400 = 600 (it was 850 by rows).
let mut s = Session::new();
s.new_workbook();
let times: Vec<u32> = (1u32..).filter(|t| !matches!(t % 10, 3 | 6)).take(800).collect();
let mut rows = vec![json!(["t", "y"])];
rows.extend(times.iter().map(|t| json!([t, (t % 7) as f64 + f64::from(*t) * 0.5])));
s.execute("range.setValues", json!({"range": "A1", "values": rows})).unwrap();
let start = times[400];
let err = s.execute("data.forecastSheet", json!({"range": "A1:B801", "forecastStart": start, "periods": 601})).expect_err("past the cap");
assert!(err.to_string().contains("from 1 to 600"), "{err}");
}
#[test]
fn history_and_periods_are_capped() {
use super::{MAX_HISTORY, MAX_PERIODS, MAX_WORK, default_periods, max_periods};
+12 -21
View File
@@ -194,25 +194,16 @@ pub(super) fn autofit_cols(s: &mut Session, p: &Json) -> Result<Json> {
let measure = s.text_width.clone();
edit(s, |cx| {
let Some(sh) = cx.wb.sheet(si) else { return Ok(Json::Null) };
let mut widths = Vec::new();
let mut measured: std::collections::HashMap<StyleId, std::collections::HashMap<String, f32>> = Default::default();
let mut distinct = 0usize;
// Selected columns, then one pass over the populated cells (the store is row-oriented, so a
// scan per column would revisit every row for each selected column, empty or not).
let cols: std::collections::BTreeSet<u32> = lines.iter().flat_map(|&(a, b)| a..=b.min(a + 16383)).collect();
let mut maxes: std::collections::BTreeMap<u32, f32> = Default::default();
if let Some(u) = sh.used_range() {
// One pass per run of adjacent selected columns, so columns between distant selections
// (A and XFD) are never visited.
let mut runs: Vec<(u32, u32)> = Vec::new();
for &c in &cols {
match runs.last_mut() {
Some((_, end)) if *end + 1 == c => *end = c,
_ => runs.push((c, c)),
}
}
for (first, last) in runs {
let span = RangeRef::new(CellRef::new(u.start.row, first), CellRef::new(u.end.row, last));
for (cell, _) in sh.cells.iter_range(span) {
let used = sh.used_range();
for (a, b) in lines {
for c in a..=b.min(a + 16383) {
let Some(u) = used else { continue };
let col = RangeRef::new(CellRef::new(u.start.row, c), CellRef::new(u.end.row, c));
let mut max: f32 = 0.0;
for (cell, _) in sh.cells.iter_range(col) {
if sh.merge_at(cell).is_some_and(|m| !m.is_single()) {
continue;
}
@@ -235,14 +226,14 @@ pub(super) fn autofit_cols(s: &mut Session, p: &Json) -> Result<Json> {
}
None => approx_text_width(line, st.font.size, st.font.bold),
};
let w = w + 6.0 + st.align.indent as f32 * 9.0 + sh.filter_button_space(cell);
let max = maxes.entry(cell.col).or_insert(0.0);
*max = max.max(w);
max = max.max(w + 6.0 + st.align.indent as f32 * 9.0 + sh.filter_button_space(cell));
}
}
if max > 0.0 {
widths.push((c, max.clamp(8.0, 1785.0)));
}
}
}
let widths: Vec<(u32, f32)> = maxes.into_iter().filter(|&(_, m)| m > 0.0).map(|(c, m)| (c, m.clamp(8.0, 1785.0))).collect();
let sh = cx.sheet_mut(si)?;
for (c, w) in widths {
let e = sh.cols.entry(c).or_default();
+21 -1
View File
@@ -163,6 +163,8 @@ fn protect_sheet(s: &mut Session, p: &Json) -> Result<Json> {
let what = d.wb.sheet(sheet).map(|sh| sh.name.clone()).unwrap_or_default();
let prot = SheetProtection { password: str_param(p, "password").filter(|x| !x.is_empty()).map(|pw| new_password_hash(d, &what, pw)), ..prot };
edit(s, |cx| {
// FORMULATEXT reads a sheet's protection: its results are recalculated.
cx.recalc_all = true;
cx.sheet_mut(sheet)?.protection = Some(prot.clone());
Ok(Json::Null)
})
@@ -182,7 +184,10 @@ fn unprotect_sheet(s: &mut Session, p: &Json) -> Result<Json> {
}
}
edit(s, |cx| {
cx.sheet_mut(sheet)?.protection = None;
// Only removing protection changes what FORMULATEXT reads; a no-op unprotect skips the full recalculation.
if cx.sheet_mut(sheet)?.protection.take().is_some() {
cx.recalc_all = true;
}
Ok(Json::Null)
})
}
@@ -404,6 +409,21 @@ mod tests {
assert_eq!(at(&s), (0, "A1".to_string()));
}
#[test]
fn protecting_and_unprotecting_leave_table_headers_alone() {
let mut s = Session::new();
s.new_workbook();
s.execute("range.setValues", json!({"range": "A1", "values": [["a", "b"], [1, 2]]})).unwrap();
s.execute("insert.table", json!({"range": "A1:B2", "header": true})).unwrap();
// An imported header can hold a number: only a structural edit would turn it back into text.
let d = s.doc_mut().unwrap();
std::sync::Arc::make_mut(&mut d.wb).sheet_mut(0).unwrap().set_value(gridcraft_core::CellRef::new(0, 0), gridcraft_core::Value::Number(7.0));
s.execute("review.protectSheet", json!({})).unwrap();
s.execute("review.unprotectSheet", json!({})).unwrap();
let v = s.doc().unwrap().wb.sheet(0).unwrap().value(gridcraft_core::CellRef::new(0, 0));
assert_eq!(v, gridcraft_core::Value::Number(7.0));
}
#[test]
fn protection_that_forbids_selecting_locked_cells_is_enforced_and_kept() {
let mut s = book();
@@ -83,9 +83,6 @@ pub(crate) fn criterion_matches(v: &Value, text: &str, crit: &Value) -> bool {
}
}
/// Most (list row, criteria row) pairs one Advanced Filter tests.
const MAX_FILTER_TESTS: u64 = 50_000_000;
pub(super) fn advanced_filter(s: &mut Session, p: &Json) -> Result<Json> {
const CMD: &str = "data.advancedFilter";
let d = s.doc()?;
@@ -136,10 +133,6 @@ pub(super) fn advanced_filter(s: &mut Session, p: &Json) -> Result<Json> {
rows.push(conds);
}
}
// Each list row is tried against every criteria row, so bound their product too.
if u64::from(list.height()) * rows.len() as u64 > MAX_FILTER_TESTS {
return Err(bad(CMD, "the list and criteria ranges together are too large"));
}
let matches_row = |row: u32| -> bool {
if rows.is_empty() {
return true;
-9
View File
@@ -298,15 +298,6 @@ fn consolidate_by_position_and_label() {
assert!(s.execute("data.consolidate", json!({"sources": ["A1"], "function": "median"})).is_err());
}
#[test]
fn advanced_filter_refuses_a_list_times_criteria_product_over_the_work_limit() {
let mut s = s();
s.execute("range.setValues", json!({"range": "A1", "values": [["Name"], ["A"]]})).unwrap();
s.execute("range.setValues", json!({"range": "H1", "values": [["Name"], ["A"]]})).unwrap();
// A sparse million-row list against 9,999 blank criteria rows would be ~10 billion tests.
assert!(s.execute("data.advancedFilter", json!({"range": "A2:A1048576", "criteria": "H1:H10000"})).is_err());
}
/// A hundred thousand positional rows take one key lookup each; a tall source and a wide one
/// whose keys would meet in a table of 160 million cells are refused before it is built.
#[test]
+3 -12
View File
@@ -1281,18 +1281,9 @@ fn content_differs(b: Option<&Cell>, a: Option<&Cell>) -> bool {
/// Objects added, changed and removed between two versions of a sheet.
fn objects(b: &Sheet, a: &Sheet) -> (usize, usize, usize) {
fn count<T: PartialEq>(b: &[T], a: &[T], id: impl Fn(&T) -> u32) -> (usize, usize, usize) {
// Objects grouped by id, so each lookup is a map probe instead of a scan of the list.
fn group<'a, T>(v: &'a [T], id: &impl Fn(&T) -> u32) -> std::collections::HashMap<u32, Vec<&'a T>> {
let mut m: std::collections::HashMap<u32, Vec<&T>> = std::collections::HashMap::new();
for x in v {
m.entry(id(x)).or_default().push(x);
}
m
}
let (gb, ga) = (group(b, &id), group(a, &id));
let added = a.iter().filter(|x| !gb.contains_key(&id(x))).count();
let removed = b.iter().filter(|x| !ga.contains_key(&id(x))).count();
let changed = b.iter().filter(|x| ga.get(&id(x)).is_some_and(|ys| ys.iter().any(|y| *y != *x))).count();
let added = a.iter().filter(|x| !b.iter().any(|y| id(y) == id(x))).count();
let removed = b.iter().filter(|x| !a.iter().any(|y| id(y) == id(x))).count();
let changed = b.iter().filter(|x| a.iter().any(|y| id(y) == id(x) && y != *x)).count();
(added, changed, removed)
}
let parts = [
+12 -6
View File
@@ -6481,14 +6481,20 @@ fn clearing_links_and_notes_keeps_allow_edit_ranges() {
}
#[test]
fn replace_does_not_search_hidden_formula_source() {
fn formulatext_does_not_reveal_a_hidden_formula() {
let mut s = s();
s.execute("cell.set", json!({"cell": "A1", "input": "=SEQUENCE(3)*7"})).unwrap();
s.execute("home.formatCells", json!({"range": "A1", "style": {"protection": {"locked": true, "hidden": true}}})).unwrap();
s.execute("cell.set", json!({"cell": "A1", "input": "=1+1"})).unwrap();
s.execute("home.formatCells", json!({"range": "A1", "style": {"protection": {"locked": false, "hidden": true}}})).unwrap();
s.execute("home.formatCells", json!({"range": "B1", "style": {"protection": {"locked": false}}})).unwrap();
s.execute("cell.set", json!({"cell": "B1", "input": "=FORMULATEXT(A1)"})).unwrap();
assert_eq!(v(&s, "B1"), Value::from("=1+1"), "not protected yet");
s.execute("review.protectSheet", json!({})).unwrap();
let hit = s.execute("edit.replace", json!({"what": "SEQUENCE", "with": "X"})).map_err(|e| e.to_string());
let miss = s.execute("edit.replace", json!({"what": "NOSUCHTEXT", "with": "X"})).map_err(|e| e.to_string());
assert_eq!(hit, miss);
// Without entering the formula again: protecting recalculates what reads the protection.
assert_eq!(v(&s, "B1"), Value::Error(gridcraft_core::CellError::NA));
s.execute("cell.set", json!({"cell": "B1", "input": "=FORMULATEXT(A1)"})).unwrap();
assert_eq!(v(&s, "B1"), Value::Error(gridcraft_core::CellError::NA));
s.execute("review.unprotectSheet", json!({})).unwrap();
assert_eq!(v(&s, "B1"), Value::from("=1+1"), "unprotecting shows it again without re-entering");
}
#[test]
+48 -11
View File
@@ -70,27 +70,43 @@ fn cell_criterion(v: &Value) -> Option<Criterion> {
/// when the label is unknown, criterion).
type CriteriaRow = (usize, Vec<(usize, Option<usize>, Criterion)>);
/// Most condition tests one selection may make (records times conditions): every record can be
/// tested against every criteria row when none matches, so a modest database and criteria table
/// multiply into billions of tests. Past it the function is `#NUM!`, as lifted lookups are; 5e8
/// is a few seconds of matching.
const SELECTION_WORK_BUDGET: usize = 500_000_000;
/// Rows (1-based within the database, excluding the header) of matching records.
fn matching_records(db: &Array, crit: &Array, ctx: &mut dyn Ctx) -> Vec<usize> {
fn matching_records(db: &Array, crit: &Array, ctx: &mut dyn Ctx) -> R<Vec<usize>> {
// The database column under each criteria label, found once per label rather than per cell.
let columns: Vec<Option<usize>> = (0..crit.cols)
.map(|c| match crit.get(0, c) {
Some(Value::Text(t)) => find_column(db, t),
Some(Value::Number(n)) => find_column(db, &gridcraft_core::number_to_text(*n)),
_ => None,
})
.collect();
// Per criteria row: its index and a list of (criteria column, database column or None when
// the label is unknown, criterion).
let mut rows: Vec<CriteriaRow> = Vec::new();
let mut tests = 0usize;
for r in 1..crit.rows {
let mut conds = Vec::new();
for c in 0..crit.cols {
let Some(cell) = crit.get(r, c) else { continue };
let Some(cr) = cell_criterion(cell) else { continue };
let col = match crit.get(0, c) {
Some(Value::Text(t)) => find_column(db, t),
Some(Value::Number(n)) => find_column(db, &gridcraft_core::number_to_text(*n)),
_ => None,
};
conds.push((c, col, cr));
conds.push((c, columns.get(c).copied().flatten(), cr));
}
tests = tests.saturating_add(conds.len().max(1));
rows.push((r, conds));
}
let all_match = rows.is_empty();
(1..db.rows)
// An empty criteria row matches every record without testing a condition, so nothing is charged.
let all_match = rows.is_empty() || rows.iter().any(|(_, conds)| conds.is_empty());
// Computed criteria (a formula evaluated per record) count like any other test.
if !all_match && db.rows.saturating_sub(1).saturating_mul(tests) > SELECTION_WORK_BUDGET {
return Err(CellError::Num);
}
Ok((1..db.rows)
.filter(|&rec| {
all_match
|| rows.iter().any(|(r, conds)| {
@@ -102,7 +118,7 @@ fn matching_records(db: &Array, crit: &Array, ctx: &mut dyn Ctx) -> Vec<usize> {
})
})
})
.collect()
.collect())
}
struct Selection<'a> {
@@ -123,7 +139,7 @@ fn select<'a>(a: &'a [Arg], ctx: &mut dyn Ctx) -> R<Selection<'a>> {
let db = as_array(dbv);
let crit = as_array(critv);
let field = field_index(&db, a)?;
let records = matching_records(&db, &crit, ctx);
let records = matching_records(&db, &crit, ctx)?;
Ok(Selection { db, field, records })
}
@@ -352,6 +368,27 @@ mod tests {
is_err(d("DSUM", n(9.0)), CellError::Value);
}
/// Security: records times criteria conditions past the selection budget is `#NUM!` before
/// any record is tested; a nonmatching criteria table used to test every record against
/// every criteria row.
#[test]
fn selection_work_is_bounded() {
use gridcraft_core::Array;
let column = |label: &str, len: usize, v: Value| {
let mut data = vec![tv(label)];
data.extend(std::iter::repeat_n(v, len));
Value::from(Array::column(data))
};
let db = column("Tree", 100_000, tv("Apple"));
let wide = column("Tree", 5_001, tv("=Plum"));
is_err(ev("DCOUNTA", vec![rf(db.clone()), t("Tree"), rf(wide)]), CellError::Num);
let narrow = column("Tree", 2, tv("=Apple"));
close(ev("DCOUNTA", vec![rf(db.clone()), t("Tree"), rf(narrow)]), 100_000.0);
// Empty criteria rows test no condition: the first one matches every record, so they cost nothing.
let empty_rows = column("Tree", 5_001, Value::Empty);
close(ev("DCOUNTA", vec![rf(db), t("Tree"), rf(empty_rows)]), 100_000.0);
}
#[test]
fn simple_criteria() {
let c = |cells: Vec<Value>| arr(vec![vec![tv("Tree"), tv("Height")], cells]);
+22 -1
View File
@@ -416,13 +416,17 @@ fn workday_core(a: &[Arg], c: &mut dyn Ctx, mask: [bool; 7], hol: Option<&Arg>)
if days == 0 {
return num_val(start as f64);
}
// The holidays on working days, still sorted (`holidays` sorts them), so each step counts
// the ones it passed with two binary searches rather than a scan of every holiday: a run of
// consecutive holidays takes one step per holiday.
let hs: Vec<i64> = hs.into_iter().filter(|&h| !mask[weekday0(sys, h)]).collect();
let dir = days.signum();
let mut cur = start;
let mut rem = days.abs();
while rem > 0 {
let next = advance(sys, cur, rem, dir, &mask, per_week);
let (lo, hi) = if dir > 0 { (cur + 1, next) } else { (next, cur - 1) };
rem = hs.iter().filter(|&&h| h >= lo && h <= hi && !mask[weekday0(sys, h)]).count() as i64;
rem = hs.partition_point(|&h| h <= hi).saturating_sub(hs.partition_point(|&h| h < lo)) as i64;
cur = next;
if !(0..=MAX_SERIAL as i64).contains(&cur) {
return Err(CellError::Num);
@@ -755,6 +759,23 @@ mod tests {
is_err(ev("TIMEVALUE", vec![t("x")]), CellError::Value);
}
/// Security: WORKDAY counts the holidays each step passes by binary search, so a long run of
/// consecutive holidays is linear work, not one scan of every holiday per holiday.
#[test]
fn workday_over_many_consecutive_holidays() {
let start = d(2000.0, 1.0, 3.0);
let hol: Vec<f64> = (1..=200_000).map(|i| start + f64::from(i)).collect();
let out = ev("WORKDAY.INTL", vec![n(start), n(1.0), n(11.0), av(col(&hol))]);
// Sundays are the only weekend (11): 200,000 days of holidays are skipped, landing on the
// first working day after them.
let after = start + 200_001.0;
let sunday = ev("WEEKDAY", vec![n(after), n(2.0)]) == nv(7.0);
close(out, if sunday { after + 1.0 } else { after });
// Holidays on a weekend don't push the result out.
let fri = d(2026.0, 10.0, 9.0);
close(ev("WORKDAY", vec![n(fri), n(1.0), av(col(&[fri + 1.0, fri + 3.0]))]), fri + 4.0);
}
#[test]
fn day_counts() {
close(ev("DAYS", vec![n(d(2021.0, 3.0, 15.0)), n(d(2021.0, 2.0, 1.0))]), 42.0);
+45 -14
View File
@@ -171,13 +171,17 @@ fn sort(args: &[Arg], _c: &mut dyn Ctx) -> R<Value> {
let a = if by_col { a.transpose() } else { a };
let idx = if has(args, 1) { int_list(&arg(args, 1)?.value)? } else { vec![1] };
let ord = if has(args, 2) { int_list(&arg(args, 2)?.value)? } else { vec![1] };
if idx.is_empty() || idx.iter().any(|&i| i < 1 || i as usize > a.cols) || ord.iter().any(|&o| o != 1 && o != -1) {
if idx.is_empty() || idx.iter().any(|&i| i < 1 || i > a.cols as i64) || ord.iter().any(|&o| o != 1 && o != -1) {
return Err(CellError::Value);
}
if ord.len() > 1 && ord.len() != idx.len() {
return Err(CellError::Value);
}
let orders: Vec<i64> = (0..idx.len()).map(|i| *ord.get(i).or(ord.first()).unwrap_or(&1)).collect();
// A column repeated in sort_index can't break a tie its first use left, so only first uses
// stay: the keys copied per row are then at most the row itself, however long sort_index is.
let mut seen = std::collections::HashSet::new();
let (idx, orders): (Vec<i64>, Vec<i64>) = idx.into_iter().zip(orders).filter(|(i, _)| seen.insert(*i)).unzip();
let mut lines: Vec<(Vec<Value>, Vec<Value>)> =
rows_of(&a).into_iter().map(|r| (idx.iter().map(|&i| r.get(i as usize - 1).cloned().unwrap_or(Value::Empty)).collect(), r)).collect();
lines.sort_by(|x, y| cmp_keys(&x.0, &y.0, &orders));
@@ -928,9 +932,16 @@ fn pivotby(args: &[Arg], ctx: &mut dyn Ctx) -> R<Value> {
let ckeys: Vec<Vec<Value>> = p.rows.iter().map(|&r| key_row(&cf, r)).collect();
let rk = distinct_keys(&rkeys, if row_sort.unsigned_abs() as usize <= rf.cols { row_sort } else { 0 });
let ck = distinct_keys(&ckeys, if col_sort.unsigned_abs() as usize <= cf.cols { col_sort } else { 0 });
// Source rows of every row key, column key and pair, built in one pass, so a total reads
// its rows instead of rescanning the whole source once per total.
let mut cell: HashMap<(String, String), Vec<usize>> = HashMap::new();
let mut by_row: HashMap<String, Vec<usize>> = HashMap::new();
let mut by_col: HashMap<String, Vec<usize>> = HashMap::new();
for ((r, c), &i) in rkeys.iter().zip(ckeys.iter()).zip(p.rows.iter()) {
cell.entry((row_key(r), row_key(c))).or_default().push(i);
let (r, c) = (row_key(r), row_key(c));
by_row.entry(r.clone()).or_default().push(i);
by_col.entry(c.clone()).or_default().push(i);
cell.entry((r, c)).or_default().push(i);
}
let mut agg = |rows: &[usize], c: usize| -> Value {
if rows.is_empty() {
@@ -940,14 +951,12 @@ fn pivotby(args: &[Arg], ctx: &mut dyn Ctx) -> R<Value> {
aggregate_with(f, &vs, ctx)
};
let rows_for = |r: Option<&Vec<Value>>, c: Option<&Vec<Value>>| -> Vec<usize> {
let rkey = r.map(|k| row_key(k));
let ckey = c.map(|k| row_key(k));
p.rows
.iter()
.zip(rkeys.iter().zip(ckeys.iter()))
.filter(|(_, (a, b))| rkey.as_ref().is_none_or(|k| *k == row_key(a)) && ckey.as_ref().is_none_or(|k| *k == row_key(b)))
.map(|(&i, _)| i)
.collect()
match (r, c) {
(Some(r), Some(c)) => cell.get(&(row_key(r), row_key(c))).cloned().unwrap_or_default(),
(Some(r), None) => by_row.get(&row_key(r)).cloned().unwrap_or_default(),
(None, Some(c)) => by_col.get(&row_key(c)).cloned().unwrap_or_default(),
(None, None) => p.rows.clone(),
}
};
let mut cols: Vec<Option<&Vec<Value>>> = ck.iter().map(Some).collect();
if col_total > 0 {
@@ -997,10 +1006,7 @@ fn pivotby(args: &[Arg], ctx: &mut dyn Ctx) -> R<Value> {
}
};
for c in &cols {
let rows = match (r, c) {
(Some(rk), Some(ck)) => cell.get(&(row_key(rk), row_key(ck))).cloned().unwrap_or_default(),
_ => rows_for(r, *c),
};
let rows = rows_for(r, *c);
for vc in 0..vals.cols {
line.push(agg(&rows, vc));
}
@@ -1204,6 +1210,21 @@ mod tests {
assert_eq!(rows_of(&r), vec![vec![nv(0.0), nv(5.0)], vec![nv(1.0), nv(2.0)], vec![nv(1.0), nv(1.0)]]);
}
/// Security: a column repeated in sort_index keys the sort once. The keys copied per row stay
/// within the row, however long sort_index is, and a repeat (even with the other order) can't
/// change the result, since its first use already ordered the rows.
#[test]
fn sort_drops_repeated_sort_indices() {
let m = || av(arr(vec![vec![nv(1.0), nv(2.0)], vec![nv(1.0), nv(1.0)], vec![nv(0.0), nv(5.0)]]));
let want = vec![vec![nv(0.0), nv(5.0)], vec![nv(1.0), nv(2.0)], vec![nv(1.0), nv(1.0)]];
let r = ev("SORT", vec![m(), av(row(&[1.0, 1.0, 2.0, 1.0])), av(row(&[1.0, -1.0, -1.0, 1.0]))]);
assert_eq!(rows_of(&r), want);
let idx = vec![1.0; 1_000_000];
let tall = av(gridcraft_core::Array::filled(2_000, 1, nv(1.0)).into());
let r = ev("SORT", vec![tall, av(row(&idx))]);
assert_eq!(rows_of(&r).len(), 2_000);
}
#[test]
fn sortby() {
let r = ev("SORTBY", vec![av(col(&[1.0, 2.0, 3.0])), av(col(&[30.0, 10.0, 20.0]))]);
@@ -1447,6 +1468,16 @@ mod tests {
);
}
/// Security: PIVOTBY totals read their source rows from indexes built in the grouping pass,
/// rather than rescanning the source once per total, so many distinct keys stay linear.
#[test]
fn pivotby_totals_with_many_groups() {
let keys: Vec<f64> = (0..20_000).map(f64::from).collect();
let r = rows_of(&ev("PIVOTBY", vec![av(col(&keys)), av(col(&vec![1.0; 20_000])), av(col(&vec![2.0; 20_000])), t("SUM")]));
assert_eq!(r.get(1), Some(&vec![nv(0.0), nv(2.0), nv(2.0)]));
assert_eq!(r.last(), Some(&vec![tv("Total"), nv(40_000.0), nv(40_000.0)]));
}
#[test]
fn pivotby_size_check_counts_column_header_rows() {
// One row, n column fields and n value columns: n header rows + value-name row + body row + total row,
+62 -18
View File
@@ -30,17 +30,27 @@ fn check_len(s: String) -> R<Value> {
/// Pushes `piece` onto `out`, failing once the result exceeds the cell limit.
fn push_capped(out: &mut String, piece: &str) -> R<()> {
out.push_str(piece);
if out.len() > MAX_TEXT * 4 {
// Checked before copying, so one oversized piece is refused without being appended.
if out.len().saturating_add(piece.len()) > MAX_TEXT * 4 {
return Err(CellError::Value);
}
out.push_str(piece);
Ok(())
}
/// `v.to_text()` without copying text values.
fn piece(v: &Value) -> R<std::borrow::Cow<'_, str>> {
match v {
Value::Text(t) => Ok(std::borrow::Cow::Borrowed(t)),
Value::Array(a) => a.data.first().map_or(Ok(std::borrow::Cow::Borrowed("")), piece),
_ => v.to_text().map(std::borrow::Cow::Owned),
}
}
fn concatenate(a: &[Arg], _c: &mut dyn Ctx) -> R<Value> {
let mut out = String::new();
for x in a {
push_capped(&mut out, &scalar(x).to_text()?)?;
push_capped(&mut out, &piece(&scalar(x))?)?;
}
check_len(out)
}
@@ -49,31 +59,49 @@ fn concat(a: &[Arg], _c: &mut dyn Ctx) -> R<Value> {
let mut out = String::new();
for x in a {
for v in as_array(&x.value).iter() {
push_capped(&mut out, &v.to_text()?)?;
push_capped(&mut out, &piece(v)?)?;
}
}
check_len(out)
}
/// The error `v.to_text()` would give, found without building any text.
fn text_error(v: &Value) -> Option<CellError> {
match v {
Value::Error(e) => Some(*e),
Value::Array(a) => a.data.first().and_then(text_error),
_ => None,
}
}
fn textjoin(a: &[Arg], _c: &mut dyn Ctx) -> R<Value> {
let delims: Vec<String> = as_array(&arg(a, 0)?.value).iter().map(|v| v.to_text()).collect::<R<_>>()?;
// Errors are found first, in argument order, so they win as before; text is then built one
// piece at a time into the capped output, so many references to one long text (or a long
// delimiter list) stop at the cap instead of first copying every piece.
let delims = as_array(&arg(a, 0)?.value);
if let Some(e) = delims.iter().find_map(text_error) {
return Err(e);
}
let ignore = boolean(a, 1)?;
let mut items = Vec::new();
for x in a.iter().skip(2) {
for v in as_array(&x.value).iter() {
let s = v.to_text()?;
if ignore && s.is_empty() {
continue;
}
items.push(s);
}
let items: Vec<_> = a.iter().skip(2).map(|x| as_array(&x.value)).collect();
if let Some(e) = items.iter().flat_map(|x| x.iter()).find_map(text_error) {
return Err(e);
}
let mut out = String::new();
for (i, s) in items.iter().enumerate() {
if i > 0 && !delims.is_empty() {
push_capped(&mut out, &delims[(i - 1) % delims.len()])?;
let mut joined = 0usize;
for v in items.iter().flat_map(|x| x.iter()) {
let s = piece(v)?;
if ignore && s.is_empty() {
continue;
}
push_capped(&mut out, s)?;
if joined > 0
&& !delims.data.is_empty()
&& let Some(d) = delims.data.get((joined - 1) % delims.data.len())
{
push_capped(&mut out, &piece(d)?)?;
}
push_capped(&mut out, &s)?;
joined += 1;
}
check_len(out)
}
@@ -1183,6 +1211,22 @@ mod tests {
assert_eq!(rows_of(&ev("CONCATENATE", vec![av(row_t(&["a", "b"])), t("x")])), vec![vec![tv("ax"), tv("bx")]]);
}
/// Security: TEXTJOIN builds its result piece by piece and stops at the text cap, rather
/// than first copying every input (100,000 references to one 32,767-character text were
/// 3 GB of copies). Errors among the inputs still win, in argument order.
#[test]
fn textjoin_stops_at_the_text_cap() {
use gridcraft_core::Array;
let long = tv(&"x".repeat(32_767));
let many = || av(Array::filled(100_000, 1, long.clone()).into());
is_err(ev("TEXTJOIN", vec![t(","), b(true), many()]), CellError::Value);
is_err(ev("TEXTJOIN", vec![av(Array::filled(1, 100_000, long.clone()).into()), b(false), t("a"), t("b")]), CellError::Value);
is_err(ev("TEXTJOIN", vec![t(","), b(true), many(), e(CellError::Div0)]), CellError::Div0);
is_err(ev("TEXTJOIN", vec![av(arr(vec![vec![tv(","), Value::Error(CellError::NA)]])), b(true), t("a")]), CellError::NA);
// Delimiters cycle and are converted only where used.
is_text(ev("TEXTJOIN", vec![av(arr(vec![vec![nv(1.0), tv(";")]])), b(false), t("a"), t("b"), t("c"), t("d")]), "a1b;c1d");
}
fn row_t(xs: &[&str]) -> Value {
arr(vec![xs.iter().map(|x| tv(x)).collect()])
}
+5
View File
@@ -50,6 +50,11 @@ impl SheetApp {
self.session.mode = gridcraft_engine::Mode::Edit;
return;
}
// A hidden formula's source isn't loaded into an edit (a blank edit would also write over
// it on commit): no edit starts. Typing still replaces the cell.
if text.is_none() && gridcraft_engine::display::formula_hidden(&d.wb, sh, at) {
return;
}
let locale = self.ui.language.formula_locale();
let current = if sh.cell_pictures.contains_key(&at) {
String::new()