mirror of https://github.com/storytold/gridcraft.git synced 2026-11-03 08:51:30 -06:00

Files

T

PranavandFredApps aedbbba896 Data: Analysis ToolPak two-factor Anova, z-Test, Fourier Analysis, Random Number Generation and Sampling (#495 )

* Data: Analysis ToolPak two-factor Anova (with and without replication) and z-Test

Builds on the ToolPak of #320. data.analysisToolPak gains three tools:
Anova: Two-Factor Without Replication (rows and columns as factors, the
residual as the error term), Anova: Two-Factor With Replication (samples of
rowsPerSample rows, with the interaction and within-cell terms) and z-Test:
Two Sample for Means (known variances, normal tails and critical values from
NORM.S.DIST and NORM.S.INV). Each has a form in the Data Analysis dialog.
Tests check the reports against Montgomery's hardness-testing and
battery-life examples.

* Data: Analysis ToolPak Fourier Analysis

data.analysisToolPak gains Fourier Analysis: the discrete Fourier transform
or its inverse of one row or column of 2^k values (at most 4096), by an
iterative radix-2 FFT. Input cells are read as IMREAL and IMAGINARY read
them, so complex text such as 3+4i works, and the results are written as
COMPLEX text. Tests check known transforms (1, 2, 3, 4; a sampled cosine),
the inverse round trip and the input limits.

* Data: Analysis ToolPak Random Number Generation and Sampling

data.analysisToolPak gains the last two of Excel's 19 tools. Random Number
Generation draws uniform, normal, Bernoulli, binomial, Poisson and discrete
numbers, or writes a patterned sequence, in one or more columns; binomial and
Poisson draws invert tables built from BINOM.DIST and POISSON.DIST. Sampling
takes every period-th value or a random sample with replacement. Both use a
seeded xoshiro256** generator: the same seed repeats the numbers, and a run
without one reports the seed it chose. Output is capped at 1,000,000 numbers.

* Data: Analysis ToolPak two-factor Anova without replication: Labels caption names the first column

With Labels ticked, Anova: Two-Factor Without Replication takes the column
names from the first row and the row names from the first cell of every data
row. The form's caption read "Labels in first row", so a user with column
headers but no row names who ticked it would lose the first data column. The
caption now reads "Labels in first row and first column", and the command's
params doc says the same. A dialog test checks the caption.

* Data: Analysis ToolPak docs: timestamp lines and revision history for the 19 tools

docs/gaps.md (§13) and docs/target-app-parity.md (what-if and analysis) gain
their Last updated line and a revision-history row for the Analysis ToolPak
reaching all 19 of Excel's tools.

* Data: Analysis ToolPak new tools refuse options that are not numbers

Random Number Generation (from, to, mean, standardDeviation, p, lambda, step,
and the whole-number options) and z-Test (hypothesizedDifference) read their
options with a default for anything that did not parse, so a form value typed
as "0,3" or a typo quietly ran with the default. A blank or absent option
still takes the default; any other value that is not a finite number is now
an error naming the option.

* Data: Analysis ToolPak refuses an Alpha that is not a number

alpha() took any value that did not parse as the default 0.05, so a typed
"0,01" ran z-Test, both two-factor Anovas (and the existing t-tests, F-Test
and single-factor Anova, which share it) at the wrong significance level
without a word. A blank or absent Alpha still means 0.05; any other value that
is not a finite number is now an error.

* Data: Analysis ToolPak Random Number Generation form hides the count for a patterned sequence

A patterned sequence takes its length from From, To, step and the two repeat
counts and ignores Number of Random Numbers, so a count typed there did
nothing. The form shows the count only for the random distributions, and the
patterned options carry a note that the repeats set how many values are
written.

* Data: Analysis ToolPak Random Seed caption says a blank seed draws a new sequence

Random Number Generation and random Sampling labelled the seed "blank for
none", but a blank seed makes the engine choose a new one, so each run draws
different numbers. The caption now reads "blank for a new sequence".

* Data: Analysis ToolPak form scrolls its options in a short window

The Data Analysis window cannot be resized, and the patterned Random Number
Generation form (seven distribution choices and five pattern fields, each
label above its box) is taller than a 600-point window, so OK ended below the
window's edge. The tool's options now sit in a vertical scroll area capped to
the window's height less the title, output options and buttons, which always
stay in view.

* Data: Analysis ToolPak docs: Forecast Sheet is no longer listed as missing

docs/target-app-parity.md's what-if and analysis row still listed Forecast
Sheet as missing, while data.forecastSheet is in the catalog and docs/gaps.md
§13 lists it as present. The row now lists it with the other analysis tools,
and today's revision-history row says so.

* Data: Analysis ToolPak seed applies only where numbers are drawn

A patterned sequence and periodic sampling draw nothing, but the form showed
the Random Seed field for both, and Random Number Generation read the seed for
every distribution: a seed left in the form that was not a whole number
refused a patterned run, and a valid one was reported as the seed used. The
seed field now shows only for the random distributions and random sampling,
and a patterned run neither reads nor reports a seed.

* Data: Analysis ToolPak Fourier Analysis keeps small parts beside large ones

The cleanup after the transform zeroed every part at or below 1e-13 of the
largest output, so a one-value input such as 1e14 + 1i lost its imaginary
part although a transform of one value performs no arithmetic, and a small
harmonic beside a large DC term was lost the same way. The cutoff is now the
rounding bound of the transform itself, 8 · ε · log2(N) · Σ(|re_n| + |im_n|),
divided by N for the inverse (whose outputs are divided by N): it depends on
the input and the number of butterfly stages, never on the largest output,
and is zero for N = 1. An input sum that overflows disables the cleanup
instead of zeroing everything. Two regression tests cover the one-value case
and an eight-point round trip of 1e6 + 1e-7 cos(2πn/8).

* Data: Analysis ToolPak binomial and Poisson tables grow with the distribution, not the trials

Random Number Generation tabulated a binomial or Poisson distribution by
evaluating BINOM.DIST or POISSON.DIST through worksheet-function dispatch
for every outcome from 0 upward, so a million trials with p = 1 meant
1,000,001 calls and a table of as many entries. The table is now built from
one value at the mode and the recurrence pmf(k + 1) = pmf(k) · (n − k)/(k + 1)
· p/(1 − p) (λ/(k + 1) for Poisson), outward in both directions: the left
tail ends where its terms vanish, the right tail once the cumulative
probability reaches 1 within f64 resolution, so the table is as wide as the
outcomes that matter (about 23,000 entries for Binomial(1,000,000, 0.5)).
p = 0 and p = 1 are exact cases with one outcome. The trials limit and the
draws are unchanged. Tests check the recurrence against BINOM.DIST and
POISSON.DIST to 1e-12 relative, the table width at a million trials, and
that the largest inputs complete in milliseconds with the right means.

---------

Co-authored-by: FredApps <FredApps@users.noreply.github.com>

2026-11-02 07:03:40 +01:00

17 KiB

Raw Blame History

Gaps

Last reviewed: 2026-10-11 · Last updated: 2026-11-01 · Change: minor (§13: Analysis ToolPak has all 19 tools; earlier: §4: chart data tables implemented with XLSX round-trip and print/PDF; earlier: §13: Solver Options; earlier: §13: Solver Load/Save Model and Reset All) · Target: Microsoft Excel (Microsoft 365)

Every known shortfall against Excel, one entry each, ranked by user impact (what blocks someone using GridCraft for real work first). This is the work list: pick from the top, check the open PRs first (50 open on 2026-10-10; several already address entries here), and update the entry when you land work. Estimates are Opus 5.5 agent-hours. Area detail lives in the linked parity docs; the summary is in target-app-parity.md.

Kinds: F feature · U UI/UX · FF file format · H hardware · L localization · P performance · S stability.

Beta blockers

No alpha blockers: every core workflow passes the alpha gate.

1. XLSX output not proven in Excel or on real files (FF)

Missing: a run over a real-world XLSX corpus, and confirmation that Excel opens what we save with no repair prompt (PivotTables, charts, CF, validation, dynamic arrays, tables).
Evidence: round-trip tests cover our own files only (crates/xlsx/src/tests/); PivotTable round-trip noted as "unverified in real Excel" since 10-07; #64 (dynamic-array flag) shows the class of bug.
Impact: the main format; a repair prompt or silent loss destroys trust.
Estimate: 15–25 h, plus owner time in Excel. file-format-parity.md

2. Large workbooks don't open; recalc is single-threaded (P, H)

Missing: streaming read for ~100 MB XLSX; multi-threaded recalc; 100k-dependent edit under 100 ms (was ~0.4 s); whole-sheet operations bounded.
Evidence: #175; no threading in crates/calc; PR #152 (whole-sheet Fill/Remove Duplicates out of memory), PR #151 (current region past 10,000 rows).
Impact: finance and data users routinely have big workbooks.
Estimate: 25–40 h. target-app-parity.md, hardware-parity.md

3. Some unmodelled XLSX content is dropped on save (FF)

Missing: slicers and their caches, timelines, the data model, query tables attached to tables, unmodelled extLst entries, chart details. Pass-through (#330) keeps custom XML, external links, connections, sheet query tables, printer settings, form and ActiveX controls, OLE objects and sheet backgrounds; never checked in Excel (repair prompts unknown).
Evidence: crates/xlsx/src/passthrough.rs (what is kept, and why the rest isn't).
Impact: open-edit-save still simplifies workbooks with slicers, timelines or Power Pivot.
Estimate: 6–10 h. file-format-parity.md

4. Charts lose options on round trip; shallow chart model (F, FF)

Missing: axis scaling, label options, error bars, per-point formatting.
Added: data tables beneath column, bar, line, area and compatible combo charts, with legend keys, border controls, live source values, native XLSX round-trip and print/PDF.
Evidence: Chart holds only title, legend position, two flags and axis titles (crates/model/src/features.rs); PRs #158, #161 open; charts can't be resized (#190). Smooth lines (#62), Switch Row/Column (#116) and trendlines (read, written, drawn) landed. Multi-level category axes (year beside month) are read, drawn and written back. (crates/model/src/features.rs); all 2-D kinds survive XLSX round trip (#158, #161); charts can't be resized (#190). Smooth lines (#62), Switch Row/Column (#116) and trendlines (read, written, drawn) landed.
Impact: charts are in most business workbooks.
Estimate: 25–35 h for beta (40–60 h full). chart-parity.md

5. Text in non-Latin scripts doesn't render or export (L)

Missing: CJK font fallback, complex shaping (Arabic, Devanagari, Thai), bidi, non-Latin PDF.
Evidence: system-font fallback for CJK, Thai and Hebrew landed in #38 (closed #32); #142 (Chinese text) awaits confirmation; #54 (symbols as boxes on Fedora); no shaping for Arabic or Devanagari; PDF writer uses 14 standard fonts only.
Impact: blocks most of the world's users even with an English UI.
Estimate: 12–20 h. localization-parity.md

6. Calculation correctness bugs (F)

Missing: nothing known. The four this entry tracked have landed: 15-significant-digit equality (#57, so =0.1+0.2=0.3 is TRUE and MATCH and COUNTIF agree), spill updates after structural edits and cleared blockers (#133, #139; PRs #148, #157), 3-D span deletion (#140), and qualified names after a sheet rename (#134). Iterative calculation for circular references landed too, with its settings in File › Options › Formulas.
Impact: wrong numbers are worse than missing features, so this entry is kept until the oracle suite of §15 can say the category is clear rather than only these five reports being so.
Estimate: — function-parity.md

7. Legacy `.xls` only partly read; encrypted workbooks can't be saved with a password (FF)

Evidence: #121: Excel 97-2003 (BIFF8) .xls files now open (values, formulas, number formats, merges, sizes, names, and cell formatting: fonts, fills, borders, alignment); exact theme colours (XFExt), charts, pictures, comments, conditional formats, validation and BIFF5 files are not read, and .xls is never written. Password-encrypted .xlsx workbooks (Agile and Standard encryption) open with their password since #11, but are saved without it: GridCraft can't write an encrypted file yet. Encrypted .xls (FILEPASS) and RC4-encrypted files are still refused with a message.
Impact: common in finance, government and anything older than 2007; encrypted files are routine in business.
Estimate: 14–21 h (.xls objects 10–15, encryption 4–6). file-format-parity.md

8. PivotTables: no slicers, calculated items, PivotCharts; unverified in Excel (F)

Evidence: parity-checklist.md missing insert.slicer, insert.timeline, insert.pivotChart, table.insertSlicer; calculated items and GETPIVOTDATA missing (calculated fields and values on rows done). Show Details (double-click a value, subtotal or grand total cell) lists its source records on a new sheet. The "Enable show details" option (pivot.options enableShowDetails, saved as enableDrill) turns it off; it has no PivotTable Options dialog yet (command only).
Impact: pivots are the analyst's main tool.
Estimate: 20–30 h for beta (25–40 h full).

9. Keyboard-first users: key tips, shortcuts, context menus (U)

Next/previous sheet navigation now skips Hidden and VeryHidden worksheets without activating a hidden sheet at the workbook boundary. Other keyboard gaps below remain.
Evidence: #33, #49 (key tips: initial Home-tab support landed in #43 on 10-10, the rest of the ribbon not yet), #110, #28 (selection), 59 command shortcuts vs Excel's 200+.
Estimate: 12–20 h. ui-parity.md

10. Crashes and freezes reported from the field (S)

Evidence: #120 (Intel UHD crash on Windows), #117 (error on close, macOS), #166 (freeze opening from Finder; PRs #164, #167), #76 (flashing on Windows 11), #130 (caught panic in table styles).
Estimate: 8–12 h plus an autosave/recovery soak.

After beta

11. No localization at all (L)

Catalog: ribbon-only in five languages (#31, #104, #82; ~12% of strings each), and 943 strings (~43%) in Japanese (10-17) and Brazilian Portuguese, including the dialog labels, messages, File view and menus. The remaining dialogs, menus and messages are English; en-US function names and separators (#45); German in PR #163. Estimate: 100–165 h including #5. localization-parity.md

12. Get & Transform, external data, data model (F)

No Power Query editor, connections, refresh, From Web/database/folder sources, relationships or data model (data.fromWeb, data.relationships missing). Estimate: 60–100 h.

13. Analysis tools: Solver, Analysis ToolPak, Forecast Sheet (F)

Forecast Sheet (data.forecastSheet) and the FORECAST.ETS family are in, including Forecast Start (hindcast, #785).
Solver (data.solver, #327) has Simplex LP with integer/binary branch and bound, a local nonlinear method (augmented Lagrangian, not GRG), the Evolutionary method, and Answer and Limits reports. Report refusal leaves the solution, saved model and history unchanged; a successful solve and its reports share one Undo step (#1050). Load/Save Model writes the model on cells of its sheet and reads it back; Reset All removes it. Solver Options sets the evaluation limit, Integer Optimality and the Evolutionary options, which are saved with the model. AllDifferent constraints and the Sensitivity and Population reports remain.
The Analysis ToolPak (data.analysisToolPak) has all 19 of Excel's tools; their chart output remains. Estimate: 12–20 h.

14. Page Layout view and print polish (U, F)

Page Layout view landed (#329): paper view with rulers and in-place header/footer editing. Margins are dragged on the rulers (#792). Remaining: pictures in headers/footers, Different First Page / Odd & Even pages. Desktop now submits directly through craft-print with printer choice, copies, duplex and grayscale; browser printing stays available. Print ranges and driver tray/options UI remain. Estimate: 5–10 h.

15. Remaining functions and an oracle suite (F)

18 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. Estimate: 16–26 h.
20 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. Estimate: 17–29 h.
21 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. Estimate: 18–30 h.
18 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. Estimate: 18–30 h. function-parity.md

16. Collaboration and review (F)

No co-authoring, Show Changes, comment navigation, notes ↔ comments conversion; cloud storage requested (#86). Estimate: 35–55 h (co-authoring scope needs the owner).

17. Insert objects (F)

Ink strokes now survive XLSX save/reopen as editable polylines. Inserted icons still lose their identity/artwork on save (#551); native Office InkML import is not implemented.
SmartArt, WordArt, equations, signature line, objects, 3-D models, screenshot; 8 shape kinds vs ~170; pictures in cells (#61, PR #105). Estimate: 25–40 h.

18. Accessibility (U)

AccessKit on, but no accessible grid/ribbon tree, no high contrast, no checker. Estimate: 12–18 h. ui-parity.md

19. XLSB/ODS beyond values; minor formats (FF)

XLSB and ODS import values only, no write; no .prn, .dif, .slk, XML Spreadsheet 2003; CSV writes raw instead of displayed values; templates saved with the workbook content type. Estimate: 15–23 h.

20. Multiple windows; pen pressure (H)

Desktop windows done: a native window per workbook window, New Window on the same workbook, Switch Windows, Arrange All. Left: Arrange All across monitors, one window on the web, no pressure or tilt in ink. Estimate: 5–8 h. hardware-parity.md

21. Automation and add-ins (F)

VBA out of scope (dropped on open, #6); no Office Scripts compatibility; no plugin or custom function API. Estimate: 25–40 h plus an owner decision.

22. AI features (F)

No in-app assistant, natural-language formulas or Analyze Data; agent control via MCP is already strong. Estimate: 30–50 h; model choice needs the owner.

23. Mobile (platform)

No iOS, iPadOS or Android layout. Estimate: 30–60 h.

24. Fonts and look (U)

Font weights beyond regular/bold (#168), system font list (#4), Calibri metrics (#29), macOS chrome (#169, #170), Linux title bar (#54), icon consistency across apps (#173). Estimate: 7–12 h.

Revision history

Date	Change	Summary
2026-11-01	minor	§13: the Analysis ToolPak has all 19 of Excel's tools (two-factor Anova with and without replication, z-Test, Fourier Analysis, Random Number Generation, Sampling); its chart output remains. The stale "`data.solver`, `data.analysisToolPak` missing" and "no Forecast Start" sentences removed
2026-11-01	minor	§4: chart data tables implemented for column, bar, line, area and compatible combo charts, with legend keys and border settings, live source values, XLSX round-trip and print/PDF; other chart gaps remain.
2026-10-31	minor	§13: Solver Options (Max Evaluations, Integer Optimality and the Evolutionary options), saved with the model
2026-10-31	minor	§13: Solver Load/Save Model (`data.solverSaveModel`, `data.solverLoadModel`) and Reset All (`data.solverReset`), with buttons in the Solver Parameters dialog
2026-10-24	minor	§8: PivotTable Enable show details: `pivot.options` `enableShowDetails` turns Show Details off for a PivotTable, read and written in XLSX as `enableDrill`
2026-10-23	minor	§8: PivotTable Show Details: double-clicking a value cell lists its source records on a new sheet (#2364)
2026-10-22	minor	§20: a native window per workbook window, New Window on the same workbook and Arrange All on the desktop; left: Arrange All across monitors, the web's one window, pen pressure (estimate 10–14 h → 5–8 h)
2026-10-20	minor	§11: the Japanese catalog covers the dialogs, messages and menus — 943 keys (~43%), matching the pt-BR coverage; a test keeps the two catalogs in step
2026-10-17	minor	§6: the five reports it tracked are closed and their PRs merged (#57, #133, #139, #140, #134; PRs #148, #155, #156, #157); the entry stays open for §15's oracle suite
2026-10-15	minor	§6: Maximum Iterations and Maximum Change in File › Options › Formulas; dropped the stale "no Options dialog yet"
2026-10-15	minor	§13: Solver prepares reports before committing the solution, model and report sheets together; report refusal preserves the workbook and Undo/Redo (#1050)
2026-10-13	trivial	§4: multi-level category axes landed
2026-10-12	minor	§9: next/previous sheet commands retain the current visible sheet and selection when there is no visible destination
2026-10-12	minor	rows auto-fit oversized fonts on open and whole-row/column formatting (#833)
2026-10-12	minor	Page Layout view: dragging margins on the rulers (#792) removed from the §14 remainder
2026-10-12	minor	§13: Forecast Sheet Forecast Start (hindcast) landed (#785)
2026-10-12	minor	§17: ink geometry survives XLSX saves; the icon portion of #551 remains
2026-10-12	minor	§8: values on rows (Σ Values in Rows, `dataOnRows`) laid out, moved and round-tripped
2026-10-12	minor	Desktop File > Print now discovers printers and submits PDF jobs through pinned craft-print with printer choice, copies, duplex and grayscale; browser printing and PDF export remain available. Remaining print parity includes page ranges and driver-specific tray/options UI; physical printer fidelity needs hardware validation.
2026-10-11	minor	§6: iterative calculation for circular references landed (Maximum Iterations, Maximum Change)
2026-10-11	minor	§15: trim-reference operators (`A1:.E10`, `A1.:E10`, `A1.:.E10`) parse and evaluate
2026-10-11	minor	§4: trendlines (six types, forecast, equation and R²) read, written and drawn
2026-10-11	minor	§7: password-encrypted workbooks open with their password; saving with one remains (#11)
2026-10-11	minor	§16: comment navigation, Show Comments, Show All Notes and Convert to Comments landed (#281)
2026-10-11	minor	§7: `.xls` (BIFF8) import of values, formulas and number formats (#121)
2026-10-11	minor	§3: XLSX pass-through keeps custom XML, external links, connections, query tables, printer settings, controls and OLE objects (#330, #324)
2026-10-11	minor	§13: Forecast Sheet landed (#322)
2026-10-11	minor	§15: ODDFPRICE, ODDFYIELD, ODDLPRICE, ODDLYIELD added (18 functions missing)
2026-10-11	minor	§15: PERCENTOF and EUROCONVERT added (20 functions missing)
2026-10-11	minor	§15: BAHTTEXT added (21 functions missing)
2026-10-11	minor	§15: TRIMRANGE added (21 functions missing; trim-reference operators still missing)
2026-10-11	minor	§16: Allow Edit Ranges landed (#307)
2026-10-11	minor	§8: calculated fields added
2026-10-11	minor	§8: PivotCharts added
2026-10-11	minor	§13, §15: FORECAST.ETS family landed (#314)
2026-10-10	minor	Updated for the 10-10 merges (#17, #31, #38, #62, #82, #104, #116) and the new chart resize bug (#190)
2026-10-10	minor	Noted that no gap blocks the alpha gate
2026-10-10	major	Created: 24 ranked gaps with evidence, from the full re-measure and the 60 open issues

17 KiB Raw Blame History Unescape Escape