- macro.save and a stopped recording count the macros kept in a newer format with the readable
ones, so a save never holds more than the next open keeps; the message names the kept count
(kept_macros_count_toward_the_limit_and_keep_their_names, which saves and reopens as xlsx and
JSON).
- The names of kept macros are taken: macro.save and macro.rename refuse them, unique_name and
next_name step around them (same test).
- macro.save no longer replaces by default: `replace` defaults to false, the error names the
existing macro and says to pass `replace: true`, and a `replace` that is not a boolean is
refused (saving_over_a_macro_needs_replace).
- macro.rename runs the whole macro check, size included, so a longer name can't take a macro
past what an xlsx file holds (renaming_checks_the_macro_size).
- Docs: macros.md, the management commands and kept macros.
- `ok` in `macro.check` meant only that the dry run hit no error. It now says whether
`macro.run` with the same parameters would go ahead without a question, and `errors` says
why not: a failing step, a Stop step that rolls back, data written over under `ask` or `fail`,
a shift that can't make room (the refusals `make_room` makes, now in `shift_refusal`), or a
run past the step or parameter limit once the dry runs before it count. `macro.check` takes
`onConflict` and `overwrite` as `macro.run` does, reports the rule it used, and refuses the
parameters `macro.run` refuses.
- After a partial footprint, the question under `ask` says where the check stopped and that
overwriting also lets the later steps write over data unchecked.
- The dry runs and the run share the parameter budget as they share the step budget.
- A dry run puts back the Go To list (`edit.goTo` with `record: true`).
- The footprint groups the touch log by sheet once, instead of filtering a million entries once
per sheet.
- Docs: what `ok` means; RAND's sequence moves on after a check, as after any recalculation.
- Tests: `the_check_says_whether_the_run_would_go_ahead`,
`the_question_after_a_partial_footprint_says_what_the_answer_covers`,
`a_dry_run_puts_back_the_go_to_list`, the shared parameter budget, and `ok` under `ask` in
`conflicts_ask_fail_or_overwrite`.
- A run that rolls back after writing a file no longer says "Nothing was changed.": the message
says the workbook was put back and names the step whose file stays (`StepError::wrote_files`,
also after a Stop with rollback).
- `macro.run` refuses an input the macro doesn't have (a misspelt input bound the selection),
an input that isn't reference text, and `onConflict`, `overwrite`, `approve` or `timeLimit` of
the wrong kind, which it used to ignore; `null` still counts as not given.
- A step no longer copies itself, and Repeat and If no longer copy their bodies, on every pass:
a body is copied once per run. A run passes its commands at most 256 MB of parameters, all
steps together, which bounds a run's work on the web build, where there is no clock.
- Every step except an approved file step runs inside the scripts' file sandbox, so a command
that touches a file under a parameter other than `path` can't do so from a macro.
- Docs: moving cells that hold data counts as writing over them.
- Tests: `run_parameters_that_would_be_ignored_are_refused`,
`a_run_passes_a_bounded_amount_of_parameters`, and the rollback message in
`each_step_is_checked_for_writes_over_data`.
Inputs, the output and a Repeat's range were bound once, by sheet index and coordinates. A step
that inserted a sheet in front of an input's sheet sent every later token to the new sheet, and
rows inserted above an input left the tokens on the cells that moved into its place; the run
wrote into the wrong cells and reported success.
- Commands record the structural edits they make as they rewrite references (`Ctx::edits`; while
a macro step runs, `DocState::edits` collects them).
- After each step the machine matches the sheets as the run's guard does and moves the inputs,
the output and the ranges of the Repeats in progress by those edits, with the formula engine's
own adjustment.
- A step that deletes an input's or the output's cells, or their sheet, or changes the size of a
range a Repeat is going over, stops the run, which rolls back.
- Test: `inputs_and_the_output_follow_their_cells_when_a_step_moves_them` (a sheet inserted in
front, rows inserted above, an input deleted, a Repeat's range grown).
A footprint is not proof of what the run will do. RAND gives the run other values than the dry
run before it (the recalculation's random sequence goes on), and NOW moves on, so an If can take
another branch in the run and write over data the dry run never reached; under `fail` the run
went ahead and wrote over it.
- Unless the rule is `overwrite`, every run checks each step as it goes, as a run after a
partial footprint already did; a write the footprint didn't see stops the run, which asks under
`ask` and fails otherwise, and rolls back.
- Test: `a_run_that_takes_another_path_than_its_check_is_still_checked` (an If on `=RAND()` over
40 runs under `fail`; before, one of them wrote over B1).
Split from the pull request below at the review's request; the code is the runner's footprint
as reviewed there, unchanged.
- `macro.run` first runs the macro in dry-run mode on the real workbook (unless the rule is
`overwrite`) and puts everything back: the workbook, its selection and history through the
dry run's undo group, and the session state a dry run may touch (`Saved`: the active
workbook, open documents, the saved mark, selections, unlocked ranges, trust, clipboard,
preferences, mode, journal and recording, queued requests, format painter, find, ruler,
hidden windows, drawing tool and the comment panes).
- Cells it would write outside its inputs that hold data, and sheets and objects it would
delete, are conflicts: `ask` asks before anything runs, `fail` fails, and `shiftDown` and
`shiftRight` insert cells over the block of every written cell and check again. A shift is
refused when it would move an input, when the macro deletes sheets or objects, or when the
touch log is full.
- A dry run stops before a file step and before Solver; the footprint is then partial, and the
run checks each step as it goes.
- The dry runs and the run share one clock, one time limit and one step budget.
- `macro.check` reports the inputs, the output anchor, approval needed and the footprint.
- Tests: the footprint tests that moved with this code.
The review asked for the runner to be split. This pull request keeps the policy, the input
checks, the step machine and `macro.run`; working out before a run where a macro writes (the
footprint dry run, shifting data to make room, `macro.check`) moves to the pull request above it.
- Unless the conflict rule is `overwrite`, every run now checks each step as it runs, which the
run did before only after a partial footprint: a step that writes over cells outside the inputs
that held data when the run started, or deletes sheets or objects that were there, stops the
run, which asks (`ask`) or fails, and the run rolls back.
- `shiftDown` and `shiftRight` need the footprint, so here they stop the run as `fail` does.
- Moved: `Footprint`, `Saved`, `footprint`, `diff`, `make_room`, `macro.check`, the dry-run
branches of the machine, `policy::previewable`, `DocState::touched` and the touch log,
`Session::journal_dropped` and `truncate_journal`, and the tests of those.
- Test: `each_step_is_checked_for_writes_over_data` (the review's counterexample: a permitted
file step or Solver, then a write over B1, under `fail` and `ask`).
Main adds UiRequest::InsertName (Paste Name, Use in Formula), which puts a name in the cell
editor; a macro step drops it, as it drops a request to start editing a cell.
The recorder expanded an automate.runScript step from the workbook as it
stood when the recording stopped, so a script edited or deleted during the
recording changed the saved macro, or dropped the step. The run now leaves
the steps it ran with the recorder (Recorder::ran, by step index, within
the byte budget), and the conversion reads those, falling back to the
workbook only when they could not be kept.
Every path that adds a macro runs macros::check, and an xlsx save of a
macro that skipped it fails (the stash refuses the oversized item). A
JSON save wrote such a macro instead, and the next open dropped it with
a warning. json_bytes now checks each macro first, so the save fails in
either format with the check's message and the macro stays in the open
workbook. Tests that build hostile JSON files write them directly.
The stash reads a legacy _sc_<key> name as data whether or not it is
hidden, so writing key macro_0 removed a person's visible _sc_macro_0
name with no word. The index allocator steps around that key as well;
this test fails without it.
One file with a visible name of the storage shape, a macro in a newer
format and one whose input declaration breaks the rules, saved and
reopened xlsx then JSON and JSON then xlsx: the visible name stays, the
newer macro comes back whole and numbered around the name, the broken
one is left out once, and the readable one survives.
A macro whose format version is above this build's was dropped on open,
so the next save removed it from the file, and the warning said only
that it couldn't be opened. It is now kept as the file held it
(Workbook::macros_unread, shared between snapshots), never read, listed
or run, and written back unchanged by JSON and xlsx saves, numbered
after the readable macros and around visible names of the storage
shape. Kept macros count toward the 1,000-macro limit after the
readable ones; one too large for an xlsx item is left out. The warning
says it is kept and saved back; damaged and over-limit macros keep
their own warning.
check_limits capped the name, description, notes, step count, depth and
input names, and left the shortcut, permissions, output anchor, default
range, column types, command ids, Repeat ranges, If cells and values and
Stop messages bounded only by the item size; the input declaration was
stored unchecked (rows [5, 2], a cell input with cols [2, 5]).
Now each of them has a limit (docs/macros.md lists them), one-line
fields refuse control characters, and a command id is letters, digits,
. and _ (every registered id passes, tested). An If value may be null
(Empty and NotEmpty ignore it); Repeat.max stays the runner's to bound.
An input's least rows and columns can't pass its most, a cell input is
one row and one column with no header and at most one column type, and
no input lists more column types than columns. check_limits takes every
struct apart field by field, so a new field doesn't compile there until
its limit is decided. A macro from a file that breaks a rule is left
out with the usual warning, never changed to fit.
The JSON reader replaces a workbook with an empty sheet list by a new
one before macros::load reads the file's macro entries, so they were
lost with no warning. The entries are carried across the reset. The
reset itself, which also starts the names and styles afresh, is
unchanged.
with_names numbered macros 0..N, and stash::set_many refuses to write a
hidden name beside a visible one of the same name, so a visible
_gc_macro_0_0 (from a file, or defined after the file opened, when the
stored names are no longer in the workbook) made every xlsx save of a
workbook with macros fail. The index is GridCraft's choice: macros now
take the free indices from 0 up, skipping every key a visible
workbook-scope name of the storage shape uses (compared as the stash
reads names), in their order. No arithmetic depends on a number from
the file.
Six fixes from a review of the recorder, each with a regression test (four
fail against the old recorder; the other two bound work that grew with the
input):
- A step that names another sheet with its sheet parameter (cell.set and
about fifty other commands) was tokenized as if on the input's sheet, so a
recorded cell.set {sheet: Two, cell: C4} ran on Two at the input's
relative place. Such a step keeps its references, like a sheet-qualified
one.
- Inlining recorded runs of other macros and scripts stops at the macro's
step limit and counts the rest in dropped, instead of expanding every run
first (a thousand runs of a long macro made millions of steps).
- macro.record {cancel: true} ends a recording without saving it: the way
out of a recording whose macro is too large for the workbook, which could
neither be saved nor ended.
- Formula tokenization reads each reference in the formula and matches it,
instead of listing every column and row of the example (an example as
large as a sheet made a million candidates per formula).
- A selection of several areas keeps its active cell as it is; a token for
the active cell alone put it outside the areas on playback.
- Steps recorded in another workbook are counted in dropped.
The touch log keeps at most a million writes; past that, a write that kept a
cell's value is missing from the footprint, so a shift derived from the
block of written cells could split a row the macro writes. The footprint
now says when the log filled up (logFull in macro.check), and a shift is
refused then, as it is when the macro deletes sheets or objects. Regression
test with a million-cell fill in a dry run.
Every nested command a dry run executes is journaled, so a long macro pushes
the journal past its limit and the journal drops its oldest entries. The
restore cut the journal back to its old length, which then kept entries from
the dry run and pointed Record Actions at the wrong place. The session now
counts the entries its journal has dropped (journal_dropped) and truncates
with its size bookkeeping (truncate_journal); the restore keeps the old
length less what was dropped meanwhile and moves the recording's start back
by the same count. Regression test with 12,000 nested steps.
Four fixes from a review of the runner, each with a regression test that
fails without it:
- A shift inserts cells over the block of every written cell, which moves
that block's columns (or rows) from there on. When a bound input lay in
the way, its data moved and sheared while the run kept the old coordinates
and reported success. The shift is now refused, naming the input and its
place. The output is not counted: it is where the macro writes, and the
second footprint catches data a shift moved into it.
- A dry run or a refused run left a clean workbook dirty: view.zoom,
sheet.activate and the view modes move the saved mark to the current
workbook on a clean document, and the rollback put the workbook back but
not the mark. Saved keeps and restores it.
- overwrite: true answers the question however ask was given; with an
explicit onConflict: ask the answer was ignored and the question came back
forever.
- The dry runs and the run share one step budget as they share one clock;
each phase counted its own 100,000 steps before. ran reports the run's
own steps.
Input.col("date").data resolved to the whole column, header included, because
every method cleared the header state. col and cols keep it; the other
methods still clear it, and only data keeps the header row for col("…").
The token test asks for D3:D11, cols(2..3).data and col(2).data.col(1).
Six fixes from a review of the resolver, each with a regression test that
fails without it:
- A bare Row or Column under a general reference key (to, from, at, area,
source, target, destination, location) stays text: Change Spelling's
replacement word and a pivot field's area were being rewritten into the
current row's A1 range. Row.col(2), {{Row}} and {"$ref": "Row"} still
resolve there, and the bare word still resolves under cell and range keys.
- {{{{ in a string is a literal {{, so text that needs the braces can be
written; an unknown token is still an error.
- Input.data on a range that is only a header row, and Name: of a table
with no data rows, are refused instead of reading the header.
- below(0), below(-3) and right(0) are refused like resize(0, 1) is, instead
of becoming 1.
- A defined name on a sheet whose name holds a dollar sign resolves: the
dollar anchors are stripped from the reference part only.
- quote_sheet is the engine's one rule (gridcraft_core::quote_sheet), which
quotes a sheet named like a cell reference, an R1C1 token or a boolean and
leaves a period unquoted.
docs/macros.md says which keys take a bare token and how to write a literal
brace.
A macro could declare an input called Q1, AB12 or Sum1, and a step written
with plain A1 text, as a recorded one is, would then resolve to the input's
range instead of the cell. check_limits refuses such names; the docs say so.
Regression test.
- Workbook::macros is a shared list (Macros, a copy-on-write newtype): a
workbook copy made for a cell edit or an undo snapshot holds the same list,
and the undo budget charges a list once a snapshot owns its own copy.
Before, every mutating command copied every step of every macro, and a
hundred undo entries kept a hundred copies the budget never saw.
- Saving to xlsx drops the last save's items and writes the macros in one
pass each (stash::set_many), instead of a scan of every name per macro.
- A hidden _gc_macro_<x>_<k> name with a key this build did not write is
kept on open and on save, as a visible one is, instead of being dropped
without a warning.
- macros::check is the one check for every path that adds a macro (JSON, a
file, built in code): the model's limits and the xlsx size limit.
Regression tests for each.
The depth check refused any step list at the fifth level, so a fourth Repeat
or If could not even hold an empty body: three levels nested, not the four
MAX_MACRO_DEPTH and the docs state. The check now counts the Repeat or If
steps themselves, in the model's limits and in the parser's pass over the
JSON. The test asks for four and refuses a fifth.
Stopping Record Actions saves a script into the workbook, which the debugger lock refuses while a
macro there is paused. The refusal came after Record Actions had taken its recording, so the
recording was lost. Stopping is now refused before anything changes, and stopping again after
the run saves the script.
`col("Amount")` read the header through the cell's display text, which is the formula while
Show Formulas is on, so a header such as ="Amo"&"unt" matched nothing. The header is now the
cell's value as formatted, whatever the sheet shows.
A visible workbook-scope name such as `_gc_macro_0_0` matched the storage key and was read as a
macro and removed when the workbook opened. Macro storage is written hidden, so only hidden names
are read back; a person's own name of that shape stays a name.
Session::execute checked the debugger lock only for top-level undoable commands. A command that is not undoable could run edits as nested commands and skip it: stopping Record Actions while a macro in the same workbook was being debugged saved the script inside the run's undo group, and stopping the run without keeping its changes deleted the script again. The lock now applies at every depth, and only the debugged run's own steps go ahead.
`Name:` read a name only up to the first period, so `Name:Sales.Q1` looked up `Sales` and
treated `Q1` as a method, and a workbook name called Sales.Q1 could not be used in a macro,
alone or as `Name:Sales.Q1.cell(1,1)`.
The name now runs through periods, and the longest defined name or table that exists and ends
at a period (or the end) is used; what follows is read as methods. So `Name:Data.row` is a
name called Data.row, and `Name:Sales.offset(1,0)` still moves Sales. When no name matches,
the error names the text before the first method, and a bad method after a dotted name is
still refused. Tests cover direct resolution, reference parameters, `$ref` and `{{...}}`,
with and without a chained method.
Opening a workbook gave each clashing macro name a suffix by trying " (2)", " (3)"... in turn,
and each try scanned every macro accepted so far. A file with 1,000 macros of the same
255-character name made about 300 million name comparisons.
Load now keeps a set of the names in use (ASCII case-insensitive, as find compares them) and,
for each shortened name and suffix width, the next suffix to try, so each suffix is looked up
about once. The result is unchanged: the smallest free suffix, within the name length limit,
and the renamed macro still passes the final checks. unique_name keeps its signature. A test
opens, saves and reopens 1,000 same-name macros in JSON and xlsx and keeps them all, and
another counts lookups instead of timing them.
prxnvt