Compare commits

...
Author SHA1 Message Date
Dylan M. Taylor 4251dad939 Merge remote-tracking branch 'origin/switch-craft-libs-csv' into codex/reviewfix-3842 2026-11-12 05:41:06 +00:00
Dylan M. Taylor 1b7e4f00dc Merge remote-tracking branch 'origin/switch-craft-libs-xls' into codex/reviewfix-3841 2026-11-12 05:40:59 +00:00
Dylan M. Taylor 4deedbb549 warn about dropped XLS rules 2026-11-12 05:40:53 +00:00
Dylan M. Taylor d3954beacf Merge remote-tracking branch 'origin/switch-craft-libs-csv' into codex/propagate-3842 2026-11-12 05:37:21 +00:00
Dylan M. Taylor 26e7112e0f Merge remote-tracking branch 'origin/switch-craft-libs-xls' into codex/propagate-3840 2026-11-12 05:37:12 +00:00
Dylan M. Taylor 3d31729de3 test(xls): expect malformed encryption error 2026-11-12 05:36:34 +00:00
Dylan M. Taylor d0c567992e Merge current CSV switch into encrypted workbook switch 2026-11-12 05:18:19 +00:00
Dylan M. Taylor a440cfac28 Merge remote-tracking branch 'origin/switch-craft-libs-offcrypto' into codex/gridcraft-3842 2026-11-12 05:14:54 +00:00
Dylan M. Taylor 92a79db048 GridCraft: update encrypted XLS switch to current shared crates 2026-11-12 05:14:47 +00:00
Dylan M. Taylor 652c08a89b Merge remote-tracking branch 'origin/switch-craft-libs-csv' into codex/gridcraft-3842 2026-11-12 05:14:07 +00:00
Dylan M. Taylor aeb3fb19ce Merge latest CSV branch updates 2026-11-12 05:13:54 +00:00
Dylan M. Taylor e4ec0d7ed1 GridCraft: update CSV switch to current shared crates 2026-11-12 05:13:10 +00:00
Dylan M. Taylor 23487b0cb5 Merge current XLS switch into CSV switch 2026-11-12 05:12:51 +00:00
Dylan M. Taylor 0407b63663 GridCraft: update XLS import to current shared crate 2026-11-12 05:11:53 +00:00
Dylan M. Taylor c913bf6615 tests: give the xls fixture its compound-file byte order mark
craft-cfb refuses a header without the 0xFFFE mark at offset 28
([MS-CFB] 2.2), so craft-xls's sniff called the fixture LegacyBinary.
The old hand-written parser never read that field, which is why the test
only broke with the switch. The fixture is a real workbook otherwise:
sniff now says Xls and read_xls parses it.
2026-11-12 05:08:20 +00:00
Dylan M. Taylor 1bd4ca8946 Test xls import with a valid compound file header 2026-11-12 05:05:01 +00:00
Dylan M. Taylor 0431741359 tests: give the xls fixture its compound-file byte order mark
craft-cfb refuses a header without the 0xFFFE mark at offset 28
([MS-CFB] 2.2), so craft-xls's sniff called the fixture LegacyBinary.
The old hand-written parser never read that field, which is why the test
only broke with the switch. The fixture is a real workbook otherwise:
sniff now says Xls and read_xls parses it.
2026-11-12 04:37:07 +00:00
Dylan M. Taylor 9c87c22662 xlsx: encrypted workbooks and compound files through craft-offcrypto and craft-cfb
Decrypting password-protected workbooks ([MS-OFFCRYPTO], Agile and Standard) moved
to storytold/craft-libs as craft-offcrypto (#931), on craft-cfb. GridCraft keeps
its DecryptError and the engine's messages; crypto.rs maps the crate's errors onto
it. The private compound-file reader and its tests are gone: sniffing looks for
the encryption streams with craft_cfb::has_entry_named (with the same fallback for
a damaged directory), and the test builders encrypt with craft-offcrypto and write
compound files with craft-cfb.

One difference, as craft-offcrypto has it: the dataIntegrity HMAC key is used at
the length it decrypts to (up to the hash size); GridCraft accepted only a
salt-size or hash-size key. Both verify what Office writes.
2026-11-12 01:34:24 +00:00
Dylan M. Taylor c1adf9b239 xlsx, engine: CSV decoding, parsing and quoting, and SYLK, through craft-csv
Text decoding (byte order marks, UTF-8, Windows-1256/1252), delimiter sniffing,
the record parser and field quoting moved to storytold/craft-libs as craft-csv
(#889), with the SYLK reader as craft_csv::sylk. GridCraft keeps typed values,
date-order inference, the used-range limit and the sheet writer in csv.rs, and in
io/sylk.rs the workbook a SYLK file becomes: formulas parsed (an unparsable one
is skipped and counted, as before), error values, number-format styles and column
widths.
2026-11-12 01:17:21 +00:00
Dylan M. Taylor 21df49c636 xlsx: read .xls workbooks through craft-xls
The BIFF8 reader moved to storytold/craft-libs as craft-xls (#878). GridCraft
keeps the mapping onto its model in xls.rs: formulas parsed, Excel's implicit @
for legacy formulas, pixel widths and heights, styles interned per XF, sheet-name
checks, print areas and titles, calculation settings and GridCraft's warning text.

The workbook now opens on the sheet it was saved on when that sheet is visible,
else on the first visible sheet (from #3551 by @Morzilla, whose test is kept).
RString cells are read as text.

Every craft-libs crate moves to one revision, 119415f4.
2026-11-12 00:58:43 +00:00
FredApps 7e0d96d74b CI: nightly release dry run (#3820)
release.yml also runs nightly on main with craft-common-actions' dry-run: every release build
except macOS, with nothing signed, pushed or drafted. Breakage that only the release build hits
(riscv64, Flatpak, FreeBSD, packaging) then shows up the next morning instead of on release
day, as it did on 11 Nov 2026.
2026-11-11 23:04:17 +01:00
Dylan M. Taylor 572bec289b engine: work limits for Advanced Filter, Remove Duplicates, filters, AutoFit, macro footprints; range sizes saturate (#3295)
* engine: bound Advanced Filter, Remove Duplicates, AutoFilter, Circle Invalid Data; one-pass AutoFit; saturating range dimensions

* address review: AutoFit scans each run of adjacent selected columns separately

* address review: report a truncated circle-invalid scan, no spill scan for plain cells in Replace, one pass for many AutoFit runs

* address review: warn in the app when Circle Invalid Data stopped early; Replace test requires success

* address review: the scan note joins an open-time notice; say cell visits
2026-11-11 16:10:10 -05:00
Dylan M. Taylor 55eb394044 engine: index merges and bound the page in HTML export and clipboard HTML (#3266)
* engine: index merges and bound the page in HTML export and clipboard HTML

Save as Web Page and the clipboard's HTML scanned every merge of the
sheet for every cell, counted a merge's span over all the rows the file
gave it, placed each picture by summing every custom row before it, and
grew the page without a byte limit. Merges are now indexed once over
the exported rectangle (overlapping merges past a few visits per cell
are refused), spans use prefix counts clipped to the table, anchors use
LineIndex, and the page stops at 256 MiB with an error.

* address review: check the web page size after each chart and before returning it

* address review: bound the merge index, escape cell text within the page's room, size images before encoding

* address review: escape image attributes within the page's room
2026-11-11 16:09:43 -05:00
Dylan M. Taylor 9e9f13114f check, xtask stats: visited directories keyed by (dev, inode); stats reads only regular files (#3287)
* Directory walkers key visited directories by (dev, inode); stats skips non-regular or oversized .rs targets

* address review: record a walked directory only within the depth limit; stats skips symlinked sources and bounds the read

* address review: rescan a directory reached from a shallower path, skipping files already collected

* xtask stats: test the directory walk (links, identity, size limit)

* address review: create the test directories atomically and privately

* xtask stats tests: no unreachable! in the directory helper

* gridcraft check: unit-test that each workbook is listed once whatever path reaches it
2026-11-11 16:09:17 -05:00
Dylan M. Taylor 35e9815603 Security: one work budget across every call of a lifted function (#3197)
* functions: one work budget across every call of a lifted function

A function lifted over a computed array runs once per element, and only the
six lookups counted their repeated work against LIFTED_SCAN_BUDGET. Criteria
aggregates rescanned their ranges per criterion, CUMIPMT/CUMPRINC stepped
through up to 1e7 periods per element, and array-producing functions such as
MUNIT built a whole result per element before keeping its first cell.

The budget now covers all of it: COUNTIF/SUMIF/AVERAGEIF(S)/MAXIFS/MINIFS
charge their range cells per call, the period loops of every element are
added up before any runs, and array results charge their cells as they are
built. Past the budget the call is #NUM!, as lifted lookups already are.

* address review: charge numeric-text periods of lifted CUMIPMT/CUMPRINC to the work budget

* address review: charge only calls that reach the cumulative period loop

* address review: charge the numeric text a lifted period call parses

* address review: SUMIF/AVERAGEIF charge only the sum-range cells they read; weight cumulative period work
2026-11-11 16:08:50 -05:00
Dylan M. Taylor 6ea8d11522 print: bound the print area to the grid; render charts once and budget picture and chart output (#3314)
* print: hold the print area to the grid; render each chart once and budget picture and chart output across pages

* address review: budget chart path points and drop charts that do not fit the budget
2026-11-11 16:08:24 -05:00
Dylan M. Taylor 7c3af8a421 fix(formula): bound the wrappers stacked on one path of a parsed formula (#3813) 2026-11-11 16:07:58 -05:00
Dylan M. Taylor ca7ea0131e fix(mcp): send the control token only to a loopback address (#3814) 2026-11-11 16:07:32 -05:00
Dylan M. Taylor aa0347cc2e fix(engine): calculated pivot fields may not build arrays from scalars (#3812) 2026-11-11 16:07:06 -05:00
Dylan M. Taylor 5cf52c5fbb chore: bump version to 0.19.1 for the Flatpak update fix (#3809) 2026-11-11 14:55:53 -05:00
Dylan M. Taylor 0c5b1e85e6 sonar: stop counting the locale table as duplicated code (#3805) 2026-11-11 14:33:16 -05:00
Dylan M. Taylor 7f6a79fd98 Fix Flatpak permissions for update checks and self-updates (#3803)
* fix(flatpak): allow release checks and self-update helpers

* test: keep Flatpak permission parsing within strict Clippy rules
2026-11-11 14:24:13 -05:00
Dylan M. Taylor 6dbfbc4058 Merge pull request #3795 from storytold/chore/craft-update-6e03f65
Updates: take craft-update from craft-libs main at 6e03f65bf
2026-11-11 12:44:31 -05:00
Dylan M. Taylor e258db5f19 Updates: take craft-update from craft-libs main at 6e03f65bf 2026-11-11 12:44:25 -05:00
Dylan M. Taylor 763816d27c ci: run Harden-Runner on every job (#3786)
Adds step-security/harden-runner (audit, pinned to v2.22.4 like
craft-common-actions) as the first step of each job that did not have it,
so every workflow in the org's public repos reports its egress to
StepSecurity. Audit, not block: block-mode allow lists come from each
job's observed traffic.
2026-11-11 12:11:59 -05:00
61 changed files with 1619 additions and 4704 deletions
+4
View File
@@ -27,6 +27,10 @@ jobs:
name: Windows installation policy (no app build)
runs-on: windows-latest
steps:
- name: Harden runner
uses: step-security/harden-runner@1d88384fd173af6ecb012dfd0264a306b711241a # v2.22.4
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
+5
View File
@@ -1,6 +1,10 @@
name: Release
on:
# A nightly dry run on main: every release build except macOS, nothing signed or released
# (craft-common-actions release.yml `dry-run`). Finds release-only breakage before release day.
schedule:
- cron: "56 1 * * *"
push:
branches: [release]
workflow_dispatch:
@@ -23,6 +27,7 @@ jobs:
# Intentionally tracked at @main (not SHA-pinned) so org-wide CI improvements, security fixes, and build optimizations propagate without PR churn across all craftapp repos.
uses: storytold/craft-common-actions/.github/workflows/release.yml@main
with:
dry-run: ${{ github.event_name == 'schedule' }}
app-name: gridcraft
app-title: GridCraft
craft-fonts-ref: b076417b5880bd4154bd22f00695b1d28c6bea83
Generated
+141 -35
View File
@@ -739,6 +739,12 @@ dependencies = [
"cc",
]
[[package]]
name = "cmov"
version = "0.5.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
[[package]]
name = "cmpv2"
version = "0.2.0"
@@ -757,7 +763,7 @@ version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7b77c319abfd5219629c45c34c89ba945ed3c5e49fcde9d16b6c3885f118a730"
dependencies = [
"const-oid",
"const-oid 0.9.6",
"der",
"spki",
"x509-cert",
@@ -827,6 +833,12 @@ version = "0.9.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8"
[[package]]
name = "const-oid"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be95d4a884358a21a199689323c29e9256b4a1b315e22f183d155422041d3a9a"
[[package]]
name = "core-foundation"
version = "0.9.4"
@@ -907,15 +919,28 @@ dependencies = [
"libc",
]
[[package]]
name = "craft-cfb"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
[[package]]
name = "craft-csv"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
dependencies = [
"thiserror 2.0.21",
]
[[package]]
name = "craft-fonts-build"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=1b34eba41e62138fc2438d2734523b302840b45d#1b34eba41e62138fc2438d2734523b302840b45d"
source = "git+https://github.com/storytold/craft-libs?rev=119415f46f601e20b9bc6b1b4d6979ceda86d88e#119415f46f601e20b9bc6b1b4d6979ceda86d88e"
[[package]]
name = "craft-i18n"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=1b34eba41e62138fc2438d2734523b302840b45d#1b34eba41e62138fc2438d2734523b302840b45d"
source = "git+https://github.com/storytold/craft-libs?rev=119415f46f601e20b9bc6b1b4d6979ceda86d88e#119415f46f601e20b9bc6b1b4d6979ceda86d88e"
dependencies = [
"fixed_decimal",
"icu_decimal",
@@ -935,10 +960,27 @@ dependencies = [
"writeable",
]
[[package]]
name = "craft-offcrypto"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
dependencies = [
"aes",
"base64 0.23.1",
"craft-cfb",
"craft-xml",
"getrandom 0.3.4",
"hmac",
"sha1 0.11.0",
"sha2 0.11.1",
"thiserror 2.0.21",
"zeroize",
]
[[package]]
name = "craft-print"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=1b34eba41e62138fc2438d2734523b302840b45d#1b34eba41e62138fc2438d2734523b302840b45d"
source = "git+https://github.com/storytold/craft-libs?rev=119415f46f601e20b9bc6b1b4d6979ceda86d88e#119415f46f601e20b9bc6b1b4d6979ceda86d88e"
dependencies = [
"craft-winprint-gdi",
"log",
@@ -949,10 +991,15 @@ dependencies = [
"winsafe",
]
[[package]]
name = "craft-sheet"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
[[package]]
name = "craft-ui"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=1b34eba41e62138fc2438d2734523b302840b45d#1b34eba41e62138fc2438d2734523b302840b45d"
source = "git+https://github.com/storytold/craft-libs?rev=119415f46f601e20b9bc6b1b4d6979ceda86d88e#119415f46f601e20b9bc6b1b4d6979ceda86d88e"
dependencies = [
"craft-fonts-build",
"craft-i18n",
@@ -972,7 +1019,7 @@ dependencies = [
[[package]]
name = "craft-update"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=1b34eba41e62138fc2438d2734523b302840b45d#1b34eba41e62138fc2438d2734523b302840b45d"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
dependencies = [
"chrono",
"dunce",
@@ -982,6 +1029,7 @@ dependencies = [
"image",
"log",
"plist",
"rustix 1.1.5",
"semver",
"serde",
"serde_json",
@@ -998,7 +1046,7 @@ dependencies = [
[[package]]
name = "craft-web-api"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=fd5e2feff6ea25bed371a99cc732b3968d621505#fd5e2feff6ea25bed371a99cc732b3968d621505"
source = "git+https://github.com/storytold/craft-libs?rev=119415f46f601e20b9bc6b1b4d6979ceda86d88e#119415f46f601e20b9bc6b1b4d6979ceda86d88e"
dependencies = [
"js-sys",
"serde_json",
@@ -1009,7 +1057,7 @@ dependencies = [
[[package]]
name = "craft-web-host"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=fd5e2feff6ea25bed371a99cc732b3968d621505#fd5e2feff6ea25bed371a99cc732b3968d621505"
source = "git+https://github.com/storytold/craft-libs?rev=119415f46f601e20b9bc6b1b4d6979ceda86d88e#119415f46f601e20b9bc6b1b4d6979ceda86d88e"
dependencies = [
"wasm-bindgen",
"web-sys",
@@ -1018,7 +1066,7 @@ dependencies = [
[[package]]
name = "craft-window"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=1b34eba41e62138fc2438d2734523b302840b45d#1b34eba41e62138fc2438d2734523b302840b45d"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
dependencies = [
"egui",
"objc2 0.6.5",
@@ -1034,6 +1082,28 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "craft-xls"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
dependencies = [
"craft-cfb",
"craft-sheet",
"md-5",
"miniz_oxide 0.9.1",
"sha1 0.10.7",
"thiserror 2.0.21",
]
[[package]]
name = "craft-xml"
version = "0.1.0"
source = "git+https://github.com/storytold/craft-libs?rev=bb57d422d0bc842cbde430955825553ab7b976f4#bb57d422d0bc842cbde430955825553ab7b976f4"
dependencies = [
"quick-xml 0.41.0",
"thiserror 2.0.21",
]
[[package]]
name = "crc32fast"
version = "1.5.2"
@@ -1114,6 +1184,15 @@ dependencies = [
"hybrid-array",
]
[[package]]
name = "ctutils"
version = "0.4.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "261af4dc7a06ea8a527a8717d8cbba3417530d6ea14e1edb3de6b67806be474f"
dependencies = [
"cmov",
]
[[package]]
name = "cursor-icon"
version = "1.2.0"
@@ -1126,7 +1205,7 @@ version = "0.7.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb"
dependencies = [
"const-oid",
"const-oid 0.9.6",
"der_derive",
"flagset",
"pem-rfc7468",
@@ -1191,7 +1270,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
dependencies = [
"block-buffer 0.12.1",
"const-oid 0.10.3",
"crypto-common 0.2.2",
"ctutils",
]
[[package]]
@@ -1882,9 +1963,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"r-efi 5.3.0",
"wasip2",
"wasm-bindgen",
]
[[package]]
@@ -2022,7 +2105,7 @@ dependencies = [
[[package]]
name = "gridcraft"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"arboard",
"chrono",
@@ -2050,7 +2133,7 @@ dependencies = [
[[package]]
name = "gridcraft-calc"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-core",
"gridcraft-formula",
@@ -2062,7 +2145,7 @@ dependencies = [
[[package]]
name = "gridcraft-chart"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-calc",
"gridcraft-core",
@@ -2073,7 +2156,7 @@ dependencies = [
[[package]]
name = "gridcraft-cli"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-engine",
"gridcraft-mcp",
@@ -2086,7 +2169,7 @@ dependencies = [
[[package]]
name = "gridcraft-core"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"chrono",
"js-sys",
@@ -2097,8 +2180,9 @@ dependencies = [
[[package]]
name = "gridcraft-engine"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"craft-csv",
"gridcraft-calc",
"gridcraft-chart",
"gridcraft-core",
@@ -2120,7 +2204,7 @@ dependencies = [
[[package]]
name = "gridcraft-formula"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-core",
"serde",
@@ -2129,7 +2213,7 @@ dependencies = [
[[package]]
name = "gridcraft-functions"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-core",
"regex",
@@ -2137,7 +2221,7 @@ dependencies = [
[[package]]
name = "gridcraft-math"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"ratex-font",
"ratex-layout",
@@ -2147,7 +2231,7 @@ dependencies = [
[[package]]
name = "gridcraft-mcp"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-engine",
"log",
@@ -2156,7 +2240,7 @@ dependencies = [
[[package]]
name = "gridcraft-model"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-core",
"gridcraft-formula",
@@ -2169,18 +2253,18 @@ dependencies = [
[[package]]
name = "gridcraft-numfmt"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-core",
]
[[package]]
name = "gridcraft-pdf"
version = "0.19.0"
version = "0.19.1"
[[package]]
name = "gridcraft-sandbox"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"landlock",
"libc",
@@ -2191,7 +2275,7 @@ dependencies = [
[[package]]
name = "gridcraft-ui-egui"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"craft-ui",
"craft-update",
@@ -2218,7 +2302,7 @@ dependencies = [
[[package]]
name = "gridcraft-web"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"craft-web-api",
"craft-web-host",
@@ -2237,16 +2321,18 @@ dependencies = [
[[package]]
name = "gridcraft-xlsx"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"aes",
"craft-cfb",
"craft-csv",
"craft-offcrypto",
"craft-xls",
"gridcraft-core",
"gridcraft-formula",
"gridcraft-functions",
"gridcraft-math",
"gridcraft-model",
"quick-xml 0.41.0",
"sha1 0.11.0",
"sha2 0.11.1",
"thiserror 2.0.21",
"zip 3.0.0",
@@ -2351,6 +2437,15 @@ version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
[[package]]
name = "hmac"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f"
dependencies = [
"digest 0.11.3",
]
[[package]]
name = "http"
version = "1.5.0"
@@ -2991,6 +3086,16 @@ version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4050469837a6ff301cd14c1f8f24f88549e6d548f24f64e2148eb0f72cebc51f"
[[package]]
name = "md-5"
version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf"
dependencies = [
"cfg-if",
"digest 0.10.7",
]
[[package]]
name = "memchr"
version = "2.8.3"
@@ -4803,7 +4908,7 @@ checksum = "da90c3d9af638898a5fdc347479b18f950bb0dde11ecf2244f94cd28135da53b"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
"const-oid",
"const-oid 0.9.6",
"der",
"digest 0.10.7",
"pem",
@@ -4877,7 +4982,7 @@ dependencies = [
"chrono",
"cmpv2",
"cms",
"const-oid",
"const-oid 0.9.6",
"der",
"hex",
"rand 0.10.3",
@@ -4916,7 +5021,7 @@ dependencies = [
"base64 0.22.1",
"chrono",
"cms",
"const-oid",
"const-oid 0.9.6",
"hex",
"pem",
"rustls-pki-types",
@@ -6519,7 +6624,7 @@ dependencies = [
[[package]]
name = "winprint"
version = "0.2.1"
source = "git+https://github.com/storytold/winprint.rs?rev=51934db382f225454cf152a8657eb03789be63c4#51934db382f225454cf152a8657eb03789be63c4"
source = "git+https://github.com/storytold/winprint.rs?rev=bc06a7c48b768eb27a828c7986f6d2868b245ee0#bc06a7c48b768eb27a828c7986f6d2868b245ee0"
dependencies = [
"flate2",
"fmt-derive",
@@ -6534,6 +6639,7 @@ dependencies = [
"windows",
"windows-core",
"windows-implement",
"windows-numerics",
"xml",
]
@@ -6640,7 +6746,7 @@ version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1301e935010a701ae5f8655edc0ad17c44bad3ac5ce8c39185f75453b720ae94"
dependencies = [
"const-oid",
"const-oid 0.9.6",
"der",
"sha1 0.10.7",
"signature",
@@ -6708,7 +6814,7 @@ checksum = "e450f9b2ed1dff33c94c12589a87338689467b9c4f5d8a5710bd09a847d2c8a7"
[[package]]
name = "xtask"
version = "0.19.0"
version = "0.19.1"
dependencies = [
"gridcraft-engine",
"serde_json",
+14 -8
View File
@@ -4,7 +4,7 @@ members = ["crates/*", "apps/*", "xtask"]
default-members = ["crates/*", "apps/gridcraft", "apps/gridcraft-cli", "xtask"]
[workspace.package]
version = "0.19.0"
version = "0.19.1"
edition = "2024"
license = "MIT OR Apache-2.0"
rust-version = "1.95"
@@ -26,10 +26,19 @@ gridcraft-ui-egui = { path = "crates/ui-egui" }
gridcraft-mcp = { path = "crates/mcp" }
gridcraft-pdf = { path = "crates/pdf" }
gridcraft-math = { path = "crates/math" }
# Every craft-libs crate (craft-update and craft-window here, craft-ui in ui-egui, craft-print in
# the app) comes from one craft-libs revision.
# Every craft-libs crate (craft-update, craft-window and the file-format crates here, craft-ui in
# ui-egui, craft-print in the app, craft-web-api and craft-web-host in the web app) comes from one
# craft-libs revision.
# craft-update: app-specific pretend versions apply to startup and manual checks.
craft-update = { git = "https://github.com/storytold/craft-libs", rev = "1b34eba41e62138fc2438d2734523b302840b45d", version = "=0.1.0", features = ["egui"] }
craft-update = { git = "https://github.com/storytold/craft-libs", rev = "bb57d422d0bc842cbde430955825553ab7b976f4", version = "=0.1.0", features = ["egui"] }
# Compound files ([MS-CFB]) and password-encrypted OOXML packages ([MS-OFFCRYPTO]).
craft-cfb = { git = "https://github.com/storytold/craft-libs", rev = "bb57d422d0bc842cbde430955825553ab7b976f4", version = "0.1.0" }
craft-offcrypto = { git = "https://github.com/storytold/craft-libs", rev = "bb57d422d0bc842cbde430955825553ab7b976f4", version = "0.1.0" }
# Delimited text (CSV, TSV) and SYLK; `crates/xlsx/src/csv.rs` and `crates/engine/src/io/sylk.rs`
# turn them into workbooks.
craft-csv = { git = "https://github.com/storytold/craft-libs", rev = "bb57d422d0bc842cbde430955825553ab7b976f4", version = "0.1.0" }
# Excel 97-2003 (.xls, BIFF8) workbooks; `crates/xlsx/src/xls.rs` maps them onto the model.
craft-xls = { git = "https://github.com/storytold/craft-libs", rev = "bb57d422d0bc842cbde430955825553ab7b976f4", version = "0.1.0" }
# foundation
thiserror = "2"
@@ -44,9 +53,6 @@ regex = { version = "1", default-features = false, features = ["std", "unicode",
quick-xml = "0.41"
zip = { version = "3", default-features = false, features = ["deflate"] }
sha2 = { version = "0.11", default-features = false }
# Decrypting password-protected workbooks ([MS-OFFCRYPTO]): AES and SHA-1, pure Rust (wasm-safe).
aes = { version = "0.9", default-features = false }
sha1 = { version = "0.11", default-features = false }
image = { version = "0.25", default-features = false, features = ["png", "jpeg"] }
# equations: RaTeX (MIT) lays LaTeX out as positioned glyphs, rules and paths. Exact pins: it is
@@ -72,7 +78,7 @@ skrifa = { version = "0.44", default-features = false, features = ["std"] }
epaint_default_fonts = "0.37"
# One native window per workbook: the window list, Window menu titles and the immediate-viewport
# driver. Same craft-libs revision as craft-update above.
craft-window = { version = "0.1.0", git = "https://github.com/storytold/craft-libs", rev = "1b34eba41e62138fc2438d2734523b302840b45d", default-features = false, features = ["egui"] }
craft-window = { version = "0.1.0", git = "https://github.com/storytold/craft-libs", rev = "bb57d422d0bc842cbde430955825553ab7b976f4", default-features = false, features = ["egui"] }
[workspace.lints.rust]
unsafe_code = "forbid"
+38 -18
View File
@@ -214,7 +214,7 @@ pub fn main(args: &[String]) -> ExitCode {
Err(_) => return usage(format!("--max-report needs a whole number, not `{n}`")),
}
}
let Discovery { files, failures: mut reports } = collect_inputs(&a.positional);
let Discovery { files, failures: mut reports, .. } = collect_inputs(&a.positional);
reports.extend(files.iter().map(|p| {
let path = p.to_string_lossy().into_owned();
match gridcraft_engine::io::read_file(&path) {
@@ -233,6 +233,9 @@ pub fn main(args: &[String]) -> ExitCode {
#[derive(Default)]
struct Discovery {
files: Vec<PathBuf>,
/// Identity (dev+inode) of each collected file, so a rescan reached by a shallower path doesn't add it twice.
#[cfg(unix)]
file_ids: std::collections::HashSet<(u64, u64)>,
failures: Vec<FileReport>,
}
@@ -257,11 +260,11 @@ impl Discovery {
/// (`~$…`) are skipped.
fn collect_inputs(inputs: &[String]) -> Discovery {
let mut out = Discovery::default();
let mut seen = std::collections::HashSet::new();
let mut seen = std::collections::HashMap::new();
for i in inputs {
let p = PathBuf::from(i);
if p.is_dir() {
if !walk(&p, 0, &mut out, &mut seen) {
if !walk(&p, 0, false, &mut out, &mut seen) {
break;
}
} else if !out.add_file(p) {
@@ -277,19 +280,29 @@ fn collect_inputs(inputs: &[String]) -> Discovery {
out
}
/// Returns false only when the global file budget stops traversal.
fn walk(dir: &Path, depth: usize, out: &mut Discovery, seen: &mut std::collections::HashSet<u64>) -> bool {
// A directory is visited once by identity (dev+inode on unix): several links to the same
// directory are aliases, not exponential branches of the scan. Other platforms keep the
// depth cap only.
/// Returns false only when the global file budget stops traversal. `seen` holds the shallowest
/// depth each directory was scanned from; `rescan` is set below a directory reached again from a
/// shallower path (its depth-limited subtree may now fit), where already collected files are skipped.
#[cfg_attr(not(unix), allow(unused_mut))]
fn walk(dir: &Path, depth: usize, mut rescan: bool, out: &mut Discovery, seen: &mut std::collections::HashMap<(u64, u64), usize>) -> bool {
// A directory is visited once by identity (dev+inode on unix) unless reached from a shallower
// path: several links to the same directory are aliases, not exponential branches of the scan
// (each directory is scanned at most MAX_DEPTH + 1 times). Other platforms keep the depth cap only.
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt as _;
if let Ok(m) = std::fs::metadata(dir)
&& let Some(id) = m.dev().checked_mul(1 << 60).and_then(|d| d.checked_add(m.ino()))
&& !seen.insert(id)
{
return true;
if let Ok(m) = std::fs::metadata(dir) {
let identity = (m.dev(), m.ino());
// A directory past the depth limit is not scanned, so it is not recorded: asked for
// by name later, it is walked from depth 0.
match seen.get(&identity) {
Some(&shallowest) if shallowest <= depth => return true,
Some(_) => rescan = true,
None => {}
}
if depth <= MAX_DEPTH {
seen.insert(identity, depth);
}
}
}
if depth > MAX_DEPTH {
@@ -324,13 +337,20 @@ fn walk(dir: &Path, depth: usize, out: &mut Discovery, seen: &mut std::collectio
}
};
if metadata.is_dir() {
if !walk(&p, depth + 1, out, seen) {
if !walk(&p, depth + 1, rescan, out, seen) {
return false;
}
} else if p.extension().and_then(std::ffi::OsStr::to_str).is_some_and(|e| e.eq_ignore_ascii_case("xlsx") || e.eq_ignore_ascii_case("xlsm")) {
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt as _;
if !out.file_ids.insert((metadata.dev(), metadata.ino())) && rescan {
continue;
}
}
if !out.add_file(p) {
return false;
}
} else if p.extension().and_then(std::ffi::OsStr::to_str).is_some_and(|e| e.eq_ignore_ascii_case("xlsx") || e.eq_ignore_ascii_case("xlsm"))
&& !out.add_file(p)
{
return false;
}
}
true
+43 -2
View File
@@ -11,8 +11,8 @@ fn discovery_reports_a_directory_read_error() {
let path = std::env::temp_dir().join(format!("gridcraft-check-not-a-directory-{}", std::process::id()));
std::fs::write(&path, "not a directory").unwrap();
let mut discovery = Discovery::default();
let mut seen = std::collections::HashSet::new();
assert!(walk(&path, 0, &mut discovery, &mut seen));
let mut seen = std::collections::HashMap::new();
assert!(walk(&path, 0, false, &mut discovery, &mut seen));
assert!(discovery.files.is_empty());
assert_eq!(discovery.failures.len(), 1);
assert_eq!(discovery.failures[0].path, path.to_string_lossy());
@@ -31,6 +31,47 @@ fn discovery_reports_omissions_only_after_the_file_limit() {
assert!(discovery.failures[0].error.as_ref().unwrap().contains("file limit"));
}
/// A new directory under the temp dir: `create_dir` fails on a name somebody else made first.
#[cfg(unix)]
fn fresh_dir(tag: &str) -> PathBuf {
let mut n = 0u32;
loop {
let dir = std::env::temp_dir().join(format!("gridcraft-check-{tag}-{}-{n}", std::process::id()));
match std::fs::create_dir(&dir) {
Ok(()) => return dir,
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => n += 1,
Err(e) => panic!("{e}"),
}
}
}
/// A directory named again by a deeper path, or by a link back to a parent, lists each workbook once.
#[cfg(unix)]
#[test]
fn discovery_lists_each_workbook_once_whatever_path_reaches_it() {
let dir = fresh_dir("once");
let sub = dir.join("sub");
std::fs::create_dir(&sub).unwrap();
std::fs::write(dir.join("a.xlsx"), b"x").unwrap();
std::fs::write(sub.join("b.xlsx"), b"x").unwrap();
std::os::unix::fs::symlink(&dir, sub.join("loop")).unwrap();
let names = |d: &Discovery| {
let mut v: Vec<String> = d.files.iter().map(|p| p.file_name().unwrap().to_string_lossy().into_owned()).collect();
v.sort();
v
};
// The link back to the top is one directory, not another branch of the scan.
let found = collect_inputs(&[dir.to_string_lossy().into_owned()]);
assert_eq!(names(&found), ["a.xlsx", "b.xlsx"]);
// `sub` named after its parent is reached from a shallower depth: scanned again, nothing listed twice.
let found = collect_inputs(&[dir.to_string_lossy().into_owned(), sub.to_string_lossy().into_owned()]);
assert_eq!(names(&found), ["a.xlsx", "b.xlsx"]);
// Named first, the same `sub` is the shallowest path and the parent's scan skips it.
let found = collect_inputs(&[sub.to_string_lossy().into_owned(), dir.to_string_lossy().into_owned()]);
assert_eq!(names(&found), ["a.xlsx", "b.xlsx"]);
let _ = std::fs::remove_dir_all(dir);
}
fn at(a1: &str) -> CellRef {
CellRef::parse(a1).unwrap()
}
+38
View File
@@ -205,6 +205,44 @@ fn check_reports_a_depth_limited_scan_while_checking_readable_files() {
let _ = std::fs::remove_dir_all(dir);
}
#[cfg(unix)]
#[test]
fn a_directory_past_the_depth_limit_is_still_scanned_when_named_as_an_input() {
let dir = tmpdir("check-depth-named");
// The first folder past the limit (33 levels down): the scan from the top reports it unreached.
let mut deep = dir.clone();
for _ in 0..33 {
deep.push("d");
}
std::fs::create_dir_all(&deep).unwrap();
ok(&["run", "--out", deep.join("deep.xlsx").to_str().unwrap(), "--quiet"]);
let out = run(&["check", dir.to_str().unwrap(), deep.to_str().unwrap(), "--json"]);
let report: serde_json::Value = serde_json::from_slice(&out.stdout).unwrap();
// The first scan reports the subtree it could not reach; the named directory is checked.
assert_eq!(report["totals"]["opened"], 1, "{report}");
let _ = std::fs::remove_dir_all(dir);
}
#[cfg(unix)]
#[test]
fn a_directory_first_reached_at_the_depth_limit_is_completed_when_named_as_an_input() {
let dir = tmpdir("check-depth-shallower");
// The 32nd folder is scanned from the top, but its child (and the workbook in it) is past the limit.
let mut limit = dir.clone();
for _ in 0..32 {
limit.push("d");
}
let child = limit.join("c");
std::fs::create_dir_all(&child).unwrap();
ok(&["run", "--out", limit.join("at_limit.xlsx").to_str().unwrap(), "--quiet"]);
ok(&["run", "--out", child.join("below.xlsx").to_str().unwrap(), "--quiet"]);
let out = run(&["check", dir.to_str().unwrap(), limit.to_str().unwrap(), "--json"]);
let report: serde_json::Value = serde_json::from_slice(&out.stdout).unwrap();
// Named from depth 0 the subtree is read: the workbook below the limit is checked, the one at it only once.
assert_eq!(report["totals"]["opened"], 2, "{report}");
let _ = std::fs::remove_dir_all(dir);
}
/// A script line whose parameters are under a key `run` does not read used to run the command
/// with no parameters at all, and the run still exited 0: three mistaken lines made Sheet2,
/// Sheet3 and Sheet4 instead of the named sheets, and nothing said so (#2256).
+2 -2
View File
@@ -18,8 +18,8 @@ gridcraft-engine = { workspace = true }
gridcraft-ui-egui = { workspace = true }
# The host messages (storytold/craft-libs): one contract and one bridge for the web builds.
# The page command API (storytold/craft-libs#413): window.gridcraft over the control channel.
craft-web-api = { git = "https://github.com/storytold/craft-libs", rev = "fd5e2feff6ea25bed371a99cc732b3968d621505", version = "0.1.0" }
craft-web-host = { git = "https://github.com/storytold/craft-libs", rev = "fd5e2feff6ea25bed371a99cc732b3968d621505", version = "0.1.0" }
craft-web-api = { git = "https://github.com/storytold/craft-libs", rev = "119415f46f601e20b9bc6b1b4d6979ceda86d88e", version = "0.1.0" }
craft-web-host = { git = "https://github.com/storytold/craft-libs", rev = "119415f46f601e20b9bc6b1b4d6979ceda86d88e", version = "0.1.0" }
egui = { workspace = true }
log = { workspace = true }
serde_json = { workspace = true }
+1 -1
View File
@@ -35,7 +35,7 @@ winit = { version = "0.30", default-features = false, features = ["x11", "waylan
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
# Same craft-libs revision as the workspace's craft-update and craft-window (see the root
# Cargo.toml).
craft-print = { git = "https://github.com/storytold/craft-libs", rev = "1b34eba41e62138fc2438d2734523b302840b45d" }
craft-print = { git = "https://github.com/storytold/craft-libs", rev = "119415f46f601e20b9bc6b1b4d6979ceda86d88e" }
craft-update = { workspace = true }
rfd = { workspace = true }
arboard = { workspace = true, features = ["image-data"] }
+9 -2
View File
@@ -147,10 +147,10 @@ impl RangeRef {
RangeRef::new(CellRef::new(0, 0), CellRef::new(MAX_ROWS - 1, MAX_COLS - 1))
}
pub fn height(&self) -> u32 {
self.end.row - self.start.row + 1
self.end.row.saturating_sub(self.start.row).saturating_add(1)
}
pub fn width(&self) -> u32 {
self.end.col - self.start.col + 1
self.end.col.saturating_sub(self.start.col).saturating_add(1)
}
pub fn count(&self) -> u64 {
self.height() as u64 * self.width() as u64
@@ -321,6 +321,13 @@ pub fn split_areas(reference: &str) -> Vec<&str> {
mod tests {
use super::*;
#[test]
fn hostile_range_dimensions_saturate_instead_of_wrapping() {
let r = RangeRef { start: CellRef::new(0, 0), end: CellRef::new(u32::MAX, u32::MAX) };
assert_eq!((r.height(), r.width()), (u32::MAX, u32::MAX));
assert_eq!(r.count(), u64::from(u32::MAX) * u64::from(u32::MAX));
}
#[test]
fn areas_split_only_at_commas_outside_quoted_sheet_names() {
assert_eq!(split_areas("B2:B4,D1"), ["B2:B4", "D1"]);
+1
View File
@@ -21,6 +21,7 @@ web-time = { workspace = true }
thiserror = { workspace = true }
log = { workspace = true }
gridcraft-xlsx = { workspace = true }
craft-csv = { workspace = true }
gridcraft-chart = { workspace = true }
gridcraft-pdf = { workspace = true }
gridcraft-math = { workspace = true }
+1 -1
View File
@@ -98,7 +98,7 @@ pub fn specs() -> Vec<CommandSpec> {
validation
),
cmd!(query "data.validate", "Check Value Against Validation", [], None, "{cell?, input} → {ok, message?}", has_doc, validate_cmd),
cmd!(noundo "data.circleInvalid", "Circle Invalid Data", ["Data", "Data Tools", "Data Validation"], None, "{} → invalid cells", has_doc, circle_invalid),
cmd!(noundo "data.circleInvalid", "Circle Invalid Data", ["Data", "Data Tools", "Data Validation"], None, "{} → {invalid: [cell], truncated: bool (true when the 100,000-cell scan allowance ended before every range was checked)}", has_doc, circle_invalid),
CommandSpec {
shortcut_mac: Some("Cmd+Shift+K"),
..cmd!(
+2 -1
View File
@@ -184,7 +184,8 @@ pub(crate) fn grow_autofilter(sh: &mut Sheet, at: CellRef) {
pub(crate) fn apply_filter(wb: &mut Workbook, sheet: usize) {
let Some(sh) = wb.sheet(sheet) else { return };
let Some(af) = sh.autofilter.clone() else { return };
let r = af.range;
// An imported range can name rows past the grid; none of those exist to hide.
let r = RangeRef::new(af.range.start, CellRef::new(af.range.end.row.min(gridcraft_core::MAX_ROWS - 1), af.range.end.col));
let mut hide = Vec::new();
// Precompute per-criterion top10 threshold and average once, not per row.
let mut col_stats: std::collections::HashMap<usize, (Option<f64>, f64)> = std::collections::HashMap::new();
+7
View File
@@ -5,6 +5,9 @@ use super::{
// ---------------------------------------------------------------- tools
/// Most cells (rows times compared columns) one Remove Duplicates call reads.
const MAX_COMPARED_CELLS: u64 = 50_000_000;
pub(super) fn remove_duplicates(s: &mut Session, p: &Json) -> Result<Json> {
let (sheet, r) = data_target(s, p, "data.removeDuplicates")?;
let d = s.doc()?;
@@ -24,6 +27,10 @@ pub(super) fn remove_duplicates(s: &mut Session, p: &Json) -> Result<Json> {
if cols.is_empty() || cols.iter().any(|c| !(r.start.col..=r.end.col).contains(c)) {
return Err(bad("data.removeDuplicates", "`columns` must name at least one column of the range"));
}
// Rows are compared one by one, so a sparse sheet's enclosing rectangle is bounded like a list.
if r.count() > MAX_COMPARED_CELLS {
return Err(bad("data.removeDuplicates", "the range is too large"));
}
let start = r.start.row + header as u32;
let mut removed = 0;
edit(s, |cx| {
+8 -2
View File
@@ -337,11 +337,17 @@ pub(super) fn circle_invalid(s: &mut Session, _: &Json) -> Result<Json> {
let sheet = d.wb.active_sheet();
let Some(sh) = d.wb.sheet(sheet) else { return Err(EngineError::NoDocument) };
let mut bad_cells = Vec::new();
// One scan allowance for the whole command: duplicate or overlapping ranges don't multiply it.
let mut budget = 100_000usize;
// Cells were left unchecked because the allowance ran out: said in the result.
let mut truncated = false;
for dv in &sh.validations {
for r in &dv.ranges {
let r = sh.used_range().and_then(|u| r.intersection(&u));
let Some(r) = r else { continue };
for c in r.iter().take(100_000) {
truncated |= r.count() > budget as u64;
for c in r.iter().take(budget) {
budget -= 1;
let text = sh.input_text(c);
if !text.is_empty() && check_validation(&d.wb, sheet, c, &text).is_some() {
bad_cells.push(c.a1());
@@ -349,5 +355,5 @@ pub(super) fn circle_invalid(s: &mut Session, _: &Json) -> Result<Json> {
}
}
}
Ok(json!({"invalid": bad_cells}))
Ok(json!({"invalid": bad_cells, "truncated": truncated}))
}
+5 -1
View File
@@ -199,7 +199,11 @@ pub(super) fn replace(s: &mut Session, p: &Json) -> Result<Json> {
let mut total = 0usize;
for si in sheets {
let cells: Vec<(CellRef, String)> =
cx.wb.sheet(si).map(|sh| sh.cells.iter().map(|(c, _)| (c, sh.input_text(c))).collect()).unwrap_or_default();
// A hidden formula's source is not searched: a count or error would reveal its text.
cx.wb
.sheet(si)
.map(|sh| sh.cells.iter().filter(|(c, cell)| cell.formula.is_none() && cell.py.is_none() || !crate::display::formula_hidden(&cx.wb, sh, *c)).map(|(c, _)| (c, sh.input_text(c))).collect())
.unwrap_or_default();
for (c, text) in cells {
if (!all && c != active) || selection.as_ref().is_some_and(|sel| !sel.contains(c)) {
continue;
+15
View File
@@ -891,6 +891,21 @@ mod tests {
assert!((0.75..=1.15).contains(&ratio), "Arabic {} vs Latin {}", width(0), width(1));
}
#[test]
fn autofit_of_many_separate_columns_fits_only_the_selected_ones() {
let mut s = crate::Session::new();
s.new_workbook();
let long = "a much longer text than the others";
// Columns A, C, E, G, I, K are selected (six runs), the ones between hold long text too.
let row: Vec<_> = (0..11).map(|c| if c % 2 == 0 { json!("x") } else { json!(long) }).collect();
s.execute("range.setValues", json!({"range": "A1", "values": [row]})).unwrap();
s.execute("home.autofitColumnWidth", json!({"cols": "A:A,C:C,E:E,G:G,I:I,K:K"})).unwrap();
let doc = s.doc().unwrap();
let size = |c: u32| doc.wb.active().and_then(|sh| sh.cols.get(&c)).and_then(|e| e.size);
assert!([0, 2, 4, 6, 8, 10].iter().all(|&c| size(c).is_some()));
assert!([1, 3, 5, 7, 9].iter().all(|&c| size(c).is_none()));
}
#[test]
fn arabic_word_ligatures_are_wider_than_letters() {
let em = |t: &str| approx_text_width(t, 72.0, false) / 96.0;
+31 -12
View File
@@ -194,16 +194,35 @@ pub(super) fn autofit_cols(s: &mut Session, p: &Json) -> Result<Json> {
let measure = s.text_width.clone();
edit(s, |cx| {
let Some(sh) = cx.wb.sheet(si) else { return Ok(Json::Null) };
let mut widths = Vec::new();
let mut measured: std::collections::HashMap<StyleId, std::collections::HashMap<String, f32>> = Default::default();
let mut distinct = 0usize;
let used = sh.used_range();
for (a, b) in lines {
for c in a..=b.min(a + 16383) {
let Some(u) = used else { continue };
let col = RangeRef::new(CellRef::new(u.start.row, c), CellRef::new(u.end.row, c));
let mut max: f32 = 0.0;
for (cell, _) in sh.cells.iter_range(col) {
// Selected columns, then one pass over the populated cells (the store is row-oriented, so a
// scan per column would revisit every row for each selected column, empty or not).
let cols: std::collections::BTreeSet<u32> = lines.iter().flat_map(|&(a, b)| a..=b.min(a + 16383)).collect();
let mut maxes: std::collections::BTreeMap<u32, f32> = Default::default();
if let Some(u) = sh.used_range() {
// One pass per run of adjacent selected columns, so columns between distant selections
// (A and XFD) are never visited.
let mut runs: Vec<(u32, u32)> = Vec::new();
for &c in &cols {
match runs.last_mut() {
Some((_, end)) if *end + 1 == c => *end = c,
_ => runs.push((c, c)),
}
}
// Every run is a pass over all the stored rows: many runs (alternating columns) are one
// pass over their whole span instead, the unselected columns skipped.
if runs.len() > 4
&& let (Some(&(first, _)), Some(&(_, last))) = (runs.first(), runs.last())
{
runs = vec![(first, last)];
}
for (first, last) in runs {
let span = RangeRef::new(CellRef::new(u.start.row, first), CellRef::new(u.end.row, last));
for (cell, _) in sh.cells.iter_range(span) {
if !cols.contains(&cell.col) {
continue;
}
if sh.merge_at(cell).is_some_and(|m| !m.is_single()) {
continue;
}
@@ -226,14 +245,14 @@ pub(super) fn autofit_cols(s: &mut Session, p: &Json) -> Result<Json> {
}
None => approx_text_width(line, st.font.size, st.font.bold),
};
max = max.max(w + 6.0 + st.align.indent as f32 * 9.0 + sh.filter_button_space(cell));
let w = w + 6.0 + st.align.indent as f32 * 9.0 + sh.filter_button_space(cell);
let max = maxes.entry(cell.col).or_insert(0.0);
*max = max.max(w);
}
}
if max > 0.0 {
widths.push((c, max.clamp(8.0, 1785.0)));
}
}
}
let widths: Vec<(u32, f32)> = maxes.into_iter().filter(|&(_, m)| m > 0.0).map(|(c, m)| (c, m.clamp(8.0, 1785.0))).collect();
let sh = cx.sheet_mut(si)?;
for (c, w) in widths {
let e = sh.cols.entry(c).or_default();
+64 -3
View File
@@ -32,8 +32,69 @@ use header_footer::draw_header_footer;
pub use header_footer::{
HF_FIELDS, HfContext, HfSection, codes_to_fields, expand_header_footer, fields_to_codes, hf_date_time, join_header_footer, split_header_footer,
};
/// Decoded pictures (bytes, width, height) keyed by (sheet, picture); `None` = undecodable.
type PicCache = std::collections::HashMap<(usize, usize), Option<(Vec<u8>, u32, u32)>>;
type DecodedPicture = (Vec<u8>, u32, u32);
/// What one PDF render shares across the pages an object spans.
#[derive(Default)]
struct PicCache {
/// Decoded pictures (bytes, width, height) keyed by (sheet, picture); `None` = undecodable.
decoded: std::collections::HashMap<(usize, usize), Option<DecodedPicture>>,
/// Laid-out charts (with their [`chart_cost`]) keyed by (sheet, chart), rendered once however
/// many pages they touch. Only charts that can still be drawn are kept.
charts: std::collections::HashMap<(usize, usize), (Vec<gridcraft_chart::Prim>, usize)>,
/// Charts too big for what was left of the budget: never drawn, so not rendered again.
rejected_charts: std::collections::HashSet<(usize, usize)>,
/// Bytes of picture pixels embedded so far, and chart cost units emitted so far. Each page an
/// object touches gets its own copy in the PDF, so a workbook-controlled anchor over thousands
/// of pages would multiply them without bound.
image_bytes: usize,
chart_prims: usize,
}
impl PicCache {
/// The chart's primitives for one more page, or `None` once they no longer fit
/// [`MAX_CHART_PRIMS`]. `render` runs at most once per chart; a chart that doesn't fit is
/// dropped at once and remembered, so its vectors are never retained.
fn chart(&mut self, key: (usize, usize), render: impl FnOnce() -> Vec<gridcraft_chart::Prim>) -> Option<&[gridcraft_chart::Prim]> {
if self.rejected_charts.contains(&key) {
return None;
}
let (prims, cost) = match self.charts.remove(&key) {
Some(cached) => cached,
None => {
let prims = render();
let cost = chart_cost(&prims);
(prims, cost)
}
};
if self.chart_prims.saturating_add(cost) > MAX_CHART_PRIMS {
self.rejected_charts.insert(key);
return None;
}
self.chart_prims += cost;
Some(&self.charts.entry(key).or_insert((prims, cost)).0)
}
}
/// What a chart's drawing costs in the PDF: one unit per primitive plus one per path point (a
/// smoothed line holds thousands of points, each written out) and per point of a wedge's outline.
fn chart_cost(prims: &[gridcraft_chart::Prim]) -> usize {
use gridcraft_chart::Prim;
prims
.iter()
.map(|p| match p {
Prim::Line { pts, .. } | Prim::Polygon { pts, .. } => pts.len().saturating_add(1),
// At most 400 steps along each of the outer and inner arcs (see `draw_prims`).
Prim::Wedge { .. } => 803,
_ => 1,
})
.fold(0, usize::saturating_add)
}
/// Most picture pixel bytes (RGBA, before PDF packing) one render embeds across all pages.
const MAX_EMBEDDED_IMAGE_BYTES: usize = 512 * 1024 * 1024;
/// Most chart cost units (primitives and path points, see [`chart_cost`]) one render draws across all pages.
const MAX_CHART_PRIMS: usize = 4_000_000;
#[cfg(test)]
use layout::bands;
@@ -102,7 +163,7 @@ pub fn render_pdf(wb: &Workbook, layouts: &[SheetLayout], file: &str, path: &str
let mut printed_blocks = Vec::new();
// Decoded pictures are reused across the pages their anchor covers: a floating picture
// spanning 40 pages was decoded (and pushed into the PDF) 40 times (#2389).
let mut pics = PicCache::new();
let mut pics = PicCache::default();
for lay in layouts {
let Some(sh) = wb.sheet(lay.sheet) else { continue };
let mut cf = if sh.cond_formats.is_empty() { None } else { Some(crate::cf::CfCache::new()) };
+22
View File
@@ -224,6 +224,12 @@ pub fn layout_sheet(wb: &Workbook, si: usize, opts: &PrintOptions, range_overrid
Some(r) => trim_full(sh, r),
None => print_range(sh).unwrap_or(RangeRef::cell(CellRef::new(0, 0))),
};
// A print area from a file or JSON can name coordinates past the grid; pagination walks every
// line of the area (zero-height rows never fill a band), so it is held to the grid first.
let area = RangeRef::new(
CellRef::new(area.start.row.min(MAX_ROWS - 1), area.start.col.min(MAX_COLS - 1)),
CellRef::new(area.end.row.min(MAX_ROWS - 1), area.end.col.min(MAX_COLS - 1)),
);
let title_rows = title_span(st.title_rows, MAX_ROWS);
let title_cols = title_span(st.title_cols, MAX_COLS);
let digits = format!("{}", area.end.row + 1).len() as f32;
@@ -379,3 +385,19 @@ pub(super) fn segments(titles: Option<(u32, u32)>, band: (u32, u32)) -> Vec<(u32
pub(super) fn seg_size(sh: &Sheet, seg: (u32, u32), rows: bool) -> f32 {
if rows { (seg.0..=seg.1).map(|r| sh.row_height(r)).sum() } else { (seg.0..=seg.1).map(|c| sh.col_width(c)).sum() }
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_print_area_past_the_grid_is_held_to_the_grid() {
let mut wb = Workbook::new();
let sh = wb.sheet_mut(0).unwrap();
sh.default_row_height = 0.0;
sh.print.print_area = Some(RangeRef { start: CellRef::new(0, 0), end: CellRef::new(u32::MAX, u32::MAX) });
let lay = layout_sheet(&wb, 0, &PrintOptions::default(), None).unwrap();
assert_eq!((lay.area.end.row, lay.area.end.col), (MAX_ROWS - 1, MAX_COLS - 1));
assert!(lay.pages.len() <= MAX_PAGES);
}
}
+16 -6
View File
@@ -88,9 +88,14 @@ pub(super) fn draw_objects(pg: &mut Page, wb: &Workbook, si: usize, sh: &Sheet,
let (x, y, w, h) = place(r);
// About 150 dpi is plenty for print.
let (mw, mh) = (((w / 72.0) * 150.0).ceil().clamp(1.0, 4000.0) as u32, ((h / 72.0) * 150.0).ceil().clamp(1.0, 4000.0) as u32);
let decoded = pics.entry((si, imi)).or_insert_with(|| decode_image(&im.data, mw, mh));
let decoded = pics.decoded.entry((si, imi)).or_insert_with(|| decode_image(&im.data, mw, mh));
if let Some((px, pw, ph)) = decoded.as_ref() {
pg.image_rgba(Rect::new(x, y, w, h), px, [*pw, *ph]);
// Every page the picture touches embeds its own copy: past the budget it is left out.
let bytes = px.len();
if pics.image_bytes.saturating_add(bytes) <= super::MAX_EMBEDDED_IMAGE_BYTES {
pics.image_bytes += bytes;
pg.image_rgba(Rect::new(x, y, w, h), px, [*pw, *ph]);
}
}
}
for shp in &sh.shapes {
@@ -120,15 +125,20 @@ pub(super) fn draw_objects(pg: &mut Page, wb: &Workbook, si: usize, sh: &Sheet,
}
draw_shape(pg, wb, shp, b, (x, y, w, h));
}
for ch in &sh.charts {
for (chi, ch) in sh.charts.iter().enumerate() {
let r = anchor_rect(sh, &ch.anchor);
if !visible(r) {
continue;
}
let (x, y, _, _) = place(r);
let data = gridcraft_chart::resolve(wb, si, ch);
let prims = gridcraft_chart::render(ch, &data, r.2 as f32, r.3 as f32, &PdfMeasure);
draw_prims(pg, &prims, Xf { ox: x, oy: y, k: b.k });
// As for pictures, each page gets its own copy of the drawing commands.
let prims = pics.chart((si, chi), || {
let data = gridcraft_chart::resolve(wb, si, ch);
gridcraft_chart::render(ch, &data, r.2 as f32, r.3 as f32, &PdfMeasure)
});
if let Some(prims) = prims {
draw_prims(pg, prims, Xf { ox: x, oy: y, k: b.k });
}
}
pg.restore();
}
+24
View File
@@ -938,3 +938,27 @@ fn equations_print_with_the_symbol_font() {
// A glyph no standard font has is a [?] box.
assert!(contains(&pdf, "(?) Tj"));
}
#[test]
fn chart_budget_counts_path_points_and_drops_charts_that_do_not_fit() {
use gridcraft_chart::Prim;
let line = |n: usize| Prim::Line { pts: vec![[0.0, 0.0]; n], color: [0, 0, 0, 255], width: 1.0, dash: false };
// A smoothed line is its points, not one unit.
assert_eq!(chart_cost(&[line(4000), Prim::Circle { cx: 0.0, cy: 0.0, r: 1.0, fill: None, stroke: None }]), 4002);
// Room for exactly two copies of a 1,000-point chart.
let mut pics = PicCache { chart_prims: MAX_CHART_PRIMS - 2002, ..Default::default() };
let mut renders = 0;
for expect in [true, true, false, false] {
let got = pics.chart((0, 0), || {
renders += 1;
vec![line(1000)]
});
assert_eq!(got.is_some(), expect);
}
assert_eq!(renders, 1, "a rejected chart is not rendered again");
assert!(pics.charts.is_empty(), "and its primitives are not kept");
// A chart too big for what is left is dropped without being cached at all.
let mut pics = PicCache { chart_prims: MAX_CHART_PRIMS - 10, ..Default::default() };
assert!(pics.chart((0, 1), || vec![line(1000)]).is_none());
assert!(pics.charts.is_empty() && pics.rejected_charts.contains(&(0, 1)));
}
@@ -83,6 +83,9 @@ pub(crate) fn criterion_matches(v: &Value, text: &str, crit: &Value) -> bool {
}
}
/// Most (list row, criteria row) pairs one Advanced Filter tests.
const MAX_FILTER_TESTS: u64 = 50_000_000;
pub(super) fn advanced_filter(s: &mut Session, p: &Json) -> Result<Json> {
const CMD: &str = "data.advancedFilter";
let d = s.doc()?;
@@ -133,6 +136,10 @@ pub(super) fn advanced_filter(s: &mut Session, p: &Json) -> Result<Json> {
rows.push(conds);
}
}
// Each list row is tried against every criteria row, so bound their product too.
if u64::from(list.height()) * rows.len() as u64 > MAX_FILTER_TESTS {
return Err(bad(CMD, "the list and criteria ranges together are too large"));
}
let matches_row = |row: u32| -> bool {
if rows.is_empty() {
return true;
+9
View File
@@ -298,6 +298,15 @@ fn consolidate_by_position_and_label() {
assert!(s.execute("data.consolidate", json!({"sources": ["A1"], "function": "median"})).is_err());
}
#[test]
fn advanced_filter_refuses_a_list_times_criteria_product_over_the_work_limit() {
let mut s = s();
s.execute("range.setValues", json!({"range": "A1", "values": [["Name"], ["A"]]})).unwrap();
s.execute("range.setValues", json!({"range": "H1", "values": [["Name"], ["A"]]})).unwrap();
// A sparse million-row list against 9,999 blank criteria rows would be ~10 billion tests.
assert!(s.execute("data.advancedFilter", json!({"range": "A2:A1048576", "criteria": "H1:H10000"})).is_err());
}
/// A hundred thousand positional rows take one key lookup each; a tall source and a wide one
/// whose keys would meet in a table of 160 million cells are refused before it is built.
#[test]
+106 -21
View File
@@ -3,7 +3,7 @@
use std::fmt::Write as _;
use gridcraft_core::{CellRef, RangeRef};
use gridcraft_model::{Sheet, Workbook};
use gridcraft_model::{LineIndex, Sheet, Workbook};
use crate::{EngineError, Result};
@@ -611,6 +611,24 @@ pub(crate) fn esc(s: &str) -> String {
s.replace('&', "&amp;").replace('<', "&lt;").replace('>', "&gt;").replace('"', "&quot;")
}
/// [`esc`], or `None` once the escaped text would pass `limit` bytes (nothing larger is built).
fn esc_within(s: &str, limit: usize) -> Option<String> {
let mut out = String::with_capacity(s.len().min(limit));
for c in s.chars() {
match c {
'&' => out.push_str("&amp;"),
'<' => out.push_str("&lt;"),
'>' => out.push_str("&gt;"),
'"' => out.push_str("&quot;"),
c => out.push(c),
}
if out.len() > limit {
return None;
}
}
Some(out)
}
fn html_cell_css(wb: &Workbook, sh: &Sheet, c: CellRef, dir: CellDir) -> String {
let st = wb.styles.get(sh.style_id(c));
let mut css = String::new();
@@ -686,6 +704,30 @@ fn table_dir(sh: &Sheet) -> &'static str {
const CLIPBOARD_HTML_MAX_BYTES: usize = 4 * 1024 * 1024;
const CLIPBOARD_HTML_MAX_CELLS: u64 = 10_000;
/// Most cells the merge index holds: one entry per merged cell inside the exported range.
const MERGE_INDEX_MAX_ENTRIES: usize = 1_000_000;
/// The merge covering each cell of `rect`, as `Sheet::merge_at` would find it (the first in the
/// sheet's list), built once so a table lookup is one probe instead of a scan of every merge.
/// `None` when overlapping merges would cost more than a few visits per cell of `rect`, or merged
/// cells fill more than [`MERGE_INDEX_MAX_ENTRIES`]: a hostile file's merge list can't multiply the
/// work or the memory of a bounded export.
fn merge_index(sh: &Sheet, rect: RangeRef) -> Option<std::collections::HashMap<CellRef, RangeRef>> {
let mut map = std::collections::HashMap::new();
let mut left = rect.count().saturating_mul(4);
for m in &sh.merges {
let Some(part) = m.intersection(&rect) else { continue };
left = left.checked_sub(part.count())?;
for c in part.iter() {
map.entry(c).or_insert(*m);
}
if map.len() > MERGE_INDEX_MAX_ENTRIES {
return None;
}
}
Some(map)
}
/// An HTML fragment for the same range as `plain_text`, previously produced by
/// `display::range_text`. Large copies keep plain text only; never return a partial table.
pub(crate) fn range_html(wb: &Workbook, sheet: usize, r: RangeRef, plain_text: &str) -> Option<String> {
@@ -694,6 +736,7 @@ pub(crate) fn range_html(wb: &Workbook, sheet: usize, r: RangeRef, plain_text: &
return None;
}
let sh = wb.sheet(sheet)?;
let merges = merge_index(sh, r)?;
let mut out = ClipboardHtml(String::new());
write!(out, "<table{} style=\"border-collapse:collapse;font-family:Calibri,Carlito,Arial,sans-serif;font-size:11pt\">", table_dir(sh)).ok()?;
for row in r.start.row..=r.end.row {
@@ -703,7 +746,7 @@ pub(crate) fn range_html(wb: &Workbook, sheet: usize, r: RangeRef, plain_text: &
out.write_str("<tr>").ok()?;
for col in r.start.col..=r.end.col {
let c = CellRef::new(row, col);
let span = if let Some(merged) = sh.merges.iter().find(|m| m.contains(c)).and_then(|m| m.intersection(&r)) {
let span = if let Some(merged) = merges.get(&c).and_then(|m| m.intersection(&r)) {
let first_row = (merged.start.row..=merged.end.row).find(|&row| !sh.is_row_hidden(row))?;
if c != CellRef::new(first_row, merged.start.col) {
continue;
@@ -766,6 +809,14 @@ const HTML_EXPORT_MAX_ROWS: u32 = 100_001;
/// tall, wide one) would otherwise be gigabytes of `<td></td>` before finishing (#2387).
const HTML_EXPORT_MAX_CELLS: u64 = 4_000_000;
/// Most bytes of one web page. Every cell may hold long text (shared across many cells in the
/// file), so the cell count alone doesn't bound the page; past this the export fails instead.
const HTML_EXPORT_MAX_BYTES: usize = 256 * 1024 * 1024;
fn html_too_big() -> EngineError {
EngineError::Other(format!("the web page would be over {} MiB. Save as .xlsx or CSV instead.", HTML_EXPORT_MAX_BYTES / (1024 * 1024)))
}
/// A sheet as a standalone HTML table (Save as Web Page). A sheet with pictures gets the sheet's
/// own column widths and row heights so each picture and chart (inline SVG) can sit at its anchor.
pub fn to_html(wb: &Workbook, sheet: usize) -> Result<String> {
@@ -808,6 +859,24 @@ pub fn to_html(wb: &Workbook, sheet: usize) -> Result<String> {
r.width()
)));
}
let merges = merge_index(sh, r)
.ok_or_else(|| EngineError::Other("the sheet's merged cells overlap too much, or cover too many cells, for a web page.".into()))?;
// Visible rows and columns of the table before each line, so a merge's span is counted
// in constant time whatever the file says its size is.
let visible_before = |hidden: &dyn Fn(u32) -> bool, from: u32, to: u32| {
let mut v = Vec::with_capacity((to - from) as usize + 2);
v.push(0u32);
for line in from..=to {
v.push(v.last().copied().unwrap_or(0) + u32::from(!hidden(line)));
}
v
};
let rows_before = visible_before(&|y| sh.is_row_hidden(y), r.start.row, r.end.row);
let cols_before = visible_before(&|x| sh.is_col_hidden(x), r.start.col, r.end.col);
let visible = |before: &[u32], from: u32, a: u32, b: u32| {
let at = |line: u32| before.get((line - from) as usize).copied().unwrap_or(0);
at(b + 1) - at(a)
};
if fixed {
out.push_str("<colgroup>");
for col in (r.start.col..=r.end.col).filter(|&c| !sh.is_col_hidden(c)) {
@@ -826,27 +895,31 @@ pub fn to_html(wb: &Workbook, sheet: usize) -> Result<String> {
}
for col in r.start.col..=r.end.col {
let c = CellRef::new(row, col);
if sh.merges.iter().any(|m| m.contains(c) && m.start != c) || (fixed && sh.is_col_hidden(col)) {
let merge = merges.get(&c);
if merge.is_some_and(|m| m.start != c) || (fixed && sh.is_col_hidden(col)) {
continue;
}
let text = crate::display::cell_text(wb, sh, c);
let dir = CellDir::of(sh, c, &text);
let css = html_cell_css(wb, sh, c, dir);
let span = sh
.merges
.iter()
.find(|m| m.start == c)
let span = merge
.map(|m| {
if fixed {
let rows = (m.start.row..=m.end.row).filter(|&y| !sh.is_row_hidden(y)).count();
let cols = (m.start.col..=m.end.col).filter(|&x| !sh.is_col_hidden(x)).count();
// Spans count the table's visible lines: the merge is clipped to the table.
let rows = visible(&rows_before, r.start.row, m.start.row, m.end.row.min(r.end.row));
let cols = visible(&cols_before, r.start.col, m.start.col, m.end.col.min(r.end.col));
format!(" rowspan=\"{rows}\" colspan=\"{cols}\"")
} else {
format!(" rowspan=\"{}\" colspan=\"{}\"", m.height(), m.width())
}
})
.unwrap_or_default();
let _ = write!(out, "<td style=\"{css}\"{}{span}>{}</td>", dir.attr(), esc(&text));
// The text is escaped only as far as the page still has room for it.
let Some(escaped) = esc_within(&text, HTML_EXPORT_MAX_BYTES.saturating_sub(out.len())) else { return Err(html_too_big()) };
let _ = write!(out, "<td style=\"{css}\"{}{span}>{escaped}</td>", dir.attr());
if out.len() > HTML_EXPORT_MAX_BYTES {
return Err(html_too_big());
}
}
out.push_str("</tr>");
}
@@ -854,36 +927,48 @@ pub fn to_html(wb: &Workbook, sheet: usize) -> Result<String> {
out.push_str("</table>");
if fixed {
// Sheet units are CSS pixels; the table starts at the used range, not at A1.
let (left, top) = used.map_or((0.0, 0.0), |r| (sh.col_left(r.start.col), sh.row_top(r.start.row)));
// Indexed positions: each anchor would otherwise sum every custom row and column before it.
let ri = LineIndex::new(&sh.rows, sh.default_row_height, gridcraft_core::MAX_ROWS);
let ci = LineIndex::new(&sh.cols, sh.default_col_width, gridcraft_core::MAX_COLS);
let (left, top) = used.map_or((0.0, 0.0), |r| (ci.start(r.start.col), ri.start(r.start.row)));
let place = |a: &gridcraft_model::Anchor| {
format!(
"position:absolute;left:{}px;top:{}px;width:{}px;height:{}px",
sh.col_left(a.cell.col) - left + f64::from(a.dx),
sh.row_top(a.cell.row) - top + f64::from(a.dy),
ci.start(a.cell.col) - left + f64::from(a.dx),
ri.start(a.cell.row) - top + f64::from(a.dy),
a.width.max(0.0),
a.height.max(0.0)
)
};
for im in &sh.images {
let _ = write!(
out,
"<img src=\"data:{};base64,{}\" alt=\"{}\" style=\"{}\">",
esc(&im.mime),
base64_encode(&im.data),
esc(&im.alt),
place(&im.anchor)
);
// The base64 text is a third longer than the picture: refused before it is built.
if out.len().saturating_add(im.data.len().div_ceil(3).saturating_mul(4)) > HTML_EXPORT_MAX_BYTES {
return Err(html_too_big());
}
// The attributes are escaped only as far as the page has room for them.
let room = HTML_EXPORT_MAX_BYTES.saturating_sub(out.len());
let (Some(mime), Some(alt)) = (esc_within(&im.mime, room), esc_within(&im.alt, room)) else { return Err(html_too_big()) };
let _ = write!(out, "<img src=\"data:{mime};base64,{}\" alt=\"{alt}\" style=\"{}\">", base64_encode(&im.data), place(&im.anchor));
if out.len() > HTML_EXPORT_MAX_BYTES {
return Err(html_too_big());
}
}
let mut svg_budget = crate::chart_svg::MAX_PAGE_SVG_BYTES;
for ch in &sh.charts {
if let Some(svg) = crate::chart_svg::chart_svg(wb, sheet, ch, &place(&ch.anchor), svg_budget)? {
svg_budget -= svg.len();
out.push_str(&svg);
if out.len() > HTML_EXPORT_MAX_BYTES {
return Err(html_too_big());
}
}
}
out.push_str("</div>");
}
out.push_str("</body></html>");
if out.len() > HTML_EXPORT_MAX_BYTES {
return Err(html_too_big());
}
Ok(out)
}
+37 -301
View File
@@ -1,156 +1,57 @@
//! SYLK (`.slk`) import: the symbolic link text format older spreadsheets and some legacy systems
//! export. One record per line, fields separated by `;` (`;;` is a literal semicolon):
//!
//! - `ID` starts the file; `E` ends it.
//! - `C;Y<row>;X<col>;K<value>;E<formula>` is a cell. `Y` and `X` may be left out to stay on the
//! current row or column. `K` is a number, `"text"`, `TRUE`/`FALSE` or an error; `E` is a
//! formula in R1C1 notation; `S;R<row>;C<col>` shares the formula of another cell.
//! - `P;P<code>` adds a number format to the file's list; `F;P<n>;Y<row>;X<col>` gives a cell the
//! n-th format (0-based), and `F;W<first> <last> <width>` sets column widths in characters.
//!
//! Other records (fonts, options, window settings) are skipped. Formulas are converted to A1 with
//! the same relative and absolute references, as Excel does when it opens the file.
//! SYLK (`.slk`) import. `craft_csv::sylk` reads the file (values, formulas rewritten from R1C1 to
//! A1, number formats, column widths); this module makes it a one-sheet workbook: formulas parsed
//! (a formula GridCraft can't parse is skipped and counted), error values recognised, formats
//! interned as styles and widths converted from characters.
use std::collections::HashMap;
use std::sync::Arc;
use gridcraft_core::{CellError, CellRef, MAX_COLS, MAX_ROWS, Value, col_to_letters};
use craft_csv::sylk;
use gridcraft_core::{CellError, CellRef, Value};
use gridcraft_model::{Cell, Formula, NumFmt, Sheet, Style, Workbook};
use crate::{EngineError, Result};
/// The largest SYLK file read, as for other text imports: past it the file is refused.
const MAX_BYTES: usize = 512 << 20;
/// Column-width records kept from one file; each can cover every column.
const MAX_WIDTH_RECORDS: usize = 1024;
/// Reads a SYLK file into a one-sheet workbook. Returns the workbook and its warnings.
pub fn read(name: &str, bytes: &[u8]) -> Result<(Workbook, Vec<String>)> {
if bytes.len() > MAX_BYTES {
return Err(EngineError::Other(format!("We can't open '{name}': the SYLK file is larger than {} MB.", MAX_BYTES >> 20)));
}
let text = match std::str::from_utf8(bytes) {
Ok(s) => s.to_string(),
// Older exports are in a single-byte code page: keep each byte as its Latin-1 character.
Err(_) => bytes.iter().map(|&b| char::from(b)).collect(),
};
let text = text.strip_prefix('\u{feff}').unwrap_or(&text);
if !text.starts_with("ID") {
return Err(EngineError::Other(format!("We can't open '{name}': it isn't a SYLK file (it doesn't start with an ID record).")));
}
let file = sylk::read(bytes).map_err(|e| match e {
sylk::Error::TooLarge => EngineError::Other(format!("We can't open '{name}': the SYLK file is larger than {} MB.", sylk::MAX_BYTES >> 20)),
_ => EngineError::Other(format!("We can't open '{name}': it isn't a SYLK file (it doesn't start with an ID record).")),
})?;
let mut wb = Workbook::new();
let mut sheet = Sheet::new("Sheet1");
let mut formats: Vec<String> = Vec::new();
// Cells' number formats by index into `formats`, applied once every format is known.
let mut cell_formats: Vec<(CellRef, usize)> = Vec::new();
let mut formulas: HashMap<CellRef, String> = HashMap::new();
let mut widths: Vec<(u32, u32, f64)> = Vec::new();
let (mut row, mut col) = (0u32, 0u32);
let mut skipped = 0usize;
for line in text.lines() {
let fields = split_fields(line);
let Some((kind, rest)) = fields.split_first() else { continue };
match kind.as_str() {
"C" => {
let (mut value, mut formula, mut shared) = (None, None, None::<(Option<u32>, Option<u32>)>);
let mut bad = false;
for f in rest {
let (tag, arg) = f.split_at(f.chars().next().map_or(0, char::len_utf8));
match tag {
"Y" => match position(arg, MAX_ROWS) {
Some(r) => row = r,
None => bad = true,
},
"X" => match position(arg, MAX_COLS) {
Some(c) => col = c,
None => bad = true,
},
"K" => value = Some(parse_value(arg)),
"E" => formula = Some(arg.to_string()),
"S" => shared = Some((None, None)),
// In a shared-formula cell, R and C name the cell whose formula it shares.
"R" if shared.is_some() => shared = shared.map(|(_, c)| (position(arg, MAX_ROWS), c)),
"C" if shared.is_some() => shared = shared.map(|(r, _)| (r, position(arg, MAX_COLS))),
_ => {}
}
}
if bad {
skipped += 1;
continue;
}
let at = CellRef::new(row, col);
if let Some((Some(r), Some(c))) = shared {
formula = formulas.get(&CellRef::new(r, c)).cloned();
}
let mut cell = sheet.cell(at).cloned().unwrap_or_else(|| Cell::value(Value::Empty));
if let Some(v) = value {
cell.value = v;
}
if let Some(r1c1) = formula {
let a1 = r1c1_formula_to_a1(&r1c1, at);
let f = Formula::new(&a1);
if f.expr.is_some() {
cell.formula = Some(Arc::new(f));
formulas.insert(at, r1c1);
} else {
skipped += 1;
}
}
sheet.set_cell(at, cell);
let mut skipped = file.skipped;
let mut styles = HashMap::new();
for (&(row, col), c) in &file.cells {
let value = match &c.value {
sylk::Value::Number(n) => Value::Number(*n),
sylk::Value::Text(s) => Value::Text(Arc::from(s.as_str())),
sylk::Value::Bool(b) => Value::Bool(*b),
sylk::Value::Error(s) => CellError::parse(s).map_or_else(|| Value::Text(Arc::from(s.as_str())), Value::Error),
_ => Value::Empty,
};
let mut cell = Cell::value(value);
if let Some(a1) = &c.formula {
let f = Formula::new(a1);
if f.expr.is_some() {
cell.formula = Some(Arc::new(f));
} else {
skipped += 1;
}
"F" => {
let mut format = None;
for f in rest {
let (tag, arg) = f.split_at(f.chars().next().map_or(0, char::len_utf8));
match tag {
"Y" => row = position(arg, MAX_ROWS).unwrap_or(row),
"X" => col = position(arg, MAX_COLS).unwrap_or(col),
"P" => format = arg.parse::<usize>().ok(),
"W" => {
let mut parts = arg.split_whitespace();
if let (Some(first), Some(last), Some(width)) = (parts.next(), parts.next(), parts.next())
&& let (Some(first), Some(last), Ok(width)) =
(position(first, MAX_COLS), position(last, MAX_COLS), width.parse::<f64>())
&& first <= last
&& (0.0..=255.0).contains(&width)
// A file of width records can't make the import loop over every column
// again and again.
&& widths.len() < MAX_WIDTH_RECORDS
{
widths.push((first, last, width));
}
}
_ => {}
}
}
if let Some(format) = format {
cell_formats.push((CellRef::new(row, col), format));
}
}
// Number formats only: `P;F…` and `P;E…` are fonts.
"P" => {
if let Some(code) = rest.first().and_then(|f| f.strip_prefix('P')) {
formats.push(code.to_string());
}
}
"E" => break,
_ => {}
}
if let Some(code) = &c.format {
cell.style = *styles
.entry(code.as_str())
.or_insert_with(|| wb.styles.intern(Style { num_fmt: NumFmt(Arc::from(code.as_str())), ..Style::default() }));
}
sheet.set_cell(CellRef::new(row, col), cell);
}
for (first, last, width) in widths {
let size = crate::cmd::format::chars_to_points(width) as f32;
for c in first..=last {
for w in &file.column_widths {
let size = crate::cmd::format::chars_to_points(w.width) as f32;
for c in w.first..=w.last {
sheet.cols.entry(c).or_default().size = Some(size);
}
}
let mut wb = Workbook::new();
let mut styles = HashMap::new();
for (at, index) in cell_formats {
let Some(code) = formats.get(index).filter(|code| !code.is_empty() && *code != "General") else { continue };
let style = *styles.entry(index).or_insert_with(|| wb.styles.intern(Style { num_fmt: NumFmt(Arc::from(code.as_str())), ..Style::default() }));
let mut cell = sheet.cell(at).cloned().unwrap_or_else(|| Cell::value(Value::Empty));
cell.style = style;
sheet.set_cell(at, cell);
}
if let Some(first) = wb.sheet_mut(0) {
sheet.name = std::mem::take(&mut first.name);
*first = sheet;
@@ -161,168 +62,3 @@ pub fn read(name: &str, bytes: &[u8]) -> Result<(Workbook, Vec<String>)> {
}
Ok((wb, warnings))
}
/// A record's fields: `;` separates them and `;;` is a literal semicolon.
fn split_fields(line: &str) -> Vec<String> {
let mut fields = vec![String::new()];
let mut chars = line.trim_end_matches('\r').chars().peekable();
while let Some(c) = chars.next() {
if c == ';' {
if chars.peek() == Some(&';') {
chars.next();
if let Some(f) = fields.last_mut() {
f.push(';');
}
} else {
fields.push(String::new());
}
} else if let Some(f) = fields.last_mut() {
f.push(c);
}
}
fields
}
/// A 1-based row or column number → 0-based, within `max`.
fn position(arg: &str, max: u32) -> Option<u32> {
arg.trim().parse::<u32>().ok().filter(|n| (1..=max).contains(n)).map(|n| n - 1)
}
/// A `K` value: `"text"` (everything between the first and the last quote, as written), a
/// boolean, an error or a number.
fn parse_value(arg: &str) -> Value {
if let Some(text) = arg.strip_prefix('"') {
let text = text.strip_suffix('"').unwrap_or(text);
return Value::Text(Arc::from(text));
}
if arg.eq_ignore_ascii_case("TRUE") || arg.eq_ignore_ascii_case("FALSE") {
return Value::Bool(arg.eq_ignore_ascii_case("TRUE"));
}
if let Some(e) = CellError::parse(arg) {
return Value::Error(e);
}
match arg.trim().parse::<f64>() {
Ok(n) if n.is_finite() => Value::Number(n),
_ => Value::Text(Arc::from(arg)),
}
}
/// One part of an R1C1 reference: `R`/`C` followed by an absolute number, a bracketed offset or
/// nothing (the cell's own row or column).
#[derive(Clone, Copy, Debug, PartialEq)]
enum Part {
Absolute(u32),
Relative(i64),
}
impl Part {
/// The 0-based index and whether it is absolute, or `None` when it falls off the sheet.
fn resolve(self, base: u32, max: u32) -> Option<(u32, bool)> {
match self {
Part::Absolute(n) => (1..=max).contains(&n).then(|| (n - 1, true)),
Part::Relative(k) => i64::from(base).checked_add(k).and_then(|v| u32::try_from(v).ok()).filter(|v| *v < max).map(|v| (v, false)),
}
}
}
/// Parses the number after `R` or `C` at `i`: digits, `[offset]` or nothing. Returns the part and
/// the index after it, or `None` when the text isn't a reference part.
fn parse_part(chars: &[char], i: usize) -> Option<(Part, usize)> {
match chars.get(i) {
Some('[') => {
let close = chars[i..].iter().position(|&c| c == ']')? + i;
let inner: String = chars[i + 1..close].iter().collect();
Some((Part::Relative(inner.parse().ok()?), close + 1))
}
Some(c) if c.is_ascii_digit() => {
let end = chars[i..].iter().position(|c| !c.is_ascii_digit()).map_or(chars.len(), |p| p + i);
let n: String = chars[i..end].iter().collect();
Some((Part::Absolute(n.parse().unwrap_or(u32::MAX)), end))
}
_ => Some((Part::Relative(0), i)),
}
}
fn is_word_char(c: char) -> bool {
c.is_alphanumeric() || matches!(c, '_' | '.' | '\\' | '?' | '$')
}
/// Rewrites the R1C1 references in a SYLK formula as A1 references for the cell at `at`:
/// `RC[-1]` → `B2`, `R1C1` → `$A$1`, `R[-2]C:R[-1]C` → `D2:D3`, a whole column `C4` → `$D:$D` and
/// a whole row `R3` → `$3:$3`. Text in quotes and quoted sheet names are left alone; a reference
/// off the sheet becomes `#REF!`.
pub(crate) fn r1c1_formula_to_a1(formula: &str, at: CellRef) -> String {
let chars: Vec<char> = formula.chars().collect();
let mut out = String::with_capacity(formula.len());
let mut i = 0;
while i < chars.len() {
let c = chars[i];
if c == '"' || c == '\'' {
// Copy the quoted run as it is; a doubled quote stays inside it.
let mut j = i + 1;
while j < chars.len() && !(chars[j] == c && chars.get(j + 1) != Some(&c)) {
j += if chars[j] == c { 2 } else { 1 };
}
let stop = (j + 1).min(chars.len());
out.extend(&chars[i..stop]);
i = stop;
continue;
}
let starts_word = i == 0 || !is_word_char(chars[i - 1]);
if starts_word
&& matches!(c, 'R' | 'r' | 'C' | 'c')
&& let Some((text, next)) = reference_at(&chars, i, at)
{
out.push_str(&text);
i = next;
continue;
}
// Copy the rest of a word as it is, so `ROUND` or `Sheet2` isn't read as a reference.
if is_word_char(c) {
let end = (i..chars.len()).find(|&j| !is_word_char(chars[j])).unwrap_or(chars.len());
out.extend(&chars[i..end]);
i = end;
} else {
out.push(c);
i += 1;
}
}
out
}
/// The A1 text of the R1C1 reference starting at `i`, and the index after it, or `None` when the
/// word there isn't a reference (a function or a name).
fn reference_at(chars: &[char], i: usize, at: CellRef) -> Option<(String, usize)> {
let mut j = i;
let mut row = None;
let mut col = None;
if matches!(chars.get(j), Some('R' | 'r')) {
let (part, next) = parse_part(chars, j + 1)?;
row = Some(part);
j = next;
}
if matches!(chars.get(j), Some('C' | 'c')) {
let (part, next) = parse_part(chars, j + 1)?;
col = Some(part);
j = next;
}
if row.is_none() && col.is_none() {
return None;
}
// A reference ends the word: `ROUND(`, `RATE` or `R2D2` are names.
if chars.get(j).is_some_and(|&c| is_word_char(c) || c == '(' || c == '[' || c == '!') {
return None;
}
let row_text = |p: Part| p.resolve(at.row, MAX_ROWS).map(|(r, abs)| format!("{}{}", if abs { "$" } else { "" }, r + 1));
let col_text = |p: Part| p.resolve(at.col, MAX_COLS).map(|(c, abs)| format!("{}{}", if abs { "$" } else { "" }, col_to_letters(c)));
// One side of a range (`R2:R4`, `C1:C3`) is written alone; a lone row or column is the whole of it.
let prev = chars[..i].iter().rev().find(|c| !c.is_whitespace());
let in_range = prev == Some(&':') || chars[j..].iter().find(|c| !c.is_whitespace()) == Some(&':');
let text = match (row, col) {
(Some(r), Some(c)) => row_text(r).zip(col_text(c)).map(|(r, c)| format!("{c}{r}")),
(Some(r), None) => row_text(r).map(|r| if in_range { r } else { format!("{r}:{r}") }),
(None, Some(c)) => col_text(c).map(|c| if in_range { c } else { format!("{c}:{c}") }),
(None, None) => None,
};
Some((text.unwrap_or_else(|| "#REF!".into()), j))
}
+72 -1
View File
@@ -763,7 +763,7 @@ fn sylk_r1c1_formulas_become_a1() {
("R[-5]C", "#REF!"),
("\"RC\"&R1C1", "\"RC\"&$A$1"),
] {
assert_eq!(sylk::r1c1_formula_to_a1(r1c1, b2), a1, "{r1c1}");
assert_eq!(craft_csv::sylk::r1c1_formula_to_a1(r1c1, (b2.row, b2.col)), a1, "{r1c1}");
}
}
@@ -810,3 +810,74 @@ fn reading_a_fifo_is_refused_without_blocking() {
assert!(err.contains("only regular files"), "{err}");
let _ = std::fs::remove_dir_all(&dir);
}
/// Merges far from what is copied or exported, in a hostile number, are indexed once instead of
/// being scanned for every cell; a merge running past the table is clipped to it.
#[test]
fn html_merge_lookups_and_spans_stay_bounded() {
let mut wb = book_with_picture("A1", (0, 0));
let sh = wb.sheet_mut(0).unwrap();
for i in 0..200_000u32 {
sh.merges.push(RangeRef::new(CellRef::new(500_000 + i * 2, 5), CellRef::new(500_000 + i * 2, 6)));
}
sh.merges.push(RangeRef::new(CellRef::new(0, 0), CellRef::new(gridcraft_core::MAX_ROWS - 1, 1)));
sh.set_value(CellRef::new(2, 1), gridcraft_core::Value::Number(2.0));
let html = copied(&wb, "A1:C50");
assert!(html.contains("rowspan=\"50\" colspan=\"2\""), "{html}");
let page = to_html(&wb, 0).unwrap();
// The table is A1:B3 (the used range): the whole-height merge spans its three rows only.
assert!(page.contains("rowspan=\"3\" colspan=\"2\""), "{page}");
// Merges stacked over one another past a few per cell are refused, not walked.
let sh = wb.sheet_mut(0).unwrap();
sh.merges = vec![RangeRef::new(CellRef::new(0, 0), CellRef::new(2, 1)); 100];
assert!(to_html(&wb, 0).is_err());
assert!(range_html(&wb, 0, RangeRef::parse("A1:B3").unwrap(), "").is_none());
}
/// Thousands of pictures over thousands of custom rows: each is placed through a row index.
#[test]
fn html_export_places_many_pictures_over_many_custom_rows() {
let mut wb = book_with_picture("A1", (0, 0));
let sh = wb.sheet_mut(0).unwrap();
for row in 0..50_000 {
sh.rows.insert(row, gridcraft_model::LineInfo { size: Some(30.0), ..Default::default() });
}
let image = sh.images[0].clone();
for i in 0..5_000u32 {
let mut im = image.clone();
im.anchor.cell = CellRef::new(40_000 + i, 0);
sh.images.push(im);
}
let page = to_html(&wb, 0).unwrap();
assert!(page.contains(&format!("top:{}px", 30 * 44_999)), "last picture placed by the index");
}
/// One long text shown by many cells: the page fails at its byte limit instead of growing on.
#[test]
fn html_export_refuses_a_page_past_its_byte_limit() {
let mut wb = Workbook::new();
let sh = wb.sheet_mut(0).unwrap();
let text = gridcraft_core::Value::text("x".repeat(32_000));
for row in 0..10_000 {
sh.set_value(CellRef::new(row, 0), text.clone());
}
let err = to_html(&wb, 0).unwrap_err();
assert!(err.to_string().contains("MiB"), "{err}");
}
#[test]
fn escaping_stops_at_the_room_left() {
assert_eq!(esc_within("a&b<c>\"d", 100).as_deref(), Some(esc("a&b<c>\"d").as_str()));
assert_eq!(esc_within("&&&", 14), None, "three `&amp;` are 15 bytes");
assert_eq!(esc_within("&&&", 15).as_deref(), Some("&amp;&amp;&amp;"));
}
#[test]
fn the_merge_index_holds_a_bounded_number_of_cells() {
let mut sh = Sheet::new("S");
let rect = RangeRef::new(CellRef::new(0, 0), CellRef::new(1_100_000, 0));
sh.merges.push(RangeRef::new(CellRef::new(0, 0), CellRef::new(999_999, 0)));
assert_eq!(merge_index(&sh, rect).map(|m| m.len()), Some(1_000_000));
sh.merges = vec![RangeRef::new(CellRef::new(0, 0), CellRef::new(1_000_000, 0))];
assert!(merge_index(&sh, rect).is_none());
}
+12 -3
View File
@@ -1281,9 +1281,18 @@ fn content_differs(b: Option<&Cell>, a: Option<&Cell>) -> bool {
/// Objects added, changed and removed between two versions of a sheet.
fn objects(b: &Sheet, a: &Sheet) -> (usize, usize, usize) {
fn count<T: PartialEq>(b: &[T], a: &[T], id: impl Fn(&T) -> u32) -> (usize, usize, usize) {
let added = a.iter().filter(|x| !b.iter().any(|y| id(y) == id(x))).count();
let removed = b.iter().filter(|x| !a.iter().any(|y| id(y) == id(x))).count();
let changed = b.iter().filter(|x| a.iter().any(|y| id(y) == id(x) && y != *x)).count();
// Objects grouped by id, so each lookup is a map probe instead of a scan of the list.
fn group<'a, T>(v: &'a [T], id: &impl Fn(&T) -> u32) -> std::collections::HashMap<u32, Vec<&'a T>> {
let mut m: std::collections::HashMap<u32, Vec<&T>> = std::collections::HashMap::new();
for x in v {
m.entry(id(x)).or_default().push(x);
}
m
}
let (gb, ga) = (group(b, &id), group(a, &id));
let added = a.iter().filter(|x| !gb.contains_key(&id(x))).count();
let removed = b.iter().filter(|x| !ga.contains_key(&id(x))).count();
let changed = b.iter().filter(|x| ga.get(&id(x)).is_some_and(|ys| ys.iter().any(|y| *y != *x))).count();
(added, changed, removed)
}
let parts = [
+15
View File
@@ -153,6 +153,11 @@ pub fn compile(formula: &str, field: &dyn Fn(&str) -> Option<String>) -> Result<
Expr::Name(n) if n.starts_with(PREFIX) => {}
Expr::Name(n) => problem = Some(format!("There's no field named '{n}'.")),
Expr::Ref(_) | Expr::Struct(_) => problem = Some("A calculated field can't refer to cells or tables; use field names.".into()),
// Evaluated once per group and per report, with only the first value kept: Excel does not
// allow functions that need arrays in a calculated field, and these build one from scalars.
Expr::Call(f, _) if matches!(f.as_str(), "SEQUENCE" | "RANDARRAY" | "MAKEARRAY" | "EXPAND" | "MUNIT") => {
problem = Some(format!("A calculated field can't use {f}: it builds an array."))
}
_ => {}
}
});
@@ -332,6 +337,16 @@ mod tests {
assert!(compile(&format!("={}", "1+".repeat(MAX_FORMULA_LEN)), &fields).is_err());
}
#[test]
fn a_calculated_field_cannot_build_arrays_from_scalars() {
for f in
["=SEQUENCE(5000,1000)", "=SUM(RANDARRAY(1000,1000))", "=MAKEARRAY(9,9,LAMBDA(r,c,r*c))+Sales", "=expand(Sales,1000,1000)", "=MUNIT(500)"]
{
assert!(compile(f, &fields).unwrap_err().contains("builds an array"), "{f}");
}
assert!(compile("=SUM({1,2,3})*Sales", &fields).is_ok(), "an array constant is bounded by the formula's own length");
}
#[test]
fn deep_and_cyclic_chains_are_bounded() {
// 300 calculated fields, each using the previous one, and a two-field cycle.
+28
View File
@@ -3612,6 +3612,8 @@ fn xls_fixture() -> Vec<u8> {
let mut put = |at: usize, v: &[u8]| b[at..at + v.len()].copy_from_slice(v);
put(0, &[0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1]);
put(26, &3u16.to_le_bytes());
// The byte order mark [MS-CFB] 2.2 requires; craft-cfb refuses a header without it.
put(28, &0xFFFEu16.to_le_bytes());
put(30, &9u16.to_le_bytes());
put(32, &6u16.to_le_bytes());
put(44, &1u32.to_le_bytes());
@@ -6480,6 +6482,19 @@ fn clearing_links_and_notes_keeps_allow_edit_ranges() {
assert!(sh.hyperlinks.is_empty() && sh.comments.is_empty());
}
#[test]
fn replace_does_not_search_hidden_formula_source() {
let mut s = s();
s.execute("cell.set", json!({"cell": "A1", "input": "=SEQUENCE(3)*7"})).unwrap();
s.execute("home.formatCells", json!({"range": "A1", "style": {"protection": {"locked": true, "hidden": true}}})).unwrap();
s.execute("review.protectSheet", json!({})).unwrap();
// Both give the same answer, and it is not an error: a hidden formula's source is not searched.
let hit = s.execute("edit.replace", json!({"what": "SEQUENCE", "with": "X"})).unwrap();
let miss = s.execute("edit.replace", json!({"what": "NOSUCHTEXT", "with": "X"})).unwrap();
assert_eq!(hit, miss);
assert_eq!(hit["replaced"], json!(0), "{hit}");
}
#[test]
fn formulatext_does_not_reveal_a_hidden_formula() {
let mut s = s();
@@ -6556,6 +6571,19 @@ fn pivot_new_sheet_destination_respects_workbook_structure_protection() {
assert!(s.execute("insert.pivotTable", json!({"source": "A1:B3", "location": "New"})).is_err());
}
#[test]
fn circle_invalid_says_when_its_scan_allowance_ended_early() {
let mut s = s();
s.execute("range.setValues", json!({"range": "A1", "values": [[-1]]})).unwrap();
s.execute("range.setValues", json!({"range": "A200000", "values": [[-1]]})).unwrap();
s.execute("data.validation", json!({"range": "A1:A5", "type": "whole", "operator": "greater", "formula1": "0"})).unwrap();
let out = s.execute("data.circleInvalid", json!({})).unwrap();
assert_eq!((out["invalid"].as_array().map(Vec::len), out["truncated"].as_bool()), (Some(1), Some(false)), "{out}");
s.execute("data.validation", json!({"range": "A1:A200000", "type": "whole", "operator": "greater", "formula1": "0"})).unwrap();
let out = s.execute("data.circleInvalid", json!({})).unwrap();
assert_eq!(out["truncated"].as_bool(), Some(true), "{out}");
}
#[test]
fn history_charges_a_changed_sheets_comment_text() {
let mut s = s();
+24
View File
@@ -69,11 +69,35 @@ fn parse_top(src: &str, locale: crate::FormulaLocale, allow_union: bool) -> Resu
let mut p = Parser { toks, i: 0, depth: 0, input: locale == crate::FormulaLocale::Es };
let e = p.expr(0, allow_union)?;
match p.peek() {
Tok::Eof if wrapper_height(&e) > MAX_DEPTH => Err(ParseError { msg: "formula is nested too deeply".into(), pos: 0 }),
Tok::Eof => Ok(e),
t => Err(p.err(format!("unexpected {}", describe(t)))),
}
}
/// The most unary operators, parentheses and invocations on one path of `e`. The parser bounds
/// each level's postfix wrappers by its own depth, so levels inside levels can still stack them
/// far past [`MAX_DEPTH`]; code that maps over the tree recurses once per wrapper.
fn wrapper_height(e: &Expr) -> usize {
let mut most = 0;
let mut pending = vec![(e, 0usize)];
while let Some((e, h)) = pending.pop() {
most = most.max(h);
match e {
Expr::Unary(_, a) | Expr::Paren(a) => pending.push((a, h + 1)),
Expr::Invoke(callee, args) => {
pending.push((callee, h + 1));
pending.extend(args.iter().map(|a| (a, h + 1)));
}
Expr::Binary(_, a, b) => pending.extend([(&**a, h), (&**b, h)]),
Expr::Call(_, args) => pending.extend(args.iter().map(|a| (a, h))),
Expr::Array(rows) => pending.extend(rows.iter().flatten().map(|a| (a, h))),
_ => {}
}
}
most
}
fn describe(t: &Tok) -> String {
match t {
Tok::RParen => "`)`".into(),
+16
View File
@@ -576,3 +576,19 @@ fn column_starting_with_at_round_trips() {
assert_eq!(parse(&text).ok(), Some(Expr::Struct(s)), "printed as {text}");
}
}
#[test]
fn wrappers_stacked_across_levels_are_bounded_as_a_whole() {
// Each level adds 40 percents inside the parser's per-level bound, so no level is too deep alone.
let stacked = |levels: usize| {
let mut f = "1".to_string();
for _ in 0..levels {
f = format!("({f}){}", "%".repeat(40));
}
f
};
assert!(parse(&stacked(5)).is_ok());
assert!(parse(&stacked(7)).is_err(), "7 x 41 wrappers on one path");
assert!(parse(&format!("{}1{}", "(".repeat(200), ")".repeat(200))).is_ok());
assert!(parse(&format!("={}", vec!["A1%"; 100].join("+"))).is_ok(), "siblings are not stacked");
}
+1
View File
@@ -13,6 +13,7 @@ mod day_count;
mod depreciation;
mod securities;
mod tvm;
pub(crate) use tvm::cumulative_args;
use day_count::{coupdaybs, coupdays, coupdaysnc, coupncd, coupnum, couppcd};
use depreciation::{amordegrc, amorlinc, db, ddb, sln, syd, vdb};
+8 -1
View File
@@ -116,7 +116,9 @@ pub(super) fn ppmt(a: &[Arg], _c: &mut dyn Ctx) -> R<Value> {
num_val(pmt_calc(rate, np, pv, fv, typ) - ipmt_calc(rate, per, np, pv, fv, typ))
}
fn cumulative(a: &[Arg], principal: bool) -> R<Value> {
/// The validated (rate, periods, pv, start, end, type) of CUMIPMT/CUMPRINC: every check that
/// refuses a call before its period loop, shared with the lifted-call work preflight.
pub(crate) fn cumulative_args(a: &[Arg]) -> R<(f64, f64, f64, f64, f64, f64)> {
let (rate, np, pv) = (num(a, 0)?, num(a, 1)?, num(a, 2)?);
let (start, end, typ) = (num(a, 3)?.trunc(), num(a, 4)?.trunc(), num(a, 5)?);
let np_i = np.trunc();
@@ -126,6 +128,11 @@ fn cumulative(a: &[Arg], principal: bool) -> R<Value> {
if end - start > 1e7 || np_i > 1e15 {
return Err(CellError::Num);
}
Ok((rate, np_i, pv, start, end, typ))
}
fn cumulative(a: &[Arg], principal: bool) -> R<Value> {
let (rate, np_i, pv, start, end, typ) = cumulative_args(a)?;
let p = pmt_calc(rate, np_i, pv, 0.0, typ);
let mut sum = 0.0;
let mut per = start;
+195 -26
View File
@@ -63,26 +63,86 @@ const SCANS: &[&str] = &["VLOOKUP", "HLOOKUP", "LOOKUP", "MATCH", "XLOOKUP", "XM
/// `XMATCH(SEQUENCE(1048576), -SEQUENCE(1048576))` runs for hours and hangs opening the file.
/// Past the budget the call is `#NUM!`, as MMULT is past 2e9 multiply-adds. 5e8 keeps the worst
/// case to a few seconds and still admits 5,000 needles over a 100,000-row table.
///
/// The same budget covers all the work of a lifted call added up across its elements: criteria
/// aggregates rescanning their ranges, period loops, and array results built per element.
const LIFTED_SCAN_BUDGET: usize = 500_000_000;
/// The length of the longest vector a lifted call of `spec` searches per needle: 0 unless `spec`
/// is one of the [`SCANS`] lookups and a non-lifted argument is an array.
/// Criteria aggregates: every call tests each cell of its ranges against its criteria, so a call
/// lifted over `n` criteria does `n` x (range cells) work, like the [`SCANS`] lookups.
const CRITERIA: &[&str] = &["COUNTIF", "COUNTIFS", "SUMIF", "SUMIFS", "AVERAGEIF", "AVERAGEIFS", "MAXIFS", "MINIFS"];
/// Cumulative TVM functions that step through every period from `start_period` (argument 3) to
/// `end_period` (argument 4) on each call.
const PERIOD_LOOPS: &[&str] = &["CUMIPMT", "CUMPRINC"];
/// Budget units one period of a [`PERIOD_LOOPS`] call costs: each period computes a payment and a
/// balance with powers, many times a scanned cell.
const PERIOD_COST: usize = 16;
/// The work every lifted call of `spec` repeats over its non-lifted arguments: the longest vector
/// a [`SCANS`] lookup searches, or every cell a [`CRITERIA`] aggregate tests. 0 for other
/// functions.
fn scan_len(spec: &FnSpec, args: &[Arg]) -> usize {
if !SCANS.contains(&spec.name) {
let scans = SCANS.contains(&spec.name);
if !scans && !CRITERIA.contains(&spec.name) {
return 0;
}
args.iter()
.enumerate()
.filter(|(i, _)| !spec.is_scalar_arg(*i))
.filter_map(|(_, a)| match &a.value {
Value::Array(arr) => Some(arr.rows.max(arr.cols)),
_ => None,
})
.max()
.unwrap_or(0)
// SUMIF and AVERAGEIF read the optional third range only at the coordinates of the first.
let first = args.first().and_then(|a| if let Value::Array(arr) = &a.value { Some(arr.data.len()) } else { None });
let lens = args.iter().enumerate().filter(|(i, _)| !spec.is_scalar_arg(*i)).filter_map(|(i, a)| match &a.value {
Value::Array(arr) if scans => Some(arr.rows.max(arr.cols)),
Value::Array(arr) if i == 2 && matches!(spec.name, "SUMIF" | "AVERAGEIF") => Some(arr.data.len().min(first.unwrap_or(usize::MAX))),
Value::Array(arr) => Some(arr.data.len()),
_ => None,
});
if scans { lens.max().unwrap_or(0) } else { lens.fold(0, usize::saturating_add) }
}
/// The work of one lifted call that depends on its own (lifted) scalar arguments: the number of
/// periods a [`PERIOD_LOOPS`] function steps through. 0 for other functions.
fn call_cost(spec: &FnSpec, cell_args: &[Arg]) -> usize {
if !PERIOD_LOOPS.contains(&spec.name) {
return 0;
}
// Coercing numeric text reads all of it, even in a call the function then refuses, so each
// character is charged as well: of the arguments it coerces in order, up to the first that fails.
let mut parse = 0usize;
for (i, a) in cell_args.iter().enumerate().take(6) {
if let Value::Text(t) = &a.value {
parse = parse.saturating_add(t.len());
}
if crate::util::num(cell_args, i).is_err() {
break;
}
}
// The function's own validation and coercion (text like "10000000" counts), so a call it
// refuses before its loop costs nothing and no loop goes uncharged.
let periods = crate::financial::cumulative_args(cell_args).map_or(0, |(.., start, end, _)| (end - start + 1.0) as usize);
parse.saturating_add(periods.saturating_mul(PERIOD_COST))
}
pub(crate) fn call_lifted(spec: &FnSpec, args: &[Arg], ctx: &mut dyn Ctx) -> Value {
call_lifted_within(spec, args, ctx, LIFTED_SCAN_BUDGET)
}
/// Sets the lifted arguments of `cell_args` to their elements at (`r`, `c`).
fn element_args(spec: &FnSpec, args: &[Arg], r: usize, c: usize, cell_args: &mut [Arg]) {
for (i, a) in args.iter().enumerate() {
if spec.is_scalar_arg(i)
&& let Value::Array(arr) = &a.value
&& let Some(slot) = cell_args.get_mut(i)
{
let v = arr.get_broadcast(r, c);
slot.value = if let Value::Array(inner) = &v { inner.data.first().cloned().unwrap_or(Value::Empty) } else { v };
// A blank element of a written array argument is given, as a scalar blank is.
slot.blank_value = !slot.from_ref && matches!(slot.value, Value::Empty);
}
}
}
/// [`call_lifted`] with every call's work added up against `budget` (see [`LIFTED_SCAN_BUDGET`]).
fn call_lifted_within(spec: &FnSpec, args: &[Arg], ctx: &mut dyn Ctx, budget: usize) -> Value {
let rejected = if spec.value_args.is_empty() { None } else { reject_multi_cell_refs(spec, args) };
let args = rejected.as_deref().unwrap_or(args);
let mut rows = 0usize;
@@ -101,27 +161,42 @@ pub(crate) fn call_lifted(spec: &FnSpec, args: &[Arg], ctx: &mut dyn Ctx) -> Val
return (spec.imp)(args, ctx);
}
let needles = rows.saturating_mul(cols);
if needles > MAX_CELLS || needles.saturating_mul(scan_len(spec, args)) > LIFTED_SCAN_BUDGET {
// Every call is charged its repeated scan (at least 1), its own period loop, and the cells of
// an array result (built in full, though only its first cell is kept), so lifting a scalar
// argument can't multiply what one call may do into hours of work:
// `COUNTIF(big_range, SEQUENCE(1e4))`, `CUMIPMT(0.1, 1e7, 1, 1, SEQUENCE(1e4,,1e7,0), 0)`,
// `MUNIT(SEQUENCE(1e4,,3000,0))`.
let per_call = scan_len(spec, args).max(1);
let mut cell_args: Vec<Arg> = args.to_vec();
let mut spent = needles.saturating_mul(per_call);
if needles > MAX_CELLS || spent > budget {
return Value::Error(CellError::Num);
}
let mut cell_args: Vec<Arg> = args.to_vec();
// Period loops are known before any call runs, so a call over budget does none of them.
if PERIOD_LOOPS.contains(&spec.name) {
for r in 0..rows {
for c in 0..cols {
element_args(spec, args, r, c, &mut cell_args);
spent = spent.saturating_add(call_cost(spec, &cell_args));
if spent > budget {
return Value::Error(CellError::Num);
}
}
}
}
let mut data = Vec::with_capacity(rows * cols);
for r in 0..rows {
for c in 0..cols {
for (i, a) in args.iter().enumerate() {
if spec.is_scalar_arg(i)
&& let Value::Array(arr) = &a.value
&& let Some(slot) = cell_args.get_mut(i)
{
let v = arr.get_broadcast(r, c);
slot.value = if let Value::Array(inner) = &v { inner.data.first().cloned().unwrap_or(Value::Empty) } else { v };
// A blank element of a written array argument is given, as a scalar blank is.
slot.blank_value = !slot.from_ref && matches!(slot.value, Value::Empty);
}
}
element_args(spec, args, r, c, &mut cell_args);
let out = (spec.imp)(&cell_args, ctx);
data.push(match out {
Value::Array(a) => a.data.first().cloned().unwrap_or(Value::Empty),
Value::Array(a) => {
spent = spent.saturating_add(a.data.len());
if spent > budget {
return Value::Error(CellError::Num);
}
a.data.first().cloned().unwrap_or(Value::Empty)
}
v => v,
});
}
@@ -206,6 +281,100 @@ mod tests {
assert_eq!(rows_of(&out), vec![vec![nv(-1.0)]; 1000]);
}
/// Security: lifting a criteria aggregate rescans its range for every criterion, so criteria
/// x range cells past the budget is `#NUM!` before any scan runs.
#[test]
fn lifted_criteria_aggregates_stop_past_their_work_budget() {
use gridcraft_core::Array;
let tall = |n: usize| av(Array::filled(n, 1, nv(-1.0)).into());
is_err(ev("COUNTIF", vec![tall(1_000_000), tall(501)]), CellError::Num);
is_err(ev("SUMIF", vec![tall(1_000_000), tall(501)]), CellError::Num);
is_err(ev("COUNTIFS", vec![tall(1_000_000), tall(501)]), CellError::Num);
// Two ranges of 250,000 cells each count together.
is_err(ev("SUMIFS", vec![tall(250_000), tall(250_000), tall(1001)]), CellError::Num);
// SUMIF reads its sum range only where the criteria range has cells: a one-cell range over a
// million-cell sum range is one cell of work per criterion.
let one = av(Array::filled(1, 1, nv(-1.0)).into());
assert_eq!(rows_of(&ev("SUMIF", vec![one, tall(501), tall(1_000_000)])).len(), 501);
let counts = ev("COUNTIF", vec![av(col(&[1.0, 2.0, 2.0])), av(col(&[2.0, 1.0, 5.0]))]);
assert_eq!(rows_of(&counts), vec![vec![nv(2.0)], vec![nv(1.0)], vec![nv(0.0)]]);
for name in super::CRITERIA.iter().chain(super::PERIOD_LOOPS) {
assert!(crate::lookup(name).is_some(), "{name} isn't registered");
}
}
/// Security: CUMIPMT and CUMPRINC loop over their periods once per lifted element; the
/// periods of all elements are added up first, so an over-budget call runs none of them.
#[test]
fn lifted_period_loops_stop_past_their_work_budget() {
use gridcraft_core::Array;
let ends = |len: usize, end: f64| av(Array::filled(len, 1, nv(end)).into());
let args = |len: usize, end: f64| vec![n(0.01), n(1e7), n(1000.0), n(1.0), ends(len, end), n(0.0)];
// 51 elements of 1e7 periods each is past 5e8.
is_err(ev("CUMIPMT", args(51, 1e7)), CellError::Num);
is_err(ev("CUMPRINC", args(51, 1e7)), CellError::Num);
// A period costs more than a scanned cell, so fewer elements are enough.
is_err(ev("CUMIPMT", args(49, 1e7)), CellError::Num);
// Numeric text is coerced by the function, so its periods are charged the same.
let text_ends = av(Array::filled(51, 1, Value::text("10000000")).into());
is_err(ev("CUMIPMT", vec![n(0.01), n(1e7), n(1000.0), n(1.0), text_ends, n(0.0)]), CellError::Num);
let out = ev("CUMIPMT", args(2, 12.0));
assert_eq!(rows_of(&out).len(), 2);
assert!(matches!(rows_of(&out).first().and_then(|r| r.first()), Some(Value::Number(x)) if *x < 0.0));
}
/// Security: numeric text is parsed once per lifted element even when the call is refused; the
/// characters read are charged, so a long text repeated down a column can't cost gigabytes of parsing.
#[test]
fn lifted_period_loops_charge_the_text_they_parse() {
use gridcraft_core::Array;
// 20,000 elements x 32,000 characters is past 5e8 though no call reaches its loop (rate is negative).
let long = Value::text(format!("{}5", "0".repeat(32_000)));
let ends = av(Array::filled(20_000, 1, long).into());
is_err(ev("CUMIPMT", vec![n(-0.01), n(10.0), n(1000.0), n(1.0), ends, n(0.0)]), CellError::Num);
// A short text period costs next to nothing.
let ends = av(Array::filled(3, 1, Value::text("12")).into());
assert_eq!(rows_of(&ev("CUMIPMT", vec![n(0.01), n(20.0), n(1000.0), n(1.0), ends, n(0.0)])).len(), 3);
}
/// Text after the first argument that fails to coerce is never read, so it is not charged: the
/// elements keep their own errors instead of one budget `#NUM!`.
#[test]
fn lifted_period_loops_charge_only_the_text_they_reach() {
use gridcraft_core::Array;
let rates = av(Array::filled(20_000, 1, Value::text("x")).into());
let long = Value::text(format!("{}5", "0".repeat(32_000)));
let out = ev("CUMIPMT", vec![rates, n(10.0), n(1000.0), n(1.0), av(long), n(0.0)]);
assert!(rows_of(&out).iter().all(|r| r.first() == Some(&Value::Error(CellError::Value))));
}
/// A call the function refuses before its loop (here: end period past the period count) is
/// not charged, so rejected elements can't push a valid one over the budget.
#[test]
fn lifted_period_loops_do_not_charge_rejected_calls() {
use gridcraft_core::Array;
let mut ends = vec![nv(1e7 + 1.0); 60];
ends.push(nv(12.0));
let ends = av(Array::new(ends.len(), 1, ends).unwrap_or_else(|| panic!("shape")).into());
let out = ev("CUMIPMT", vec![n(0.01), n(1e7), n(1000.0), n(1.0), ends, n(0.0)]);
let rows = rows_of(&out);
assert_eq!(rows.len(), 61);
assert!(matches!(rows.last().and_then(|r| r.first()), Some(Value::Number(x)) if *x < 0.0));
}
/// Security: an array-producing function lifted over a computed array builds each element's
/// whole result before keeping one cell; their cells count against the budget.
#[test]
fn lifted_array_results_count_against_the_budget() {
use gridcraft_core::Array;
let spec = crate::lookup("MUNIT").unwrap_or_else(|| panic!("MUNIT isn't registered"));
let dims = |n: usize| [av(Array::filled(n, 1, nv(10.0)).into())];
// Ten 10x10 identity matrices are 1,000 cells.
let out = super::call_lifted_within(spec, &dims(10), &mut TestCtx::default(), 1_010);
assert_eq!(rows_of(&out), vec![vec![nv(1.0)]; 10]);
is_err(super::call_lifted_within(spec, &dims(11), &mut TestCtx::default(), 1_010), CellError::Num);
}
#[test]
fn value_params_reject_multi_cell_references() {
let s = FnSpec { value_args: &[true, false], ..spec() };
+9
View File
@@ -53,6 +53,15 @@ impl Remote {
self.conn = None;
let mut last = std::io::Error::new(std::io::ErrorKind::NotFound, format!("cannot resolve {}", self.addr));
for sa in self.addr.to_socket_addrs()? {
// The token goes over plain TCP, so only to this machine: a network peer could read it,
// and the app's control port listens on loopback only.
if !sa.ip().is_loopback() {
last = std::io::Error::new(
std::io::ErrorKind::PermissionDenied,
format!("{sa} is not a loopback address: the control token is sent only to 127.0.0.1 or ::1"),
);
continue;
}
match TcpStream::connect_timeout(&sa, Duration::from_secs(2)) {
Ok(s) => {
s.set_nodelay(true).ok();
+3
View File
@@ -482,6 +482,9 @@ fn remote_backend_over_tcp() {
drop(s);
handle.join().unwrap();
assert!(Remote::connect_with_token("127.0.0.1:1", "x".into()).is_err());
// A network address is refused before anything, the token included, is sent.
let err = Remote::connect_with_token("192.0.2.1:7979", "x".into()).err().unwrap();
assert_eq!(err.kind(), std::io::ErrorKind::PermissionDenied, "{err}");
}
#[test]
+1 -1
View File
@@ -11,7 +11,7 @@ workspace = true
[dependencies]
# The shared capture slot (`craft_ui::capture`) comes from the one craft-libs revision every
# craft-libs dependency uses (see the root Cargo.toml).
craft-ui = { git = "https://github.com/storytold/craft-libs", rev = "1b34eba41e62138fc2438d2734523b302840b45d" }
craft-ui = { git = "https://github.com/storytold/craft-libs", rev = "119415f46f601e20b9bc6b1b4d6979ceda86d88e" }
gridcraft-engine = { workspace = true }
gridcraft-chart = { workspace = true }
# `monochrome_emoji_fonts` restores the Noto Emoji and emoji-icon faces that 0.37 moved behind
+16 -1
View File
@@ -1,5 +1,5 @@
use super::{
INTERFACE_SCALE_MAX, INTERFACE_SCALE_MIN, Json, NoticeKind, SheetApp, UiRequest, backstage, clamp_interface_scale, clean_error, dialogs,
INTERFACE_SCALE_MAX, INTERFACE_SCALE_MIN, Json, Notice, NoticeKind, SheetApp, UiRequest, backstage, clamp_interface_scale, clean_error, dialogs,
formula_locale, i18n, json, now_ms, printing,
};
@@ -96,6 +96,21 @@ impl SheetApp {
{
self.deliver_pdf(r);
}
// The scan stopped at its cell allowance: say that some cells were not checked.
if id == "data.circleInvalid"
&& let Ok(r) = &r
&& r.get("truncated").and_then(Json::as_bool) == Some(true)
{
let scan = "The scan stopped after 100,000 cell visits; some validated cells were not checked for invalid data.";
// An open-time warning that is still shown keeps its text and action; the scan note joins it.
match self.notice.as_mut() {
Some(n) => {
n.text = format!("{} {scan}", n.text);
n.kind = NoticeKind::Warning;
}
None => self.notice = Some(Notice { text: scan.into(), kind: NoticeKind::Warning, action: None }),
}
}
r
}
+27
View File
@@ -0,0 +1,27 @@
//! Release and source Flatpaks must allow release checks and host update helpers.
fn finish_args(manifest: &str) -> Vec<&str> {
manifest
.split("finish-args:")
.nth(1)
.unwrap_or_default()
.lines()
.take_while(|line| line.is_empty() || line.starts_with(' ') || line.starts_with('#'))
.filter_map(|line| line.trim().strip_prefix("- "))
.map(|arg| arg.trim_matches(['\'', '"']))
.collect()
}
#[test]
fn flatpak_update_permissions() {
let manifests = [
include_str!("../../../packaging/linux/flatpak/ai.storyteller.gridcraft.bundle.yml"),
include_str!("../../../packaging/linux/flatpak/ai.storyteller.gridcraft.yml"),
];
let args = manifests.map(finish_args);
assert_eq!(args[0], args[1], "source and release permissions must agree");
for permissions in args {
for required in ["--share=network", "--talk-name=org.freedesktop.Flatpak"] {
assert!(permissions.contains(&required), "missing update permission: {required}");
}
}
}
+22
View File
@@ -1396,6 +1396,28 @@ fn macro_workbook_bytes() -> Vec<u8> {
gridcraft_engine::io::save_bytes(&wb, "macros.xlsm").unwrap()
}
#[test]
fn circle_invalid_data_warns_when_its_scan_was_cut_short() {
let mut h = harness(blank());
h.state_mut().run("range.setValues", json!({"range": "A1", "values": [[-1]]})).unwrap();
h.state_mut().run("range.setValues", json!({"range": "A200000", "values": [[-1]]})).unwrap();
h.state_mut().run("data.validation", json!({"range": "A1:A200000", "type": "whole", "operator": "greater", "formula1": "0"})).unwrap();
h.state_mut().run("data.circleInvalid", json!({})).unwrap();
let n = h.state().notice.clone().expect("a notice was shown");
assert!(n.text.contains("100,000 cell visits"), "{}", n.text);
// An open-time warning keeps its action when the scan note joins it.
h.state_mut().notice = Some(gridcraft_ui_egui::Notice {
text: "Macros.".into(),
kind: gridcraft_ui_egui::NoticeKind::Info,
action: Some(("More".into(), "macros".into())),
});
h.state_mut().run("data.circleInvalid", json!({})).unwrap();
let n = h.state().notice.clone().unwrap();
assert!(n.text.starts_with("Macros. ") && n.text.contains("cell visits"), "{}", n.text);
assert_eq!(n.action.as_ref().map(|a| a.1.as_str()), Some("macros"));
assert_eq!(n.kind, gridcraft_ui_egui::NoticeKind::Warning);
}
#[test]
fn opening_a_macro_workbook_shows_a_notice() {
// The reader's "macros kept but not executed" warning must reach a dismissible info bar,
+4 -2
View File
@@ -18,11 +18,13 @@ gridcraft-core = { workspace = true }
gridcraft-formula = { workspace = true }
gridcraft-math = { workspace = true }
gridcraft-model = { workspace = true }
craft-cfb = { workspace = true }
craft-csv = { workspace = true }
craft-offcrypto = { workspace = true }
craft-xls = { workspace = true }
thiserror = { workspace = true }
quick-xml = { workspace = true }
zip = { workspace = true }
aes = { workspace = true }
sha1 = { workspace = true }
sha2 = { workspace = true }
[dev-dependencies]
-455
View File
@@ -1,455 +0,0 @@
//! Just enough of the Compound File Binary (CFB/OLE2) container to tell what a non-zip
//! spreadsheet file is: Excel wraps password-protected workbooks in a CFB holding an
//! `EncryptedPackage` stream, while Excel 97-2003 `.xls` (and other legacy Office files) are
//! CFB files without one. Every read is bounds-checked and every loop bounded.
/// The CFB signature at the start of the file.
pub(crate) const SIGNATURE: [u8; 8] = [0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1];
const END_OF_CHAIN: u32 = 0xFFFF_FFFE;
/// Sector ids at or above this are markers (free, end of chain, FAT/DIFAT sector), not sectors.
const MAX_REG_SECT: u32 = 0xFFFF_FFFA;
/// Directory sectors followed at most (4096 entries with 512-byte sectors).
const MAX_DIR_SECTORS: usize = 1024;
/// DIFAT sectors followed at most.
const MAX_DIFAT_SECTORS: usize = 1024;
/// Stream names that mark an encrypted (password-protected) OOXML package.
const ENCRYPTED_STREAMS: [&str; 2] = ["EncryptedPackage", "EncryptionInfo"];
fn u16_at(b: &[u8], at: usize) -> Option<u16> {
let s = b.get(at..at.checked_add(2)?)?;
Some(u16::from_le_bytes([*s.first()?, *s.get(1)?]))
}
fn u32_at(b: &[u8], at: usize) -> Option<u32> {
let s = b.get(at..at.checked_add(4)?)?;
Some(u32::from_le_bytes([*s.first()?, *s.get(1)?, *s.get(2)?, *s.get(3)?]))
}
/// Whether the compound file holds an encryption stream. Walks the directory through the FAT;
/// when the structure can't be followed (truncated or malformed), falls back to scanning the
/// bytes for the UTF-16LE stream names.
pub(crate) fn is_encrypted(bytes: &[u8]) -> bool {
match directory_names(bytes) {
Some(names) => names.iter().any(|n| ENCRYPTED_STREAMS.iter().any(|e| n.eq_ignore_ascii_case(e))),
None => ENCRYPTED_STREAMS.iter().any(|e| contains_utf16(bytes, e)),
}
}
fn contains_utf16(hay: &[u8], needle: &str) -> bool {
let n: Vec<u8> = needle.encode_utf16().flat_map(u16::to_le_bytes).collect();
!n.is_empty() && hay.windows(n.len()).any(|w| w == n.as_slice())
}
/// Names of every directory entry, or `None` when the directory can't be read.
fn directory_names(b: &[u8]) -> Option<Vec<String>> {
if !b.starts_with(&SIGNATURE) {
return None;
}
let shift = u16_at(b, 30)?;
if shift != 9 && shift != 12 {
return None;
}
let size = 1usize << shift;
let per = size / 4;
let sector = |id: u32| -> Option<&[u8]> {
let start = (id as usize).checked_add(1)?.checked_mul(size)?;
b.get(start..start.checked_add(size)?)
};
// The DIFAT: the FAT's sector ids, 109 in the header then chained DIFAT sectors.
let mut difat: Vec<u32> = (0..109).filter_map(|i| u32_at(b, 76 + i * 4)).filter(|&s| s < MAX_REG_SECT).collect();
let mut next = u32_at(b, 68)?;
for _ in 0..MAX_DIFAT_SECTORS {
if next >= MAX_REG_SECT {
break;
}
let s = sector(next)?;
difat.extend((0..per - 1).filter_map(|i| u32_at(s, i * 4)).filter(|&x| x < MAX_REG_SECT));
next = u32_at(s, (per - 1) * 4)?;
}
let fat_next = |id: u32| -> Option<u32> {
let idx = id as usize;
let fat_sector = *difat.get(idx / per)?;
u32_at(sector(fat_sector)?, (idx % per) * 4)
};
let mut names = Vec::new();
let mut id = u32_at(b, 48)?;
for _ in 0..MAX_DIR_SECTORS {
if id == END_OF_CHAIN {
return Some(names);
}
if id >= MAX_REG_SECT {
return None;
}
let s = sector(id)?;
for e in s.as_chunks::<128>().0 {
let len = usize::from(u16_at(e, 64)?);
if !(2..=64).contains(&len) {
continue;
}
let units: Vec<u16> = (0..(len - 2) / 2).filter_map(|i| u16_at(e, i * 2)).collect();
names.push(String::from_utf16_lossy(&units));
}
id = fat_next(id)?;
}
// A directory longer than the bound (or a cyclic chain): report what was read.
Some(names)
}
/// Directory entry type of a stream object ([MS-CFB] 2.6.1).
const STREAM_OBJECT: u8 = 2;
/// "No entry" in the directory tree (NOSTREAM).
const NO_STREAM: u32 = 0xFFFF_FFFF;
/// Streams larger than this are refused (the format allows more, a workbook never needs it).
const MAX_STREAM: u64 = 1 << 30;
/// One directory entry ([MS-CFB] 2.6).
struct DirEntry {
name: String,
kind: u8,
left: u32,
right: u32,
child: u32,
start: u32,
size: u64,
}
/// A parsed compound file: the FAT, the mini FAT and the directory, all bounds-checked.
/// Implemented from Microsoft's public [MS-CFB] specification.
pub(crate) struct Compound<'a> {
bytes: &'a [u8],
sector_size: usize,
mini_sector_size: usize,
mini_cutoff: u64,
fat: Vec<u32>,
mini_fat: Vec<u32>,
entries: Vec<DirEntry>,
}
impl<'a> Compound<'a> {
/// Parses the header, FAT, mini FAT and directory. Errors on anything that can't be followed.
pub(crate) fn open(b: &'a [u8]) -> Result<Compound<'a>, String> {
let bad = |m: &str| format!("damaged compound file: {m}");
if !b.starts_with(&SIGNATURE) {
return Err(bad("missing signature"));
}
let major = u16_at(b, 26).ok_or_else(|| bad("truncated header"))?;
let shift = u16_at(b, 30).ok_or_else(|| bad("truncated header"))?;
if shift != 9 && shift != 12 {
return Err(bad("unsupported sector size"));
}
let mini_shift = u16_at(b, 32).ok_or_else(|| bad("truncated header"))?;
if mini_shift != 6 {
return Err(bad("unsupported mini sector size"));
}
let sector_size = 1usize << shift;
let per = sector_size / 4;
// Sectors that physically exist after the header sector: every chain is bounded by it.
let sectors = (b.len() / sector_size).saturating_sub(1);
let sector = |id: u32| -> Option<&[u8]> {
let start = (id as usize).checked_add(1)?.checked_mul(sector_size)?;
b.get(start..start.checked_add(sector_size)?)
};
// DIFAT: 109 FAT sector ids in the header, then chained DIFAT sectors.
let mut fat_sectors: Vec<u32> = (0..109).filter_map(|i| u32_at(b, 76 + i * 4)).filter(|&s| s < MAX_REG_SECT).collect();
let mut next = u32_at(b, 68).ok_or_else(|| bad("truncated header"))?;
let mut hops = 0usize;
while next < MAX_REG_SECT {
hops += 1;
if hops > sectors.min(MAX_DIFAT_SECTORS * 64) {
return Err(bad("DIFAT chain loops"));
}
let s = sector(next).ok_or_else(|| bad("DIFAT sector out of range"))?;
fat_sectors.extend((0..per - 1).filter_map(|i| u32_at(s, i * 4)).filter(|&x| x < MAX_REG_SECT));
next = u32_at(s, (per - 1) * 4).ok_or_else(|| bad("truncated DIFAT sector"))?;
}
if fat_sectors.len() > sectors {
return Err(bad("more FAT sectors than the file holds"));
}
let mut fat = Vec::with_capacity(fat_sectors.len().saturating_mul(per));
for id in fat_sectors {
let s = sector(id).ok_or_else(|| bad("FAT sector out of range"))?;
fat.extend((0..per).filter_map(|i| u32_at(s, i * 4)));
}
let mut cf = Compound { bytes: b, sector_size, mini_sector_size: 64, mini_cutoff: 4096, fat, mini_fat: vec![], entries: vec![] };
if let Some(c) = u32_at(b, 56) {
cf.mini_cutoff = u64::from(c);
}
// Directory.
let dir_start = u32_at(b, 48).ok_or_else(|| bad("truncated header"))?;
let dir = cf.read_chain(dir_start, u64::MAX)?;
for e in dir.as_chunks::<128>().0 {
let len = usize::from(u16_at(e, 64).unwrap_or(0)).min(64);
let units: Vec<u16> = (0..len.saturating_sub(2) / 2).filter_map(|i| u16_at(e, i * 2)).collect();
let mut size = u32_at(e, 120).map(u64::from).unwrap_or(0);
if major >= 4 {
size |= u64::from(u32_at(e, 124).unwrap_or(0)) << 32;
}
cf.entries.push(DirEntry {
name: String::from_utf16_lossy(&units),
kind: e.get(66).copied().unwrap_or(0),
left: u32_at(e, 68).unwrap_or(NO_STREAM),
right: u32_at(e, 72).unwrap_or(NO_STREAM),
child: u32_at(e, 76).unwrap_or(NO_STREAM),
start: u32_at(e, 116).unwrap_or(END_OF_CHAIN),
size,
});
}
if cf.entries.first().is_none_or(|r| r.kind != 5) {
return Err(bad("missing root entry"));
}
// Mini FAT.
let mini_start = u32_at(b, 60).ok_or_else(|| bad("truncated header"))?;
if mini_start < MAX_REG_SECT {
let raw = cf.read_chain(mini_start, u64::MAX)?;
cf.mini_fat = (0..raw.len() / 4).filter_map(|i| u32_at(&raw, i * 4)).collect();
}
Ok(cf)
}
/// Follows a FAT chain from `start`, reading at most `limit` bytes. A chain longer than the
/// FAT (so a cycle) or pointing outside the file is an error. The file may end inside the
/// chain's last sector (some writers don't pad it, and Excel opens those files): that sector
/// reads short, and the caller checks the stream's declared size against what was read.
fn read_chain(&self, start: u32, limit: u64) -> Result<Vec<u8>, String> {
let limit = usize::try_from(limit.min(MAX_STREAM)).unwrap_or(usize::MAX);
let mut out = Vec::new();
let mut id = start;
let mut seen = vec![false; self.fat.len()];
let mut cut = false;
while id != END_OF_CHAIN && out.len() < limit {
if cut {
return Err("damaged compound file: sector out of range (the file may be truncated)".into());
}
if id >= MAX_REG_SECT {
return Err("damaged compound file: broken sector chain".into());
}
match seen.get_mut(id as usize) {
Some(v) if !*v => *v = true,
Some(_) => return Err("damaged compound file: sector chain loops".into()),
None => return Err("damaged compound file: sector missing from the FAT".into()),
}
let start = (id as usize).checked_add(1).and_then(|i| i.checked_mul(self.sector_size));
let s = start
.and_then(|s| self.bytes.get(s..s.checked_add(self.sector_size)?.min(self.bytes.len())))
.ok_or("damaged compound file: sector out of range (the file may be truncated)")?;
cut = s.len() < self.sector_size;
out.extend_from_slice(s);
id = *self.fat.get(id as usize).ok_or("damaged compound file: sector missing from the FAT")?;
}
Ok(out)
}
/// Follows a mini FAT chain inside the mini stream container. Like `read_chain`, the
/// container may end inside the chain's last mini sector.
fn read_mini_chain(&self, container: &[u8], start: u32, limit: usize) -> Result<Vec<u8>, String> {
let mut out = Vec::new();
let mut id = start;
let mut seen = vec![false; self.mini_fat.len()];
let mut cut = false;
while id != END_OF_CHAIN && out.len() < limit {
if cut {
return Err("damaged compound file: mini sector out of range".into());
}
if id >= MAX_REG_SECT {
return Err("damaged compound file: broken mini sector chain".into());
}
match seen.get_mut(id as usize) {
Some(v) if !*v => *v = true,
Some(_) => return Err("damaged compound file: mini sector chain loops".into()),
None => return Err("damaged compound file: mini sector missing from the mini FAT".into()),
}
let at = (id as usize).checked_mul(self.mini_sector_size);
let s = at
.and_then(|a| container.get(a..a.checked_add(self.mini_sector_size)?.min(container.len())))
.ok_or("damaged compound file: mini sector out of range")?;
cut = s.len() < self.mini_sector_size;
out.extend_from_slice(s);
id = *self.mini_fat.get(id as usize).ok_or("damaged compound file: mini sector missing from the mini FAT")?;
}
Ok(out)
}
/// The contents of the stream called `name` (case-insensitive) directly under the root
/// storage, or `None` when there is no such stream.
/// The directory entry of the stream called `name` (case-insensitive) directly under the
/// root storage.
fn root_entry(&self, name: &str) -> Result<Option<&DirEntry>, String> {
let Some(root) = self.entries.first() else { return Ok(None) };
// Walk the root's children (a red-black tree through left/right siblings), bounded.
let mut stack = vec![root.child];
let mut seen = 0usize;
while let Some(i) = stack.pop() {
if i == NO_STREAM {
continue;
}
seen += 1;
if seen > self.entries.len() {
return Err("damaged compound file: directory tree loops".into());
}
let Some(e) = self.entries.get(i as usize) else { continue };
if e.kind == STREAM_OBJECT && e.name.eq_ignore_ascii_case(name) {
return Ok(Some(e));
}
stack.push(e.left);
stack.push(e.right);
}
Ok(None)
}
/// Whether the root storage holds a stream called `name`, without reading it.
pub(crate) fn has_root_stream(&self, name: &str) -> bool {
matches!(self.root_entry(name), Ok(Some(_)))
}
/// The contents of the stream called `name` (case-insensitive) directly under the root
/// storage, or `None` when there is no such stream.
pub(crate) fn root_stream(&self, name: &str) -> Result<Option<Vec<u8>>, String> {
let Some(root) = self.entries.first() else { return Ok(None) };
let found = self.root_entry(name)?;
let Some(e) = found else { return Ok(None) };
if e.size > MAX_STREAM {
return Err("the workbook stream is larger than 1 GiB".into());
}
let size = usize::try_from(e.size).map_err(|_| "stream too large".to_string())?;
let mut data = if e.size < self.mini_cutoff {
let container = self.read_chain(root.start, root.size)?;
self.read_mini_chain(&container, e.start, size)?
} else {
self.read_chain(e.start, e.size)?
};
if data.len() < size {
return Err("damaged compound file: a stream is shorter than its declared size (the file may be truncated)".into());
}
data.truncate(size);
Ok(Some(data))
}
}
#[cfg(test)]
pub(crate) mod tests {
use super::*;
/// A minimal valid compound file (512-byte sectors: FAT at sector 0, directory at sector 1)
/// whose directory holds a root entry plus the given stream names.
pub(crate) fn build(streams: &[&str]) -> Vec<u8> {
let mut b = vec![0u8; 512 * 3];
b[..8].copy_from_slice(&SIGNATURE);
b[24..26].copy_from_slice(&0x3Eu16.to_le_bytes()); // minor version
b[26..28].copy_from_slice(&3u16.to_le_bytes()); // major version
b[28..30].copy_from_slice(&0xFFFEu16.to_le_bytes()); // byte order
b[30..32].copy_from_slice(&9u16.to_le_bytes()); // sector shift
b[32..34].copy_from_slice(&6u16.to_le_bytes()); // mini sector shift
b[44..48].copy_from_slice(&1u32.to_le_bytes()); // FAT sectors
b[48..52].copy_from_slice(&1u32.to_le_bytes()); // first directory sector
b[60..64].copy_from_slice(&END_OF_CHAIN.to_le_bytes()); // no mini FAT
b[68..72].copy_from_slice(&END_OF_CHAIN.to_le_bytes()); // no DIFAT sectors
for i in 0..109 {
b[76 + i * 4..80 + i * 4].copy_from_slice(&0xFFFF_FFFFu32.to_le_bytes());
}
b[76..80].copy_from_slice(&0u32.to_le_bytes()); // FAT lives in sector 0
// FAT (sector 0, at byte 512): sector 0 is a FAT sector, sector 1 ends its chain.
for i in 0..128 {
b[512 + i * 4..516 + i * 4].copy_from_slice(&0xFFFF_FFFFu32.to_le_bytes());
}
b[512..516].copy_from_slice(&0xFFFF_FFFDu32.to_le_bytes());
b[516..520].copy_from_slice(&END_OF_CHAIN.to_le_bytes());
// Directory (sector 1, at byte 1024).
for (k, name) in std::iter::once("Root Entry").chain(streams.iter().copied()).take(4).enumerate() {
let e = 1024 + k * 128;
let units: Vec<u16> = name.encode_utf16().take(31).collect();
for (i, u) in units.iter().enumerate() {
b[e + i * 2..e + i * 2 + 2].copy_from_slice(&u.to_le_bytes());
}
b[e + 64..e + 66].copy_from_slice(&((units.len() as u16 + 1) * 2).to_le_bytes());
b[e + 66] = if k == 0 { 5 } else { 2 };
}
b
}
#[test]
fn reads_directory_names() {
let b = build(&["EncryptionInfo", "EncryptedPackage"]);
assert_eq!(directory_names(&b), Some(vec!["Root Entry".into(), "EncryptionInfo".into(), "EncryptedPackage".into()]));
assert!(is_encrypted(&b));
assert!(!is_encrypted(&build(&["Workbook", "\u{5}SummaryInformation"])));
}
#[test]
fn hostile_headers_never_panic() {
let mut b = build(&["Workbook"]);
for at in [30usize, 48, 68, 76, 512, 516, 1024 + 64] {
for v in [0u32, 1, 7, 0xFFFF_FFFA, 0xFFFF_FFFF, 0x7FFF_FFFF] {
let mut c = b.clone();
c[at..at + 4].copy_from_slice(&v.to_le_bytes());
let _ = is_encrypted(&c);
for cut in [0usize, 8, 100, 600, 1100] {
let _ = is_encrypted(&c[..cut.min(c.len())]);
}
}
}
// A directory chain that loops back on itself stops at the bound.
b[516..520].copy_from_slice(&1u32.to_le_bytes());
assert!(!is_encrypted(&b));
}
/// `compound_file` with one regular-sector stream of `len` bytes (FAT in sector 0, directory
/// in sector 1, the stream from sector 2), cut right after the stream's last byte, the way
/// some writers leave the final sector unpadded.
fn unpadded(len: usize) -> (Vec<u8>, Vec<u8>) {
let data: Vec<u8> = (0..len).map(|i| (i % 251) as u8).collect();
let mut file = crate::testing::compound_file(&[("Workbook", &data)]);
file.truncate(512 * 3 + len);
(file, data)
}
#[test]
fn an_unpadded_last_sector_still_reads() {
let (file, data) = unpadded(5000);
assert_ne!(file.len() % 512, 0);
let cf = Compound::open(&file).unwrap();
assert_eq!(cf.root_stream("Workbook").unwrap().unwrap(), data);
// Bytes the stream needs that the file doesn't hold are still damage.
let cf = Compound::open(&file[..file.len() - 1]).unwrap();
assert!(cf.root_stream("Workbook").is_err());
}
#[test]
fn an_unpadded_last_mini_sector_still_reads() {
let data: Vec<u8> = (0..1000u32).map(|i| (i % 251) as u8).collect();
let mut file = crate::testing::compound_file(&[("Workbook", &data)]);
// Header, FAT, directory, mini FAT, then the mini stream holding the stream's bytes.
file.truncate(512 * 4 + data.len());
let cf = Compound::open(&file).unwrap();
assert_eq!(cf.root_stream("Workbook").unwrap().unwrap(), data);
let cf = Compound::open(&file[..file.len() - 1]).unwrap();
assert!(cf.root_stream("Workbook").is_err());
}
#[test]
fn a_cut_sector_inside_a_chain_is_damage() {
let (mut file, _) = unpadded(5000);
// The cut sector (11) points on to the directory sector and the stream claims those
// bytes too: the partial sector would splice 392 bytes into the middle of the stream.
file[512 + 11 * 4..512 + 12 * 4].copy_from_slice(&1u32.to_le_bytes());
file[1024 + 128 + 120..1024 + 128 + 124].copy_from_slice(&5512u32.to_le_bytes());
let cf = Compound::open(&file).unwrap();
assert!(cf.root_stream("Workbook").is_err());
}
#[test]
fn a_cut_mini_sector_inside_a_chain_is_damage() {
let small = [7u8; 100];
let data: Vec<u8> = (0..1000u32).map(|i| (i % 251) as u8).collect();
let mut file = crate::testing::compound_file(&[("Small", &small), ("Workbook", &data)]);
// "Small" holds mini sectors 0-1, "Workbook" 2-17; the mini stream starts at sector 3.
file.truncate(512 * 4 + 128 + data.len());
assert_eq!(Compound::open(&file).unwrap().root_stream("Workbook").unwrap().unwrap(), data);
// The cut mini sector (17) points on to sector 0 and the stream claims those bytes too.
file[1536 + 17 * 4..1536 + 18 * 4].copy_from_slice(&0u32.to_le_bytes());
file[1024 + 2 * 128 + 120..1024 + 2 * 128 + 124].copy_from_slice(&1064u32.to_le_bytes());
let cf = Compound::open(&file).unwrap();
assert!(cf.root_stream("Workbook").is_err());
}
}
+19 -511
View File
@@ -1,42 +1,7 @@
//! Password-encrypted workbooks. Excel wraps the ordinary OOXML package in a compound file with
//! two streams: `EncryptionInfo` (how the key comes from the password, and a verifier that tells
//! a wrong password) and `EncryptedPackage` (the package, encrypted with AES). Agile encryption
//! (Excel 2010 onward) and Standard encryption (Excel 2007) are read; RC4, Extensible and
//! certificate-only encryption are reported as unsupported.
//!
//! Clean-room from Microsoft's public [MS-OFFCRYPTO] specification. Every read is
//! bounds-checked, and the costly inputs (spin count, stated sizes) are capped.
use aes::cipher::{BlockCipherDecrypt, KeyInit};
use sha2::Digest;
use crate::cfb::Compound;
use crate::xml::El;
/// Spin counts above this are refused (the cap sheet-protection hashes use too).
const MAX_SPIN_COUNT: u32 = 10_000_000;
/// Standard encryption always iterates this many times.
const STANDARD_SPIN_COUNT: u32 = 50_000;
/// Agile packages are encrypted in segments of this many bytes, each with its own IV.
pub(crate) const SEGMENT: usize = 4096;
/// AES block size.
const BLOCK: usize = 16;
/// Agile block keys ([MS-OFFCRYPTO] 2.3.4.13): one per value derived from the password.
pub(crate) const BLOCK_VERIFIER_INPUT: [u8; 8] = [0xFE, 0xA7, 0xD2, 0x76, 0x3B, 0x4B, 0x9E, 0x79];
pub(crate) const BLOCK_VERIFIER_HASH: [u8; 8] = [0xD7, 0xAA, 0x0F, 0x6D, 0x30, 0x61, 0x34, 0x4E];
pub(crate) const BLOCK_KEY: [u8; 8] = [0x14, 0x6E, 0x0B, 0xE7, 0xAB, 0xAC, 0xD0, 0xD6];
/// The two dataIntegrity block keys ([MS-OFFCRYPTO] 2.3.4.14), one per HMAC field.
pub(crate) const BLOCK_HMAC_KEY: [u8; 8] = [0x5F, 0xB2, 0xAD, 0x01, 0x0C, 0xB9, 0xE1, 0xF6];
pub(crate) const BLOCK_HMAC_VALUE: [u8; 8] = [0xA0, 0x67, 0x7F, 0x02, 0xB2, 0x2C, 0x84, 0x33];
/// The password key encryptor's URI (the other kind is certificate-based).
pub(crate) const PASSWORD_ENCRYPTOR: &str = "http://schemas.microsoft.com/office/2006/keyEncryptor/password";
/// Standard encryption header flags ([MS-OFFCRYPTO] 2.3.1).
pub(crate) const F_CRYPTOAPI: u32 = 0x04;
const F_EXTERNAL: u32 = 0x10;
pub(crate) const F_AES: u32 = 0x20;
//! two streams, `EncryptionInfo` and `EncryptedPackage`. `craft-offcrypto` reads them (Agile and
//! Standard encryption; RC4, Extensible, certificate-only and rights-managed files are named as
//! unsupported); this module keeps GridCraft's error type.
/// Why an encrypted workbook couldn't be decrypted.
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
@@ -50,489 +15,32 @@ pub enum DecryptError {
Malformed(String),
}
fn malformed(m: impl Into<String>) -> DecryptError {
DecryptError::Malformed(m.into())
impl From<craft_offcrypto::Error> for DecryptError {
fn from(e: craft_offcrypto::Error) -> Self {
use craft_offcrypto::Error as E;
match e {
E::WrongPassword | E::PasswordRequired => DecryptError::WrongPassword,
E::Unsupported(scheme) => DecryptError::Unsupported(scheme),
E::Malformed(detail) | E::TooLarge(detail) => DecryptError::Malformed(detail),
e => DecryptError::Malformed(e.to_string()),
}
}
}
/// The plain OOXML package (a zip) inside an encrypted workbook.
pub fn decrypt(bytes: &[u8], password: &str) -> Result<Vec<u8>, DecryptError> {
let (info, package) = streams(bytes)?;
decrypt_streams(&info, &package, password)
craft_offcrypto::decrypt(bytes, Some(password)).map_err(DecryptError::from)
}
/// Whether the right password could decrypt this encrypted workbook, checked without one. An
/// error names a scheme no password opens (RC4, rights management…) or damage, so callers can
/// say so instead of asking for a password first.
pub fn check_encryption(bytes: &[u8]) -> Result<(), DecryptError> {
let (info, _) = streams(bytes)?;
match scheme(&info)? {
Scheme::Agile(xml) => agile_descriptor(xml).map(|_| ()),
Scheme::Standard(info) => standard_header(info).map(|_| ()),
}
craft_offcrypto::check(bytes).map_err(DecryptError::from)
}
/// The `EncryptionInfo` and `EncryptedPackage` streams. Rights management (IRM) keeps the
/// package but describes its encryption elsewhere, with no `EncryptionInfo`.
fn streams(bytes: &[u8]) -> Result<(Vec<u8>, Vec<u8>), DecryptError> {
let cf = Compound::open(bytes).map_err(malformed)?;
let info = cf.root_stream("EncryptionInfo").map_err(malformed)?;
let package = cf.root_stream("EncryptedPackage").map_err(malformed)?.ok_or_else(|| malformed("no EncryptedPackage stream"))?;
let info = info.ok_or_else(|| DecryptError::Unsupported("rights-management".into()))?;
Ok((info, package))
/// Whether a compound file holds an encryption stream, at any depth. A damaged directory falls
/// back to a search for the stream names, so the file is still reported as encrypted.
pub(crate) fn is_encrypted(bytes: &[u8]) -> bool {
["EncryptedPackage", "EncryptionInfo"].iter().any(|n| craft_cfb::has_entry_named(bytes, n, &craft_cfb::Limits::default()))
}
/// Decrypts from the two streams' contents.
pub(crate) fn decrypt_streams(info: &[u8], package: &[u8], password: &str) -> Result<Vec<u8>, DecryptError> {
match scheme(info)? {
Scheme::Agile(xml) => agile(xml, package, password),
Scheme::Standard(info) => standard(info, package, password),
}
}
/// The two schemes read, with the part of `EncryptionInfo` each one parses.
enum Scheme<'a> {
/// The XML descriptor after the version and reserved field.
Agile(&'a [u8]),
/// Everything after the version.
Standard(&'a [u8]),
}
fn scheme(info: &[u8]) -> Result<Scheme<'_>, DecryptError> {
let short = || malformed("truncated EncryptionInfo");
let major = u16_at(info, 0).ok_or_else(short)?;
let minor = u16_at(info, 2).ok_or_else(short)?;
match (major, minor) {
(4, 4) => Ok(Scheme::Agile(info.get(8..).ok_or_else(short)?)),
(2..=4, 2) => Ok(Scheme::Standard(info.get(4..).ok_or_else(short)?)),
(3 | 4, 3) => Err(DecryptError::Unsupported("Extensible".into())),
_ => Err(DecryptError::Unsupported(format!("version {major}.{minor}"))),
}
}
fn u16_at(b: &[u8], at: usize) -> Option<u16> {
let s = b.get(at..at.checked_add(2)?)?;
Some(u16::from_le_bytes([*s.first()?, *s.get(1)?]))
}
fn u32_at(b: &[u8], at: usize) -> Option<u32> {
let s = b.get(at..at.checked_add(4)?)?;
Some(u32::from_le_bytes([*s.first()?, *s.get(1)?, *s.get(2)?, *s.get(3)?]))
}
/// The plaintext size an `EncryptedPackage` states, and the ciphertext after it. A size larger
/// than the ciphertext is damage, so it never drives an allocation.
fn package_parts(package: &[u8]) -> Result<(usize, &[u8]), DecryptError> {
let head: [u8; 8] = package.get(..8).and_then(|h| h.try_into().ok()).ok_or_else(|| malformed("truncated EncryptedPackage"))?;
let data = package.get(8..).unwrap_or_default();
let size = usize::try_from(u64::from_le_bytes(head)).unwrap_or(usize::MAX);
if size > data.len() {
return Err(malformed("the package is shorter than its stated size"));
}
Ok((size, data))
}
// ---- hashing -------------------------------------------------------------------------------
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum Hash {
Sha1,
Sha256,
Sha384,
Sha512,
}
impl Hash {
fn from_name(name: &str) -> Result<Hash, DecryptError> {
Ok(match name {
"SHA1" | "SHA-1" => Hash::Sha1,
"SHA256" | "SHA-256" => Hash::Sha256,
"SHA384" | "SHA-384" => Hash::Sha384,
"SHA512" | "SHA-512" => Hash::Sha512,
other => return Err(DecryptError::Unsupported(format!("{other} hash"))),
})
}
/// The digest of the concatenated `parts`.
pub(crate) fn digest(self, parts: &[&[u8]]) -> Vec<u8> {
fn run<D: Digest>(parts: &[&[u8]]) -> Vec<u8> {
let mut d = D::new();
for p in parts {
d.update(p);
}
d.finalize().to_vec()
}
match self {
Hash::Sha1 => run::<sha1::Sha1>(parts),
Hash::Sha256 => run::<sha2::Sha256>(parts),
Hash::Sha384 => run::<sha2::Sha384>(parts),
Hash::Sha512 => run::<sha2::Sha512>(parts),
}
}
/// The digest's length in bytes.
fn digest_len(self) -> usize {
match self {
Hash::Sha1 => 20,
Hash::Sha256 => 32,
Hash::Sha384 => 48,
Hash::Sha512 => 64,
}
}
/// The bytes the algorithm's compression function takes: 128 for SHA-384/512, else 64.
fn block_size(self) -> usize {
match self {
Hash::Sha384 | Hash::Sha512 => 128,
Hash::Sha1 | Hash::Sha256 => 64,
}
}
}
/// H0 = H(salt ‖ UTF-16LE(password)), then Hn = H(LE32(n - 1) ‖ Hn-1) for `spin` rounds.
pub(crate) fn iterated_hash(hash: Hash, password: &str, salt: &[u8], spin: u32) -> Vec<u8> {
let pw: Vec<u8> = password.encode_utf16().flat_map(u16::to_le_bytes).collect();
let mut h = hash.digest(&[salt, &pw]);
for i in 0..spin {
h = hash.digest(&[&i.to_le_bytes(), &h]);
}
h
}
/// H(a ‖ b), truncated or padded with 0x36 to `len`: Agile's derived keys and IVs.
pub(crate) fn derive(hash: Hash, a: &[u8], b: &[u8], len: usize) -> Vec<u8> {
let mut k = hash.digest(&[a, b]);
k.resize(len, 0x36);
k
}
/// HMAC-SHA-`hash` ([RFC2104]), as Agile's dataIntegrity uses.
pub(crate) fn hmac(hash: Hash, key: &[u8], message: &[u8]) -> Vec<u8> {
let bs = hash.block_size();
let mut k = key.to_vec();
if k.len() > bs {
k = hash.digest(&[&k]);
}
k.resize(bs, 0);
let pad = |x: u8| k.iter().map(|b| b ^ x).collect::<Vec<u8>>();
let inner = hash.digest(&[&pad(0x36), message]);
hash.digest(&[&pad(0x5C), &inner])
}
/// The Standard encryption key ([MS-OFFCRYPTO] 2.3.4.7): SHA-1 iterated 50,000 times, the
/// block-0 hash, then the 0x36 / 0x5C expansion, truncated to `key_len`.
pub(crate) fn standard_key(password: &str, salt: &[u8], key_len: usize) -> Vec<u8> {
let h = iterated_hash(Hash::Sha1, password, salt, STANDARD_SPIN_COUNT);
let h = Hash::Sha1.digest(&[&h, &0u32.to_le_bytes()]);
let expand = |pad: u8| {
let mut buf = [pad; 64];
for (b, x) in buf.iter_mut().zip(&h) {
*b ^= x;
}
Hash::Sha1.digest(&[&buf])
};
let mut key = expand(0x36);
key.extend(expand(0x5C));
key.truncate(key_len);
key
}
// ---- AES -----------------------------------------------------------------------------------
enum Aes {
A128(aes::Aes128),
A192(aes::Aes192),
A256(aes::Aes256),
}
impl Aes {
fn new(key: &[u8]) -> Result<Aes, DecryptError> {
let bad = |_| malformed("bad AES key");
Ok(match key.len() {
16 => Aes::A128(aes::Aes128::new_from_slice(key).map_err(bad)?),
24 => Aes::A192(aes::Aes192::new_from_slice(key).map_err(bad)?),
32 => Aes::A256(aes::Aes256::new_from_slice(key).map_err(bad)?),
n => return Err(malformed(format!("a {n}-byte AES key"))),
})
}
fn decrypt_block(&self, block: &mut [u8; BLOCK]) {
let b = aes::Block::cast_from_core_mut(block);
match self {
Aes::A128(c) => c.decrypt_block(b),
Aes::A192(c) => c.decrypt_block(b),
Aes::A256(c) => c.decrypt_block(b),
}
}
/// ECB: each block on its own. `data` must be whole blocks.
fn decrypt_ecb(&self, data: &mut [u8]) -> Result<(), DecryptError> {
let (blocks, rest) = data.as_chunks_mut::<BLOCK>();
if !rest.is_empty() {
return Err(malformed("ciphertext isn't whole AES blocks"));
}
blocks.iter_mut().for_each(|b| self.decrypt_block(b));
Ok(())
}
/// CBC: each block XORed with the previous ciphertext block (the IV for the first).
fn decrypt_cbc(&self, iv: &[u8], data: &mut [u8]) -> Result<(), DecryptError> {
let mut prev: [u8; BLOCK] = iv.try_into().map_err(|_| malformed("an IV that isn't one AES block"))?;
let (blocks, rest) = data.as_chunks_mut::<BLOCK>();
if !rest.is_empty() {
return Err(malformed("ciphertext isn't whole AES blocks"));
}
for b in blocks {
let cipher = *b;
self.decrypt_block(b);
b.iter_mut().zip(prev).for_each(|(x, p)| *x ^= p);
prev = cipher;
}
Ok(())
}
}
// ---- Standard encryption -------------------------------------------------------------------
/// What a Standard `EncryptionInfo` holds: the key size and the password verifier.
struct StandardHeader<'a> {
key_len: usize,
salt: &'a [u8],
verifier: &'a [u8],
verifier_hash: &'a [u8],
}
/// `info` starts after the version: flags, header size, header, verifier.
fn standard_header(info: &[u8]) -> Result<StandardHeader<'_>, DecryptError> {
let short = || malformed("truncated Standard encryption header");
let flags = u32_at(info, 0).ok_or_else(short)?;
if flags & F_EXTERNAL != 0 {
return Err(DecryptError::Unsupported("external".into()));
}
if flags & F_AES == 0 || flags & F_CRYPTOAPI == 0 {
return Err(DecryptError::Unsupported("RC4".into()));
}
let header_size = usize::try_from(u32_at(info, 4).ok_or_else(short)?).map_err(|_| short())?;
let header = info.get(8..8usize.checked_add(header_size).ok_or_else(short)?).ok_or_else(short)?;
let alg_id = u32_at(header, 8).ok_or_else(short)?;
let alg_hash = u32_at(header, 12).ok_or_else(short)?;
let key_bits = u32_at(header, 16).ok_or_else(short)?;
let key_len = match (alg_id, key_bits) {
(0x660E | 0, 128) => 16,
(0x660F, 192) => 24,
(0x6610, 256) => 32,
(0x660E..=0x6610, _) => return Err(malformed(format!("AES with a {key_bits}-bit key"))),
_ => return Err(DecryptError::Unsupported(format!("cipher 0x{alg_id:04X}"))),
};
if alg_hash != 0x8004 && alg_hash != 0 {
return Err(DecryptError::Unsupported(format!("hash 0x{alg_hash:04X}")));
}
// Verifier: salt size, salt, encrypted verifier, verifier hash size, encrypted hash.
let v = info.get(8 + header_size..).ok_or_else(short)?;
if u32_at(v, 0) != Some(16) {
return Err(malformed("a Standard salt that isn't 16 bytes"));
}
let salt = v.get(4..20).ok_or_else(short)?;
let verifier = v.get(20..36).ok_or_else(short)?;
let verifier_hash = v.get(40..72).ok_or_else(short)?;
Ok(StandardHeader { key_len, salt, verifier, verifier_hash })
}
fn standard(info: &[u8], package: &[u8], password: &str) -> Result<Vec<u8>, DecryptError> {
let h = standard_header(info)?;
let (mut verifier, mut verifier_hash) = (h.verifier.to_vec(), h.verifier_hash.to_vec());
let aes = Aes::new(&standard_key(password, h.salt, h.key_len))?;
aes.decrypt_ecb(&mut verifier)?;
aes.decrypt_ecb(&mut verifier_hash)?;
if verifier_hash.get(..20) != Some(Hash::Sha1.digest(&[&verifier]).as_slice()) {
return Err(DecryptError::WrongPassword);
}
let (size, data) = package_parts(package)?;
let mut out = data.get(..size.div_ceil(BLOCK) * BLOCK).ok_or_else(|| malformed("the package isn't whole AES blocks"))?.to_vec();
aes.decrypt_ecb(&mut out)?;
out.truncate(size);
Ok(out)
}
// ---- Agile encryption ----------------------------------------------------------------------
/// The cipher parameters shared by `keyData` and a password `encryptedKey` element.
pub(crate) struct Params {
pub(crate) hash: Hash,
pub(crate) salt: Vec<u8>,
pub(crate) key_len: usize,
}
impl Params {
fn read(e: &El) -> Result<Params, DecryptError> {
let cipher = e.attr("cipherAlgorithm").unwrap_or("AES");
if cipher != "AES" {
return Err(DecryptError::Unsupported(cipher.into()));
}
let chaining = e.attr("cipherChaining").unwrap_or("ChainingModeCBC");
if chaining != "ChainingModeCBC" {
return Err(DecryptError::Unsupported(format!("AES {chaining}")));
}
if e.attr_u32("blockSize") != Some(16) {
return Err(malformed("an AES block size that isn't 16"));
}
let key_len = match e.attr_u32("keyBits") {
Some(128) => 16,
Some(192) => 24,
Some(256) => 32,
other => return Err(malformed(format!("AES with keyBits {}", other.unwrap_or(0)))),
};
let hash = Hash::from_name(e.attr("hashAlgorithm").unwrap_or("SHA1"))?;
let salt = base64(e.attr("saltValue").ok_or_else(|| malformed("no saltValue"))?)?;
if salt.is_empty() || salt.len() > 64 {
return Err(malformed("a salt that isn't 1 to 64 bytes"));
}
Ok(Params { hash, salt, key_len })
}
/// An IV: H(salt ‖ block key) for a block key, else the salt, sized to one AES block.
pub(crate) fn iv(&self, block_key: Option<&[u8]>) -> Vec<u8> {
match block_key {
Some(k) => derive(self.hash, &self.salt, k, BLOCK),
None => {
let mut iv = self.salt.clone();
iv.resize(BLOCK, 0x36);
iv
}
}
}
}
/// What an Agile descriptor holds: the package's cipher, the password key encryptor with its
/// three encrypted values (verifier input, verifier hash, package key), and the package's
/// integrity pair, which Excel always writes and the reader requires.
struct AgileDescriptor {
key_data: Params,
key: Params,
spin: u32,
verifier_input: Vec<u8>,
verifier_hash: Vec<u8>,
key_value: Vec<u8>,
hmac_key: Vec<u8>,
hmac_value: Vec<u8>,
}
/// `xml` is the XML encryption descriptor after the 8-byte version and reserved field.
fn agile_descriptor(xml: &[u8]) -> Result<AgileDescriptor, DecryptError> {
let root = crate::xml::parse(xml).map_err(|e| malformed(e.to_string()))?;
let key_data = Params::read(root.child("keyData").ok_or_else(|| malformed("no keyData"))?)?;
let encryptor = root
.child("keyEncryptors")
.and_then(|k| k.kids("keyEncryptor").find(|e| e.attr("uri") == Some(PASSWORD_ENCRYPTOR)))
.ok_or_else(|| DecryptError::Unsupported("certificate".into()))?
.child("encryptedKey")
.ok_or_else(|| malformed("no encryptedKey"))?;
let key = Params::read(encryptor)?;
let spin = encryptor.attr_u32("spinCount").ok_or_else(|| malformed("no spinCount"))?;
if spin > MAX_SPIN_COUNT {
return Err(malformed(format!("a spin count of {spin}, more than the {MAX_SPIN_COUNT} allowed")));
}
let field = |name: &str| base64(encryptor.attr(name).ok_or_else(|| malformed(format!("no {name}")))?);
// A descriptor without dataIntegrity can't prove its ciphertext is the one the password
// protects: the verifier only authenticates the key, not the package ([MS-OFFCRYPTO] 2.3.4.14).
let integrity = root.child("dataIntegrity").ok_or_else(|| malformed("no dataIntegrity"))?;
let hmac_field = |attr: &str| {
let v = base64(integrity.attr(attr).ok_or_else(|| malformed(format!("no {attr}")))?)?;
if v.is_empty() || v.len() % BLOCK != 0 {
return Err(malformed(format!("a whole-block {attr}")));
}
Ok(v)
};
Ok(AgileDescriptor {
key_data,
key,
spin,
verifier_input: field("encryptedVerifierHashInput")?,
verifier_hash: field("encryptedVerifierHashValue")?,
key_value: field("encryptedKeyValue")?,
hmac_key: hmac_field("encryptedHmacKey")?,
hmac_value: hmac_field("encryptedHmacValue")?,
})
}
/// Decrypts an Agile package ([MS-OFFCRYPTO] 2.3.4.10): checks the password against the
/// verifier, unwraps the package key, verifies the dataIntegrity HMAC, then decrypts the segments.
fn agile(xml: &[u8], package: &[u8], password: &str) -> Result<Vec<u8>, DecryptError> {
let descriptor = agile_descriptor(xml)?;
let AgileDescriptor { key_data, key, spin, verifier_input, verifier_hash, key_value, hmac_key, hmac_value } = descriptor;
let h = iterated_hash(key.hash, password, &key.salt, spin);
let iv = key.iv(None);
let field = |data: &[u8], block_key: &[u8]| -> Result<Vec<u8>, DecryptError> {
let mut data = data.to_vec();
Aes::new(&derive(key.hash, &h, block_key, key.key_len))?.decrypt_cbc(&iv, &mut data)?;
Ok(data)
};
let input = field(&verifier_input, &BLOCK_VERIFIER_INPUT)?;
let expected = field(&verifier_hash, &BLOCK_VERIFIER_HASH)?;
let input = input.get(..key.salt.len()).unwrap_or(&input);
let actual = key.hash.digest(&[input]);
if expected.get(..actual.len()) != Some(actual.as_slice()) {
return Err(DecryptError::WrongPassword);
}
let secret = field(&key_value, &BLOCK_KEY)?;
let aes = Aes::new(secret.get(..key_data.key_len).ok_or_else(|| malformed("a short encryptedKeyValue"))?)?;
// Package integrity, [MS-OFFCRYPTO] 2.3.4.14: both HMAC fields are encrypted with the
// package key (the verifier above only authenticates the password), computed over the whole
// EncryptedPackage stream, StreamSize field included. The specification keys the HMAC with a
// random salt-size array, but Excel's key is as long as the hash (64 bytes for SHA-512), so
// both lengths are tried.
let mut mac_field = hmac_key;
aes.decrypt_cbc(&key_data.iv(Some(&BLOCK_HMAC_KEY)), &mut mac_field)?;
let mut declared = hmac_value;
aes.decrypt_cbc(&key_data.iv(Some(&BLOCK_HMAC_VALUE)), &mut declared)?;
let verified = [key_data.hash.digest_len(), key_data.salt.len()].into_iter().any(|len| {
let Some(key) = mac_field.get(..len) else { return false };
let mac = hmac(key_data.hash, key, package);
declared.get(..mac.len()) == Some(mac.as_slice())
});
if !verified {
return Err(malformed("the decrypted data failed its integrity check"));
}
let (size, data) = package_parts(package)?;
let mut out = Vec::with_capacity(size);
for (i, segment) in data.chunks(SEGMENT).enumerate() {
if out.len() >= size {
break;
}
let index = u32::try_from(i).map_err(|_| malformed("too many segments"))?;
let start = out.len();
out.extend_from_slice(segment);
let block = out.get_mut(start..).unwrap_or_default();
aes.decrypt_cbc(&key_data.iv(Some(&index.to_le_bytes())), block)?;
}
out.truncate(size);
Ok(out)
}
// ---- base64 --------------------------------------------------------------------------------
/// Standard base64 (whitespace ignored), as Agile's XML uses for every binary value.
fn base64(s: &str) -> Result<Vec<u8>, DecryptError> {
let mut out = Vec::with_capacity(s.len() / 4 * 3);
let (mut buf, mut bits) = (0u32, 0u32);
for c in s.bytes().filter(|c| !c.is_ascii_whitespace() && *c != b'=') {
let v = match c {
b'A'..=b'Z' => c - b'A',
b'a'..=b'z' => c - b'a' + 26,
b'0'..=b'9' => c - b'0' + 52,
b'+' => 62,
b'/' => 63,
_ => return Err(malformed("invalid base64")),
};
buf = (buf << 6) | u32::from(v);
bits += 6;
if bits >= 8 {
bits -= 8;
out.push((buf >> bits) as u8);
buf &= (1 << bits) - 1;
}
}
Ok(out)
}
#[cfg(test)]
mod tests;
-250
View File
@@ -1,250 +0,0 @@
//! Decryption tests. The expected keys and IVs below were computed independently with
//! msoffcrypto-tool 6.0.0 (MIT) for the synthetic password and salts used here; the encrypted
//! files are built in code by `crate::testing`. No Excel-produced file is used.
use super::*;
use crate::testing::{agile_info, encrypt_agile, encrypt_agile_with_hmac_key, encrypt_standard, with_agile_header};
const PASSWORD: &str = "Pass1234";
/// 16 zero bytes (whole AES blocks), a stand-in for a real integrity field where the values
/// can't be reached anyway (scheme errors, descriptor-parse damage).
const HMAC_STUB: &str = "AAAAAAAAAAAAAAAAAAAAAA==";
fn hex(s: &str) -> Vec<u8> {
(0..s.len()).step_by(2).map(|i| u8::from_str_radix(&s[i..i + 2], 16).unwrap()).collect()
}
fn salt() -> Vec<u8> {
(0u8..16).collect()
}
#[test]
fn aes_decrypts_the_fips_197_vectors() {
let mut b: [u8; 16] = hex("69c4e0d86a7b0430d8cdb78070b4c55a").try_into().unwrap();
Aes::new(&hex("000102030405060708090a0b0c0d0e0f")).unwrap().decrypt_block(&mut b);
assert_eq!(b.to_vec(), hex("00112233445566778899aabbccddeeff"));
let mut b: [u8; 16] = hex("8ea2b7ca516745bfeafc49904b496089").try_into().unwrap();
Aes::new(&hex("000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f")).unwrap().decrypt_block(&mut b);
assert_eq!(b.to_vec(), hex("00112233445566778899aabbccddeeff"));
}
#[test]
fn agile_keys_match_msoffcrypto() {
let h = iterated_hash(Hash::Sha512, PASSWORD, &salt(), 1000);
assert_eq!(derive(Hash::Sha512, &h, &BLOCK_VERIFIER_INPUT, 32), hex("c57def04938e1e83497271f6ec099e890bfd2bd9cf05ff208f8b2405df361ae6"));
assert_eq!(derive(Hash::Sha512, &h, &BLOCK_KEY, 32), hex("4ab08cf2ed95199d58d04f19331049a8a07cc1ff508431a1dd0785c84e261d03"));
let h = iterated_hash(Hash::Sha1, PASSWORD, &salt(), 1000);
assert_eq!(derive(Hash::Sha1, &h, &BLOCK_VERIFIER_INPUT, 16), hex("88c7567d028eae33a9a55f65729dcd2b"));
let key_data = Params { hash: Hash::Sha512, salt: (16u8..32).collect(), key_len: 32 };
assert_eq!(key_data.iv(Some(&1u32.to_le_bytes())), hex("da22ccba062aff577d207b2dee4e165b"));
}
#[test]
fn agile_pads_short_hashes_with_0x36() {
// SHA-1 gives 20 bytes; an AES-256 key needs 32, so 12 bytes of 0x36 follow.
let k = derive(Hash::Sha1, b"a", b"b", 32);
assert_eq!(k[..20], Hash::Sha1.digest(&[b"ab"])[..]);
assert_eq!(k[20..], [0x36; 12]);
}
#[test]
fn standard_keys_match_msoffcrypto() {
assert_eq!(standard_key(PASSWORD, &salt(), 16), hex("ae0b402aeb23a779471a130843e431d4"));
assert_eq!(standard_key(PASSWORD, &salt(), 32), hex("ae0b402aeb23a779471a130843e431d47b416314b5146b491cb815283663d26f"));
}
#[test]
fn base64_decodes_and_rejects_junk() {
assert_eq!(base64("AAECAwQFBgcICQoLDA0ODw==").unwrap(), salt());
assert_eq!(base64(" AAEC\nAw== ").unwrap(), vec![0, 1, 2, 3]);
assert!(matches!(base64("AA*C"), Err(DecryptError::Malformed(_))));
}
/// A payload that isn't a whole number of segments or blocks.
fn payload(len: usize) -> Vec<u8> {
(0..len).map(|i| (i * 7 + i / 251) as u8).collect()
}
#[test]
fn agile_round_trips() {
let plain = payload(10_000);
let (info, package) = encrypt_agile(&plain, PASSWORD, 1000);
assert_eq!(decrypt_streams(&info, &package, PASSWORD).unwrap(), plain);
}
#[test]
fn standard_round_trips() {
let plain = payload(10_000);
let (info, package) = encrypt_standard(&plain, PASSWORD);
assert_eq!(decrypt_streams(&info, &package, PASSWORD).unwrap(), plain);
}
#[test]
fn non_ascii_passwords_round_trip() {
// UTF-16LE, surrogate pairs included: "Pässwörd€" plus an emoji outside the BMP.
let pw = "P\u{e4}ssw\u{f6}rd\u{20ac}\u{1F511}";
let (info, package) = encrypt_agile(&payload(100), pw, 1000);
assert_eq!(decrypt_streams(&info, &package, pw).unwrap(), payload(100));
assert_eq!(decrypt_streams(&info, &package, "Passwrd"), Err(DecryptError::WrongPassword));
let (info, package) = encrypt_standard(&payload(100), pw);
assert_eq!(decrypt_streams(&info, &package, pw).unwrap(), payload(100));
}
#[test]
fn a_wrong_password_is_reported_as_such() {
let (info, package) = encrypt_agile(&payload(100), PASSWORD, 1000);
assert_eq!(decrypt_streams(&info, &package, "pass1234"), Err(DecryptError::WrongPassword));
assert_eq!(decrypt_streams(&info, &package, ""), Err(DecryptError::WrongPassword));
let (info, package) = encrypt_standard(&payload(100), PASSWORD);
assert_eq!(decrypt_streams(&info, &package, "Pass12345"), Err(DecryptError::WrongPassword));
}
#[test]
fn unsupported_schemes_are_named() {
let unsupported = |info: &[u8]| match decrypt_streams(info, &[0; 8], PASSWORD) {
Err(DecryptError::Unsupported(s)) => s,
other => panic!("expected Unsupported, got {other:?}"),
};
let (mut rc4, _) = encrypt_standard(&payload(10), PASSWORD);
rc4[4..8].copy_from_slice(&F_CRYPTOAPI.to_le_bytes()); // fCryptoAPI without fAES
assert_eq!(unsupported(&rc4), "RC4");
assert_eq!(unsupported(&[3, 0, 3, 0, 0, 0, 0, 0]), "Extensible");
assert_eq!(unsupported(&[1, 0, 1, 0, 0, 0, 0, 0]), "version 1.1");
let des = agile_info(PASSWORD, 1000, HMAC_STUB, HMAC_STUB).replace("cipherAlgorithm=\"AES\"", "cipherAlgorithm=\"3DES\"");
assert_eq!(unsupported(&with_agile_header(&des)), "3DES");
let cert = agile_info(PASSWORD, 1000, HMAC_STUB, HMAC_STUB).replace("keyEncryptor/password", "keyEncryptor/certificate");
assert_eq!(unsupported(&with_agile_header(&cert)), "certificate");
}
#[test]
fn damaged_files_are_malformed_not_panics() {
let is_malformed = |r: Result<Vec<u8>, DecryptError>| matches!(r, Err(DecryptError::Malformed(_)));
let plain = payload(5000);
for (info, package) in [encrypt_agile(&plain, PASSWORD, 1000), encrypt_standard(&plain, PASSWORD)] {
// Every truncation of either stream.
for cut in (0..info.len()).step_by(7) {
let _ = decrypt_streams(&info[..cut], &package, PASSWORD);
}
for cut in (0..package.len()).step_by(97) {
let _ = decrypt_streams(&info, &package[..cut], PASSWORD);
}
// A stated size past the ciphertext never drives an allocation.
let mut huge = package.clone();
huge[..8].copy_from_slice(&u64::MAX.to_le_bytes());
assert!(is_malformed(decrypt_streams(&info, &huge, PASSWORD)));
// Ciphertext that isn't whole AES blocks.
let mut ragged = package.clone();
ragged.truncate(ragged.len() - 3);
let len = (ragged.len() - 8) as u64;
ragged[..8].copy_from_slice(&len.to_le_bytes());
assert!(is_malformed(decrypt_streams(&info, &ragged, PASSWORD)));
}
let xml = agile_info(PASSWORD, 1000, HMAC_STUB, HMAC_STUB);
let bad = |s: String| is_malformed(decrypt_streams(&with_agile_header(&s), &[0; 8], PASSWORD));
assert!(bad(xml.replace("spinCount=\"1000\"", "spinCount=\"4000000000\"")));
assert!(bad(xml.replace("saltValue=\"", "saltValue=\"*")));
assert!(bad(xml.replace("keyBits=\"256\"", "keyBits=\"100\"")));
assert!(bad(xml.replace("blockSize=\"16\"", "blockSize=\"8\"")));
assert!(bad(xml.replace("<keyData", "<other")));
assert!(bad("<encryption".into()));
}
#[test]
fn non_ascii_passwords_hash_as_utf16le_like_msoffcrypto() {
// "Pässwörd€" plus an emoji outside the BMP, a surrogate pair in UTF-16.
let h = iterated_hash(Hash::Sha512, "P\u{e4}ssw\u{f6}rd\u{20ac}\u{1F511}", &salt(), 1000);
assert_eq!(
h,
hex("2bca96ca0a7e8d88d9314a2eb1819a1ece94ff9ce34266ff82f891a88c4240c4c9f39c729b7c5c410c27fc8e2b7ad90d8c005eb68a603377b8a4ab858041c76a")
);
}
#[test]
fn hmac_matches_the_rfc2104_test_vectors() {
use crate::crypto::hmac as mac;
let h = |v: Vec<u8>| v.iter().map(|b| format!("{b:02x}")).collect::<String>();
// RFC 2202 test case 1 (HMAC-SHA-1) and HMAC-SHA-256 vectors.
assert_eq!(h(mac(Hash::Sha1, &[0x0b; 20], b"Hi There")), "b617318655057264e28bc0b6fb378c8ef146be00");
assert_eq!(h(mac(Hash::Sha256, &[0x0b; 20], b"Hi There")), "b0344c61d8db38535ca8afceaf0bf12b881dc200c9833da726e9376c2e32cff7");
// RFC 4231 test case 5 (HMAC-SHA-256): a key longer than the block size is hashed first.
assert_eq!(
h(mac(Hash::Sha256, &[0xaa; 131], b"Test Using Larger Than Block-Size Key - Hash Key First")),
"60e431591ee0b67f0d8a26aacbf5b77f8e0bc6213728c5140546040f0ee37f54"
);
}
/// Strips the `<dataIntegrity …/>` element from an Agile descriptor.
fn without_integrity(info: &[u8]) -> Vec<u8> {
let xml = std::str::from_utf8(&info[8..]).unwrap().to_string();
let start = xml.find("<dataIntegrity").unwrap();
let end = start + xml[start..].find("/>").unwrap() + 2;
let stripped = format!("{}{}", &xml[..start], &xml[end..]);
with_agile_header(&stripped)
}
#[test]
fn integrity_is_required_and_enforced() {
let plain = payload(300);
let (info, package) = encrypt_agile(&plain, PASSWORD, 1000);
// An attacker can't just delete the integrity element to get the package verified.
assert!(matches!(decrypt_streams(&without_integrity(&info), &package, PASSWORD),
Err(DecryptError::Malformed(m)) if m.contains("dataIntegrity")));
// An empty or ragged field is damage, not a pass.
for bad in [("encryptedHmacKey", ""), ("encryptedHmacKey", "AA=="), ("encryptedHmacValue", "")] {
let xml =
agile_info(PASSWORD, 1000, HMAC_STUB, HMAC_STUB).replace(&format!("{}=\"{HMAC_STUB}\"", bad.0), &format!("{}=\"{}\"", bad.0, bad.1));
let r = decrypt_streams(&with_agile_header(&xml), &package, PASSWORD);
assert!(matches!(r, Err(DecryptError::Malformed(_))), "{}=\"{}\"", bad.0, bad.1);
}
}
#[test]
fn modified_ciphertext_fails_the_integrity_check() {
let plain = payload(300);
let (info, package) = encrypt_agile(&plain, PASSWORD, 1000);
// The correct password proves the verifier can't vouch for a modified package.
for at in [8usize, 100, package.len() - 5] {
let mut tampered = package.clone();
tampered[at] ^= 0x80;
assert!(
matches!(decrypt_streams(&info, &tampered, PASSWORD),
Err(DecryptError::Malformed(m)) if m.contains("integrity")),
"byte {at}"
);
}
// And neither can the descriptor: a swapped integrity value is refused.
let xml = String::from_utf8_lossy(&info[8..]).to_string();
let at = xml.find("encryptedHmacValue=\"").unwrap() + "encryptedHmacValue=\"".len();
let mut xml: Vec<u8> = xml.into_bytes();
xml[at] ^= b'A' ^ b'B';
assert!(matches!(decrypt_streams(&with_agile_header(std::str::from_utf8(&xml).unwrap()), &package, PASSWORD), Err(DecryptError::Malformed(_))));
}
#[test]
fn integrity_matches_the_stream_not_the_plaintext() {
// The MAC covers the whole EncryptedPackage stream, StreamSize field included: changing the
// stated size alone must fail, while the actual bytes round-trip.
let plain = payload(4200);
let (info, package) = encrypt_agile(&plain, PASSWORD, 1000);
let mut ours = package.clone();
ours[..8].copy_from_slice(&((plain.len() + 1) as u64).to_le_bytes());
assert!(matches!(decrypt_streams(&info, &ours, PASSWORD), Err(DecryptError::Malformed(m)) if m.contains("integrity")));
assert_eq!(decrypt_streams(&info, &package, PASSWORD).unwrap(), plain);
}
#[test]
fn integrity_keys_as_long_as_the_hash_or_the_salt_verify() {
// Excel keys the HMAC with 64 bytes (#2712); [MS-OFFCRYPTO] 2.3.4.14 describes a salt-size
// key. Both verify, and neither lets a modified package through.
let plain = payload(5000);
for key in [vec![7u8; 64], vec![7u8; 16]] {
let (info, package) = encrypt_agile_with_hmac_key(&plain, PASSWORD, 1000, &key);
assert_eq!(decrypt_streams(&info, &package, PASSWORD).unwrap(), plain, "{}-byte key", key.len());
let mut tampered = package.clone();
tampered[100] ^= 1;
assert!(matches!(decrypt_streams(&info, &tampered, PASSWORD), Err(DecryptError::Malformed(m)) if m.contains("integrity")));
}
// Another length (a key that matches neither) fails.
let (info, package) = encrypt_agile_with_hmac_key(&plain, PASSWORD, 1000, &[7u8; 32]);
assert!(matches!(decrypt_streams(&info, &package, PASSWORD), Err(DecryptError::Malformed(m)) if m.contains("integrity")));
}
+10 -251
View File
@@ -1,4 +1,6 @@
//! CSV import and export (RFC 4180 with the usual real-world leniency).
//! CSV import and export (RFC 4180 with the usual real-world leniency). Decoding, delimiter
//! sniffing, the record parser and field quoting are `craft-csv`'s; this module turns records into
//! typed cells (with each column's date order) and writes a sheet.
use std::sync::Arc;
@@ -12,241 +14,11 @@ use crate::{CsvOptions, IoError, ReadReport};
/// The largest CSV or tab-separated file written (bytes).
pub const MAX_CSV_BYTES: usize = 512 << 20;
/// Windows-1252 code points for bytes 0x80–0x9F (others map to Latin-1).
const CP1252: [u32; 32] = [
0x20AC, 0x81, 0x201A, 0x0192, 0x201E, 0x2026, 0x2020, 0x2021, 0x02C6, 0x2030, 0x0160, 0x2039, 0x0152, 0x8D, 0x017D, 0x8F, 0x90, 0x2018, 0x2019,
0x201C, 0x201D, 0x2022, 0x2013, 0x2014, 0x02DC, 0x2122, 0x0161, 0x203A, 0x0153, 0x9D, 0x017E, 0x0178,
];
pub use craft_csv::{decode_text, sniff_delimiter};
/// Windows-1256 code points for bytes 0x80–0xFF (unicode.org `CP1256.TXT`).
pub(crate) const CP1256: [u32; 128] = [
0x20AC, 0x067E, 0x201A, 0x0192, 0x201E, 0x2026, 0x2020, 0x2021, 0x02C6, 0x2030, 0x0679, 0x2039, 0x0152, 0x0686, 0x0698, 0x0688, 0x06AF, 0x2018,
0x2019, 0x201C, 0x201D, 0x2022, 0x2013, 0x2014, 0x06A9, 0x2122, 0x0691, 0x203A, 0x0153, 0x200C, 0x200D, 0x06BA, 0x00A0, 0x060C, 0x00A2, 0x00A3,
0x00A4, 0x00A5, 0x00A6, 0x00A7, 0x00A8, 0x00A9, 0x06BE, 0x00AB, 0x00AC, 0x00AD, 0x00AE, 0x00AF, 0x00B0, 0x00B1, 0x00B2, 0x00B3, 0x00B4, 0x00B5,
0x00B6, 0x00B7, 0x00B8, 0x00B9, 0x061B, 0x00BB, 0x00BC, 0x00BD, 0x00BE, 0x061F, 0x06C1, 0x0621, 0x0622, 0x0623, 0x0624, 0x0625, 0x0626, 0x0627,
0x0628, 0x0629, 0x062A, 0x062B, 0x062C, 0x062D, 0x062E, 0x062F, 0x0630, 0x0631, 0x0632, 0x0633, 0x0634, 0x0635, 0x0636, 0x00D7, 0x0637, 0x0638,
0x0639, 0x063A, 0x0640, 0x0641, 0x0642, 0x0643, 0x00E0, 0x0644, 0x00E2, 0x0645, 0x0646, 0x0647, 0x0648, 0x00E7, 0x00E8, 0x00E9, 0x00EA, 0x00EB,
0x0649, 0x064A, 0x00EE, 0x00EF, 0x064B, 0x064C, 0x064D, 0x064E, 0x00F4, 0x064F, 0x0650, 0x00F7, 0x0651, 0x00F9, 0x0652, 0x00FB, 0x00FC, 0x200E,
0x200F, 0x06D2,
];
/// Evidence that bytes which aren't UTF-8 are Windows-1256 Arabic rather than Windows-1252.
#[derive(Default)]
struct Cp1256Evidence {
/// High bytes that decode to different characters in the two code pages, apart from invisible
/// marks and ordinals after a digit.
differing: usize,
/// Of those, Arabic letters and Arabic punctuation (، ؛ ؟) in Windows-1256.
arabic: usize,
/// Arabic bytes next to an ASCII letter: Western accents sit inside Latin words, Arabic doesn't.
arabic_touching_latin: usize,
/// Accented Latin bytes the two code pages share, next to another high byte: Hebrew
/// Windows-1255 letters land there, Arabic and Western text rarely do.
shared_letters_in_runs: usize,
}
impl Cp1256Evidence {
fn of(bytes: &[u8]) -> Self {
let mut e = Self::default();
for (i, &b) in bytes.iter().enumerate() {
if b < 0x80 {
continue;
}
let before = i.checked_sub(1).and_then(|j| bytes.get(j)).copied();
let after = bytes.get(i + 1).copied();
let next_to = |pred: fn(u8) -> bool| before.is_some_and(pred) || after.is_some_and(pred);
if same_in_cp1252_and_cp1256(b) {
if is_shared_latin_letter(b) && next_to(|x| x >= 0x80) {
e.shared_letters_in_runs += 1;
}
continue;
}
if is_invisible_mark(b) || is_ordinal_after_digit(b, before) {
continue;
}
e.differing += 1;
if is_cp1256_arabic(b) {
e.arabic += 1;
if next_to(|x| x.is_ascii_alphabetic()) {
e.arabic_touching_latin += 1;
}
}
}
e
}
/// Thresholds chosen against month, day and language names from 95 glibc locales (Western in
/// Windows-1252, Arabic, Persian and Urdu in 1256, Hebrew and Yiddish in 1255) plus CSV samples.
fn is_arabic(&self) -> bool {
self.arabic >= 2
&& self.arabic * 10 >= self.differing * 9
&& self.arabic_touching_latin * 10 <= self.arabic * 3
&& self.shared_letters_in_runs * 20 <= self.arabic
}
}
fn same_in_cp1252_and_cp1256(b: u8) -> bool {
matches!(
b,
0x80 | 0x82..=0x89 | 0x8B..=0x8C | 0x91..=0x97 | 0x99 | 0x9B..=0x9C | 0xA0 | 0xA2..=0xA9 | 0xAB..=0xB9 | 0xBB..=0xBE | 0xD7 | 0xE0 | 0xE2 | 0xE7..=0xEB
| 0xEE..=0xEF | 0xF4 | 0xF7 | 0xF9 | 0xFB..=0xFC
)
}
/// ZWNJ, ZWJ, LRM and RLM: common inside Persian words and around Arabic cells, but no evidence
/// either way.
fn is_invisible_mark(b: u8) -> bool {
matches!(b, 0x9D | 0x9E | 0xFD | 0xFE)
}
/// `º` and `ª` after a digit (`1º`, `2ª`) are Western ordinals, though Windows-1256 reads the bytes
/// as `؛` and `ھ`; either way they are no evidence.
fn is_ordinal_after_digit(b: u8, before: Option<u8>) -> bool {
matches!(b, 0xAA | 0xBA) && before.is_some_and(|x| x.is_ascii_digit())
}
fn is_shared_latin_letter(b: u8) -> bool {
matches!(b, 0xE0 | 0xE2 | 0xE7..=0xEB | 0xEE..=0xEF | 0xF4 | 0xF9 | 0xFB..=0xFC)
}
fn is_cp1256_arabic(b: u8) -> bool {
matches!(
b,
0x81 | 0x8A | 0x8D..=0x90 | 0x98 | 0x9A | 0x9F | 0xA1 | 0xAA | 0xBA | 0xBF..=0xD6 | 0xD8..=0xDB | 0xDD..=0xDF | 0xE1 | 0xE3..=0xE6 | 0xEC..=0xED | 0xFF
)
}
/// Decodes CSV bytes with BOM-aware UTF-8 and UTF-16 handling. Without a BOM, valid UTF-8 is kept.
/// Other bytes decode as Windows-1256 when they read as Arabic (see [`Cp1256Evidence`]), else as
/// Windows-1252.
pub fn decode_text(bytes: &[u8]) -> String {
if let Some(rest) = bytes.strip_prefix(&[0xEF, 0xBB, 0xBF]) {
return String::from_utf8_lossy(rest).into_owned();
}
let utf16 = |b: &[u8], le: bool| -> String {
let units = b.as_chunks::<2>().0.iter().map(|c| if le { u16::from_le_bytes(*c) } else { u16::from_be_bytes(*c) });
char::decode_utf16(units).map(|r| r.unwrap_or('\u{FFFD}')).collect()
};
if let Some(rest) = bytes.strip_prefix(&[0xFF, 0xFE]) {
return utf16(rest, true);
}
if let Some(rest) = bytes.strip_prefix(&[0xFE, 0xFF]) {
return utf16(rest, false);
}
match std::str::from_utf8(bytes) {
Ok(s) => s.to_string(),
Err(_) => {
let use_windows_1256 = Cp1256Evidence::of(bytes).is_arabic();
bytes
.iter()
.map(|&b| match b {
0x80..=0xFF if use_windows_1256 => {
char::from_u32(CP1256.get((b - 0x80) as usize).copied().unwrap_or(0xFFFD)).unwrap_or('\u{FFFD}')
}
0x80..=0x9F => char::from_u32(CP1252.get((b - 0x80) as usize).copied().unwrap_or(0xFFFD)).unwrap_or('\u{FFFD}'),
_ => b as char,
})
.collect()
}
}
}
/// Picks comma, semicolon or tab: the candidate with the most consistent count per line
/// (outside quotes) over the first non-blank lines. A candidate must appear on every such line, otherwise comma wins.
pub fn sniff_delimiter(text: &str, quote: char) -> u8 {
let mut best = (b',', 0usize, 0usize);
for d in [',', ';', '\t'] {
let mut counts = vec![];
let mut n = 0usize;
let mut in_q = false;
let mut blank = true;
for ch in text.chars().take(64 * 1024) {
if !in_q && ch == '\n' {
// Blank lines carry no delimiter evidence.
if !blank {
counts.push(n);
}
n = 0;
blank = true;
if counts.len() >= 20 {
break;
}
continue;
}
if ch != '\r' {
blank = false;
}
if ch == quote {
in_q = !in_q;
} else if !in_q && ch == d {
n += 1;
}
}
if !blank {
counts.push(n);
}
let min = counts.iter().copied().min().unwrap_or(0);
let total: usize = counts.iter().sum();
if min > 0 && (min, total) > (best.1, best.2) {
best = (d as u8, min, total);
}
}
best.0
}
/// Splits CSV text into records of fields. A record keeps at most [`MAX_COLS`] fields; the rest are
/// dropped as they are read, so a row of endless delimiters cannot allocate a string per field.
pub(crate) fn parse_records(text: &str, delim: char, quote: char, mut on_record: impl FnMut(Vec<String>) -> bool) {
let mut field = String::new();
let mut record: Vec<String> = vec![];
let mut in_q = false;
let mut chars = text.chars().peekable();
let mut field_started_quoted = false;
while let Some(c) = chars.next() {
if in_q {
if c == quote {
if chars.peek() == Some(&quote) {
field.push(quote);
chars.next();
} else {
in_q = false;
}
} else {
field.push(c);
}
continue;
}
if c == quote && field.is_empty() && !field_started_quoted {
in_q = true;
field_started_quoted = true;
} else if c == delim {
let f = std::mem::take(&mut field);
if record.len() < MAX_COLS as usize {
record.push(f);
}
field_started_quoted = false;
} else if c == '\r' || c == '\n' {
if c == '\r' && chars.peek() == Some(&'\n') {
chars.next();
}
let f = std::mem::take(&mut field);
if record.len() < MAX_COLS as usize {
record.push(f);
}
field_started_quoted = false;
if !on_record(std::mem::take(&mut record)) {
return;
}
} else {
field.push(c);
}
}
if !field.is_empty() || !record.is_empty() || field_started_quoted {
if record.len() < MAX_COLS as usize {
record.push(field);
}
on_record(record);
}
/// [`craft_csv::parse_records`] with a record kept to the grid's [`MAX_COLS`] fields.
fn parse_records(text: &str, delim: char, quote: char, on_record: impl FnMut(Vec<String>) -> bool) {
craft_csv::parse_records(text, delim, quote, MAX_COLS as usize, on_record);
}
/// How many rows and bytes of a text file are read to decide each column's date order.
@@ -587,25 +359,12 @@ pub fn write_csv_limited(sheet: &Sheet, delimiter: u8, limit: usize, mut cell_te
out.push(delim);
}
let t = cell_text(CellRef::new(row, col));
let quoted = t.contains(delim) || t.contains('"') || t.contains('\n') || t.contains('\r');
// The exact size of the field as written (the delimiter was pushed above), so a row
// never grows past the cap unseen and no doubled-quote copy of the text is built.
let field_len = if quoted { t.len() + t.matches('"').count() + 2 } else { t.len() };
if out.len().saturating_add(field_len) > limit {
// never grows past the cap unseen.
if out.len().saturating_add(craft_csv::field_len(&t, delim)) > limit {
return Err(too_large(format!("more than {} MiB (stopped at row {})", limit >> 20, row + 1)));
}
if quoted {
out.push('"');
for ch in t.chars() {
if ch == '"' {
out.push('"');
}
out.push(ch);
}
out.push('"');
} else {
out.push_str(&t);
}
craft_csv::push_field(&mut out, &t, delim);
}
out.push_str("\r\n");
if out.len() > limit {
+2 -3
View File
@@ -12,7 +12,6 @@
#![forbid(unsafe_code)]
mod array_formula;
mod cfb;
mod chart;
mod chartex;
mod controls;
@@ -272,8 +271,8 @@ pub fn sniff(bytes: &[u8]) -> Format {
// CFB/OLE2 compound file: a password-protected workbook when it holds an encryption
// stream, an Excel 97-2003 workbook when it holds a workbook stream, otherwise another
// legacy binary (.doc…).
if bytes.starts_with(&cfb::SIGNATURE) {
return if cfb::is_encrypted(bytes) {
if bytes.starts_with(&craft_cfb::SIGNATURE) {
return if crypto::is_encrypted(bytes) {
Format::Encrypted
} else if xls::is_xls(bytes) {
Format::Xls
+6 -263
View File
@@ -1,282 +1,25 @@
//! Test support: builds password-encrypted workbooks in code, so tests here and in the crates
//! above (engine, UI) never need an encrypted file on disk. Compiled for this crate's tests and,
//! elsewhere, only with the `test-support` feature (a dev-dependency feature, never shipped).
//!
//! Salts and keys are fixed so the output is deterministic. This is not an encryptor for real
//! use: a fixed key protects nothing.
//! The encryption is `craft-offcrypto`'s and the compound file `craft-cfb`'s.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use aes::cipher::{BlockCipherEncrypt, KeyInit};
use crate::crypto::{
BLOCK_HMAC_KEY, BLOCK_HMAC_VALUE, BLOCK_KEY, BLOCK_VERIFIER_HASH, BLOCK_VERIFIER_INPUT, F_AES, F_CRYPTOAPI, Hash, PASSWORD_ENCRYPTOR, Params,
SEGMENT, derive, hmac, iterated_hash, standard_key,
};
/// The spin count `encrypt_workbook` uses: Excel's default.
pub const SPIN_COUNT: u32 = 100_000;
const PASSWORD_SALT: [u8; 16] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15];
const KEY_DATA_SALT: [u8; 16] = [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31];
const SECRET_KEY: [u8; 32] =
[64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95];
const VERIFIER: [u8; 16] = [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47];
/// The dataIntegrity HMAC key, as long as the hash: Excel writes 64 bytes for SHA-512, not the
/// salt-size key [MS-OFFCRYPTO] 2.3.4.14 describes.
const HMAC_KEY: [u8; 64] = [
112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169,
170, 171, 172, 173, 174, 175,
];
/// `package` (normally an `.xlsx` zip) as Excel stores it with a password: Agile encryption
/// (AES-256, SHA-512) inside a compound file.
pub fn encrypt_workbook(package: &[u8], password: &str) -> Vec<u8> {
let (info, encrypted) = encrypt_agile(package, password, SPIN_COUNT);
compound_file(&[("EncryptionInfo", &info), ("EncryptedPackage", &encrypted)])
craft_offcrypto::encrypt_with_spin_count(package, password, SPIN_COUNT).unwrap()
}
/// The same with Standard encryption (AES-128, SHA-1), as Excel 2007 wrote it.
pub fn encrypt_workbook_standard(package: &[u8], password: &str) -> Vec<u8> {
let (info, encrypted) = encrypt_standard(package, password);
compound_file(&[("EncryptionInfo", &info), ("EncryptedPackage", &encrypted)])
craft_offcrypto::encrypt_standard(package, password, 128).unwrap()
}
/// A version 3 compound file (512-byte sectors) holding `streams` under the root, laid out as
/// Excel lays it out: streams under 4096 bytes in the mini stream, larger ones in regular
/// sectors. Up to 109 FAT sectors (about 7 MB of streams).
/// A version 3 compound file (512-byte sectors) holding `streams` under the root: streams under
/// 4096 bytes in the mini stream, larger ones in regular sectors.
pub fn compound_file(streams: &[(&str, &[u8])]) -> Vec<u8> {
const SECTOR: usize = 512;
const MINI: usize = 64;
const CUTOFF: usize = 4096;
const FREE: u32 = 0xFFFF_FFFF;
const END: u32 = 0xFFFF_FFFE;
const FAT_SECTOR: u32 = 0xFFFF_FFFD;
let is_mini = |d: &[u8]| d.len() < CUTOFF;
// The mini stream: small streams in 64-byte mini sectors, chained by the mini FAT.
let mut container = vec![];
let mut mini_fat: Vec<u32> = vec![];
let mut mini_starts = vec![];
for (_, d) in streams {
if !is_mini(d) || d.is_empty() {
mini_starts.push(END);
continue;
}
let first = mini_fat.len();
let n = d.len().div_ceil(MINI);
for k in 0..n {
mini_fat.push(if k + 1 == n { END } else { (first + k + 1) as u32 });
}
mini_starts.push(first as u32);
container.extend_from_slice(d);
container.resize((first + n) * MINI, 0);
}
// Regular-sector blobs, in order: directory, mini FAT, mini stream, then each large stream.
let dir_len = (streams.len() + 1).div_ceil(4) * SECTOR;
let mut blobs: Vec<Vec<u8>> = vec![vec![0; dir_len], mini_fat.iter().flat_map(|v| v.to_le_bytes()).collect(), container.clone()];
blobs.extend(streams.iter().map(|(_, d)| if is_mini(d) { vec![] } else { d.to_vec() }));
let counts: Vec<usize> = blobs.iter().map(|b| b.len().div_ceil(SECTOR)).collect();
let body: usize = counts.iter().sum();
let mut fat_sectors = 1;
while fat_sectors * 128 < fat_sectors + body {
fat_sectors += 1;
}
assert!(fat_sectors <= 109, "the test writer has no DIFAT sectors");
let mut fat = vec![FREE; fat_sectors * 128];
fat[..fat_sectors].fill(FAT_SECTOR);
let mut starts = vec![];
let mut next = fat_sectors;
for &n in &counts {
if n == 0 {
starts.push(END);
continue;
}
starts.push(next as u32);
for k in 0..n {
fat[next + k] = if k + 1 == n { END } else { (next + k + 1) as u32 };
}
next += n;
}
let put16 = |b: &mut [u8], at: usize, v: u16| b[at..at + 2].copy_from_slice(&v.to_le_bytes());
let put32 = |b: &mut [u8], at: usize, v: u32| b[at..at + 4].copy_from_slice(&v.to_le_bytes());
// Directory: the root (owning the mini stream), then the streams chained as right siblings.
let dir = &mut blobs[0];
for k in 0..dir_len / 128 {
for at in [68, 72, 76] {
put32(dir, k * 128 + at, FREE);
}
}
let mut entry = |k: usize, name: &str, kind: u8, start: u32, size: usize| {
let e = k * 128;
let units: Vec<u8> = name.encode_utf16().flat_map(u16::to_le_bytes).collect();
dir[e..e + units.len()].copy_from_slice(&units);
put16(dir, e + 64, units.len() as u16 + 2);
dir[e + 66] = kind;
dir[e + 67] = 1; // black
if kind == 5 && !streams.is_empty() {
put32(dir, e + 76, 1);
}
if kind == 2 && k < streams.len() {
put32(dir, e + 72, k as u32 + 1);
}
put32(dir, e + 116, start);
put32(dir, e + 120, size as u32);
};
entry(0, "Root Entry", 5, starts[2], container.len());
for (k, (name, data)) in streams.iter().enumerate() {
let start = if is_mini(data) { mini_starts[k] } else { starts[3 + k] };
entry(k + 1, name, 2, start, data.len());
}
let mut out = vec![0u8; SECTOR * (1 + fat_sectors + body)];
out[..8].copy_from_slice(&crate::cfb::SIGNATURE);
put16(&mut out, 24, 0x3E); // minor version
put16(&mut out, 26, 3); // major version
put16(&mut out, 28, 0xFFFE); // byte order
put16(&mut out, 30, 9); // 512-byte sectors
put16(&mut out, 32, 6); // 64-byte mini sectors
put32(&mut out, 44, fat_sectors as u32);
put32(&mut out, 48, starts[0]);
put32(&mut out, 56, CUTOFF as u32);
put32(&mut out, 60, starts[1]); // first mini FAT sector (END when there is none)
put32(&mut out, 64, counts[1] as u32);
put32(&mut out, 68, END); // no DIFAT sectors
for i in 0..109 {
put32(&mut out, 76 + i * 4, if i < fat_sectors { i as u32 } else { FREE });
}
for (i, v) in fat.iter().enumerate() {
put32(&mut out, SECTOR + i * 4, *v);
}
for (blob, start) in blobs.iter().zip(&starts) {
if !blob.is_empty() {
let at = SECTOR * (1 + *start as usize);
out[at..at + blob.len()].copy_from_slice(blob);
}
}
out
}
fn encrypt_block(key: &[u8], block: &mut [u8; 16]) {
let b = aes::Block::cast_from_core_mut(block);
match key.len() {
16 => aes::Aes128::new_from_slice(key).unwrap().encrypt_block(b),
32 => aes::Aes256::new_from_slice(key).unwrap().encrypt_block(b),
n => panic!("no {n}-byte AES in the test encryptor"),
}
}
fn pad(data: &[u8]) -> Vec<u8> {
let mut v = data.to_vec();
v.resize(data.len().div_ceil(16) * 16, 0);
v
}
fn encrypt_ecb(key: &[u8], data: &[u8]) -> Vec<u8> {
let mut v = pad(data);
v.as_chunks_mut::<16>().0.iter_mut().for_each(|b| encrypt_block(key, b));
v
}
fn encrypt_cbc(key: &[u8], iv: &[u8], data: &[u8]) -> Vec<u8> {
let mut v = pad(data);
let mut prev: [u8; 16] = iv.try_into().unwrap();
for b in v.as_chunks_mut::<16>().0 {
b.iter_mut().zip(prev).for_each(|(x, p)| *x ^= p);
encrypt_block(key, b);
prev = *b;
}
v
}
fn base64_encode(b: &[u8]) -> String {
const T: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
let mut s = String::new();
for c in b.chunks(3) {
let n = (u32::from(c[0]) << 16) | (u32::from(*c.get(1).unwrap_or(&0)) << 8) | u32::from(*c.get(2).unwrap_or(&0));
for i in 0..4 {
s.push(if i <= c.len() { T[(n >> (18 - 6 * i)) as usize & 63] as char } else { '=' });
}
}
s
}
fn package(size: usize, body: Vec<u8>) -> Vec<u8> {
let mut p = (size as u64).to_le_bytes().to_vec();
p.extend(body);
p
}
/// The Agile XML descriptor: AES-256, SHA-512, a password encryptor salted with 0..16, and the
/// dataIntegrity pair (`hmac_key_b64`/`hmac_value_b64`) a compliant file carries.
pub(crate) fn agile_info(password: &str, spin: u32, hmac_key_b64: &str, hmac_value_b64: &str) -> String {
let key = Params { hash: Hash::Sha512, salt: PASSWORD_SALT.to_vec(), key_len: 32 };
let h = iterated_hash(Hash::Sha512, password, &key.salt, spin);
let iv = key.iv(None);
let field = |block_key: &[u8], data: &[u8]| base64_encode(&encrypt_cbc(&derive(Hash::Sha512, &h, block_key, 32), &iv, data));
let input = field(&BLOCK_VERIFIER_INPUT, &VERIFIER);
let value = field(&BLOCK_VERIFIER_HASH, &Hash::Sha512.digest(&[&VERIFIER]));
let secret = field(&BLOCK_KEY, &SECRET_KEY);
let params = |salt: &[u8]| {
format!(
"saltSize=\"16\" blockSize=\"16\" keyBits=\"256\" hashSize=\"64\" cipherAlgorithm=\"AES\" cipherChaining=\"ChainingModeCBC\" hashAlgorithm=\"SHA512\" saltValue=\"{}\"",
base64_encode(salt)
)
};
// The dataIntegrity pair ([MS-OFFCRYPTO] 2.3.4.14), encrypted with the package key.
format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\r\n<encryption xmlns=\"http://schemas.microsoft.com/office/2006/encryption\" xmlns:p=\"http://schemas.microsoft.com/office/2006/keyEncryptor/password\"><keyData {}/><dataIntegrity encryptedHmacKey=\"{hmac_key_b64}\" encryptedHmacValue=\"{hmac_value_b64}\"/><keyEncryptors><keyEncryptor uri=\"{PASSWORD_ENCRYPTOR}\"><p:encryptedKey spinCount=\"{spin}\" {} encryptedVerifierHashInput=\"{input}\" encryptedVerifierHashValue=\"{value}\" encryptedKeyValue=\"{secret}\"/></keyEncryptor></keyEncryptors></encryption>",
params(KEY_DATA_SALT.as_slice()),
params(&PASSWORD_SALT),
)
}
/// `EncryptionInfo` for an Agile descriptor: version 4.4, reserved 0x40, then the XML.
pub(crate) fn with_agile_header(xml: &str) -> Vec<u8> {
let mut info = vec![4, 0, 4, 0, 0x40, 0, 0, 0];
info.extend_from_slice(xml.as_bytes());
info
}
/// `EncryptionInfo` and `EncryptedPackage` for `plain`, Agile (AES-256, SHA-512), with the
/// dataIntegrity HMAC over the whole EncryptedPackage stream as Excel writes it.
pub(crate) fn encrypt_agile(plain: &[u8], password: &str, spin: u32) -> (Vec<u8>, Vec<u8>) {
encrypt_agile_with_hmac_key(plain, password, spin, &HMAC_KEY)
}
/// [`encrypt_agile`] with the dataIntegrity HMAC keyed by `hmac_key`.
pub(crate) fn encrypt_agile_with_hmac_key(plain: &[u8], password: &str, spin: u32, hmac_key: &[u8]) -> (Vec<u8>, Vec<u8>) {
let key_data = Params { hash: Hash::Sha512, salt: KEY_DATA_SALT.to_vec(), key_len: 32 };
let mut body = vec![];
for (i, segment) in plain.chunks(SEGMENT).enumerate() {
body.extend(encrypt_cbc(&SECRET_KEY, &key_data.iv(Some(&(i as u32).to_le_bytes())), segment));
}
let stream = package(plain.len(), body);
let (hmac_key_b64, hmac_value_b64) = (
base64_encode(&encrypt_cbc(&SECRET_KEY, &key_data.iv(Some(&BLOCK_HMAC_KEY)), hmac_key)),
base64_encode(&encrypt_cbc(&SECRET_KEY, &key_data.iv(Some(&BLOCK_HMAC_VALUE)), &hmac(Hash::Sha512, hmac_key, &stream))),
);
(with_agile_header(&agile_info(password, spin, &hmac_key_b64, &hmac_value_b64)), stream)
}
/// `EncryptionInfo` and `EncryptedPackage` for `plain`, Standard (AES-128, SHA-1).
pub(crate) fn encrypt_standard(plain: &[u8], password: &str) -> (Vec<u8>, Vec<u8>) {
let key = standard_key(password, &PASSWORD_SALT, 16);
let mut header = vec![];
for v in [F_CRYPTOAPI | F_AES, 0, 0x660E, 0x8004, 128, 0x18, 0, 0] {
header.extend_from_slice(&u32::to_le_bytes(v));
}
header.extend("\0".encode_utf16().flat_map(u16::to_le_bytes));
let mut info = vec![4, 0, 2, 0];
info.extend_from_slice(&(F_CRYPTOAPI | F_AES).to_le_bytes());
info.extend_from_slice(&(header.len() as u32).to_le_bytes());
info.extend(header);
info.extend_from_slice(&16u32.to_le_bytes());
info.extend(PASSWORD_SALT);
info.extend(encrypt_ecb(&key, &VERIFIER));
info.extend_from_slice(&20u32.to_le_bytes());
info.extend(encrypt_ecb(&key, &Hash::Sha1.digest(&[&VERIFIER])));
(info, package(plain.len(), encrypt_ecb(&key, plain)))
craft_cfb::write_streams(craft_cfb::Version::V3, streams).unwrap()
}
+6 -32
View File
@@ -1,8 +1,8 @@
use gridcraft_core::{CellRef, DateSystem, MAX_COLS, Value};
use gridcraft_model::{Sheet, Workbook};
use crate::csv::{decode_text, parse_records, sniff_delimiter};
use crate::{CsvOptions, Format, IoError, read_csv, read_csv_with_report, sniff, write_csv, write_csv_limited};
use craft_csv::{decode_text, parse_records, sniff_delimiter};
fn v(wb: &Workbook, r: u32, c: u32) -> Value {
wb.sheet(0).unwrap().value(CellRef::new(r, c))
@@ -148,33 +148,6 @@ fn encodings() {
assert_eq!(v(&wb, 0, 1), Value::Number(1.0));
}
#[test]
fn windows_1256_table_matches_the_unicode_mapping() {
let at = |b: u8| crate::csv::CP1256.get(usize::from(b - 0x80)).and_then(|&c| char::from_u32(c));
let expected = [
(0x80, '€'),
(0x81, 'پ'),
(0x8D, 'چ'),
(0x98, 'ک'),
(0x9D, '\u{200C}'),
(0xA1, '،'),
(0xBA, '؛'),
(0xBF, '؟'),
(0xC7, 'ا'),
(0xC8, 'ب'),
(0xDC, 'ـ'),
(0xE1, 'ل'),
(0xE3, 'م'),
(0xE9, 'é'),
(0xFD, '\u{200E}'),
(0xFE, '\u{200F}'),
(0xFF, 'ے'),
];
for (b, c) in expected {
assert_eq!(at(b), Some(c), "0x{b:02X}");
}
}
#[test]
fn arabic_csv_in_windows_1256_reads_as_arabic() {
let bytes = [
@@ -302,8 +275,9 @@ fn sniffing_formats() {
assert_eq!(sniff(b"a,b\n1,2"), Format::Csv);
// An Excel password-protected file is a CFB/OLE2 compound file with an EncryptedPackage
// stream; any other compound file is a legacy binary (.xls), not an encrypted workbook.
assert_eq!(sniff(&crate::cfb::tests::build(&["EncryptionInfo", "EncryptedPackage"])), Format::Encrypted);
assert_eq!(sniff(&crate::cfb::tests::build(&["Workbook"])), Format::LegacyBinary);
let compound = crate::testing::compound_file;
assert_eq!(sniff(&compound(&[("EncryptionInfo", b"x"), ("EncryptedPackage", b"x")])), Format::Encrypted);
assert_eq!(sniff(&compound(&[("WordDocument", b"x")])), Format::LegacyBinary);
assert_eq!(sniff(&[0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1, 0, 0]), Format::LegacyBinary);
assert_eq!(sniff(&[0xD0, 0xCF, 0x11, 0xE0, 0, 0, 1, 2]), Format::Unknown);
assert_eq!(sniff(b""), Format::Unknown);
@@ -393,14 +367,14 @@ fn parser_stops_collecting_fields_at_max_cols() {
let mut text = ",".repeat((MAX_COLS + extra) as usize - 1);
text.push_str("end\nx,y\n");
let mut lens = vec![];
parse_records(&text, ',', '"', |rec| {
parse_records(&text, ',', '"', MAX_COLS as usize, |rec| {
lens.push(rec.len());
true
});
assert_eq!(lens, vec![MAX_COLS as usize, 2]);
// An unterminated last row is capped the same way.
let mut lens = vec![];
parse_records(&",".repeat(MAX_COLS as usize + extra as usize), ',', '"', |rec| {
parse_records(&",".repeat(MAX_COLS as usize + extra as usize), ',', '"', MAX_COLS as usize, |rec| {
lens.push(rec.len());
true
});
+8 -18
View File
@@ -3,24 +3,11 @@
use crate::testing::{compound_file, encrypt_workbook, encrypt_workbook_standard};
use crate::{DecryptError, decrypt};
#[test]
fn the_test_compound_writer_reads_back() {
let big: Vec<u8> = (0..3_000_000u32).map(|i| (i % 253) as u8).collect();
let mid: Vec<u8> = (0..4095u32).map(|i| i as u8).collect();
let file = compound_file(&[("Small", b"hello"), ("Big", &big), ("Empty", b""), ("Mid", &mid)]);
let cf = crate::cfb::Compound::open(&file).unwrap();
assert_eq!(cf.root_stream("Mid").unwrap().unwrap(), mid, "just under the cutoff: the mini stream");
assert_eq!(cf.root_stream("Small").unwrap().unwrap(), b"hello");
assert_eq!(cf.root_stream("big").unwrap().unwrap(), big);
assert_eq!(cf.root_stream("Empty").unwrap().unwrap(), b"");
assert_eq!(cf.root_stream("Missing").unwrap(), None);
}
#[test]
fn encrypted_files_decrypt_to_the_package() {
let package: Vec<u8> = (0..3_000_000u32).map(|i| (i * 31 % 251) as u8).collect();
let agile = encrypt_workbook(&package, "Pass1234");
assert!(crate::cfb::is_encrypted(&agile));
assert!(crate::crypto::is_encrypted(&agile));
assert_eq!(decrypt(&agile, "Pass1234").unwrap(), package);
assert_eq!(decrypt(&agile, "wrong"), Err(DecryptError::WrongPassword));
let standard = encrypt_workbook_standard(&package, "Pass1234");
@@ -49,16 +36,19 @@ fn a_gridcraft_workbook_survives_encryption() {
#[test]
fn schemes_no_password_opens_are_named_before_asking_for_one() {
use crate::check_encryption;
use crate::testing::{agile_info, with_agile_header};
let unsupported = |s: &str| Err(DecryptError::Unsupported(s.into()));
assert_eq!(check_encryption(&encrypt_workbook(b"package", "pw")), Ok(()));
assert_eq!(check_encryption(&encrypt_workbook_standard(b"package", "pw")), Ok(()));
let file = |info: &[u8]| compound_file(&[("EncryptionInfo", info), ("EncryptedPackage", &[0; 8])]);
assert_eq!(check_encryption(&file(&[3, 0, 2, 0, 4, 0, 0, 0])), unsupported("RC4"));
assert_eq!(check_encryption(&file(&[4, 0, 3, 0, 0, 0, 0, 0])), unsupported("Extensible"));
let cert =
agile_info("pw", 1000, "AAAAAAAAAAAAAAAAAAAAAA==", "AAAAAAAAAAAAAAAAAAAAAA==").replace("keyEncryptor/password", "keyEncryptor/certificate");
assert_eq!(check_encryption(&file(&with_agile_header(&cert))), unsupported("certificate"));
// An Agile descriptor whose only key encryptor is a certificate.
let agile = encrypt_workbook(b"package", "pw");
let cf = craft_cfb::CompoundFile::open(&agile).unwrap();
let info = cf.read(cf.child(0, "EncryptionInfo").unwrap()).unwrap();
let mut cert = info[..8].to_vec();
cert.extend(String::from_utf8(info[8..].to_vec()).unwrap().replace("keyEncryptor/password", "keyEncryptor/certificate").into_bytes());
assert_eq!(check_encryption(&file(&cert)), unsupported("certificate"));
assert!(matches!(check_encryption(&file(&[4, 0])), Err(DecryptError::Malformed(_))));
// Rights management (IRM) keeps an EncryptedPackage but no EncryptionInfo.
let irm = compound_file(&[("EncryptedPackage", &[0; 8])]);
+58 -12
View File
@@ -392,12 +392,12 @@ fn text_result_of_a_formula_kept_as_its_value() {
#[test]
fn refuses_encrypted_old_and_foreign_files() {
// FILEPASS in the globals: an encrypted workbook.
// FILEPASS in the globals: a damaged encryption record must be rejected as such.
let mut stream = bof(0x0005);
stream.extend(rec(0x002F, &[1, 0, 1, 0]));
stream.extend(eof());
let err = read_xls(&compound("Workbook", &stream)).unwrap_err().to_string();
assert!(err.contains("password-protected"), "{err}");
assert!(err.contains("encryption record is damaged"), "{err}");
// A BIFF5 (Excel 5.0/95) workbook keeps its data in a `Book` stream.
let book = compound("Book", &stream);
assert_eq!(sniff(&book), Format::Xls);
@@ -484,6 +484,45 @@ fn dialog_sheets_are_skipped_not_read_as_worksheets() {
assert!(report.warnings.iter().any(|w| w.contains("Dialog sheets")), "{:?}", report.warnings);
}
/// A workbook opens on the sheet it was saved on (WINDOW1's `itabCur`), as Excel does: counted
/// over all the file's sheets, skipped ones included. A hidden or skipped sheet, or no window
/// record, falls back to the first visible sheet.
#[test]
fn opens_on_the_sheet_it_was_saved_on() {
// BoundSheet: position, visibility (0 visible, 1 hidden), kind, name.
let boundsheet = |pos: u32, state: u8, name: &str| rec(0x0085, &cat(&[&pos.to_le_bytes(), &[state, 0, name.len() as u8, 0], name.as_bytes()]));
let window1 = |tab: u16| rec(0x003D, &u16s(&[0, 0, 0x4000, 0x2000, 0x0038, tab, 0, 1, 600]));
let dialog = cat(&[&bof(0x0010), &rec(0x0081, &[0x10, 0x00]), &eof()]);
let worksheet = |n: f64| cat(&[&bof(0x0010), &rec(0x0081, &[0x01, 0x04]), &rec(0x0203, &cat(&[&cell(0, 0, 0), &n.to_le_bytes()])), &eof()]);
// Dialog1 (skipped), Sheet1, Sheet2, Hidden (hidden).
let book = |window: &[u8]| {
let globals = |p: [u32; 4]| {
cat(&[
&bof(0x0005),
window,
&boundsheet(p[0], 0, "Dialog1"),
&boundsheet(p[1], 0, "Sheet1"),
&boundsheet(p[2], 0, "Sheet2"),
&boundsheet(p[3], 1, "Hidden"),
&eof(),
])
};
let (d, s1, s2, h) = (dialog.clone(), worksheet(1.0), worksheet(2.0), worksheet(3.0));
let base = globals([0; 4]).len() as u32;
let p = [base, base + d.len() as u32, base + (d.len() + s1.len()) as u32, base + (d.len() + s1.len() + s2.len()) as u32];
let stream = cat(&[&globals(p), &d, &s1, &s2, &h]);
let (wb, _) = read_xls(&compound("Workbook", &stream)).unwrap();
wb.active().map(|s| s.name.clone()).unwrap_or_default()
};
assert_eq!(book(&window1(2)), "Sheet2", "itabCur 2 is Sheet2, counting the skipped dialog sheet");
assert_eq!(book(&window1(1)), "Sheet1");
assert_eq!(book(&window1(3)), "Sheet1", "a hidden sheet falls back to the first visible one");
assert_eq!(book(&window1(0)), "Sheet1", "a skipped sheet falls back to the first visible one");
assert_eq!(book(&window1(99)), "Sheet1", "an index past the sheets falls back too");
assert_eq!(book(&[]), "Sheet1", "no window record");
assert_eq!(book(&rec(0x003D, &u16s(&[0, 0, 0, 0, 0]))), "Sheet1", "a window record too short for the index");
}
#[test]
fn sheets_share_one_record_budget() {
// Many BoundSheet records point into one long substream (at successive nested BOFs), so each
@@ -509,10 +548,13 @@ fn sheets_share_one_record_budget() {
let file = compound("Workbook", &stream);
// About n × 2,000 records in all: over a 100,000 shared allowance, though each sheet alone fits.
assert!(matches!(
crate::xls::read_xls_budgeted(&file, crate::xls::Budget::with_max_records(100_000), crate::FunctionInfo::default()),
crate::xls::read_xls_budgeted(&file, craft_xls::Limits { max_records: 100_000, ..Default::default() }, crate::FunctionInfo::default()),
Err(IoError::TooLarge(_))
));
assert!(crate::xls::read_xls_budgeted(&file, crate::xls::Budget::with_max_records(10_000_000), crate::FunctionInfo::default()).is_ok());
assert!(
crate::xls::read_xls_budgeted(&file, craft_xls::Limits { max_records: 10_000_000, ..Default::default() }, crate::FunctionInfo::default())
.is_ok()
);
// Two sheets at the same offset are refused outright.
let mut dup = bof(0x0005);
@@ -768,15 +810,17 @@ fn xls_short_window2_is_ignored() {
fn shared_formula_text_counts_against_the_text_allowance() {
let file = compound("Workbook", &sample(false));
// The smallest allowance that reads every string: nothing is left for expanded formulas.
let left = (0..1000).find(|n| crate::xls::read_xls_budgeted(&file, crate::xls::Budget::with_text_left(*n), Default::default()).is_ok()).unwrap();
let (wb, report) = crate::xls::read_xls_budgeted(&file, crate::xls::Budget::with_text_left(left), Default::default()).unwrap();
let left = (0..1000)
.find(|n| crate::xls::read_xls_budgeted(&file, craft_xls::Limits { max_text: *n, ..Default::default() }, Default::default()).is_ok())
.unwrap();
let (wb, report) = crate::xls::read_xls_budgeted(&file, craft_xls::Limits { max_text: left, ..Default::default() }, Default::default()).unwrap();
assert_eq!(ftext(&wb, "B6"), None);
assert_eq!(v(&wb, 0, "B6"), Value::Number(20.0));
assert!(report.warnings.iter().any(|w| w.contains("shared-formula text")), "{:?}", report.warnings);
// an array anchor after the cut-off still gets its formula
assert!(ftext(&wb, "C5").is_some());
// With room, the followers keep their formulas.
let (wb, _) = crate::xls::read_xls_budgeted(&file, crate::xls::Budget::default(), Default::default()).unwrap();
let (wb, _) = crate::xls::read_xls_budgeted(&file, Default::default(), Default::default()).unwrap();
assert!(ftext(&wb, "B6").is_some());
}
@@ -798,12 +842,14 @@ fn shared_formulas_do_not_starve_a_later_sheets_text() {
let p1 = globals([0; 2]).len() as u32;
let file = compound("Workbook", &cat(&[&globals([p1, p1 + one.len() as u32]), &one, &two]));
// The smallest allowance that reads every string leaves nothing for expanded formulas.
let left = (0..1000).find(|n| crate::xls::read_xls_budgeted(&file, crate::xls::Budget::with_text_left(*n), Default::default()).is_ok()).unwrap();
let (wb, _) = crate::xls::read_xls_budgeted(&file, crate::xls::Budget::with_text_left(left), Default::default()).unwrap();
let left = (0..1000)
.find(|n| crate::xls::read_xls_budgeted(&file, craft_xls::Limits { max_text: *n, ..Default::default() }, Default::default()).is_ok())
.unwrap();
let (wb, _) = crate::xls::read_xls_budgeted(&file, craft_xls::Limits { max_text: left, ..Default::default() }, Default::default()).unwrap();
assert_eq!(v(&wb, 1, "A1"), Value::text("abcdefgh"));
assert_eq!(ftext(&wb, "B6"), None);
assert_eq!(v(&wb, 0, "B6"), Value::Number(20.0));
let (wb, _) = crate::xls::read_xls_budgeted(&file, crate::xls::Budget::default(), Default::default()).unwrap();
let (wb, _) = crate::xls::read_xls_budgeted(&file, Default::default(), Default::default()).unwrap();
assert!(ftext(&wb, "B6").is_some());
}
@@ -826,7 +872,7 @@ fn xls_duplicate_sheet_names_are_refused() {
g.extend(&sheet);
compound("Workbook", &g)
};
let err = crate::xls::read_xls_budgeted(&file("One", "one"), crate::xls::Budget::default(), Default::default()).unwrap_err().to_string();
let err = crate::xls::read_xls_budgeted(&file("One", "one"), Default::default(), Default::default()).unwrap_err().to_string();
assert!(err.contains("already taken"), "{err}");
assert!(crate::xls::read_xls_budgeted(&file("One", "Two"), crate::xls::Budget::default(), Default::default()).is_ok());
assert!(crate::xls::read_xls_budgeted(&file("One", "Two"), Default::default(), Default::default()).is_ok());
}
+328 -896
View File
File diff suppressed because it is too large Load Diff
-551
View File
@@ -1,551 +0,0 @@
//! BIFF8 parsed formulas ([MS-XLS] 2.2.2, 2.5.198): reverse-Polish `Ptg` tokens back to an
//! expression tree. Anything outside the common token set is reported as unsupported, and the
//! caller keeps the cell's cached value instead.
use std::sync::Arc;
use gridcraft_core::{CellError, CellRef};
use gridcraft_formula::{Anchor, BinOp, Expr, RefKind, Reference, SheetSel, UnOp};
use super::ftab::FTAB;
/// Formula token bytes decoded at most (Excel's limit is 1,800 per formula).
const MAX_RGCE: usize = 16 * 1024;
/// Values in one array constant at most.
const MAX_ARRAY: usize = 65_536;
/// Why a formula could not be turned back into text.
#[derive(Debug, PartialEq)]
pub(super) struct Unsupported(pub &'static str);
/// Where the formula lives: refs in cell formulas store coordinates; shared formulas store
/// row/column offsets for relative parts; names store offsets for 3-D refs too.
#[derive(Clone, Copy, PartialEq)]
pub(super) enum Mode {
Cell,
Shared,
Name,
}
/// A supporting link ([MS-XLS] 2.4.271): only links to this workbook (and add-in function
/// names) are resolved; external workbooks are never followed.
#[derive(Default)]
pub(super) struct SupBook {
pub internal: bool,
pub addin: bool,
pub names: Vec<String>,
}
/// XTI ([MS-XLS] 2.5.344): a supporting link and a range of its sheets.
#[derive(Clone, Copy)]
pub(super) struct Xti {
pub book: u16,
pub first: i16,
pub last: i16,
}
/// Workbook-level tables formulas refer to.
#[derive(Default)]
pub(super) struct Links {
/// Every sheet's name in BoundSheet8 order (including skipped chart/macro sheets).
pub sheets: Vec<String>,
/// Defined names (Lbl records), in order; `None` for names that can't be referenced.
pub names: Vec<String>,
/// Per defined name: whether the import skips it (its formula can't be decoded).
pub skipped: Vec<bool>,
pub books: Vec<SupBook>,
pub xti: Vec<Xti>,
}
pub(super) fn function(iftab: u16) -> Option<(&'static str, i8)> {
FTAB.binary_search_by_key(&iftab, |(i, _, _)| *i).ok().and_then(|i| FTAB.get(i)).map(|(_, n, a)| (*n, *a))
}
fn error_code(code: u8) -> Option<CellError> {
Some(match code {
0x00 => CellError::Null,
0x07 => CellError::Div0,
0x0F => CellError::Value,
0x17 => CellError::Ref,
0x1D => CellError::Name,
0x24 => CellError::Num,
0x2A => CellError::NA,
0x2B => CellError::GettingData,
_ => return None,
})
}
pub(super) fn cell_error(code: u8) -> CellError {
error_code(code).unwrap_or(CellError::NA)
}
struct Bytes<'a> {
b: &'a [u8],
pos: usize,
}
const TRUNC: Unsupported = Unsupported("truncated formula");
impl Bytes<'_> {
fn take(&mut self, n: usize) -> Result<&[u8], Unsupported> {
let s = self.b.get(self.pos..self.pos.checked_add(n).ok_or(TRUNC)?).ok_or(TRUNC)?;
self.pos += n;
Ok(s)
}
fn u8(&mut self) -> Result<u8, Unsupported> {
self.take(1)?.first().copied().ok_or(TRUNC)
}
fn u16(&mut self) -> Result<u16, Unsupported> {
let s = self.take(2)?;
Ok(u16::from_le_bytes([*s.first().ok_or(TRUNC)?, *s.get(1).ok_or(TRUNC)?]))
}
fn u32(&mut self) -> Result<u32, Unsupported> {
Ok(u32::from(self.u16()?) | (u32::from(self.u16()?) << 16))
}
fn f64(&mut self) -> Result<f64, Unsupported> {
let a: [u8; 8] = self.take(8)?.try_into().map_err(|_| TRUNC)?;
Ok(f64::from_le_bytes(a))
}
fn chars(&mut self, cch: usize, high: bool) -> Result<String, Unsupported> {
let units: Vec<u16> = if high {
self.take(cch.checked_mul(2).ok_or(TRUNC)?)?.as_chunks::<2>().0.iter().map(|c| u16::from_le_bytes(*c)).collect()
} else {
self.take(cch)?.iter().map(|&b| u16::from(b)).collect()
};
Ok(String::from_utf16_lossy(&units))
}
}
/// A column field (ColRelU / ColRelNegU): the 14-bit column and the two relative flags.
fn col_field(v: u16) -> (u16, bool, bool) {
(v & 0x3FFF, v & 0x4000 != 0, v & 0x8000 != 0)
}
/// One end of a reference. In `relative` (RgceLocRel) form the relative parts are offsets
/// from `base`, wrapped into the BIFF8 grid (65,536 × 256).
fn anchor(row: u16, col: u16, base: CellRef, relative: bool) -> Anchor {
let (c, col_rel, row_rel) = col_field(col);
let (r, c) = if relative {
let r = if row_rel { (u32::from(row).wrapping_add(base.row)) & 0xFFFF } else { u32::from(row) };
// A 14-bit signed offset; adding modulo 256 gives the same column either way.
let c = if col_rel { (u32::from(c).wrapping_add(base.col)) & 0xFF } else { u32::from(c) & 0xFF };
(r, c)
} else {
(u32::from(row), u32::from(c) & 0xFF)
};
Anchor { row: r, col: c, row_abs: !row_rel, col_abs: !col_rel }
}
/// An area; a range spanning every BIFF8 row (or column) becomes a whole-column (row) ref.
fn area(r0: u16, r1: u16, c0: u16, c1: u16, base: CellRef, relative: bool) -> RefKind {
let a = anchor(r0, c0, base, relative);
let b = anchor(r1, c1, base, relative);
if a.row == 0 && b.row == 0xFFFF {
return RefKind::Cols(a.col, a.col_abs, b.col, b.col_abs);
}
if a.col == 0 && b.col == 0xFF {
return RefKind::Rows(a.row, a.row_abs, b.row, b.row_abs);
}
RefKind::Range(a, b)
}
impl Links {
fn sheet_sel(&self, ixti: u16) -> Result<Option<SheetSel>, Unsupported> {
let x = self.xti.get(usize::from(ixti)).ok_or(Unsupported("reference to a missing sheet link"))?;
let book = self.books.get(usize::from(x.book)).ok_or(Unsupported("reference to a missing workbook link"))?;
if !book.internal {
return Err(Unsupported("references to other workbooks (external links)"));
}
let name = |i: i16| usize::try_from(i).ok().and_then(|i| self.sheets.get(i)).cloned();
Ok(match (name(x.first), name(x.last)) {
(Some(a), Some(_)) if x.first == x.last => Some(SheetSel::Named(a)),
(Some(a), Some(b)) => Some(SheetSel::Span(a, b)),
// A deleted sheet: the reference is #REF!.
_ => None,
})
}
fn extern_name(&self, ixti: u16, index: u32) -> Result<String, Unsupported> {
let x = self.xti.get(usize::from(ixti)).ok_or(Unsupported("reference to a missing name link"))?;
let book = self.books.get(usize::from(x.book)).ok_or(Unsupported("reference to a missing workbook link"))?;
if !book.internal && !book.addin {
return Err(Unsupported("names in other workbooks (external links)"));
}
let i = usize::try_from(index).ok().and_then(|i| i.checked_sub(1)).ok_or(Unsupported("invalid name index"))?;
book.names.get(i).cloned().ok_or(Unsupported("reference to a missing name"))
}
}
/// Strips the prefixes newer functions carry in files (`_xlfn.IFERROR`).
fn function_name(name: &str) -> String {
let n = name.strip_prefix("_xlfn.").or_else(|| name.strip_prefix("_xlws.")).unwrap_or(name);
n.to_ascii_uppercase()
}
/// Expression nesting at most, as in the formula parser: deeper trees would overflow the stack
/// when printed, evaluated or dropped (a hostile file can chain thousands of PtgParen tokens).
const MAX_DEPTH: usize = 256;
/// Operand stack entries: an expression and its nesting depth.
type Stack = Vec<(Expr, usize)>;
/// Pops an operand, recording its depth in `kids` (the deepest operand this token consumed).
fn pop(stack: &mut Stack, kids: &mut usize) -> Result<Expr, Unsupported> {
let (e, d) = stack.pop().ok_or(Unsupported("malformed formula (operand missing)"))?;
*kids = (*kids).max(d);
Ok(e)
}
fn pop_n(stack: &mut Stack, n: usize, kids: &mut usize) -> Result<Vec<Expr>, Unsupported> {
let at = stack.len().checked_sub(n).ok_or(Unsupported("malformed formula (argument missing)"))?;
let popped = stack.split_off(at);
*kids = popped.iter().map(|(_, d)| *d).fold(*kids, usize::max);
Ok(popped.into_iter().map(|(e, _)| e).collect())
}
/// Pushes the result of a token one level above the operands it consumed.
fn push(stack: &mut Stack, kids: usize, e: Expr) -> Result<(), Unsupported> {
let depth = kids.saturating_add(1);
if depth > MAX_DEPTH {
return Err(Unsupported("formula nested too deeply"));
}
stack.push((e, depth));
Ok(())
}
/// Decodes `rgce` (tokens) with `rgcb` (the extra data for array constants) evaluated at
/// `base`. Returns the expression or why it can't be represented.
pub(super) fn decode(rgce: &[u8], rgcb: &[u8], base: CellRef, mode: Mode, links: &Links) -> Result<Expr, Unsupported> {
if rgce.len() > MAX_RGCE {
return Err(Unsupported("formula too long"));
}
let mut t = Bytes { b: rgce, pos: 0 };
let mut extra = Bytes { b: rgcb, pos: 0 };
let mut stack: Stack = Vec::new();
while t.pos < rgce.len() {
let ptg = t.u8()?;
let mut kids = 0usize;
// Operand classes (reference / value / array) share one token number.
let id = if ptg & 0x60 != 0 { 0x20 | (ptg & 0x1F) } else { ptg };
match id {
0x03..=0x11 => {
let op = match id {
0x03 => BinOp::Add,
0x04 => BinOp::Sub,
0x05 => BinOp::Mul,
0x06 => BinOp::Div,
0x07 => BinOp::Pow,
0x08 => BinOp::Concat,
0x09 => BinOp::Lt,
0x0A => BinOp::Le,
0x0B => BinOp::Eq,
0x0C => BinOp::Ge,
0x0D => BinOp::Gt,
0x0E => BinOp::Ne,
0x0F => BinOp::Intersect,
0x10 => BinOp::Union,
_ => BinOp::Range,
};
let b = pop(&mut stack, &mut kids)?;
let a = pop(&mut stack, &mut kids)?;
push(&mut stack, kids, Expr::Binary(op, Box::new(a), Box::new(b)))?;
}
0x12..=0x14 => {
let op = match id {
0x12 => UnOp::Plus,
0x13 => UnOp::Neg,
_ => UnOp::Percent,
};
let x = pop(&mut stack, &mut kids)?;
push(&mut stack, kids, Expr::Unary(op, Box::new(x)))?;
}
0x15 => {
let x = pop(&mut stack, &mut kids)?;
push(&mut stack, kids, Expr::Paren(Box::new(x)))?;
}
0x16 => push(&mut stack, kids, Expr::Missing)?,
0x17 => {
let cch = usize::from(t.u8()?);
let high = t.u8()? & 1 != 0;
push(&mut stack, kids, Expr::Text(Arc::from(t.chars(cch, high)?)))?;
}
0x19 => {
let kind = t.u8()?;
if kind & 0x04 != 0 {
// PtgAttrChoose: jump table of cOffset + 1 entries.
let n = usize::from(t.u16()?) + 1;
t.take(n * 2)?;
} else {
t.take(2)?;
}
if kind & 0x10 != 0 {
// PtgAttrSum: SUM of the single operand.
let x = pop(&mut stack, &mut kids)?;
push(&mut stack, kids, Expr::Call("SUM".into(), vec![x]))?;
}
// Semi (volatile), If, Goto, Space, Baxcel: no effect on the expression.
}
0x1C => {
let e = t.u8()?;
push(&mut stack, kids, Expr::Error(error_code(e).ok_or(Unsupported("unknown error constant"))?))?;
}
0x1D => push(&mut stack, kids, Expr::Bool(t.u8()? != 0))?,
0x1E => push(&mut stack, kids, Expr::Number(f64::from(t.u16()?)))?,
0x1F => {
let n = t.f64()?;
if !n.is_finite() {
return Err(Unsupported("invalid number constant"));
}
push(&mut stack, kids, Expr::Number(n))?;
}
0x20 => {
t.take(7)?;
push(&mut stack, kids, array_constant(&mut extra)?)?;
}
0x21 => {
let iftab = t.u16()?;
let (name, args) = function(iftab).ok_or(Unsupported("unknown function"))?;
let n = usize::try_from(args).map_err(|_| Unsupported("unknown function argument count"))?;
let args = pop_n(&mut stack, n, &mut kids)?;
push(&mut stack, kids, Expr::Call(name.to_string(), args))?;
}
0x22 => {
let n = usize::from(t.u8()?);
let tab = t.u16()?;
if tab & 0x8000 != 0 {
return Err(Unsupported("macro sheet commands"));
}
let mut args = pop_n(&mut stack, n, &mut kids)?;
if tab & 0x7FFF == 0x00FF {
// A function called by name (add-in or newer function): the name comes first.
if args.is_empty() {
return Err(Unsupported("malformed function call"));
}
let name = match args.remove(0) {
Expr::Name(n) => function_name(&n),
_ => return Err(Unsupported("function call through an expression")),
};
push(&mut stack, kids, Expr::Call(name, args))?;
} else {
let (name, _) = function(tab & 0x7FFF).ok_or(Unsupported("unknown function"))?;
push(&mut stack, kids, Expr::Call(name.to_string(), args))?;
}
}
0x23 => {
let i = usize::try_from(t.u32()?).ok().and_then(|i| i.checked_sub(1));
let name = i.and_then(|i| links.names.get(i)).filter(|n| !n.is_empty()).ok_or(Unsupported("reference to a missing defined name"))?;
// A cell using a name the import drops would calculate to #NAME?: keep its
// saved result instead, as for references to other workbooks.
if mode != Mode::Name && i.and_then(|i| links.skipped.get(i)) == Some(&true) {
return Err(Unsupported("defined names GridCraft skipped"));
}
push(&mut stack, kids, Expr::Name(name.clone()))?;
}
0x24 | 0x2C => {
let (r, c) = (t.u16()?, t.u16()?);
let a = anchor(r, c, base, id == 0x2C);
push(&mut stack, kids, Expr::Ref(Reference { sheet: SheetSel::Current, kind: RefKind::Cell(a) }))?;
}
0x25 | 0x2D => {
let (r0, r1, c0, c1) = (t.u16()?, t.u16()?, t.u16()?, t.u16()?);
let kind = area(r0, r1, c0, c1, base, id == 0x2D);
push(&mut stack, kids, Expr::Ref(Reference { sheet: SheetSel::Current, kind }))?;
}
0x26 => {
// PtgMemArea: a cache of the sub-expression that follows; skip its extra data.
t.take(6)?;
let n = usize::from(extra.u16()?);
extra.take(n.checked_mul(8).ok_or(TRUNC)?)?;
}
0x27 | 0x28 => {
t.take(6)?;
}
0x29 => {
t.take(2)?;
}
0x2A => {
t.take(4)?;
push(&mut stack, kids, Expr::Error(CellError::Ref))?;
}
0x2B => {
t.take(8)?;
push(&mut stack, kids, Expr::Error(CellError::Ref))?;
}
0x39 => {
let ixti = t.u16()?;
let i = t.u32()?;
push(&mut stack, kids, Expr::Name(links.extern_name(ixti, i)?))?;
}
0x3A => {
let ixti = t.u16()?;
let (r, c) = (t.u16()?, t.u16()?);
let a = anchor(r, c, base, mode == Mode::Name);
push(
&mut stack,
kids,
match links.sheet_sel(ixti)? {
Some(sheet) => Expr::Ref(Reference { sheet, kind: RefKind::Cell(a) }),
None => Expr::Error(CellError::Ref),
},
)?;
}
0x3B => {
let ixti = t.u16()?;
let (r0, r1, c0, c1) = (t.u16()?, t.u16()?, t.u16()?, t.u16()?);
let kind = area(r0, r1, c0, c1, base, mode == Mode::Name);
push(
&mut stack,
kids,
match links.sheet_sel(ixti)? {
Some(sheet) => Expr::Ref(Reference { sheet, kind }),
None => Expr::Error(CellError::Ref),
},
)?;
}
0x3C => {
t.take(6)?;
push(&mut stack, kids, Expr::Error(CellError::Ref))?;
}
0x3D => {
t.take(10)?;
push(&mut stack, kids, Expr::Error(CellError::Ref))?;
}
0x01 => return Err(Unsupported("nested shared or array formula reference")),
0x02 => return Err(Unsupported("what-if data tables")),
0x18 => return Err(Unsupported("natural-language formulas")),
_ => return Err(Unsupported("unknown formula token")),
}
if stack.len() > MAX_RGCE {
return Err(Unsupported("formula too complex"));
}
}
let (e, _) = stack.pop().ok_or(Unsupported("malformed formula (operand missing)"))?;
if !stack.is_empty() {
return Err(Unsupported("malformed formula (unused operands)"));
}
Ok(e)
}
/// PtgExtraArray ([MS-XLS] 2.5.198.59): an array constant's values in row-major order.
fn array_constant(b: &mut Bytes<'_>) -> Result<Expr, Unsupported> {
let cols = usize::from(b.u8()?) + 1;
let rows = usize::from(b.u16()?) + 1;
if cols.saturating_mul(rows) > MAX_ARRAY {
return Err(Unsupported("array constant too large"));
}
let mut out = Vec::with_capacity(rows);
for _ in 0..rows {
let mut row = Vec::with_capacity(cols);
for _ in 0..cols {
let kind = b.u8()?;
row.push(match kind {
0x00 => {
b.take(8)?;
Expr::Missing
}
0x01 => {
let n = b.f64()?;
if !n.is_finite() {
return Err(Unsupported("invalid number constant"));
}
Expr::Number(n)
}
0x02 => {
let cch = usize::from(b.u16()?);
let high = b.u8()? & 1 != 0;
Expr::Text(Arc::from(b.chars(cch, high)?))
}
0x04 => {
let v = b.u8()?;
b.take(7)?;
Expr::Bool(v != 0)
}
0x10 => {
let e = b.u8()?;
b.take(7)?;
Expr::Error(error_code(e).ok_or(Unsupported("unknown error constant"))?)
}
_ => return Err(Unsupported("unknown array constant value")),
});
}
out.push(row);
}
Ok(Expr::Array(out))
}
#[cfg(test)]
mod tests {
use super::*;
fn links() -> Links {
Links {
sheets: vec!["Sheet1".into(), "My Data".into()],
names: vec!["Rate".into()],
skipped: vec![false],
books: vec![
SupBook { internal: true, addin: false, names: vec![] },
SupBook { internal: false, addin: true, names: vec!["_xlfn.IFERROR".into()] },
SupBook::default(),
],
xti: vec![
Xti { book: 0, first: 0, last: 1 },
Xti { book: 1, first: -2, last: -2 },
Xti { book: 2, first: 0, last: 0 },
Xti { book: 0, first: -1, last: -1 },
],
}
}
fn text(rgce: &[u8], rgcb: &[u8]) -> Result<String, Unsupported> {
decode(rgce, rgcb, CellRef::new(0, 0), Mode::Cell, &links()).map(|e| gridcraft_formula::print(&e))
}
#[test]
fn decodes_common_tokens() {
// {1,"a";TRUE,#N/A} via PtgArray + PtgExtraArray.
let extra =
[&[1u8, 1, 0][..], &[1], &1f64.to_le_bytes(), &[2, 1, 0, 0, b'a'], &[4, 1, 0, 0, 0, 0, 0, 0, 0], &[0x10, 0x2A, 0, 0, 0, 0, 0, 0, 0]]
.concat();
assert_eq!(text(&[0x60, 0, 0, 0, 0, 0, 0, 0], &extra).unwrap(), "{1,\"a\";TRUE,#N/A}");
// SUM via PtgAttrSum over a 3-D span, whole column.
let area3d = [&[0x3Bu8][..], &0u16.to_le_bytes(), &[0, 0, 0xFF, 0xFF, 1, 0, 1, 0]].concat();
assert_eq!(text(&[area3d.as_slice(), &[0x19, 0x10, 0, 0]].concat(), &[]).unwrap(), "SUM('Sheet1:My Data'!$B:$B)");
// IFERROR (a newer function) called by name through PtgNameX, with a missing argument.
let namex = [&[0x39u8][..], &1u16.to_le_bytes(), &1u32.to_le_bytes()].concat();
let call = [namex.as_slice(), &[0x1E, 5, 0, 0x16, 0x22, 3, 0xFF, 0]].concat();
assert_eq!(text(&call, &[]).unwrap(), "IFERROR(5,)");
// A deleted sheet becomes #REF!; the CHOOSE jump table is skipped.
let ref_deleted = [&[0x3Au8][..], &3u16.to_le_bytes(), &[0, 0, 0, 0]].concat();
let choose = [&[0x1Eu8, 1, 0][..], &[0x19, 0x04, 1, 0, 0, 0, 0, 0], ref_deleted.as_slice(), &[0x1E, 2, 0, 0x22, 3, 100, 0]].concat();
assert_eq!(text(&choose, &[]).unwrap(), "CHOOSE(1,#REF!,2)");
// External links, data tables and unbalanced stacks are refused, never guessed.
let external = [&[0x3Au8][..], &2u16.to_le_bytes(), &[0, 0, 0, 0]].concat();
assert!(text(&external, &[]).is_err());
assert!(text(&[0x02, 0, 0, 0, 0], &[]).is_err());
assert!(text(&[0x1E, 1, 0, 0x1E, 2, 0], &[]).is_err());
assert!(text(&[0x03], &[]).is_err());
assert!(text(&[0x21, 0xFF, 0x7F], &[]).is_err());
assert!(text(&[0x60, 0, 0, 0, 0, 0, 0, 0], &[0xFF, 0xFF, 0xFF]).is_err());
}
#[test]
fn deep_nesting_is_refused_not_a_stack_overflow() {
// 1 followed by thousands of PtgParen / PtgUminus tokens: a hostile file used to build a
// 16K-deep tree that overflowed the stack when printed or dropped.
for op in [0x15u8, 0x13] {
let mut rgce = vec![0x1E, 1, 0];
rgce.resize(MAX_RGCE, op);
assert_eq!(text(&rgce, &[]), Err(Unsupported("formula nested too deeply")));
}
// Binary chains nest too: 1+1+1+... left-associated.
let mut chain = vec![0x1E, 1, 0];
while chain.len() + 4 <= MAX_RGCE {
chain.extend([0x1E, 1, 0, 0x03]);
}
assert!(text(&chain, &[]).is_err());
// Nesting up to the limit still decodes.
let mut ok = vec![0x1E, 1, 0];
ok.resize(3 + MAX_DEPTH - 1, 0x15);
assert!(text(&ok, &[]).is_ok());
}
}
-376
View File
@@ -1,376 +0,0 @@
// Generated from the Ftab table of [MS-XLS] section 2.5.198.17: (iftab, name, fixed argument
// count or -1 when the function takes a variable number of arguments).
pub(super) const FTAB: &[(u16, &str, i8)] = &[
(0, "COUNT", -1),
(1, "IF", -1),
(2, "ISNA", 1),
(3, "ISERROR", 1),
(4, "SUM", -1),
(5, "AVERAGE", -1),
(6, "MIN", -1),
(7, "MAX", -1),
(8, "ROW", -1),
(9, "COLUMN", -1),
(10, "NA", 0),
(11, "NPV", -1),
(12, "STDEV", -1),
(13, "DOLLAR", -1),
(14, "FIXED", -1),
(15, "SIN", 1),
(16, "COS", 1),
(17, "TAN", 1),
(18, "ATAN", 1),
(19, "PI", 0),
(20, "SQRT", 1),
(21, "EXP", 1),
(22, "LN", 1),
(23, "LOG10", 1),
(24, "ABS", 1),
(25, "INT", 1),
(26, "SIGN", 1),
(27, "ROUND", 2),
(28, "LOOKUP", -1),
(29, "INDEX", -1),
(30, "REPT", 2),
(31, "MID", 3),
(32, "LEN", 1),
(33, "VALUE", 1),
(34, "TRUE", 0),
(35, "FALSE", 0),
(36, "AND", -1),
(37, "OR", -1),
(38, "NOT", 1),
(39, "MOD", 2),
(40, "DCOUNT", 3),
(41, "DSUM", 3),
(42, "DAVERAGE", 3),
(43, "DMIN", 3),
(44, "DMAX", 3),
(45, "DSTDEV", 3),
(46, "VAR", -1),
(47, "DVAR", 3),
(48, "TEXT", 2),
(49, "LINEST", -1),
(50, "TREND", -1),
(51, "LOGEST", -1),
(52, "GROWTH", -1),
(53, "GOTO", 1),
(54, "HALT", -1),
(55, "RETURN", -1),
(56, "PV", -1),
(57, "FV", -1),
(58, "NPER", -1),
(59, "PMT", -1),
(60, "RATE", -1),
(61, "MIRR", 3),
(62, "IRR", -1),
(63, "RAND", 0),
(64, "MATCH", -1),
(65, "DATE", 3),
(66, "TIME", 3),
(67, "DAY", 1),
(68, "MONTH", 1),
(69, "YEAR", 1),
(70, "WEEKDAY", -1),
(71, "HOUR", 1),
(72, "MINUTE", 1),
(73, "SECOND", 1),
(74, "NOW", 0),
(75, "AREAS", 1),
(76, "ROWS", 1),
(77, "COLUMNS", 1),
(78, "OFFSET", -1),
(79, "ABSREF", 2),
(80, "RELREF", 2),
(81, "ARGUMENT", -1),
(82, "SEARCH", -1),
(83, "TRANSPOSE", 1),
(84, "ERROR", -1),
(85, "STEP", 0),
(86, "TYPE", 1),
(87, "ECHO", -1),
(88, "SET.NAME", -1),
(89, "CALLER", 0),
(90, "DEREF", 1),
(91, "WINDOWS", -1),
(92, "SERIES", -1),
(93, "DOCUMENTS", -1),
(94, "ACTIVE.CELL", 0),
(95, "SELECTION", 0),
(96, "RESULT", -1),
(97, "ATAN2", 2),
(98, "ASIN", 1),
(99, "ACOS", 1),
(100, "CHOOSE", -1),
(101, "HLOOKUP", -1),
(102, "VLOOKUP", -1),
(103, "LINKS", -1),
(104, "INPUT", -1),
(105, "ISREF", 1),
(106, "GET.FORMULA", 1),
(107, "GET.NAME", -1),
(108, "SET.VALUE", 2),
(109, "LOG", -1),
(110, "EXEC", -1),
(111, "CHAR", 1),
(112, "LOWER", 1),
(113, "UPPER", 1),
(114, "PROPER", 1),
(115, "LEFT", -1),
(116, "RIGHT", -1),
(117, "EXACT", 2),
(118, "TRIM", 1),
(119, "REPLACE", 4),
(120, "SUBSTITUTE", -1),
(121, "CODE", 1),
(122, "NAMES", -1),
(123, "DIRECTORY", -1),
(124, "FIND", -1),
(125, "CELL", -1),
(126, "ISERR", 1),
(127, "ISTEXT", 1),
(128, "ISNUMBER", 1),
(129, "ISBLANK", 1),
(130, "T", 1),
(131, "N", 1),
(132, "FOPEN", -1),
(133, "FCLOSE", 1),
(134, "FSIZE", 1),
(135, "FREADLN", 1),
(136, "FREAD", 2),
(137, "FWRITELN", 2),
(138, "FWRITE", 2),
(139, "FPOS", -1),
(140, "DATEVALUE", 1),
(141, "TIMEVALUE", 1),
(142, "SLN", 3),
(143, "SYD", 4),
(144, "DDB", -1),
(145, "GET.DEF", -1),
(146, "REFTEXT", -1),
(147, "TEXTREF", -1),
(148, "INDIRECT", -1),
(149, "REGISTER", -1),
(150, "CALL", -1),
(151, "ADD.BAR", -1),
(152, "ADD.MENU", -1),
(153, "ADD.COMMAND", -1),
(154, "ENABLE.COMMAND", -1),
(155, "CHECK.COMMAND", -1),
(156, "RENAME.COMMAND", -1),
(157, "SHOW.BAR", -1),
(158, "DELETE.MENU", -1),
(159, "DELETE.COMMAND", -1),
(160, "GET.CHART.ITEM", -1),
(161, "DIALOG.BOX", 1),
(162, "CLEAN", 1),
(163, "MDETERM", 1),
(164, "MINVERSE", 1),
(165, "MMULT", 2),
(166, "FILES", -1),
(167, "IPMT", -1),
(168, "PPMT", -1),
(169, "COUNTA", -1),
(170, "CANCEL.KEY", -1),
(171, "FOR", -1),
(172, "WHILE", 1),
(173, "BREAK", 0),
(174, "NEXT", 0),
(175, "INITIATE", 2),
(176, "REQUEST", 2),
(177, "POKE", 3),
(178, "EXECUTE", 2),
(179, "TERMINATE", 1),
(180, "RESTART", -1),
(181, "HELP", -1),
(182, "GET.BAR", -1),
(183, "PRODUCT", -1),
(184, "FACT", 1),
(185, "GET.CELL", -1),
(186, "GET.WORKSPACE", 1),
(187, "GET.WINDOW", -1),
(188, "GET.DOCUMENT", -1),
(189, "DPRODUCT", 3),
(190, "ISNONTEXT", 1),
(191, "GET.NOTE", -1),
(192, "NOTE", -1),
(193, "STDEVP", -1),
(194, "VARP", -1),
(195, "DSTDEVP", 3),
(196, "DVARP", 3),
(197, "TRUNC", -1),
(198, "ISLOGICAL", 1),
(199, "DCOUNTA", 3),
(200, "DELETE.BAR", 1),
(201, "UNREGISTER", 1),
(204, "USDOLLAR", -1),
(205, "FINDB", -1),
(206, "SEARCHB", -1),
(207, "REPLACEB", 4),
(208, "LEFTB", -1),
(209, "RIGHTB", -1),
(210, "MIDB", 3),
(211, "LENB", 1),
(212, "ROUNDUP", 2),
(213, "ROUNDDOWN", 2),
(214, "ASC", 1),
(215, "DBCS", 1),
(216, "RANK", -1),
(219, "ADDRESS", -1),
(220, "DAYS360", -1),
(221, "TODAY", 0),
(222, "VDB", -1),
(223, "ELSE", 0),
(224, "ELSE.IF", 1),
(225, "END.IF", 0),
(226, "FOR.CELL", -1),
(227, "MEDIAN", -1),
(228, "SUMPRODUCT", -1),
(229, "SINH", 1),
(230, "COSH", 1),
(231, "TANH", 1),
(232, "ASINH", 1),
(233, "ACOSH", 1),
(234, "ATANH", 1),
(235, "DGET", 3),
(236, "CREATE.OBJECT", -1),
(237, "VOLATILE", -1),
(238, "LAST.ERROR", 0),
(239, "CUSTOM.UNDO", -1),
(240, "CUSTOM.REPEAT", -1),
(241, "FORMULA.CONVERT", -1),
(242, "GET.LINK.INFO", -1),
(243, "TEXT.BOX", -1),
(244, "INFO", 1),
(245, "GROUP", 0),
(246, "GET.OBJECT", -1),
(247, "DB", -1),
(248, "PAUSE", -1),
(251, "RESUME", -1),
(252, "FREQUENCY", 2),
(253, "ADD.TOOLBAR", -1),
(254, "DELETE.TOOLBAR", 1),
(256, "RESET.TOOLBAR", 1),
(257, "EVALUATE", 1),
(258, "GET.TOOLBAR", -1),
(259, "GET.TOOL", -1),
(260, "SPELLING.CHECK", -1),
(261, "ERROR.TYPE", 1),
(262, "APP.TITLE", -1),
(263, "WINDOW.TITLE", -1),
(264, "SAVE.TOOLBAR", -1),
(265, "ENABLE.TOOL", 3),
(266, "PRESS.TOOL", 3),
(267, "REGISTER.ID", -1),
(268, "GET.WORKBOOK", -1),
(269, "AVEDEV", -1),
(270, "BETADIST", -1),
(271, "GAMMALN", 1),
(272, "BETAINV", -1),
(273, "BINOMDIST", 4),
(274, "CHIDIST", 2),
(275, "CHIINV", 2),
(276, "COMBIN", 2),
(277, "CONFIDENCE", 3),
(278, "CRITBINOM", 3),
(279, "EVEN", 1),
(280, "EXPONDIST", 3),
(281, "FDIST", 3),
(282, "FINV", 3),
(283, "FISHER", 1),
(284, "FISHERINV", 1),
(285, "FLOOR", 2),
(286, "GAMMADIST", 4),
(287, "GAMMAINV", 3),
(288, "CEILING", 2),
(289, "HYPGEOMDIST", 4),
(290, "LOGNORMDIST", 3),
(291, "LOGINV", 3),
(292, "NEGBINOMDIST", 3),
(293, "NORMDIST", 4),
(294, "NORMSDIST", 1),
(295, "NORMINV", 3),
(296, "NORMSINV", 1),
(297, "STANDARDIZE", 3),
(298, "ODD", 1),
(299, "PERMUT", 2),
(300, "POISSON", 3),
(301, "TDIST", 3),
(302, "WEIBULL", 4),
(303, "SUMXMY2", 2),
(304, "SUMX2MY2", 2),
(305, "SUMX2PY2", 2),
(306, "CHITEST", 2),
(307, "CORREL", 2),
(308, "COVAR", 2),
(309, "FORECAST", 3),
(310, "FTEST", 2),
(311, "INTERCEPT", 2),
(312, "PEARSON", 2),
(313, "RSQ", 2),
(314, "STEYX", 2),
(315, "SLOPE", 2),
(316, "TTEST", 4),
(317, "PROB", -1),
(318, "DEVSQ", -1),
(319, "GEOMEAN", -1),
(320, "HARMEAN", -1),
(321, "SUMSQ", -1),
(322, "KURT", -1),
(323, "SKEW", -1),
(324, "ZTEST", -1),
(325, "LARGE", 2),
(326, "SMALL", 2),
(327, "QUARTILE", 2),
(328, "PERCENTILE", 2),
(329, "PERCENTRANK", -1),
(330, "MODE", -1),
(331, "TRIMMEAN", 2),
(332, "TINV", 2),
(334, "MOVIE.COMMAND", -1),
(335, "GET.MOVIE", -1),
(336, "CONCATENATE", -1),
(337, "POWER", 2),
(338, "PIVOT.ADD.DATA", -1),
(339, "GET.PIVOT.TABLE", -1),
(340, "GET.PIVOT.FIELD", -1),
(341, "GET.PIVOT.ITEM", -1),
(342, "RADIANS", 1),
(343, "DEGREES", 1),
(344, "SUBTOTAL", -1),
(345, "SUMIF", -1),
(346, "COUNTIF", 2),
(347, "COUNTBLANK", 1),
(348, "SCENARIO.GET", -1),
(349, "OPTIONS.LISTS.GET", 1),
(350, "ISPMT", 4),
(351, "DATEDIF", 3),
(352, "DATESTRING", 1),
(353, "NUMBERSTRING", 2),
(354, "ROMAN", -1),
(355, "OPEN.DIALOG", -1),
(356, "SAVE.DIALOG", -1),
(357, "VIEW.GET", -1),
(358, "GETPIVOTDATA", -1),
(359, "HYPERLINK", -1),
(360, "PHONETIC", 1),
(361, "AVERAGEA", -1),
(362, "MAXA", -1),
(363, "MINA", -1),
(364, "STDEVPA", -1),
(365, "VARPA", -1),
(366, "STDEVA", -1),
(367, "VARA", -1),
(368, "BAHTTEXT", 1),
(369, "THAIDAYOFWEEK", 1),
(370, "THAIDIGIT", 1),
(371, "THAIMONTHOFYEAR", 1),
(372, "THAINUMSOUND", 1),
(373, "THAINUMSTRING", 1),
(374, "THAISTRINGLENGTH", 1),
(375, "ISTHAIDIGIT", 1),
(376, "ROUNDBAHTDOWN", 1),
(377, "ROUNDBAHTUP", 1),
(378, "THAIYEAR", 1),
(379, "RTD", -1),
];
-223
View File
@@ -1,223 +0,0 @@
//! BIFF8 record framing ([MS-XLS] 2.1.4) and the string structures ([MS-XLS] 2.5.240,
//! 2.5.293, 2.5.294) that may continue across `Continue` records.
use super::invalid;
use crate::IoError;
/// `Continue` record type.
pub(super) const CONTINUE: u16 = 0x003C;
/// Characters in one string at most (Excel's cell limit is 32,767; the SST may hold longer).
const MAX_CHARS: usize = 1 << 20;
/// `Continue` records gathered for one record at most.
const MAX_PARTS: usize = 1 << 16;
/// One record and the payloads of the `Continue` records that follow it.
pub(super) struct Record<'a> {
pub id: u16,
pub parts: Vec<&'a [u8]>,
}
/// Iterates over the records of a stream starting at `pos`.
pub(super) struct Records<'a> {
stream: &'a [u8],
pos: usize,
}
impl<'a> Records<'a> {
pub fn new(stream: &'a [u8], pos: usize) -> Self {
Records { stream, pos }
}
fn raw(&mut self) -> Result<Option<(u16, &'a [u8])>, IoError> {
if self.pos >= self.stream.len() {
return Ok(None);
}
let head = self.stream.get(self.pos..self.pos.saturating_add(4)).ok_or_else(|| invalid("truncated record header"))?;
let id = u16::from_le_bytes([head.first().copied().unwrap_or(0), head.get(1).copied().unwrap_or(0)]);
let len = usize::from(u16::from_le_bytes([head.get(2).copied().unwrap_or(0), head.get(3).copied().unwrap_or(0)]));
let start = self.pos + 4;
let data = self.stream.get(start..start.saturating_add(len)).ok_or_else(|| invalid("truncated record"))?;
self.pos = start + len;
Ok(Some((id, data)))
}
fn peek_id(&self) -> Option<u16> {
let h = self.stream.get(self.pos..self.pos.checked_add(2)?)?;
Some(u16::from_le_bytes([*h.first()?, *h.get(1)?]))
}
/// The next record with its `Continue` records, or `None` at the end of the stream.
pub fn next_record(&mut self) -> Result<Option<Record<'a>>, IoError> {
let Some((id, data)) = self.raw()? else { return Ok(None) };
let mut parts = vec![data];
while self.peek_id() == Some(CONTINUE) && parts.len() < MAX_PARTS {
if let Some((_, more)) = self.raw()? {
parts.push(more);
}
}
Ok(Some(Record { id, parts }))
}
}
/// Reads a record's payload, crossing into its `Continue` records when needed.
pub(super) struct Cursor<'a, 'b> {
parts: &'b [&'a [u8]],
part: usize,
pos: usize,
/// Bytes in every part after the current one, kept up to date as parts advance, so
/// `remaining()` is constant-time however many `Continue` records a string session crosses.
after: usize,
}
impl<'a, 'b> Cursor<'a, 'b> {
pub fn new(parts: &'b [&'a [u8]]) -> Self {
let after = parts.iter().skip(1).map(|p| p.len()).sum::<usize>();
Cursor { parts, part: 0, pos: 0, after }
}
/// Skips to the next part when the current one is used up. Returns whether a new part began.
fn settle(&mut self) -> bool {
let mut moved = false;
while self.parts.get(self.part).is_some_and(|p| self.pos >= p.len()) && self.part + 1 < self.parts.len() {
self.after = self.after.saturating_sub(self.parts[self.part + 1].len());
self.part += 1;
self.pos = 0;
moved = true;
}
moved
}
pub fn u8(&mut self) -> Result<u8, IoError> {
self.settle();
let b = self.parts.get(self.part).and_then(|p| p.get(self.pos)).copied().ok_or_else(|| invalid("record is shorter than its contents"))?;
self.pos += 1;
Ok(b)
}
pub fn u16(&mut self) -> Result<u16, IoError> {
Ok(u16::from_le_bytes([self.u8()?, self.u8()?]))
}
pub fn u32(&mut self) -> Result<u32, IoError> {
Ok(u32::from_le_bytes([self.u8()?, self.u8()?, self.u8()?, self.u8()?]))
}
pub fn f64(&mut self) -> Result<f64, IoError> {
let mut b = [0u8; 8];
for x in b.iter_mut() {
*x = self.u8()?;
}
Ok(f64::from_le_bytes(b))
}
pub fn skip(&mut self, n: usize) -> Result<(), IoError> {
for _ in 0..n {
self.u8()?;
}
Ok(())
}
/// Bytes left in the record (all parts), constant-time in the part count.
pub fn remaining(&self) -> usize {
let here = self.parts.get(self.part).map_or(0, |p| p.len().saturating_sub(self.pos));
here.saturating_add(self.after)
}
/// The rest of the record as one buffer.
pub fn rest(&mut self) -> Vec<u8> {
let mut out = Vec::with_capacity(self.remaining());
if let Some(p) = self.parts.get(self.part) {
out.extend_from_slice(p.get(self.pos..).unwrap_or(&[]));
}
for p in self.parts.iter().skip(self.part + 1) {
out.extend_from_slice(p);
}
self.part = self.parts.len();
self.pos = 0;
self.after = 0;
out
}
/// `cch` characters, compressed (one byte each) or UTF-16. When the characters run into
/// a `Continue` record, that record starts with a fresh compression flag ([MS-XLS] 2.5.293).
pub fn chars(&mut self, cch: usize, mut high: bool) -> Result<String, IoError> {
if cch > MAX_CHARS {
return Err(invalid("string too long"));
}
let mut units: Vec<u16> = Vec::with_capacity(cch.min(self.remaining()));
while units.len() < cch {
let at_end = self.parts.get(self.part).is_none_or(|p| self.pos >= p.len());
if at_end && self.settle() {
high = self.u8()? & 1 != 0;
}
units.push(if high { self.u16()? } else { u16::from(self.u8()?) });
}
Ok(String::from_utf16_lossy(&units))
}
/// ShortXLUnicodeString: 1-byte count, flags, characters.
pub fn short_string(&mut self) -> Result<String, IoError> {
let cch = usize::from(self.u8()?);
let high = self.u8()? & 1 != 0;
self.chars(cch, high)
}
/// XLUnicodeString: 2-byte count, flags, characters.
pub fn string(&mut self) -> Result<String, IoError> {
let cch = usize::from(self.u16()?);
let high = self.u8()? & 1 != 0;
self.chars(cch, high)
}
/// XLUnicodeRichExtendedString: the text; formatting runs and phonetic data are skipped.
pub fn rich_string(&mut self) -> Result<String, IoError> {
let cch = usize::from(self.u16()?);
let flags = self.u8()?;
let runs = if flags & 0x08 != 0 { usize::from(self.u16()?) } else { 0 };
let ext = if flags & 0x04 != 0 { self.u32()? as usize } else { 0 };
let text = self.chars(cch, flags & 1 != 0)?;
let extra = runs.saturating_mul(4).saturating_add(ext);
if extra > self.remaining() {
return Err(invalid("string formatting runs past the end of the record"));
}
self.skip(extra)?;
Ok(text)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn strings_continue_with_a_new_compression_flag() {
// "ab" compressed, then "ç" and "d" as UTF-16 in a Continue part.
let first: &[u8] = &[4, 0, 0, b'a', b'b'];
let second: &[u8] = &[1, 0xE7, 0, b'd', 0];
let parts = [first, second];
let mut c = Cursor::new(&parts);
assert_eq!(c.rich_string().unwrap(), "abçd");
assert_eq!(c.remaining(), 0);
let mut c = Cursor::new(&parts);
assert!(c.chars(MAX_CHARS + 1, false).is_err());
}
#[test]
fn remaining_counts_every_part_wherever_the_cursor_is() {
let parts: [&[u8]; 4] = [&[1, 2, 3], &[2, 3, 4, 5], &[3, 4, 5, 6, 7], &[4]];
let mut c = Cursor::new(&parts);
assert_eq!(c.remaining(), 13);
for b in 1..=3u64 {
assert_eq!(u64::from(c.u8().unwrap()), b, "part 0 in order");
}
assert_eq!(c.remaining(), 10, "after the first part is used up");
for _ in 0..4 {
c.u8().unwrap();
}
assert_eq!(c.remaining(), 6, "into the Continue parts");
let mut c = Cursor::new(&parts);
assert_eq!(u64::from(c.u8().unwrap()), 1);
assert_eq!(c.rest().len(), 12);
assert_eq!(c.remaining(), 0);
}
}
-406
View File
@@ -1,406 +0,0 @@
//! Cell formatting in a BIFF8 Workbook stream, from [MS-XLS]: Font (2.4.122), XF (2.4.353) with
//! CellXF (2.5.91), Palette (2.4.188) and Style (2.4.269). Every XF is
//! decoded on its own: a cell XF stores its complete formatting (the fAtr* bits only say whether
//! later changes to the parent style propagate), so nothing is inherited from `ixfParent`.
use std::collections::HashMap;
use gridcraft_model::style::Protection;
use gridcraft_model::{
Alignment, BorderLine, BorderStyle, Borders, Color, Fill, Font, HAlign, NumFmt, PatternType, Style, StyleId, StyleTable, Underline, VAlign,
VertAlign, Workbook,
};
use super::records::Cursor;
use crate::tables::{INDEXED, builtin_format};
/// Font records kept at most ([MS-XLS] FontIndex allows indexes up to 1022).
const MAX_FONTS: usize = 1024;
/// Style records kept at most.
const MAX_STYLES: usize = 65_536;
/// Characters kept from a style name (Excel allows 255).
const MAX_STYLE_NAME: usize = 255;
/// The automatic font colour (IcvFont 0x7FFF) and the system foreground/background colours.
const ICV_FONT_AUTO: u16 = 0x7FFF;
const ICV_DEFAULT_FG: u16 = 0x40;
const ICV_DEFAULT_BG: u16 = 0x41;
/// One XF record, decoded as far as its bytes allow.
#[derive(Clone, Debug, Default)]
pub(super) struct Xf {
ifnt: u16,
ifmt: u16,
flags: u16,
/// The CellXF/StyleXF data: 14 bytes when complete.
data: [u8; 14],
/// Whether all 20 bytes were present.
complete: bool,
}
/// A Style record: the XF it names and its name.
#[derive(Clone, Debug)]
pub(super) struct NamedStyle {
ixf: u16,
/// `Some(id)` for a built-in style (BuiltInStyle.istyBuiltIn), `None` for a user style.
builtin: Option<u8>,
/// A user style's name (empty for built-in styles).
name: String,
}
/// Formatting records collected from the globals substream.
#[derive(Debug, Default)]
pub(super) struct StyleRecords {
fonts: Vec<Font>,
font_colors: Vec<u16>,
xfs: Vec<Xf>,
/// Palette entries for colour indexes 8 and up (RRGGBB), when a Palette record is present.
palette: Option<Vec<u32>>,
styles: Vec<NamedStyle>,
/// Records that were shorter than their structure; the parts present were used.
pub(super) short: bool,
}
impl StyleRecords {
/// A Font record. The colour is resolved later, once the palette is known.
pub(super) fn font(&mut self, c: &mut Cursor<'_, '_>) {
if self.fonts.len() >= MAX_FONTS {
return;
}
match read_font(c) {
Some((font, icv)) => {
self.fonts.push(font);
self.font_colors.push(icv);
}
None => {
// Keep the numbering of the fonts after it.
self.short = true;
self.fonts.push(Font::default());
self.font_colors.push(ICV_FONT_AUTO);
}
}
}
/// An XF record. A short record keeps the fields it has.
pub(super) fn xf(&mut self, c: &mut Cursor<'_, '_>, max: usize) {
if self.xfs.len() >= max {
return;
}
let mut xf = Xf::default();
let head = (|| Some((c.u16().ok()?, c.u16().ok()?, c.u16().ok()?)))();
match head {
Some((ifnt, ifmt, flags)) => {
xf.ifnt = ifnt;
xf.ifmt = ifmt;
xf.flags = flags;
xf.complete = true;
for b in xf.data.iter_mut() {
match c.u8() {
Ok(v) => *b = v,
Err(_) => {
xf.complete = false;
break;
}
}
}
}
None => {
// Not even the font and format: an unformatted, locked cell.
xf.flags = 1;
}
}
if !xf.complete {
self.short = true;
}
self.xfs.push(xf);
}
/// A Palette record: colours for indexes 8 onwards.
pub(super) fn palette(&mut self, c: &mut Cursor<'_, '_>) {
let Ok(ccv) = c.u16() else {
self.short = true;
return;
};
let n = usize::from(ccv).min(56);
if usize::from(ccv) > c.remaining() / 4 {
self.short = true;
}
let mut colors = Vec::with_capacity(n);
for _ in 0..n {
let (Ok(r), Ok(g), Ok(b), Ok(_)) = (c.u8(), c.u8(), c.u8(), c.u8()) else { break };
colors.push((u32::from(r) << 16) | (u32::from(g) << 8) | u32::from(b));
}
self.palette = Some(colors);
}
/// A Style record.
pub(super) fn style(&mut self, c: &mut Cursor<'_, '_>) {
if self.styles.len() >= MAX_STYLES {
return;
}
let Ok(word) = c.u16() else {
self.short = true;
return;
};
let ixf = word & 0x0FFF;
if word & 0x8000 != 0 {
let Ok(id) = c.u8() else {
self.short = true;
return;
};
// BuiltInStyle: the style's number (0 is Normal) and outline level.
self.styles.push(NamedStyle { ixf, builtin: Some(id), name: String::new() });
} else {
match c.string() {
Ok(name) => {
let name: String = name.chars().filter(|ch| *ch != '\0').take(MAX_STYLE_NAME).collect();
if !name.is_empty() {
self.styles.push(NamedStyle { ixf, builtin: None, name });
}
}
Err(_) => self.short = true,
}
}
}
/// The 64-entry colour table: the built-in eight, then the palette (or the default one).
fn color_table(&self) -> [u32; 64] {
let mut table = INDEXED;
if let Some(p) = &self.palette {
for (slot, c) in table.iter_mut().skip(8).zip(p.iter()) {
*slot = *c;
}
}
table
}
}
/// Font record fields up to the name; `None` if the record is too short for them.
fn read_font(c: &mut Cursor<'_, '_>) -> Option<(Font, u16)> {
let height = c.u16().ok()?;
let flags = c.u16().ok()?;
let icv = c.u16().ok()?;
let weight = c.u16().ok()?;
let script = c.u16().ok()?;
let underline = c.u8().ok()?;
let family = c.u8().ok()?;
let charset = c.u8().ok()?;
c.skip(1).ok()?; // reserved
let name = c.short_string().ok()?;
let mut f = Font::default();
let name: String = name.chars().filter(|ch| *ch != '\0').take(255).collect();
if !name.trim().is_empty() {
f.name = name;
}
if height > 0 {
f.size = (f32::from(height) / 20.0).clamp(1.0, 409.0);
}
f.italic = flags & 0x0002 != 0;
f.strike = flags & 0x0008 != 0;
// Weights run 100-1000; Excel writes 400 for normal and 700 for bold.
f.bold = weight >= 600;
f.vert = match script {
1 => VertAlign::Superscript,
2 => VertAlign::Subscript,
_ => VertAlign::Baseline,
};
f.underline = match underline {
0x01 => Underline::Single,
0x02 => Underline::Double,
0x21 => Underline::SingleAccounting,
0x22 => Underline::DoubleAccounting,
_ => Underline::None,
};
f.family = Some(family);
f.charset = Some(charset);
Some((f, icv))
}
/// A colour index in an XF (IcvXF) or Font (IcvFont): 0-63 from the colour table, the system
/// and automatic colours as automatic.
fn color(icv: u16, table: &[u32; 64]) -> Color {
match icv {
ICV_DEFAULT_FG | ICV_DEFAULT_BG | ICV_FONT_AUTO => Color::Auto,
_ => table.get(usize::from(icv)).map_or(Color::Auto, |c| Color::Rgb(*c)),
}
}
/// A border line style from its BIFF8 code (dgLeft and the like); unknown codes are no border.
fn border_style(dg: u32) -> BorderStyle {
match dg {
1 => BorderStyle::Thin,
2 => BorderStyle::Medium,
3 => BorderStyle::Dashed,
4 => BorderStyle::Dotted,
5 => BorderStyle::Thick,
6 => BorderStyle::Double,
7 => BorderStyle::Hair,
8 => BorderStyle::MediumDashed,
9 => BorderStyle::DashDot,
10 => BorderStyle::MediumDashDot,
11 => BorderStyle::DashDotDot,
12 => BorderStyle::MediumDashDotDot,
13 => BorderStyle::SlantDashDot,
_ => BorderStyle::None,
}
}
/// A fill pattern from its BIFF8 code (fls); unknown codes are no pattern.
fn pattern(fls: u16) -> PatternType {
match fls {
0x01 => PatternType::Solid,
0x02 => PatternType::Gray50,
0x03 => PatternType::Gray75,
0x04 => PatternType::Gray25,
0x05 => PatternType::DarkHorizontal,
0x06 => PatternType::DarkVertical,
0x07 => PatternType::DarkDown,
0x08 => PatternType::DarkUp,
0x09 => PatternType::DarkGrid,
0x0A => PatternType::DarkTrellis,
0x0B => PatternType::LightHorizontal,
0x0C => PatternType::LightVertical,
0x0D => PatternType::LightDown,
0x0E => PatternType::LightUp,
0x0F => PatternType::LightGrid,
0x10 => PatternType::LightTrellis,
0x11 => PatternType::Gray125,
0x12 => PatternType::Gray0625,
_ => PatternType::None,
}
}
/// One border side: its style and colour; a side without a style has no colour.
fn line(dg: u32, icv: u32, table: &[u32; 64]) -> BorderLine {
let style = border_style(dg);
if style == BorderStyle::None {
return BorderLine::default();
}
// Colour 0 means "not specified": automatic.
let color = if icv == 0 { Color::Auto } else { color(icv as u16, table) };
BorderLine { style, color }
}
/// The font an XF names (FontIndex: index 4 is never written, so 5 and up count from one).
fn xf_font(ifnt: u16, fonts: &[Font]) -> Font {
let i = match ifnt {
0..=3 => usize::from(ifnt),
4 => return fonts.first().cloned().unwrap_or_default(),
_ => usize::from(ifnt) - 1,
};
fonts.get(i).or_else(|| fonts.first()).cloned().unwrap_or_default()
}
/// The number format an XF names: a custom format, else a built-in one, else General.
fn num_fmt(ifmt: u16, formats: &HashMap<u16, String>) -> NumFmt {
let code = formats.get(&ifmt).map(String::as_str).or_else(|| builtin_format(u32::from(ifmt))).unwrap_or("General");
if code.eq_ignore_ascii_case("General") { NumFmt::default() } else { NumFmt::new(code) }
}
/// The model style of one XF.
fn xf_style(xf: &Xf, fonts: &[Font], formats: &HashMap<u16, String>, table: &[u32; 64]) -> Style {
let mut s = Style {
font: xf_font(xf.ifnt, fonts),
num_fmt: num_fmt(xf.ifmt, formats),
protection: Protection { locked: xf.flags & 0x0001 != 0, hidden: xf.flags & 0x0002 != 0 },
..Style::default()
};
if !xf.complete {
return s;
}
let d = &xf.data;
let byte = |i: usize| d.get(i).copied().unwrap_or(0);
let word32 = |i: usize| u32::from_le_bytes([byte(i), byte(i + 1), byte(i + 2), byte(i + 3)]);
let (a0, rot, a2) = (byte(0), byte(1), byte(2));
s.align = Alignment {
h: match a0 & 0x07 {
1 => HAlign::Left,
2 => HAlign::Center,
3 => HAlign::Right,
4 => HAlign::Fill,
5 => HAlign::Justify,
6 => HAlign::CenterAcross,
7 => HAlign::Distributed,
_ => HAlign::General,
},
wrap: a0 & 0x08 != 0,
v: match (a0 >> 4) & 0x07 {
0 => VAlign::Top,
1 => VAlign::Center,
3 => VAlign::Justify,
4 => VAlign::Distributed,
_ => VAlign::Bottom,
},
rotation: match rot {
0..=90 => i16::from(rot),
91..=180 => -(i16::from(rot) - 90),
255 => 255,
_ => 0,
},
indent: a2 & 0x0F,
shrink: a2 & 0x10 != 0,
reading_order: None,
};
let b1 = word32(4);
let b2 = word32(8);
let mut borders = Borders {
left: line(b1 & 0x0F, (b1 >> 16) & 0x7F, table),
right: line((b1 >> 4) & 0x0F, (b1 >> 23) & 0x7F, table),
top: line((b1 >> 8) & 0x0F, b2 & 0x7F, table),
bottom: line((b1 >> 12) & 0x0F, (b2 >> 7) & 0x7F, table),
..Borders::default()
};
let diag = line((b2 >> 21) & 0x0F, (b2 >> 14) & 0x7F, table);
let which = (b1 >> 30) & 0x03;
if which & 1 != 0 {
borders.diag_down = diag;
}
if which & 2 != 0 {
borders.diag_up = diag;
}
s.borders = borders;
let fill = u16::from_le_bytes([byte(12), byte(13)]);
let fls = ((b2 >> 26) & 0x3F) as u16;
let fg = color(fill & 0x7F, table);
let bg = color((fill >> 7) & 0x7F, table);
s.fill = match pattern(fls) {
PatternType::None => Fill::default(),
PatternType::Solid => Fill::solid(fg),
p => Fill { pattern: p, fg, bg },
};
s
}
/// Imports the formatting: sets the workbook's default (Normal) font and named cell styles, and
/// returns the model style of every XF in file order.
pub(super) fn build(r: &StyleRecords, formats: &HashMap<u16, String>, wb: &mut Workbook) -> Vec<StyleId> {
let table = r.color_table();
let fonts: Vec<Font> = r.fonts.iter().zip(r.font_colors.iter()).map(|(f, icv)| Font { color: color(*icv, &table), ..f.clone() }).collect();
// The Normal style's font is what every unformatted cell shows.
let normal_xf = r.styles.iter().find(|s| s.builtin == Some(0)).map_or(0, |s| usize::from(s.ixf));
let normal_font = r.xfs.get(normal_xf).map(|xf| xf_font(xf.ifnt, &fonts)).or_else(|| fonts.first().cloned());
if let Some(font) = normal_font {
wb.styles = StyleTable::with_default(Style { font, ..Style::default() });
}
let out = r.xfs.iter().map(|xf| wb.styles.intern(xf_style(xf, &fonts, formats, &table))).collect();
// User-defined named styles. Excel 2007 and later write every built-in style into each .xls
// (about fifty, their colours approximated to the palette); GridCraft defines those itself.
let mut named: Vec<(String, Style)> = Vec::new();
// Names are compared ASCII-case-insensitively; a set keeps a hostile file's many names linear.
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::from(["normal".to_string()]);
for ns in &r.styles {
if ns.builtin.is_some() {
continue;
}
let key = ns.name.to_ascii_lowercase();
if seen.contains(&key) {
continue;
}
if let Some(xf) = r.xfs.get(usize::from(ns.ixf)) {
seen.insert(key);
named.push((ns.name.clone(), xf_style(xf, &fonts, formats, &table)));
}
}
wb.cell_styles = named;
out
}
+2 -1
View File
@@ -1,6 +1,6 @@
# File-format parity
> **Last reviewed:** 2026-11-01 · **Last updated:** 2026-11-03 · **Change:** minor (form controls: buttons, scroll bars and spin buttons read and written; earlier: a web page's first table opens as a sheet, also when the page is named `.xls`; earlier: `workbookPr filterPrivacy` kept, and honoured on save; earlier: a workbook's R1C1 reference style, `calcPr refMode`, is kept through open and save; earlier: XLSX page margins are written as their decimals, `0.3` rather than `0.30000001192092896`; earlier: Python cells in `.xlsx` as [MS-XLSX] specifies, `_xlfn._xlws.PY(…)` naming code in the Python part, with cached values, and in GridCraft JSON, with limits on open; earlier: equations in xlsx as Office Math with a marker in `descr` so an edit made in Excel reads back, read back only up to the equation source cap, and a brace with a label and a script on its other side keeps both; Symbol font in PDF; earlier: XLSX reads and writes Page Setup's page order, `pageSetup pageOrder`, and PDF export follows it; earlier: chart data tables: native c:dTable import/export; earlier: XLSX workbook-view selection, normalized tab ratios and sheet-edit regressions; fidelity estimates unchanged; earlier: formulas from Google Sheets' XLSX export open as the original formula instead of `IFERROR(__xludf.DUMMYFUNCTION(…),…)` when GridCraft calculates every function in it, and save as it; `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; earlier: `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; CSV and text open detects each column's date order from values that read only one way, warns when every date is ambiguous, and keeps a column with both kinds as text (#2765)) · **Target:** Microsoft Excel (Microsoft 365), Excel for Mac 16.113.4
> **Last reviewed:** 2026-11-01 · **Last updated:** 2026-11-12 · **Change:** minor (.xls is read by the shared craft-xls crate: the workbook opens on the sheet it was saved on, and rich-text (RString) cells are read; earlier: form controls: buttons, scroll bars and spin buttons read and written; earlier: a web page's first table opens as a sheet, also when the page is named `.xls`; earlier: `workbookPr filterPrivacy` kept, and honoured on save; earlier: a workbook's R1C1 reference style, `calcPr refMode`, is kept through open and save; earlier: XLSX page margins are written as their decimals, `0.3` rather than `0.30000001192092896`; earlier: Python cells in `.xlsx` as [MS-XLSX] specifies, `_xlfn._xlws.PY(…)` naming code in the Python part, with cached values, and in GridCraft JSON, with limits on open; earlier: equations in xlsx as Office Math with a marker in `descr` so an edit made in Excel reads back, read back only up to the equation source cap, and a brace with a label and a script on its other side keeps both; Symbol font in PDF; earlier: XLSX reads and writes Page Setup's page order, `pageSetup pageOrder`, and PDF export follows it; earlier: chart data tables: native c:dTable import/export; earlier: XLSX workbook-view selection, normalized tab ratios and sheet-edit regressions; fidelity estimates unchanged; earlier: formulas from Google Sheets' XLSX export open as the original formula instead of `IFERROR(__xludf.DUMMYFUNCTION(…),…)` when GridCraft calculates every function in it, and save as it; `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; earlier: `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; CSV and text open detects each column's date order from values that read only one way, warns when every date is ambiguous, and keeps a column with both kinds as text (#2765)) · **Target:** Microsoft Excel (Microsoft 365), Excel for Mac 16.113.4
Every format Excel opens or saves, with GridCraft's read and write support, fidelity and tests.
Part of [target-app-parity.md](target-app-parity.md). **File formats: ~50% ready, 80–120 h**
@@ -161,6 +161,7 @@ refresh. Counted under features (Get & Transform, ~10% ready), not here.
| Date | Change | Summary |
|---|---|---|
| 2026-11-12 | minor | .xls import reads the file with craft-xls (storytold/craft-libs), the reader that moved there from this repository; GridCraft keeps the mapping onto its model. The workbook now opens on the sheet it was saved on (WINDOW1 `itabCur`), falling back to the first visible sheet as before (from #3551), and cells saved as RString records (text with formatting runs) are read as text instead of skipped |
| 2026-11-10 | minor | `.xls` import: every formula except a Ctrl+Shift+Enter array is read as a legacy one with Excel's implicit `@` (a title row `=Other!A1:G1` shows its first cell instead of `#SPILL!`), one-cell Ctrl+Shift+Enter arrays stay arrays, and `@` of a 2-D range meets the formula's row and column (#410, PR #1367) |
| 2026-11-02 | minor | XLSX: buttons, scroll bars and spin buttons read and written (control properties parts, `<controls>`, VML `x:ClientData`); other form controls stay with pass-through; a button's command, fixed input or output cell travels in a hidden defined name and comes back only while the file still assigns the button's macro (or none, for a command) |
| 2026-11-03 | minor | SYLK (`.slk`) import: values, R1C1 formulas as A1 (shared formulas too) with cached results, number formats and column widths; import only (#1481) |
@@ -17,6 +17,8 @@ runtime-version: '25.08'
sdk: org.freedesktop.Sdk
command: gridcraft
finish-args:
# The shared updater inspects and updates this installation via flatpak-spawn --host.
- --talk-name=org.freedesktop.Flatpak
- --filesystem=xdg-data/ArtCraft/versions:ro
# Display: Wayland first; X11 only when no Wayland compositor is running.
- --socket=wayland
@@ -18,6 +18,8 @@ sdk-extensions:
- org.freedesktop.Sdk.Extension.rust-stable
command: gridcraft
finish-args:
# The shared updater inspects and updates this installation via flatpak-spawn --host.
- --talk-name=org.freedesktop.Flatpak
- --filesystem=xdg-data/ArtCraft/versions:ro
# Display: Wayland first; X11 only when no Wayland compositor is running.
- --socket=wayland
+4
View File
@@ -1,2 +1,6 @@
sonar.projectKey=storytold_gridcraft
sonar.organization=storytold
# The region table is one record per locale with the same field names and many shared values
# (month names, formats); that repetition is data, not copy-pasted logic.
sonar.cpd.exclusions=crates/core/src/region.rs
+69 -5
View File
@@ -13,7 +13,10 @@ struct Counts {
proptests: usize,
}
fn walk(dir: &Path, c: &mut Counts, seen: &mut std::collections::HashSet<u64>, budget: &mut usize) {
/// Larger files (or devices behind a symlink) are not source files and are not read.
const MAX_SOURCE_BYTES: u64 = 16 << 20;
fn walk(dir: &Path, c: &mut Counts, seen: &mut std::collections::HashSet<(u64, u64)>, budget: &mut usize) {
if *budget == 0 {
// Hard stop for pathological trees: two directory symlinks pointing at each other
// would otherwise be walked exponentially.
@@ -22,9 +25,9 @@ fn walk(dir: &Path, c: &mut Counts, seen: &mut std::collections::HashSet<u64>, b
// Directory identities (dev+inode on unix) make a symlink to a visited directory a
// no-op, not another exponential branch of the scan.
#[cfg(unix)]
let id = std::fs::metadata(dir).ok().and_then(|m| m.dev().checked_mul(1 << 60).and_then(|d| d.checked_add(m.ino())));
let id = std::fs::metadata(dir).ok().map(|m| (m.dev(), m.ino()));
#[cfg(not(unix))]
let id: Option<u64> = None;
let id: Option<(u64, u64)> = None;
if let Some(id) = id
&& !seen.insert(id)
{
@@ -45,7 +48,9 @@ fn walk(dir: &Path, c: &mut Counts, seen: &mut std::collections::HashSet<u64>, b
walk(&p, c, seen, budget);
}
} else if name.ends_with(".rs")
&& let Ok(s) = std::fs::read_to_string(&p)
// The entry's own metadata does not follow a symlink: only regular files are sources.
&& e.metadata().is_ok_and(|m| m.is_file() && m.len() <= MAX_SOURCE_BYTES)
&& let Some(s) = read_source(&p)
{
c.files += 1;
c.lines += s.lines().count();
@@ -56,6 +61,15 @@ fn walk(dir: &Path, c: &mut Counts, seen: &mut std::collections::HashSet<u64>, b
}
}
/// A source file's text, read up to [`MAX_SOURCE_BYTES`]: one that grew or was replaced since its
/// size was checked is skipped, not read whole.
fn read_source(p: &Path) -> Option<String> {
use std::io::Read as _;
let mut s = String::new();
std::fs::File::open(p).ok()?.take(MAX_SOURCE_BYTES + 1).read_to_string(&mut s).ok()?;
(s.len() as u64 <= MAX_SOURCE_BYTES).then_some(s)
}
/// `#[test]` attributes and `proptest!` blocks (each block may hold many
/// property tests, and macro-generated tests are only counted once — use
/// `--exact` for harness-accurate numbers).
@@ -129,11 +143,61 @@ pub fn run(_root: &Path, exact: bool) -> Result<(), String> {
#[cfg(test)]
mod tests {
use super::count_tests;
use super::{Counts, MAX_SOURCE_BYTES, count_tests, read_source, walk};
/// A new, private directory: `create_dir` fails if the name exists (even as a link), so a
/// directory somebody else made in advance is never used.
fn private_dir(tag: &str) -> std::path::PathBuf {
let mut n = 0u32;
loop {
let dir = std::env::temp_dir().join(format!("xtask-stats-{tag}-{}-{n}", std::process::id()));
let mut b = std::fs::DirBuilder::new();
#[cfg(unix)]
std::os::unix::fs::DirBuilderExt::mode(&mut b, 0o700);
match b.create(&dir) {
Ok(()) => return dir,
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {}
Err(e) => panic!("{e}"),
}
n += 1;
}
}
#[test]
fn counts_attributes_and_proptest_blocks() {
let src = "#[test]\nfn a() {}\n// #[test] commented\n #[test]\nfn b() {}\nproptest! {\n}\n";
assert_eq!(count_tests(src), (2, 1));
}
#[test]
fn a_source_past_the_size_limit_is_not_read() {
let dir = private_dir("size");
let (small, big) = (dir.join("a.rs"), dir.join("b.rs"));
std::fs::write(&small, "fn a() {}\n").unwrap();
let file = std::fs::File::create(&big).unwrap();
file.set_len(MAX_SOURCE_BYTES + 1).unwrap(); // sparse: nothing is written
drop(file);
assert_eq!(read_source(&small).as_deref(), Some("fn a() {}\n"));
assert_eq!(read_source(&big), None);
let _ = std::fs::remove_dir_all(dir);
}
#[cfg(unix)]
#[test]
fn walk_counts_regular_sources_once_and_skips_links_and_big_files() {
let dir = private_dir("walk");
let sub = dir.join("sub");
std::fs::create_dir(&sub).unwrap();
std::fs::write(dir.join("a.rs"), "#[test]\nfn a() {}\n").unwrap();
std::fs::write(sub.join("b.rs"), "fn b() {}\n").unwrap();
// A directory link back to the root is one identity, a file link is not a regular file,
// and a source over the size limit is not read.
std::os::unix::fs::symlink(&dir, sub.join("loop")).unwrap();
std::os::unix::fs::symlink(dir.join("a.rs"), sub.join("link.rs")).unwrap();
std::fs::File::create(dir.join("big.rs")).unwrap().set_len(MAX_SOURCE_BYTES + 1).unwrap();
let mut c = Counts::default();
walk(&dir, &mut c, &mut std::collections::HashSet::new(), &mut 100);
assert_eq!((c.files, c.tests), (2, 1));
let _ = std::fs::remove_dir_all(dir);
}
}