The equivalence of `header_plan` / `preview_plan` broke for the corpus's
ARW6 pair: a Compression-32766 strip was holed behind its first 64 KiB
prefix, but the header probe validates more there - `parse_tile_header`
walks the 13 stream header words of every tile, `stream_halves` reads
each stream's index table, and `first_half_shift` decodes the residual
stream's first half, which sits behind the earlier streams' data. The
probe read zeros (and 0xFF) there and failed with "ARW6 first-half row
count outside 2..=7" where the whole file decodes (first corpus case:
arw6-sony-ilce7rm6-apsc-land-hq.arw; the corpus is git-ignored, so CI
never saw it).
`tiff_plan` now asks `arw6_keep` (crates/raw/src/sparse.rs) for those
scattered ranges when `image()` reports Compression 32766 for one strip:
the tile table, then per tile, in staged rounds so each ask sits in one
`Need`, the picture/group tables, the stream header words (positions
from the new `arw6::stream_header_positions`, mirroring the group-table
walk), then with the real `parse_tile_header` run, the 13 index tables
and finally each residual first half (its length A is its index's first
entry). A layout that cannot be walked - a corrupt ARW6 - reads whole:
`arw6::decode` stops at the same validation of loaded bytes, so the
probe is still identical. `parse_tile_table`/`parse_tile_header`/`u16_be`
/`u24_be` are now `pub(crate)` for the walker, and `Want::pending_plan`
(renamed from `into_plan`) takes `&mut self` so the walk can keep asking
after the IFD walk answered.
The full-size JPEG these files carry as a `JPEGInterchangeFormat` image
(compression 6/7, where 0x0201/0x0202 are the interchange tag and its
length and `image()` therefore declines) reached no hole at all and made
the plan read it whole: together with the strip work, the header plan
now reads 26-28% -> ARW corpus average 1.7% (the two ARW6 files read
2 / 1%); nothing reads the interchange image in header mode, and under
`preview_plan` `embedded_preview` wants it, so the hole is header-mode
only (the corpus test's 0 / 0xFF fills guard both).
Tests: without the corpus, sparse.rs's synthetic single- and two-tile
ARW6 files (residual stream past the prefix, Qis::REAL) assert the plan
reads past the 512 KiB prefix and probes as the whole file does, for
both fills; with it, `sparse_corpus` passes for both real files, both
fills, both plans, and the ARW6 decode-sum corpus tests are unchanged.
Gates: fmt, clippy -D warnings, heif, parity (+ refreshed summary),
layers, assets and wasm all pass; `cargo test --workspace` has no
failure that clean main doesn't show on this Windows machine (engine
sidecar's 20 fs-fault and ui-egui's denoise-model tests fail there
identically with the fix stashed).
Co-Authored-By: GLM 5.3 Flash <noreply@z.ai>
The 2668-entry table read off the black-box oracle's output is gone.
arw6_curve.rs now builds the 4096-entry table at first use from our own
measurement: piecewise linear through knots u_k = 128 k, c_k = u_k for
k <= 11 and 356.89 + 655.646 ln(k - 6) above, rounded half up, fitted to
the 7 same-exposure lossless (LJ92) / ARW6 pairs of the CC0 ILCE-7RM6 set.
A unit test pins the whole table (sum 27776336, weighted 92792980288),
its anchors, the identity through 1408 and strict monotonicity.
The corpus sums become regression values of the new output; the 12-bit
codes before the curve stay verified against the oracle (Phase 0).
docs/arw6-compression.md describes the curve, how it was measured, its
uncertainty, the structure and x2-scale evidence, why it is not in the
file, and the after-the-fact comparison with the oracle's curve; the
tracker and docs no longer call the output bit-exact against LibRaw.
Co-Authored-By: Claude <noreply@anthropic.com>
docs/superpowers/ held the working spec and plan, which pointed at the
local plan/ folder. The ARW6 module docs now cite the sections of
docs/arw6-compression.md, which gains the facts they relied on from the
spec: the NP field of word 0, the stream header word layout and how the
first half's residual lines are counted.
Co-Authored-By: Claude <noreply@anthropic.com>
`cargo xtask corpus --download` now fetches one full-frame Compressed and
one APS-C Compressed HQ file (about 94 MB) instead of all 12 (about
630 MB); the corpus test checks those two and the lossless-sibling
regression test is gone, since nothing would download those files.
docs/arw6-compression.md keeps the full table with source paths and
SHA-256, says which two xtask fetches and how to fetch the rest with
`curl -L`, and records the CC0 1.0 statement of RawDB's set record
(accessed 2026-10-10).
Co-Authored-By: Claude <noreply@anthropic.com>
The RawDB entries in xtask carry their SHA-256 as the third field, which
`cargo xtask corpus --download` checks; the corpus test keeps only the
sensor sum checks.
Co-Authored-By: Claude <noreply@anthropic.com>
