Nikon bodies from before ~2016 record no CropArea, so the decoder kept
their whole sensor data, less trailing dark columns. Where that data
holds masked lines above black or optical black, they were developed as
a coloured or dark bar on that edge: the D7000 and D5100's 36 columns at
~600 (#976), the D800's 14 (#914, the top in portrait), the D600 / D610's
8 (#1236), the D3200's 46, and others.
MEASURED_BODIES (crates/raw/src/vendor/nef.rs) now gives 44 more model /
data size / depth groups the frame of their camera JPEG inside the data,
next to the D4, D7100 and D90 entries already there: D7000, D5100, D800,
D800E, Df, the D4's 12-bit mode, D3, D3S, D700, D300, D300S, D600, D610,
D810, D4S, D5600, Coolpix A, D3000, D40X, D60, D80, D2Xs, D200, D3200
and Nikon 1 V1; and, on best-guess frames short of the evidence rule,
each with a comment naming its evidence, the D3100 (3 of 4 clean files
agree), D5000 (4 of 5, and the D90's sensor), D40 (2 of 2), D50 and D70
(one file each, agreeing) and D70s (next to them, between pixels).
Every best-guess frame keeps the masked lines out by several pixels.
An entry matches model, data size and depth exactly. Bodies whose data
holds only picture beyond the camera frame get no entry and render as
before.
`cargo xtask raw-frame FILE...` (crates/engine/examples/raw_frame.rs)
measures a body: it aligns each raw's embedded camera JPEG with the
sensor data (half-size luminance, gradient correlation, sub-pixel peak;
weak, ambiguous or between-pixel peaks are flagged), counts the edge
lines that hold no picture and reports what the decoder's crop keeps
beyond the camera frame. A masked line is flat and its two CFA phases
agree within the noise or within 0.4% of white, so filler with a small
fixed offset between them (D90, D5000) counts. It prints entries only
with evidence: at least 3 clean files agreeing exactly, of the model
(12- and 14-bit pooled) or, with at least one of its own, with sibling
models of the same readout (data and JPEG size) that agree on the same
frame. On the D7500, which records its CropArea, it finds that area
exactly; it declines the Z f, whose JPEG is distortion-corrected. It
works for any make whose raw embeds a full-size camera JPEG: on Sony,
Canon CR2 / CR3, Pentax and Samsung files it finds the decoder's own
crop (relative to the active area).
Every Nikon raw on raw.pixls.us without CropArea was surveyed this way
(its 138 CC0 samples, plus 124 other publicly posted sample files; only
the measurements are committed). Bodies whose data holds only picture
beyond the frame need no entry (D750, D5300, D5500, D3300, D2X, D2Hs,
D3X, D5200, Coolpix P6000, Nikon 1 AW1 / J3 / V2 / V3, the D800's crop
modes); raw-parity.md lists them, and no longer claims every NEF
coding but HE decodes. measured_bodies_are_consistent checks every entry lies
inside its data and is listed once; measured_older_dx_bodies_image_area
covers the D200, D3200, Nikon 1 V1 and best-guess entries.
Fixes#976Fixes#914Fixes#1236
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve doc conflicts in gaps, parity-checklist and raw-parity; coverage
numbers regenerated with docs/raw/coverage.py, parity summary refreshed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Recheck all 54 files in the 41-model shortlist on main 25ef65c8.
Pin 40 accepted files by SHA-256 and full decoded sensor FNV-1a,
and record their camera-image correlation. Exclude eleven candidate
models after checking identity, camera module, provenance and references.
Update generated coverage and tracking documents to 265 verified models.
Keep Nikon coding limits and the own-matrix DNG rendering issue explicit.
Co-Authored-By: Codex <noreply@openai.com>
Fitted with `lightcraft-cli calibrate` to 144 usable ILCA-99M2 ARWs (of
150, seven shooting days, Standard creative style, DRO Auto) and their
in-camera JPEGs; aggregate colour statistics only.
On 30 withheld photos (profile fitted to the other 120) mean dE76 against
the camera JPEG goes 5.11 -> 4.87; 25 of 30 improve, the worst
regression is 0.24. All 150 photos stay accepted. The public raw.pixls.us
ILCA-99M2 file, not used for fitting, stays accepted (5.57 -> 5.73).
A corpus test checks that this public sample's fit keeps exactly the
built-in profile's matrix and table; the ILCE-7CR test now shares its
body. LOOK_VERSION 5 -> 6, as the profile changes the fit of this model.
Verified on 54 ILCE-7RM6 photos, the CA planes made the render worse: our
demosaic already hides about 70 % of the lens CA (the render shows 0.29x
the raw-CFA CA) while the post-demosaic warp moves R and B by 0.84x the
table, so the corner R/G and B/G CA went from rms 0.032 / max 0.125 px
(off) to 0.052 / 0.219 px (on) at 1440 px; only the 16-35 mm improved.
warp_opcode takes a lateral_ca flag; lens_opcodes passes false, so a Sony
ARW gets the one-plane distortion warp. The CA conversion stays live and
tested (FF and APS-C examples, both lossless corpus files' tables). The
follow-up is to warp the CFA colour planes before demosaic. The doc,
tracker (docs/parity-checklist.md), docs/gaps.md, docs/raw-parity.md and
whats-new say distortion applied, CA parsed and off
by default, vignetting not applied; the one-plane default costs 24 -> 45 ms
in the sample stage at 1920 px (three planes: 112 ms).
Co-Authored-By: Claude <noreply@anthropic.com>
New vendor::arw_lens_rm6 reads the raw IFD's lens tables (0x7037 distortion,
0x7032 vignetting, 0x7035 lateral CA, their flags) and turns the distortion
table into a one-plane WarpRectilinear in OpcodeList3, with the conventions
measured on 54 ILCE-7RM6 photos: knots at i/15 of the full-frame crop half
diagonal, S(r) = r (1 + v(r)/15000), frame-filling zoom, centred on the
default crop. Only the ILCE-7RM6 is enabled; vignetting is read, not applied.
The radial least-squares fit is split out as vendor::fit_radial_warp (RW2
results unchanged); its error gate can be limited to the radii the crop
shows, so a padded active area (lossless files) doesn't refuse a fit over
pixels that are never rendered.
It sits next to main's vendor::arw_lens (#498), which holds the ILCE-7RM4A
conventions measured against Sony's exports (unit 16384, table indexed at the
output radius); the two are selected by camera model (disjoint allowlists).
arw::decode chains this module's opcodes after arw_lens::distortion, with the
same active area and the same crop (the DNG-style default crop when the file
has one, which on the ILCE-7RM6 equals Sony's crop tags), and like it only
for CFA data (not the downsized linear-RGB lossless sizes, where the knot
spacing in full-frame pixels would not hold).
The corpus test checks the conversion on the two lossless RawDB files (CC0).
The APS-C one is pinned in xtask RAW_SAMPLES with its sha256; the full-frame
one is not (at most two pins per PR, and #625 pins two Compressed ARW 6.0
files), so its case runs only where the file is present.
RENDER_CACHE_VERSION 62 -> 63.
Co-Authored-By: Claude <noreply@anthropic.com>
Four CC0 EOS RP samples from raw.pixls.us (lossless RAW and C-RAW, full
frame and 1.6x crop mode) now have exact full-sensor regressions against
the DNGLab 0.8.0 black-box oracle (Windows binary, checked first against
the documented M50 RAW hash). The decoder already matched; no decoder
change was needed. Crop mode is a single 4032x2656 tile, full frame two
3192-wide tiles.
docs/cr3.md also records that the white level is Canon's specular white,
which at the -1/3-stop ISOs (e.g. 11062 at ISO 5000) is well below the
sensor's saturation point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cargo xtask ci` points each test step at target/ci/skips-<step>.jsonl,
then probes the machine: OS, CPUs, RAM, toolchain, corpus and craft-fonts,
and the GPU through the ignored `gpu_probe` test of lightcraft-gpu (run
right after the workspace tests with --workspace, so it reuses their build;
xtask stays without wgpu). The run ends with the skips grouped by need and
reason and writes target/ci/report.json (schema 1: commit, dirty flag,
machine, capabilities, step results and times, skips) and
target/ci/summary.md for a PR (`--summary-md` prints it). `--hardware`
also runs the lightcraft-gpu tests once per backend that has a hardware
adapter. Nothing is uploaded; reasons are cut free of the workspace root
and home directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One CC0 raw.pixls.us file per uncompressed Olympus ORF and Samsung SRW
body not yet pinned (23 files, 17 Adobe models) is added to RAW_SAMPLES
and pinned in crates/raw/tests/olympus_samsung_corpus.rs: own sensor
hash plus a black-box check against the camera's embedded JPEG (the
E-10 pinned only, inspected by eye).
Also: as_chunks in spots.rs for the chunks_exact_to_as_chunks lint.
Docs: evidence.tsv + coverage re-measure (252 verified), gap 10,
checklist, raw-parity, ROADMAP. Headline unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One CC0 raw.pixls.us file per Sony and Leica body that no earlier bulk
check covered (22 files, 21 Adobe models) is added to RAW_SAMPLES and
pinned in crates/raw/tests/sony_leica_corpus.rs: own sensor hash plus a
black-box check against the camera's embedded JPEG where it has a colour
one (five files pinned only, inspected by eye).
Also: as_chunks in spots.rs for the chunks_exact_to_as_chunks lint.
Docs: evidence.tsv + coverage re-measure (256 verified), gap 10,
checklist, raw-parity, ROADMAP. Headline unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sources: TIFF 6.0 and own black-box analysis of the three CC0 samples;
white-balance estimate from the embedded preview: own design.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One CC0 raw.pixls.us file per Canon CR2, Pentax and Ricoh body that
decodes (100 files, 97 Adobe models) is added to RAW_SAMPLES and pinned
in crates/raw/tests/canon_pentax_corpus.rs: own sensor hash plus a
black-box check that the decoded green follows the camera's embedded
JPEG (r 0.69-1.00; four files pinned only, inspected by eye).
Also: as_chunks in spots.rs for the chunks_exact_to_as_chunks lint.
Docs: evidence.tsv + coverage re-measure (332 verified, 50% on verified
bodies), gap 10, checklist, raw-parity, ROADMAP. Headline unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nikon bodies from before ~2016 record no CropArea, so the decoder kept
their whole sensor data, less trailing dark columns. Where that data
holds masked lines above black or optical black, they were developed as
a coloured or dark bar on that edge: the D7000 and D5100's 36 columns at
~600 (#976), the D800's 14 (#914, the top in portrait), and others.
MEASURED_BODIES (crates/raw/src/vendor/nef.rs) now gives 36 model /
data size / depth groups the frame of their camera JPEG inside the data:
D7000, D5100, D800, D800E, Df, the D4's 12-bit mode, D3, D3S, D700,
D300, D300S, D600, D610, D810, D4S, D5600, Coolpix A, D3000, D40X, D60,
D80 and D2Xs. An entry matches model, data size and depth exactly; the
measured white level becomes optional (only the D4's 14-bit mode has
one). Bodies whose data holds only picture beyond the camera frame get
no entry and render as before.
`cargo xtask raw-frame FILE...` (crates/engine/examples/raw_frame.rs)
measures a body: it aligns each raw's embedded camera JPEG with the
sensor data (half-size luminance, gradient correlation, sub-pixel peak;
weak, ambiguous or between-pixel peaks are flagged), counts the edge
lines that hold no picture and reports what the decoder's crop keeps
beyond the camera frame. It prints entries only with evidence: at least
3 clean files agreeing exactly, of the model (12- and 14-bit pooled) or,
with at least one of its own, with sibling models of the same readout
(data and JPEG size) that agree on the same frame. On the D7500, which
records its CropArea, it finds that area exactly; it declines the Z f,
whose JPEG is distortion-corrected.
It works for any make whose raw embeds a full-size camera JPEG: on
Sony, Canon CR2 / CR3, Pentax and Samsung files it finds the decoder's
own crop (relative to the active area).
docs/nikon-bodies.md surveys every Nikon raw on raw.pixls.us (98
models, 291 files; 138 CC0 samples measured): 22 models record
CropArea; of the 88 groups without it, 36 have entries, 13 are fine, 25
could not be aligned and 7 do not decode. The D3200, D3100, D200, D50,
D750 and Nikon 1 V1 / V2 are pending, each listed with the files it
needs; the doc also describes how to add a body. raw-parity.md no
longer claims every NEF coding but HE decodes.
measured_bodies_are_consistent checks every entry lies inside its data
and is listed once.
The evidence is committed in docs/nikon-bodies/: headers.json (every
file's raw size, depth and maker-note crop tags), samples.json (the 138
measured CC0 files with URL and sha256) and frames.jsonl (the tool's
per-file results). `cargo xtask corpus --download nikon` fetches the
samples (sha256 checked) so `cargo xtask raw-frame` reproduces every
count.
Fixes#976Fixes#914
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One CC0 raw.pixls.us file per Nikon and Fujifilm body that decodes is
added to RAW_SAMPLES and pinned in crates/raw/tests/body_corpus.rs: own
sensor hash plus a black-box check that the decoded green follows the
camera's embedded JPEG (r 0.77-1.00).
The sweep found COOLPIX 5700 / 8400 / 8800 decoding wrongly (two stacked
half-height fields, CMYG read as RGB): recorded as unsupported. #702 is
fixed, so the A100 evidence row is preview, not unsupported.
Also: as_chunks in spots.rs for the chunks_exact_to_as_chunks lint.
Docs: evidence.tsv + coverage re-measure (352 verified, 86% use-weighted,
58% on verified bodies), gap 10, checklist, raw-parity, target-app-parity,
ROADMAP. Headline numbers unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All 110 CC0 CR3s on raw.pixls.us decode from sensor data; none uses a
refused CRX variant. One CC0 file per further body (1D X III, 90D, 250D,
850D, M200, M50 II, M6 II, R, R3, R5, R6, R6 II, R6 III, R7, R10, RP, R50,
R50 V, G5 X II, G7 X III, SX70, V1) is added to RAW_SAMPLES and pinned in
cr3_corpus.rs, checked against the camera's embedded JPEG.
The R6 Mark III writes ColorData version 66, laid out as 64/65; it lost
its as-shot WB and white level. Render cache 61 -> 62.
Also: as_chunks in spots.rs for the chunks_exact_to_as_chunks lint.
Docs: evidence.tsv + coverage re-measure (257 verified, 92% use-weighted),
gaps 2/10, checklist rows, raw-parity, target-app-parity, ROADMAP, README.
Sources: own black-box analysis of CC0 files; ExifTool Canon tag tables.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The CRX 0x200 QP map opens with its first symbol as a 32-bit literal; every
later symbol is Rice coded with a limited-length escape: 23 zeros, the
terminating 1 and the symbol in 8 bits (the ITU-T T.87 layout for a 32-bit
limit and 8-bit symbols, without its minus-one offset). The old rule (an
escape after 16 zeros with a 16-bit literal) matched only the first symbol,
which is all the R8 and R7 maps need. R5 Mark II maps (QP 150-162) jump by up
to 12 where the Rice parameter is 0, giving 18- to 22-zero prefixes that
misparsed ("Canon CRX adaptive QP value -21087"): 174 of 306 sampled files
from a private R5 Mark II library opened as previews. Every QP map of its
33,665 C-RAW files now decodes to the exact end of its entropy stream; the six
oracle fixtures are unchanged. Established by black-box analysis of the
bitstream; no third-party decoder source consulted.
Adds the four CC0 R5 Mark II samples (raw.pixls.us 7881-7884) to the corpus
with pinned sensor hashes (the crop-mode framing is left to #754), bumps
RENDER_CACHE_VERSION to 43, and updates cr3.md, raw-parity (235 verified
models), coverage, gaps, the checklist rows, ROADMAP, target-app-parity and
What's New.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lightcraft-web-<version>.zip carried Cargo host build output next to the site: deps/ (81 files, 90.6 MB), build/ (149 files, 22.4 MB) and .fingerprint/ (297 files), ~113 MB that the app never references and that leaked build paths and crate names to anyone hosting the zip.
Cause: xtask wrote the bundle to <target>/web (SITE in packaging/web/package.sh) while building with --profile web. Cross-compiling with --target wasm32-unknown-unknown makes Cargo write the selected profile host artifacts (build scripts, proc-macros, deps/, build/, .fingerprint/) to <target>/<profile>/, so they landed in the bundle directory and package.sh copied the whole tree into the zip.
Fix: write the bundle to <target>/web-site, a directory that is not a Cargo profile name, and point SITE at it. The wasm32/web/ target artifacts and the web profile are unchanged, so the rust-cache keys and the CI web job keep working.
Guard: package.sh now fails if deps/, build/ or .fingerprint/ exists in $SITE, and a unit test asserts the bundle directory is not a Cargo profile name, so the collision cannot come back silently.
Reference (#793)
`cargo xtask corpus --download` now fetches one full-frame Compressed and
one APS-C Compressed HQ file (about 94 MB) instead of all 12 (about
630 MB); the corpus test checks those two and the lossless-sibling
regression test is gone, since nothing would download those files.
docs/arw6-compression.md keeps the full table with source paths and
SHA-256, says which two xtask fetches and how to fetch the rest with
`curl -L`, and records the CC0 1.0 statement of RawDB's set record
(accessed 2026-10-10).
Co-Authored-By: Claude <noreply@anthropic.com>
Michael Anderson