Recheck all 54 files in the 41-model shortlist on main 25ef65c8.
Pin 40 accepted files by SHA-256 and full decoded sensor FNV-1a,
and record their camera-image correlation. Exclude eleven candidate
models after checking identity, camera module, provenance and references.
Update generated coverage and tracking documents to 265 verified models.
Keep Nikon coding limits and the own-matrix DNG rendering issue explicit.
Co-Authored-By: Codex <noreply@openai.com>
Sony ARWs record shading, lateral chromatic aberration and distortion as a camera setting plus a radial
table in the raw image IFD (0x7031-0x7037), with copies in the encrypted SR2SubIFD (0x797c-0x7982).
They are now decoded for every body that records them and carried through RawInfo and RawImage as
`lens_metadata.sony` (types in crates/raw/src/lens_metadata.rs, decoder in
crates/raw/src/vendor/arw_lens.rs):
- Raw-IFD fields are the ones the TIFF reader parsed, which is what the image itself uses. A scan of
the directory adds diagnostics only: repeated tags, and why an entry the reader skipped was dropped
(values outside the file, an unknown type, past the reader's limits). SR2SubIFD fields are read
from their first entry. Only the exact layouts are accepted: SHORT[1] or non-negative SSHORT[1]
settings, SSHORT[17] headed by 16 for shading and distortion, SSHORT[33] headed by 32 for CA.
Nothing is allocated from a count in the file.
- The raw IFD's copy wins. The SR2SubIFD copy fills in only for a tag the raw IFD lacks, never for a
malformed one, and the two copies' agreement is recorded.
- Settings keep their number and their documented meaning, Unknown(value) otherwise (never Off).
- Framing evidence (stored frame, Sony and default crops, reference rectangle, coding, 3:2) and a
per-component status. Only Applied components become opcodes.
Applied corrections are unchanged: the ILCE-7RM4A distortion conversion receives exactly the table it
received before, and no other body or component gains a correction. The ILCE-7CR distortion misses its
camera JPEG at 24 mm, and shading and CA semantics are not validated on any body; see
docs/sony-lens-corrections.md.
Locating the SR2SubIFD, now done for every ARW that has one, selects the SR2Private offset, length and
key with the TIFF reader's own rules, which lightcraft-tiff now exposes and builds its reader on
(directory_entries, entry_span, ParseOptions::budget_for, decode_value). Only each field's first value
is decoded, so the directory's other entries cost no memory sized by the file: a 16 MiB file whose
SR2Private holds eight fields aliasing it peaks at 37 kB. DirectoryEntries keeps its layout private,
and its positions() iterator uses checked arithmetic.
Tests: synthetic decoder tests; a differential test of the SR2Private selection against the TIFF
reader's in both byte orders (0 to 8,193 filler entries, unreadable and unusual first occurrences of
each field, a used-up value budget); a dhat peak-heap test; public-corpus checks against the values
ExifTool prints, with pinned and anchored SHA-256s (LIGHTCRAFT_CORPUS_REQUIRED=1 makes absent files
fail); and an opt-in local inventory check (crates/raw/tests/sony_lens_inventory.rs).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verified on 54 ILCE-7RM6 photos, the CA planes made the render worse: our
demosaic already hides about 70 % of the lens CA (the render shows 0.29x
the raw-CFA CA) while the post-demosaic warp moves R and B by 0.84x the
table, so the corner R/G and B/G CA went from rms 0.032 / max 0.125 px
(off) to 0.052 / 0.219 px (on) at 1440 px; only the 16-35 mm improved.
warp_opcode takes a lateral_ca flag; lens_opcodes passes false, so a Sony
ARW gets the one-plane distortion warp. The CA conversion stays live and
tested (FF and APS-C examples, both lossless corpus files' tables). The
follow-up is to warp the CFA colour planes before demosaic. The doc,
tracker (docs/parity-checklist.md), docs/gaps.md, docs/raw-parity.md and
whats-new say distortion applied, CA parsed and off
by default, vignetting not applied; the one-plane default costs 24 -> 45 ms
in the sample stage at 1920 px (three planes: 112 ms).
Co-Authored-By: Claude <noreply@anthropic.com>
vendor::arw_lens_rm6::warp_opcode (was distortion_opcode) turns the CA table
(0x7035: red half, blue half, on the distortion knots) into the R and B
planes of the WarpRectilinear: a colour's source radius is
g (1 + c(g) / 3.36e6) at the green source radius g. The unit is provisional
(borderline measurement, about +-30 % scene bias). Each colour plane goes
through the same fit gate as G, and a table with a value beyond 2e-3 relative
(about 12 px at the full-frame corner; the 54 photos' tables stay below
5.1e-4, i.e. 1724) is refused; without a CA table, or when it is refused, the
warp keeps its single G plane, and CA alone is never applied. fit_radial_warp
now rounds its gate up to the next sample, capped at 1 (a NaN gates every
sample), so the band between the last sample inside the crop and the crop
corner is checked too (RW2 passes 1.0: unchanged).
Co-Authored-By: Claude <noreply@anthropic.com>
New vendor::arw_lens_rm6 reads the raw IFD's lens tables (0x7037 distortion,
0x7032 vignetting, 0x7035 lateral CA, their flags) and turns the distortion
table into a one-plane WarpRectilinear in OpcodeList3, with the conventions
measured on 54 ILCE-7RM6 photos: knots at i/15 of the full-frame crop half
diagonal, S(r) = r (1 + v(r)/15000), frame-filling zoom, centred on the
default crop. Only the ILCE-7RM6 is enabled; vignetting is read, not applied.
The radial least-squares fit is split out as vendor::fit_radial_warp (RW2
results unchanged); its error gate can be limited to the radii the crop
shows, so a padded active area (lossless files) doesn't refuse a fit over
pixels that are never rendered.
It sits next to main's vendor::arw_lens (#498), which holds the ILCE-7RM4A
conventions measured against Sony's exports (unit 16384, table indexed at the
output radius); the two are selected by camera model (disjoint allowlists).
arw::decode chains this module's opcodes after arw_lens::distortion, with the
same active area and the same crop (the DNG-style default crop when the file
has one, which on the ILCE-7RM6 equals Sony's crop tags), and like it only
for CFA data (not the downsized linear-RGB lossless sizes, where the knot
spacing in full-frame pixels would not hold).
The corpus test checks the conversion on the two lossless RawDB files (CC0).
The APS-C one is pinned in xtask RAW_SAMPLES with its sha256; the full-frame
one is not (at most two pins per PR, and #625 pins two Compressed ARW 6.0
files), so its case runs only where the file is present.
RENDER_CACHE_VERSION 62 -> 63.
Co-Authored-By: Claude <noreply@anthropic.com>
libtest has no skipped status and hides the output of passing tests, so a
test that returns early without a GPU, a corpus file, a craft-fonts face or
a model passed silently. `lightcraft-testkit` (the testkit slot the layering
table already reserves, dev-dependency only, std only) adds
`skip!(Need, "reason")`: it prints the note as before and, when
LIGHTCRAFT_SKIP_LOG names a file, appends one JSON line (need, reason, test,
binary, file:line). Plain `cargo test` is unchanged. The 61 existing skip
sites use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sources: TIFF 6.0 and own black-box analysis of the three CC0 samples;
white-balance estimate from the embedded preview: own design.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The 2668-entry table read off the black-box oracle's output is gone.
arw6_curve.rs now builds the 4096-entry table at first use from our own
measurement: piecewise linear through knots u_k = 128 k, c_k = u_k for
k <= 11 and 356.89 + 655.646 ln(k - 6) above, rounded half up, fitted to
the 7 same-exposure lossless (LJ92) / ARW6 pairs of the CC0 ILCE-7RM6 set.
A unit test pins the whole table (sum 27776336, weighted 92792980288),
its anchors, the identity through 1408 and strict monotonicity.
The corpus sums become regression values of the new output; the 12-bit
codes before the curve stay verified against the oracle (Phase 0).
docs/arw6-compression.md describes the curve, how it was measured, its
uncertainty, the structure and x2-scale evidence, why it is not in the
file, and the after-the-fact comparison with the oracle's curve; the
tracker and docs no longer call the output bit-exact against LibRaw.
Co-Authored-By: Claude <noreply@anthropic.com>
`cargo xtask corpus --download` now fetches one full-frame Compressed and
one APS-C Compressed HQ file (about 94 MB) instead of all 12 (about
630 MB); the corpus test checks those two and the lossless-sibling
regression test is gone, since nothing would download those files.
docs/arw6-compression.md keeps the full table with source paths and
SHA-256, says which two xtask fetches and how to fetch the rest with
`curl -L`, and records the CC0 1.0 statement of RawDB's set record
(accessed 2026-10-10).
Co-Authored-By: Claude <noreply@anthropic.com>
The RawDB entries in xtask carry their SHA-256 as the third field, which
`cargo xtask corpus --download` checks; the corpus test keeps only the
sensor sum checks.
Co-Authored-By: Claude <noreply@anthropic.com>
Fujifilm RAFs carry per-shot GeometricDistortionParams (0xf00b) and VignettingParams (0xf010)
in the raw IFD. Derived by measurement against the cameras' own JPEGs and LightCraft's
uncorrected renders: [scale, n knots, n values], scale x n = half diagonal of the recorded image,
last knot = farthest corner, distortion in percent indexed by the source radius with the
frame-filling zoom, vignetting as 100/V gain. They become a WarpRectilinear and a
FixVignetteRadial for 59 validated bodies (same path as the Sony ARW and Panasonic RW2
corrections); unmatched bodies and tables with another scale stay uncorrected.
On 67 body/lens combinations the corrected render's radial shape error against the camera JPEG
falls from a median of 0.72 % to 0.05 % of the half diagonal. The lateral CA table (0xf00f) is
real but is not applied: our demosaic already registers the colour planes and the stored shift
over-corrects. See docs/fuji-lens-corrections.md.
RENDER_CACHE_VERSION 48.
Issue #697 (ILCE-7CM2 lossless compressed M-size ARW "washed out" in 0.4
and 0.5) is the 512-vs-1024 YCbCr pedestal of #535 item 3, fixed on main
by 3100cc1a after v0.5.0. Measured on the reporter's file: 0.5.0 renders
the darkest half of the dusk scene +11 L* and blue against the camera
JPEG, with the lamplight turned white (mean dE 11.8); main is within
+3 L* (dE 8.9). Its darkest 0.1 % per channel sits at 1017/1020/1024,
confirming the 1024 pedestal on a fifth body.
- Corpus: two CC0 raw.pixls.us ILCE-7CM2 files, lossless L and the
compressed "M-size" coding, which is Bayer at 4736x3132 with black 512
(only the lossless M/S codings are YCbCr), so a size-based pedestal
rule would break it.
- Tests: corpus_sony_a7cm2_codings (model, layout, levels, crop, as-shot
gains, header/full agreement) and both files in the #535 data-floor
test.
- docs/parity.md: the ILCE-7CM2 verification on the Sony coverage row.
Keep image framing inside the validated 3:2 scope even when DefaultCrop retains native geometry. Cover all 23 Sony fixtures, preserve crop precedence, invalidate old renders, and record fresh Sony reference measurements.
Co-authored-by: Codex <noreply@openai.com>
Main (#575, #580) rewrote default_crop. The image crop now keeps main's
order for every Sony body (Sony's crop tags, then DefaultCrop, then the
recorded-size crop), so framing does not change. Only the ILCE-7RM4A
distortion warp takes its centre and normalization from DefaultCrop when
the file has one, which keeps the geometry validated against Sony's
exports (centre x 4783.5 on the A7R IVA sample). The DNG round-trip test
now checks both: Sony's crop on the image, the warp centred on the
default crop. RENDER_CACHE_VERSION 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's SR2SubIFD white balance (#575) already reads the A900's applied
levels (2688 / 1024 / 1024 / 1516, the same 2.625 / 1.480 gains), so the
hard-coded maker-note 0x0020 block at offset 188 is dropped. The centred
crop now goes through #580's recorded_size_crop (a `centred` flag, only
for the packed 12-bit layout); every other body keeps main's framing.
Adds the CC0 A900 sample to the corpus with checks for its white balance,
black level, bit depth and crop. RENDER_CACHE_VERSION 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
High Efficiency NEFs (HE / HE★, Z 8, Z 9, Z 6III, Z f) keep compression
34713 but carry a JPEG XS-based codestream (SOC + CAP, FF10 FF50) and no
maker-note 0x0096 table. They fell through to "Nikon compressed NEF
without a linearization table", so the Preview only notice gave Z f users
no hint why their raw wasn't decoded or that Lossless compressed NEFs work.
Detect the codestream before asking for the table and report "Nikon High
Efficiency NEF" with the workaround. Add a synthetic regression test, CC0
Z f HE + Lossless samples to the corpus (raw.pixls.us) with a corpus test,
and list HE NEF among the format gaps in parity.md and ROADMAP.md.
No decoder: the format has no public specification and the existing
decoders (LibRaw, rawler) are LGPL and derived from Nikon's binaries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011bbmJTxYJyYqKfNH8uXRC6
Part of #535 (items 4 and 5).
- Pre-2017 bodies without Sony crop tags shot in the camera's 16:9 mode now get
the 16:9 crop of their camera JPEG: width from FullImageSize, anchored left as
before (drops the ILCE-7SM2's 32 padding columns); height from FullImageSize
(ILCE-7SM2) or from an Exif image size of the same width (DSLR-A580, whose
FullImageSize still says 3:2); centred vertically at an even offset, as
measured by registering each camera JPEG on its raw. The aspect now matches
the JPEG, so the camera-look fit runs: ILCE-7SM2 dE 15.2 -> 4.8,
DSLR-A580 dE 26.7 -> 3.9 (vs main 88ea1a8).
- The DSC-R1's SR2 (maker-note FileFormat 1 0 0 0) stores encrypted samples and
decoded to noise; it is now Unsupported in full and header mode and imports
preview-only with that reason.
- Only these 3 of 140 Sony decodes change; render cache version 26 (23-25 are
taken by #568, #542, #575).
- Tests: crop helper (3:2 unchanged, both 16:9 sources, even offsets, hostile
sizes), synthetic Sony file with FileFormat SR2 vs ARW, corpus test of the
two crops and the SR2 refusal; three samples added to the corpus list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #535 (item 1). Stacked on #542 (SR2 black levels); supersedes its
positional keystream tables.
- White balance is WB_RGGBLevels (0x7313), the gains the camera applied:
the raw IFD's plain copy when present, else the copy in the encrypted
SR2SubIFD, which every ARW from the DSLR-A200 on carries (identical to
the plain one on all 57 CC0 files that have both). The maker note's
Tag2010 gains stay only as a last resort.
- 14 bodies had no white balance and rendered green whenever the camera
look was rejected (A450 ... A700, NEX-3/5/C3, SLT-A33/A35/A55,
ILCE-6001, UMC-R10C). On others Tag2010 missed the applied gains by up
to 100 % (ILCE-3500 at 5600 K: R -31 / B +101 % against the camera
JPEG's neutral pixels, now +4 / +0 %; ILCE-7S on Shade -9 / +15 ->
-1 / +1 %; SLT-A57 -25 / +26 -> -6 / -3 %).
- New vendor/sr2.rs decrypts the whole SR2SubIFD, found black-box:
exiftool -v5's decrypted dump as known plaintext gives the keystream
(key 11 22 33 44, the same at every position in 126 CC0 ARWs from 97
bodies, 15 layouts); an exhaustive search over lag pairs found
k[n] = k[n-63] ^ k[n-127] on 32-bit words, so the first 127 words
generate the rest, matching all 126 files (5.2 MB of keystream).
- The SR2 black level (0x7310) is now read through the same decryptor,
replacing #542's head and per-position keystream tables: one SR2
decryption path. Identical results on all 131 Sony files (every
value #542 decoded; the 51 it could not are the files' plain 0x7310).
Unit tests pin the keystream against answers recorded without the
generator: every byte #542 and #148 had recovered, the SHA-256 of the
whole 62112-byte keystream recovered from the CC0 DSLR-A700 sample, and
the encrypted WB_RGGBLevels of four CC0 files (one per layout) with the
levels ExifTool decrypts.
- Sweep (4f350c4 -> this): gains equal ExifTool's decrypted
WB_RGGBLevels on all 100 decoded CC0 CFA files; on 83 files with
>= 200 JPEG-neutral pixels, within 10 % of the camera's applied gains
71 -> 83, over 10 % 5 -> 0. Accepted fits unchanged (max 0.15 dE);
forced fallback median dE 18.50 -> 18.06, neutral-area cast on the 22
changed files 8.8 -> 1.2.
- Corpus test for five CC0 samples (A500, A33, A700, ILCE-3500 5600 K,
ILCE-7S Shade; three added to RAW_SAMPLES); the pre-2017 test's gains
move to WB_RGGBLevels (RX100 III, RX100, ILCE-7RM2: within 2.6 % of
the old Tag2010 values in Auto). Synthetic tests for decryption,
directory parsing, source precedence, black level through the full
DNGPrivateData chain, and hostile input; #542's layout tests (incl.
the A700 trap) now run through the decryptor.
- Render cache version 25.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #535 (items 2 and 3).
- SR2SubIFD black level: decrypt the directory's first entry (always the
black level, 0x7310) to find where its value sits, and decrypt it with the
keystream known for that position (1638, 2166, 2418, 2510). The fixed read at
2510 gave 354-365 on the DSLR-A450/A500/A550 and 975 on the DSLR-A700, where
ExifTool and the data floor say 512 (A700 render: dE 20.1 -> 8.8,
dL* -9.8 -> 0.0). Keystreams recovered black-box (plain 0x7310 and exiftool -v3
as known plaintext), consistent over 115 ARWs from 97 bodies, 15 layouts.
- Downsized lossless M/S ARWs (linear YCbCr): black 1024 and white 16895, i.e.
the recorded 512 plus 512, measured against same-scene full-size codings on
12 files from 4 bodies (offset 1015-1052, scale 0.98-1.02). The 512 pedestal
lifted shadows by ~3 % of white; M/S renders now match their full-size twins
(7RM5 M dE 6.7 -> 3.1).
- Only these 16 of 140 Sony files change; render cache version 24.
- Tests: synthetic SR2 blocks for every known layout (incl. the A700 trap),
YCbCr black/white; corpus test of black levels against the data floor;
A500 and A700 samples added to the corpus list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The YCbCr M coding's pedestal is corrected separately (#535, item 3), so
this test no longer depends on which change lands first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three CC0 raw.pixls.us files of one ILCE-7CR (61 MP) scene: lossless
compressed L and M (subsampled YCbCr) and compressed ARW2. A raw corpus
test pins model, colour-filter layout, black level, white level, as-shot
gains and default crop, and checks that the lossless and compressed
mosaics agree inside the crop (green block correlation 0.9976). An engine
corpus test checks each gets a fitted camera look, not the neutral
fallback. All nine public codings decode and fit (mean dE76 1.59 against
their camera JPEGs).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Samsung SRW raws whose raw IFD (private compression 32769 or 32770) stores the samples uncompressed are decoded from the sensor data: 12-bit samples packed two in three bytes (bit order told apart from the samples; NX5, NX10, NX11 and NX20, NX200, NX210, NX1000, NX1100) and 16-bit little-endian words (EX1, WB2000). CFA layout from the raw IFD's CFAPattern; framing, white balance, black and saturation levels from the maker note (value offsets relative to the note; the vectors are stored offset by its EncryptionKey); a one-direction Exif picture size crops to the centre of the framing; black from dark rows or columns when the file states none; saturation plateau below the stated level wins. Compressed codings (32770 with fewer bytes, 32772, 32773) stay preview only with that reason. srw is imported from folders; SRW takes the file-local camera look. RENDER_CACHE_VERSION 17.
Read DNG 1.6 semantic masks (SemanticName, SemanticInstanceID,
MaskSubArea; any coding the tile/strip reader decodes, e.g. ProRAW's
lossy-JPEG matte) with lightcraft_raw::semantic_masks, map them onto the
developed image, and carry Apple's sky / portrait / skin / hair / teeth /
glasses mattes in SourceInfo::mattes (SourceInfo is no longer Copy).
Sky, Subject, Background and People components sample them through
orientation, warp (Warp::frame, as geometry.rs) and crop like the image,
so preview = export, instead of the heuristics; files without mattes
render as before. Malformed mask IFDs are skipped, and only the first
MAX_MASKS mask IFDs are decoded, so a file full of them costs bounded
work. The render cache version is bumped once, in the white-balance commit.
The corpus test reads corpus/raw/dng-apple-iphone12pro-proraw.dng, the
name `cargo xtask corpus` downloads it under.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parse ProfileGainTableMap (DNG 1.6) and ProfileGainTableMap2 (DNG 1.7):
both versions, DNG 1.7 precedence (version 2 in IFD 0, then the raw IFD;
version 1 in the raw IFD, then IFD 0), u8/u16/f16/f32 gains, gamma.
Malformed tags (wrong size, zero or huge grids, non-finite or negative
gains, bad spacing or gamma) are ignored, never allocated for.
Byte order: the specification gives this tag no byte order of its own,
unlike opcode lists, so the payload is read in the file's byte order with
a big-endian fallback; a wrong-order read can't pass the size checks.
The module docs quote the DNG 1.6 "Byte Order" rule.
The map is kept in ColorData, written back by the DNG writer (version 1,
or version 2 when its gamma isn't 1) and dropped from merge outputs, whose
geometry no longer matches the capture's active area. It is not applied
to the render: Lightroom Classic renders Apple ProRAW without it.
Supersedes the parser of #273.
The CC0 iPhone 12 Pro ProRAW from raw.pixls.us joins the corpus.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compression 52546 tiles and strips decode through jxl-oxide (new
default-on `jxl` feature of lightcraft-raw): bare codestreams and
ISO-BMFF containers, 1 or 3 planes, 8-16-bit integers scaled back
exactly from the codestream's own bit depth, 16-bit floats. Edge tiles
may be coded at their nominal or in-image size. Headers are checked
against the IFD in header-only probing too. A single-chunk JPEG XL
preview IFD (RGB or grey) is offered as the embedded preview.
The decoder's memory is bounded by the tile: 16 bytes per sample (three
times that for a lossy grey tile, which the decoder expands to three
channels) plus 64 KiB; jxl-oxide 0.12 was measured at <= 10.3 bytes per
sample for lossless and <= 13.5 per colour sample for lossy tiles. A
codestream needing more (e.g. extra channels) is an error.
Lossless tiles are checked bit-exact on synthetic CFA and LinearRaw
files (zune-jpegxl as a test-only encoder); lossy tiles are unverified
on a real file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brandon Thomas