`embedded_preview_srgb` (the loupe stand-in and the grid thumbnail of an unedited raw) decoded the
largest embedded JPEG whatever the size asked for: a 512 px thumbnail of a Sony A7R VI ARW decoded
its 9984 x 6656 JPEG (5.6 MB) although the file holds a 1616 x 1080 one (0.3 MB).
`lightcraft_raw::embedded_preview_covering(bytes, min_edge)` returns the smallest of a TIFF-based
raw's JPEGs whose long edge is at least `min_edge` and that is a copy of the largest one: aspect
ratio within 1 % (not a letterboxed thumbnail of another shape) and the same number of components
(not the grey semantic masks of an iPhone ProRAW DNG); else the largest, as `embedded_preview`.
The colour-space handling stays in `decode_raw_preview` (a `covering` argument); an unsupported
raw's editing source and the camera-look fit keep the largest JPEG, so a source never depends on
its size level.
On 120 raws of every make in the corpus at 256 / 512 / 1600 px: wherever the largest JPEG is still
chosen the output is identical byte for byte; where a smaller one is (Sony, Nikon, Panasonic,
Adobe DNG) orientation matches, the size is equal or 1-2 px off, mean difference <= 4.7 / 255.
- Rebased onto #1020: its plan route (`preview_window` -> `planned_preview`) now takes the box size and picks
with `embedded_preview_covering` too, so a TIFF raw read through the sparse plan gets the whole file's JPEG.
Co-Authored-By: Claude <noreply@anthropic.com>
Verified on 54 ILCE-7RM6 photos, the CA planes made the render worse: our
demosaic already hides about 70 % of the lens CA (the render shows 0.29x
the raw-CFA CA) while the post-demosaic warp moves R and B by 0.84x the
table, so the corner R/G and B/G CA went from rms 0.032 / max 0.125 px
(off) to 0.052 / 0.219 px (on) at 1440 px; only the 16-35 mm improved.
warp_opcode takes a lateral_ca flag; lens_opcodes passes false, so a Sony
ARW gets the one-plane distortion warp. The CA conversion stays live and
tested (FF and APS-C examples, both lossless corpus files' tables). The
follow-up is to warp the CFA colour planes before demosaic. The doc,
tracker (docs/parity-checklist.md), docs/gaps.md, docs/raw-parity.md and
whats-new say distortion applied, CA parsed and off
by default, vignetting not applied; the one-plane default costs 24 -> 45 ms
in the sample stage at 1920 px (three planes: 112 ms).
Co-Authored-By: Claude <noreply@anthropic.com>
vendor::arw_lens_rm6::warp_opcode (was distortion_opcode) turns the CA table
(0x7035: red half, blue half, on the distortion knots) into the R and B
planes of the WarpRectilinear: a colour's source radius is
g (1 + c(g) / 3.36e6) at the green source radius g. The unit is provisional
(borderline measurement, about +-30 % scene bias). Each colour plane goes
through the same fit gate as G, and a table with a value beyond 2e-3 relative
(about 12 px at the full-frame corner; the 54 photos' tables stay below
5.1e-4, i.e. 1724) is refused; without a CA table, or when it is refused, the
warp keeps its single G plane, and CA alone is never applied. fit_radial_warp
now rounds its gate up to the next sample, capped at 1 (a NaN gates every
sample), so the band between the last sample inside the crop and the crop
corner is checked too (RW2 passes 1.0: unchanged).
Co-Authored-By: Claude <noreply@anthropic.com>
New vendor::arw_lens_rm6 reads the raw IFD's lens tables (0x7037 distortion,
0x7032 vignetting, 0x7035 lateral CA, their flags) and turns the distortion
table into a one-plane WarpRectilinear in OpcodeList3, with the conventions
measured on 54 ILCE-7RM6 photos: knots at i/15 of the full-frame crop half
diagonal, S(r) = r (1 + v(r)/15000), frame-filling zoom, centred on the
default crop. Only the ILCE-7RM6 is enabled; vignetting is read, not applied.
The radial least-squares fit is split out as vendor::fit_radial_warp (RW2
results unchanged); its error gate can be limited to the radii the crop
shows, so a padded active area (lossless files) doesn't refuse a fit over
pixels that are never rendered.
It sits next to main's vendor::arw_lens (#498), which holds the ILCE-7RM4A
conventions measured against Sony's exports (unit 16384, table indexed at the
output radius); the two are selected by camera model (disjoint allowlists).
arw::decode chains this module's opcodes after arw_lens::distortion, with the
same active area and the same crop (the DNG-style default crop when the file
has one, which on the ILCE-7RM6 equals Sony's crop tags), and like it only
for CFA data (not the downsized linear-RGB lossless sizes, where the knot
spacing in full-frame pixels would not hold).
The corpus test checks the conversion on the two lossless RawDB files (CC0).
The APS-C one is pinned in xtask RAW_SAMPLES with its sha256; the full-frame
one is not (at most two pins per PR, and #625 pins two Compressed ARW 6.0
files), so its case runs only where the file is present.
RENDER_CACHE_VERSION 62 -> 63.
Co-Authored-By: Claude <noreply@anthropic.com>
Move the forward-map inversion, least-squares fit and solve4 out of the RW2
decoder into vendor::warp_from_forward_map; RW2 calls it in DNG-normalised units.
Co-Authored-By: Claude <noreply@anthropic.com>
The hue/saturation table of a fitted camera look (ILCE-7RM4A, APS-C ILCE-7RM6, ...) ran
HsvTable::apply pixel by pixel: branches on the data (largest channel, hue sector),
saturating float-to-usize casts and bounds-checked table reads, ~550 instructions and 1.2
branch misses per pixel. CameraColourModel::apply now runs it 16 pixels at a time
(HueSat::apply_lanes -> HsvTable::apply_lanes, structure of arrays): per pixel the same IEEE
operations in the same order, with the branches turned into selects and the table indices
computed as exact f32 integers, so the compiler vectorises everything but the table reads,
powf and the rare rem_euclid, which run lane by lane. DNG profile tables and the other colour
models keep the per-pixel path.
Measured on a Core Ultra 9 185H P-core (busy machine, min of N):
- HsvTable alone, 72x5x5 fitted-look-shaped table, random colours, one thread:
243 -> 106 cycles (57 -> 26 ns) per pixel.
- Colour stage of a full-size load, one thread: 60 MP ILCE-7RM4A 3.80 -> 1.99 s,
26 MP APS-C ILCE-7RM6 HQ 1.88 -> 0.98 s; 4 CPUs: 1.02 -> 0.67 s, 0.56 -> 0.30 s.
Exactness: new tests compare the lanes with the one-by-one path bit for bit (to_bits) on tens
of thousands of colours (every combination of zero, -0, tiny, negative, around 1, huge, NaN and
+-inf components; grey; hues at the 0/360 degree wrap; near-black noise in the restore's blend)
for fitted, value-scaled, 2D, many-turn and degenerate tables, 16 and 5 lanes, and on an image
whose rows aren't a whole number of lanes. Stage fingerprints (decode, AHD, 2560 px and
full-size loads) of 7 Sony raws are identical to origin/main.
Sources: own design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The D1X/D1H raw SubIFD is 4028x1324 with XResolution 600 / YResolution 300:
the pixel is twice as tall as it is wide. Nothing read that, so the
picture came out about 3:1. Add a pixel aspect to RawImage/RawInfo, read
it from the raw IFD resolution ratio (NEF) and from DefaultScale (DNG),
accept only 1.5..4, and resample the rows after demosaic. developed_size()
reports the stretched size, binning is off for non-square pixels, and the
catalog and lens-profile sizing use developed_size().
header_plan skipped an ARW6's strip but its first 64 KB and then said Ready, while the ARW6
header probe (Mode::Header) validates every tile: its picture header, stream tables and the
first lines of its residual stream, which lie in the skipped bytes. The probe of the partly
read buffer then failed ("ARW6 first-half row count outside 2..=7") where the whole file
succeeds: sparse_corpus failed on arw6-sony-ilce7rm6-apsc-land-c.arw, the first ARW6 file it
reads (it stops at the first failure). Like NEF, ORF and PEF, whose header probes read
samples, an ARW6 now gets HeaderPlan::Whole from the header plan. The preview plan is
unchanged: the embedded preview reads no tile, and the 13 ARW previews in the corpus are
still read from 21 % of their bytes.
Only the test reads the header plan today (the app's partial reads use preview_plan), so no
import changes. New test: a synthetic two-tile ARW6 whose second tile lies past a first one
of bulk size (Ready before, Whole now). sparse_corpus passes on the 8 ILCE-7RM6 ARW6 files.
cargo test --workspace: 3205 passed, 1 failed, visualize::tests::
all_masks_overlay_shows_every_visible_mask_at_once, which fails on main b3d392b4 too
(PR #1456 addresses it); fmt, clippy, parity, layers, assets and wasm pass.
Sources: own design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The two halves of each stream are independent bit streams: they now decode in parallel
(`decode_stream`) instead of one after the other inside each of a tile's 13 streams.
- The vertical inverse 5/3 runs its rows in parallel. The horizontal pass lifts each row in place
(`lift_row`, scratch space reused per thread), rows in parallel, instead of allocating two vectors
per row and copying the result back; a test checks it against the vertical pass, row by row.
Best of 3 on 4 CPUs, previous and new decoder alternating: ILCE-7RM6 FF HQ 0.60 → 0.48 s
(process CPU 1.14 → 0.87 s), FF Compressed 0.62 → 0.48 s, APS-C HQ 0.34 → 0.23 s, APS-C
Compressed 0.31 → 0.24 s (docs/arw6-compression.md → Timings). The decoded mosaic is identical
(hash of every sample) on the 8 ARW6 files of the RawDB ILCE-7RM6 set and 3 private A7R VI photos.
Co-Authored-By: Claude <noreply@anthropic.com>
- The vertical inverse 5/3 lifts whole rows at once (`inverse_lift` with samples `w` values wide)
instead of gathering each column through a strided iterator and scattering it back: the same
integer arithmetic, sequential memory. The horizontal pass is the same function with w = 1.
- `0^n 1` codes are counted with one `leading_zeros` over the next MAX_DPT + 1 bits instead of bit by
bit (same bits consumed, same errors); each line reserves its values up front.
- Colour reconstruction reads plane rows as slices and runs over rows in parallel (it was one
thread per tile: 4 threads on a full-frame file whatever the machine).
Best of 5 on 4 CPUs, previous and new decoder alternating: ILCE-7RM6 FF HQ 0.56 → 0.46 s,
FF Compressed 0.54 → 0.38 s, APS-C HQ 0.34 → 0.23 s, APS-C Compressed 0.35 → 0.24 s
(docs/arw6-compression.md → Timings). The decoded mosaic is identical (hash of every sample) on
the 8 ARW6 corpus files and 2 private full-frame photos (ISO 100, ISO 12800); the pinned corpus
sums still match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three changes, each leaving the output bit for bit as it was:
- Each step runs only where the tile's output depends on it (choice ±0, homogeneity ±1, Lab ±2,
greens ±3, mosaic ±5 inside the 6 px margin), with direct neighbour indices instead of an
index clamped on every read; the clamped values at the buffer edge never reached the output.
- A candidate pixel is built in registers before its Lab conversion: writing its channels
through an index and reading the pixel back stalled on store forwarding (the hottest
instructions of the profile).
- Tiles are written into the output one band at a time, instead of all 64 MP of tiles being
collected and then copied (797 MB held twice). Tiles inside the image copy their mosaic rows
instead of reflecting and taking a modulo per sample.
Sony ILCE-7RM6 9984 × 6656 Compressed HQ raw, demosaic only, best of 5 on 4 cores:
1.85 s → 1.05 s wall, 6.3 s → 4.0 s CPU. Verified identical (f32 bits) on 7 real raws
(ILCE-7RM6 FF/APS-C, ARW6 HQ/C and lossless, ILCE-7RM4A) and, against the previous code, on
random mosaics of all four Bayer layouts at 1 × 1 … 300 × 7, 257 × 130 (tile edges, reflection).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/cr3.md gains a variant-status table: every supported and rejected CRX
coding variant, the function that gates it and the exact Unsupported text.
crates/raw/tests/cr3_variants.rs enforces the rejections (19 tests). Each
rejected variant is rebuilt as a small synthetic CR3 (container, CMP1
descriptor, CRX stream) and the test asserts the exact string, and that
probe_info (the import path) and decode return the same error. Control
fixtures decode through the same container, including both edges of the
accepted uniform quantization range (4..=44) and adaptive QP range
(131..=168), so a rejection can only come from the documented gate. The
table itself is prose and is not checked against the tests.
The doc said only QP values 131-167 were accepted; the decoder also accepts
168 (quantization value 44, step 102), so that sentence now says so.
Tests and docs only; no decoder change.
crates/raw/tests/robust.rs only mutated DNGs. Add two small valid CR3
fixtures (a 4x4 lossless CRX and a 16x16 three-level C-RAW CRX) and a
property that flips bytes in them or truncates them, then runs the existing
exercise() helper (probe, embedded_preview, decode, develop and the colour
transform) plus probe_info, the header-only description used on import:
errors or clean decodes, never a panic. Unlike random TIFF-like bytes, a
structurally valid file gets past the container walk, so the mutations land
in the CRX marker walker and entropy readers. A fixture test asserts that
both files decode as built, so the mutations hit live decode paths.
Tests only; the existing DNG property keeps its 400 cases and the new one
uses proptest's default (256, PROPTEST_CASES overrides it).
crates/raw/tests/crx_properties.rs drives lightcraft_raw::decode with
synthetic CR3 containers built around CRX streams that the test encodes
itself: reference entropy coders for the MED, Rice, MELCODE and QP-map
layouts, and the exact algebraic inverse of one 5/3 lifting line. The
tile-halo coefficients are free parameters of that inverse, so synthesis
must reconstruct any encoded plane exactly. Tests only; no decoder change.
- Wavelet round trip: random planes decode exactly across odd plane sizes
and multi-tile halo configurations (eight geometries, arbitrary halo
values).
- Quantization: every uniform quant value 4..=44 and every adaptive QP value
131..=168 dequantizes within the 14-bit range. Overshoot is clamped, and
an in-range plane lands exactly on level * step, so the step sequence
itself is observable. Quant 3 and 45 and QP 130 and 169 are rejected by
decode and probe_info alike. Random QP maps, including odd map heights,
with random per-band base and gain stay in range.
- Lossless samples outside the bit depth are Corrupt errors instead of being
clamped; in-range planes decode exactly.
- Adversarial entropy payloads planted in well-formed lossless, C-RAW,
version 0x200 and QP-map streams either error or decode deterministically
and never panic. A QP Rice prefix longer than the 23-zero escape is
Corrupt.
- Random IAD1 geometry either decodes with the active area inside the
sensor and the crop inside the active area, or is rejected.
- Decoding gives identical results across repeated and four-way parallel
invocations.
- Odd QP-map heights decode: the last QP row has no partner and is used
alone, and QP values are averaged before the step lookup. Large QP jumps
take the escape code.
The case budget is proptest's default (256 per property); set
PROPTEST_CASES to change it.
Four CC0 EOS RP samples from raw.pixls.us (lossless RAW and C-RAW, full
frame and 1.6x crop mode) now have exact full-sensor regressions against
the DNGLab 0.8.0 black-box oracle (Windows binary, checked first against
the documented M50 RAW hash). The decoder already matched; no decoder
change was needed. Crop mode is a single 4032x2656 tile, full frame two
3192-wide tiles.
docs/cr3.md also records that the white level is Canon's specular white,
which at the -1/3-stop ISOs (e.g. 11062 at ISO 5000) is well below the
sensor's saturation point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: crates/engine/src/tests_merge.rs (#1123's test landed where this branch's is; both kept),
docs/parity-checklist.md (the LR-MERGE-HDR row: both notes) and docs/whats-new.md (both entries).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015gRtdWF1E9358FfzhzYqCS
Merged HDR DNGs lost the camera's embedded lens correction: load_frame strips the lens opcodes from each
source (frames are merged uncorrected) and write_linear_dng wrote no OpcodeList3, so Optics ▸ Enable Lens
Corrections did nothing on the result (Sony ILCE-7M3/7RM4A, Panasonic/Leica, DNGs with their own opcodes).
Each raw frame now keeps the lens corrections it applies (the DNG 1.6 skip rule already applied),
re-expressed for the default crop develop keeps by the new Opcode::cropped: the centre moves into the crop
and coefficients are rescaled for the crop's farthest corner. HdrResult carries the reference frame's and the
DNG writes them as its OpcodeList3. Frames turned before merging (mixed orientations) and panoramas carry none.
Tests: cropped corrections give the crop of the corrected whole (WarpRectilinear, WarpRectilinear2 with and
without reciprocal, FixVignetteRadial); an HDR merge of cropped, corrected brackets, upright and vertical, gives
the same profile correction as its reference photo (fails without the fix). Real brackets not measured yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015gRtdWF1E9358FfzhzYqCS
The D90 writes 4352 x 2868 samples whose last 42 columns (from 4310) are
filler at a flat ~254, above its dark image samples (black 0). The
masked-column test only trims columns near zero, so they stayed in the
image and developed as a pink band on the right. Measured on 10 of 550
NEFs from one body: the camera's 4288 x 2848 JPEG sits at (10, 10), and
green saturates at 3840-3851 while red and blue run on to 4095, so one
white level of 3830 keeps clipped highlights white. The D90 joins the D4
in MEASURED_BODIES; other bodies are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sony ARWs of cameras without measured colour matrices were edited on the
relative scale (6500 K / 0 = as shot). Their SR2SubIFD stores the
multipliers of the camera's presets at 2500, 3200, 4500, 6000 and 8500 K;
lightcraft_raw::kelvin_presets reads the as-shot multipliers on the curve
through them and they take the spectral matrices' place in the #730
design (as-shot Kelvin and tint only; look, Bradford adaptation and the
pixel-identical legacy upgrade unchanged).
Covered: ILCA-99M2, ILCA-77M2, SLT-A33, DSC-RX10, ILCE-3500, ILCE-7S.
Kelvin-dial shots read back 3400 -> 3396, 5200 -> 5193, 4700 -> 4708,
5600 -> 5594 K; Shade 7493 K. Off-curve constants (isotherm tilt 0.74,
tint spacing 1) measured from the 14 Sony bodies with both spectral
matrices and presets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Masked lines are measured against the outermost line's level when it sits
just above BlackPoint (K-70, K-1), saturated outer lines count when dark
ones follow (645Z), and a frame whose interior is within the tolerance
keeps its whole area. Line means read samples with get(). Render cache
version 62: PEF frames change size.
An uncompressed Leaf Aptus 22 MOS keeps its 16-bit mosaic in IFD0
(compression 1, BlackIsZero, one sample per pixel) with no Make tag and
no CFA photometric, so `probe` returned None and the file imported as a
greyscale image of the mosaic. Leaf's data block, tag 0x8606 (LeafData
in the ExifTool EXIF tag-name documentation), now marks such an IFD0 as
a raw: OtherTiff, preview only with the reason, as the lossless-JPEG MOS
already is. A rendered RGB image that kept the block stays an image.
The Hasselblad Lunar is a Sony NEX-7 that writes an ARW under Make
HASSELBLAD; only SONY files reached the ARW reader, so it opened as its
preview. A HASSELBLAD file with Sony's raw coding (an IFD with
compression 32767 or the tone-curve tag 0x7010, which no 3FR / FFF has)
now goes there and decodes like a NEX-7: black 512, the SR2SubIFD white
balance. The Tag2010 white-balance fallback, the one table arw.rs keys
by model, takes the body from the maker note's SonyModelID (0xb001)
when the file is not Sony-branded.
Docs: the Lunar is verified in docs/raw/evidence.tsv (236 verified),
raw-parity.md, parity-checklist.md, gaps.md and whats-new.md.
RENDER_CACHE_VERSION 61 -> 62: a Lunar's thumbnails were rendered from
its preview.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The DNG CFAPattern describes the top-left of the ActiveArea, but RawImage::cfa is sensor-anchored (the crop to the active area shifts it by the area's offset). The DNG reader now moves the tag's pattern out to the sensor origin, and the DNG writer states the pattern back at the ActiveArea. A DNG whose area starts on an odd row or column (the Canon EOS 60D DNG of #1086, top 51) no longer reads RGGB as GBRG.
Cause: a floating-point CFA DNG is read as stored (no check of its float
samples). AHD and PPG clamp each green estimate between its two neighbouring
greens with `est.clamp(a.min(b), a.max(b))`; when both greens are NaN the
bounds are NaN and `f32::clamp` panics (ppg.rs:45, ahd.rs green pass). Any
float DNG with two adjacent NaN greens crashed decode + develop. Infinities
alone did not panic but turned into NaN in the demosaic arithmetic.
Fix, at two layers:
- demosaic: a `clamp_between` helper replaces the three clamps. With finite
or infinite bounds it is the same `f32::clamp` call (bit-identical
results); with NaN bounds it returns the estimate unclamped, so the panic
is gone whatever puts NaN into the mosaic.
- DNG decode: non-finite float samples are replaced where the white level is
known. NaN and -inf read as black (0), +inf as the white level (a clipped
highlight). No sensor records NaN or infinity, so a valid file is never
touched; finite samples, negative or above white included, stay as stored.
Tests: `demosaic::tests::non_finite_samples_do_not_panic` (NaN block, +/-inf
and an all-NaN mosaic through Bilinear, PPG and AHD; plus `clamp_between`
equal to the old clamp bit for bit over finite, signed-zero and infinite
values) and `non_finite_float_samples_decode_and_develop` (a float CFA DNG,
uncompressed and Deflate, with NaN and +/-inf samples: decoded values
checked, develop with all three methods gives a finite image). Both panicked
before the fix at ppg.rs:45 / ahd.rs.
Co-Authored-By: Claude <noreply@anthropic.com>
libtest has no skipped status and hides the output of passing tests, so a
test that returns early without a GPU, a corpus file, a craft-fonts face or
a model passed silently. `lightcraft-testkit` (the testkit slot the layering
table already reserves, dev-dependency only, std only) adds
`skip!(Need, "reason")`: it prints the note as before and, when
LIGHTCRAFT_SKIP_LOG names a file, appends one JSON line (need, reason, test,
binary, file:line). Plain `cargo test` is unchanged. The 61 existing skip
sites use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One CC0 raw.pixls.us file per uncompressed Olympus ORF and Samsung SRW
body not yet pinned (23 files, 17 Adobe models) is added to RAW_SAMPLES
and pinned in crates/raw/tests/olympus_samsung_corpus.rs: own sensor
hash plus a black-box check against the camera's embedded JPEG (the
E-10 pinned only, inspected by eye).
Also: as_chunks in spots.rs for the chunks_exact_to_as_chunks lint.
Docs: evidence.tsv + coverage re-measure (252 verified), gap 10,
checklist, raw-parity, ROADMAP. Headline unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
