Since #281, probe takes a Sony file as a raw only with raw evidence, and
two layouts have none of the kinds has_raw_ifd looks for: SRF (DSC-F828,
an RGB-looking IFD0 of 14-bit samples) and the DSLR-A100 (raw data
addressed by SubIFDs). Both were "not a readable photo". For Sony, IFD0
samples of 12 or 14 bits, or model DSLR-A100 with an old-style JPEG IFD0
(compression 6) and SubIFDs, now count too; an 8- or 16-bit camera TIFF,
or an exported A100 TIFF with a SubIFDs thumbnail, stays an image. The public F828 and
A100 samples open through their previews as on main 5436951; 133 other Sony files unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A grid thumbnail and a loupe preview of an unedited raw show the camera's embedded JPEG, which
sits near the start of the file — for a 24 MP Fuji RAF, 5.6 MB of a 56 MB file (measured:
DSCF9492.RAF). The loader read the file whole and threw the sensor data away, so every preview
paid for 50 MB nobody looked at. On a local disk that is merely wasteful; on a network share,
where photographers keep libraries, it is the wait: the same preview took 10 s over a GVFS/SMB
Wi-Fi share against 0.96 s after this change.
- `lightcraft_raw::bounded_preview(bytes)` locates the preview from the header alone and reports
its byte range; `embedded_preview_from_range` extracts it, byte for byte what `embedded_preview`
returns for the whole file (the pointer is read from the RAF header's fixed u32s at 84/88).
- `engine::files::preview_window` reads a 4 KiB prefix, then — because the container reader clamps
a slice to the buffer it was given, so a short prefix reports a plausible but truncated 3948
bytes — re-reads exactly the extent the header declares, and takes the window out of the prefix
rather than reading those bytes twice. `LIGHTCRAFT_PROFILE=1` prints what each preview cost.
- Formats whose preview only the whole file reveals (CR3 boxes, TIFF pointer tags past the prefix,
CRW/MRW/X3F scans), a lying pointer, a short read and a file that changes underneath us all fall
back to the whole-file path: a damaged header never costs the user their preview.
- Tests: the header range matches `embedded_preview` byte for byte, with a clamped prefix, a padded
extent, a late pointer, a lying pointer and a non-JPEG window covered; on a synthetic RAF with a
200 KB tail the quick path and the whole-file path produce identical pixels and the read stays
under half the file; `crates/engine/tests/preview_read.rs` (opt-in, `#[ignore]`) measures a real
file on a share. No raw fixture is committed.
- `docs/raw-preview-reads.md` records the mechanism and the numbers.
Co-Authored-By: DeepSeek V3.2 <noreply@deepseek.com>
The R5 Mark II's APS-C crop-mode files (5376 x 3574) state an IAD1 active area of x 132..5243,
but columns 132..~256 sit at the black level and the image runs to the right edge (measured on
raw.pixls.us 7883/7884), while their recommended crop, x 272..5359 (5088 x 3392, the camera JPEG's
size), lies on the image. The recommended crop didn't fit inside the stated active area, so the
whole active area was kept: a black band down the left and ~130 image columns lost.
When the recommended crop doesn't fit inside the stated active area, it is now the valid area.
On 7883 the render's high-pass correlation with the camera JPEG goes from -0.13 to 0.99. The
active area and crop of the other 108 decodable CC0 CR3s on raw.pixls.us are unchanged.
RENDER_CACHE_VERSION 49.
Raws without matrices in the file (ARW, NEF, RW2, RAF, CR2, CR3, PEF, SRW,
ORF) were all edited on the relative white-balance scale (6500 K / 0 = as
shot), even when lightcraft_raw::spectral has measured matrices for the
camera. Those 52 bodies (ILCE-7M3, ILCE-7M4, D850, EOS 5D Mark III...) now
edit in Kelvin like a DNG and like Lightroom.
Design:
- One decision on both sides: spectral::find_camera("Make Model") backs
catalog::relative_wb_camera (Photo::relative_wb); files::WbScaleOf decides
the same from the decoded file. A corpus test asserts they agree.
Photo::fitted_look keeps the old predicate for smart previews.
- The matrices only convert: As Shot is the file's multipliers read through
the model's ColorMatrix1/2 (spectral::as_shot_white_through). The look
(JPEG fit / profile / spectral fallback) is unchanged, and the slider
adapts from that white with Bradford. The camera-space rebalance stays
reserved for files with their own matrices.
- Presets resolve to their Kelvin values (WbMode::preset, sources cited).
- The Color panel shows the white balance in effect (effective_wb).
- XMP: crs::Target::for_file takes the camera; matrix cameras read
crs:Temperature as Kelvin (marked "scale": "kelvin"), others keep the
#510 as-shot shift.
Migration (no look changes):
- WhiteBalance.scale: WbScale { Legacy (absent in JSON), Kelvin }. A
white balance without a scale keeps its old meaning. On a spectral-Kelvin
raw (SourceInfo.legacy_relative_wb) the pipeline composes the old
adaptation (6500 -> value) with the new one (as shot -> ...) in Bradford
cone space (local::legacy_white). Von Kries scalings compose exactly, so
the matrix matches to f32 and the pixels are identical
(pipeline legacy_relative_white_balance_renders_as_before_on_a_kelvin_raw,
media old_relative_white_balance_on_a_kelvin_camera_keeps_its_look...).
- Every white-balance writer (develop.set/adjust/quickAdjust/wb/wbPick/
reset) first runs local::normalize_wb: a legacy value becomes the Custom
Kelvin value with the same look and is marked Kelvin. Partials without a
scale (old presets, develop.merge) stay legacy.
- Catalog format 7: Op::SetAsShotWb replaces the 6500/0 as-shot recorded
by older imports when the file is next decoded or reloaded (system op,
not an undo step). Format-version test updated.
- The web wire carries legacy_relative_wb.
Docs: parity (LR-EDIT-COLOR-TEMP), camera-preview-colour, xmp-interop,
lightroom-catalog-import, control-protocol, mcp, whats-new.
white_from_data anchored the plateau search on the absolute maximum, so a
handful of stray samples above saturation (Canon EOS 70D CR2: 83 of 20.6M,
up to 14426 vs. a plateau at 13583) made it fall back to 2^bits-1. The
image rendered ~0.3 EV dark and clipped highlights were never seen as
clipped. Skip up to half a plateau's worth of strays when locating the top.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The D4 records no CropArea. Its 4992 x 3292 data keeps 2 columns of optical
black left of the image and 42 right of it (then 8 empty ones), which showed
as purple stripes on both sides; the camera's own 4928 x 3280 JPEG sits at
(8, 6). Green saturates at 15546-15672 while red and blue run on to 16383,
so the data's own top left clipped highlights magenta; one level of 15520
keeps them white without moving the white balance.
Measured on 148 uncompressed 14-bit NEFs from one body by aligning the
sensor data with the embedded camera JPEG.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fujifilm RAFs carry per-shot GeometricDistortionParams (0xf00b) and VignettingParams (0xf010)
in the raw IFD. Derived by measurement against the cameras' own JPEGs and LightCraft's
uncorrected renders: [scale, n knots, n values], scale x n = half diagonal of the recorded image,
last knot = farthest corner, distortion in percent indexed by the source radius with the
frame-filling zoom, vignetting as 100/V gain. They become a WarpRectilinear and a
FixVignetteRadial for 59 validated bodies (same path as the Sony ARW and Panasonic RW2
corrections); unmatched bodies and tables with another scale stay uncorrected.
On 67 body/lens combinations the corrected render's radial shape error against the camera JPEG
falls from a median of 0.72 % to 0.05 % of the half diagonal. The lateral CA table (0xf00f) is
real but is not applied: our demosaic already registers the colour planes and the stored shift
over-corrects. See docs/fuji-lens-corrections.md.
RENDER_CACHE_VERSION 48.
`probe_info` (every file an import adds) ran the whole Nikon Huffman
decode on a compressed NEF because `nef::decode` had no header mode and
the masked-trailing-column test wanted the samples: 386 ms per 24 MP
file, the full decode time, where every other format probes in < 1 ms.
The header has no source for those columns (the D5100 and D7000 write 8
masked columns at value 0, compressed and uncompressed alike, with no
CropArea or other maker-note tag), so the probe cannot skip the test.
Instead both modes now judge the active width from the first 128 rows
(`nef::MASK_ROWS`) against the nominal white level: the header probe
decodes only those rows (`nefc::decode_rows`, the stream is sequential;
uncompressed files read only the strips holding them) and the full
decode uses the same rule, so `probe_info == RawImage::info` still
holds. On every corpus NEF/NRW the answer is the same for 8 to 512 rows
as for the whole frame. The whole-image size and truncation checks still
apply in both modes; the "lossy after split" end-of-strip rule is only
checked when the whole strip is decoded.
Release, same files: probe 66 -> 3.4 ms (D5100 lossless), 72 -> 3.2 ms
(D7000 lossy 12), 88 -> 3.1 ms (D7500 lossless 12), 92 -> 3.7 ms (Z f
lossless), 23 -> 1.5 ms (D5100 uncompressed), 54 -> 2.7 ms (B700 NRW).
Tests: a synthetic lossless NEF whose stream is cut off after two
thirds of the rows probes to the same info as the whole file while the
full decode reports the truncation; synthetic uncompressed NEFs with
masked columns in the top rows, lower rows, all or none give the same
active area in both modes. Both failed before the fix. Docs: the
`probe_info` doc comment, docs/parity.md (LR-IMP-FORMATS),
docs/gpu-pipeline.md.
Issue #697 (ILCE-7CM2 lossless compressed M-size ARW "washed out" in 0.4
and 0.5) is the 512-vs-1024 YCbCr pedestal of #535 item 3, fixed on main
by 3100cc1a after v0.5.0. Measured on the reporter's file: 0.5.0 renders
the darkest half of the dusk scene +11 L* and blue against the camera
JPEG, with the lamplight turned white (mean dE 11.8); main is within
+3 L* (dE 8.9). Its darkest 0.1 % per channel sits at 1017/1020/1024,
confirming the 1024 pedestal on a fifth body.
- Corpus: two CC0 raw.pixls.us ILCE-7CM2 files, lossless L and the
compressed "M-size" coding, which is Bayer at 4736x3132 with black 512
(only the lossless M/S codings are YCbCr), so a size-based pedestal
rule would break it.
- Tests: corpus_sony_a7cm2_codings (model, layout, levels, crop, as-shot
gains, header/full agreement) and both files in the #535 data-floor
test.
- docs/parity.md: the ILCE-7CM2 verification on the Sony coverage row.
Keep image framing inside the validated 3:2 scope even when DefaultCrop retains native geometry. Cover all 23 Sony fixtures, preserve crop precedence, invalidate old renders, and record fresh Sony reference measurements.
Co-authored-by: Codex <noreply@openai.com>
FixBadPixelsConstant listed every pixel equal to its constant, and the
FixBadPixelsList cap was checked only after a whole rectangle was pushed.
Cap both at the number the list arm already used (1 << 22).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Merge origin/main into #480 and apply the review:
- the Samsung-with-a-Pentax-maker-note -> PEF rule now also needs raw evidence; its test fixture
gets a CFA photometric, and the same RGB file without it is not a raw;
- for the Pentax family the Exif CFAPattern (0xa302) counts as raw evidence, as main's PEF reader
takes its layout from it (uncompressed / PackBits PEFs);
- Samsung's private raw compressions 32769..=32773 count for SRW (NX1 / NX500 use 32773);
- the engine fallback in files.rs is dropped (main's version kept): a corrupt raw without a
preview stays an import error; the engine regression test for camera-authored TIFFs is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/main into #555. The decode_hasselblad doc comment records how the coding rule was
established (first hypothesis recalled by the AI author from training data that includes other
converters, corrected for byte order and confirmed by a black-box search over 17,280 layout
hypotheses on the CC0 files; the code is original). The LR-IMP-FORMATS row on main's text now
lists the decoded FFF and lossless-JPEG 3FR files instead of preview-only. RENDER_CACHE_VERSION
37 -> 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's reconstruction (#607) replaces this branch's rewrite of highlight::reconstruct. Kept from the
branch: develop_binned_masked (clipping decided per sample before averaging) and the denoised twin
using the same mask. New: highlight::reconstruct_masked, #607's reconstruction with the clipped
channels told by the mask (block means as lower bounds), shared with reconstruct_with through a
generic clip test (no slowdown on full-size images), and CameraColourModel::rebuild_highlights_masked
so binned previews keep the colour model's clipped white. RENDER_CACHE_VERSION 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/main into #560 (fit_preview / camera-look refactors of #528, #583, #568, #616).
`starting_colour` now tries the per-file camera-JPEG fit (with the camera profile when there is
one) first and uses the spectral-sensitivity matrices only when that fit fails, before the neutral
fallback, so no photo with an accepted JPEG fit changes. Tests pin the order through an injectable
fit; docs, parity, ROADMAP and what's new describe it. RENDER_CACHE_VERSION 37 -> 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main (#575, #580) rewrote default_crop. The image crop now keeps main's
order for every Sony body (Sony's crop tags, then DefaultCrop, then the
recorded-size crop), so framing does not change. Only the ILCE-7RM4A
distortion warp takes its centre and normalization from DefaultCrop when
the file has one, which keeps the geometry validated against Sony's
exports (centre x 4783.5 on the A7R IVA sample). The DNG round-trip test
now checks both: Sony's crop on the image, the warp centred on the
default crop. RENDER_CACHE_VERSION 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's SR2SubIFD white balance (#575) already reads the A900's applied
levels (2688 / 1024 / 1024 / 1516, the same 2.625 / 1.480 gains), so the
hard-coded maker-note 0x0020 block at offset 188 is dropped. The centred
crop now goes through #580's recorded_size_crop (a `centred` flag, only
for the packed 12-bit layout); every other body keeps main's framing.
Adds the CC0 A900 sample to the corpus with checks for its white balance,
black level, bit depth and crop. RENDER_CACHE_VERSION 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Testing the first version on ILCE-7RM4 photos found a partly clipped sky between
bare branches leaning cyan to teal under #523's edit, where Lightroom shows white.
Where only green clipped, red and blue say such a sky is a pale cyan-blue; the
rebuild kept that colour until halfway to full clipping, and darkening and a camera
profile showed it. Colour borrowed from another surface (bright foliage below the
sky) could also push green up, with a small step where green first clips.
- highlight::reconstruct_with: the colour of a pale partly clipped pixel
(saturation relative to the colour model's white below 0.2, easing off by 0.4)
fades to the clipped neutral from the first clip on instead of from halfway;
brightness and strongly coloured pixels keep the old schedule.
- Borrowed colour counts less and less as both the pixel's own channels and the
rim's colour contradict it (0.15 to 0.4 stop); either alone can be wrong.
- Right past the rim every estimate continues the rim's colour (it used to hold
the clipped channels at their clip, for the neutral estimate only), so the first
clip has no step whatever lent the estimate. The rim is built whenever there is
clipping.
- Nothing jumps where a later channel clips: in the rim distance and the
contradiction, a clipped channel counts as the lower bound it is, and a channel
goes over to that from 0.75 of its clip; a channel about to clip is a floor
capped like the clip floor it becomes.
- Tests: a pale sky above yellow-green foliage with thin dark branches in it (no
step where green clips, no green from the foliage, the same beside the branches,
neutral well past the clip); the fringe test compares the sky beside the branch
with the sky away from it, on its way to neutral; a band whose red clips second
next to a rim of another colour renders without a step, in six layouts.
- docs/parity.md: the Highlights note. Render cache version 28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Uncompressed 16-bit words are read in the other byte order only when the
file's order leaves more than 1% of sampled words at or above
1 << max(bits, 14) and the other order leaves none. Measured on the CC0
DSC-R1 (raw.pixls.us 3221). Crop and white balance (SR2SubIFD) are not
addressed. RENDER_CACHE_VERSION 31 (adjust at merge).
Single-strip ORFs at 12.8 bits per pixel are 16-byte blocks: ten little-endian
12-bit samples in bytes 0..14 plus a zero pad byte. Main routed them through the
XZ-2 12-bit word path (stripes, magenta cast). New branch before it; the exactly
12.0 bpp XZ-2 branch is unchanged. Layout measured on CC0 files (log and scripts in
data/testing/retry-olympus-e300 and tools/retry-olympus-e300).
RENDER_CACHE_VERSION bumped to 30 (open PRs use up to 29; adjust at merge).
dev-null-undefined