Merge origin/main into #480 and apply the review:
- the Samsung-with-a-Pentax-maker-note -> PEF rule now also needs raw evidence; its test fixture
gets a CFA photometric, and the same RGB file without it is not a raw;
- for the Pentax family the Exif CFAPattern (0xa302) counts as raw evidence, as main's PEF reader
takes its layout from it (uncompressed / PackBits PEFs);
- Samsung's private raw compressions 32769..=32773 count for SRW (NX1 / NX500 use 32773);
- the engine fallback in files.rs is dropped (main's version kept): a corrupt raw without a
preview stays an import error; the engine regression test for camera-authored TIFFs is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/main into #555. The decode_hasselblad doc comment records how the coding rule was
established (first hypothesis recalled by the AI author from training data that includes other
converters, corrected for byte order and confirmed by a black-box search over 17,280 layout
hypotheses on the CC0 files; the code is original). The LR-IMP-FORMATS row on main's text now
lists the decoded FFF and lossless-JPEG 3FR files instead of preview-only. RENDER_CACHE_VERSION
37 -> 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's reconstruction (#607) replaces this branch's rewrite of highlight::reconstruct. Kept from the
branch: develop_binned_masked (clipping decided per sample before averaging) and the denoised twin
using the same mask. New: highlight::reconstruct_masked, #607's reconstruction with the clipped
channels told by the mask (block means as lower bounds), shared with reconstruct_with through a
generic clip test (no slowdown on full-size images), and CameraColourModel::rebuild_highlights_masked
so binned previews keep the colour model's clipped white. RENDER_CACHE_VERSION 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/main into #560 (fit_preview / camera-look refactors of #528, #583, #568, #616).
`starting_colour` now tries the per-file camera-JPEG fit (with the camera profile when there is
one) first and uses the spectral-sensitivity matrices only when that fit fails, before the neutral
fallback, so no photo with an accepted JPEG fit changes. Tests pin the order through an injectable
fit; docs, parity, ROADMAP and what's new describe it. RENDER_CACHE_VERSION 37 -> 38.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Single-strip ORFs at 12.8 bits per pixel are 16-byte blocks: ten little-endian
12-bit samples in bytes 0..14 plus a zero pad byte. Main routed them through the
XZ-2 12-bit word path (stripes, magenta cast). New branch before it; the exactly
12.0 bpp XZ-2 branch is unchanged. Layout measured on CC0 files (log and scripts in
data/testing/retry-olympus-e300 and tools/retry-olympus-e300).
RENDER_CACHE_VERSION bumped to 30 (open PRs use up to 29; adjust at merge).
Highlight reconstruction rebuilds clipped channels from the chromaticity of
unclipped pixels nearby. Three changes to that colour:
- an unclipped pixel counts by its brightness ((m / 0.8 clip)^4, at most 1),
filled coarse to fine by push-pull, so dark wires across a bright cloud no
longer tint it (the iPhone frame of #548 turned cyan at Highlights -100);
- pixels within 6 px of clipping (partly demosaiced from clipped samples, 25 %
greener than the sky on that frame) give no colour unless nothing else does;
- the rebuilt colour is drawn toward neutral at its brightest channel as more
channels clip ((n - 1 + s) / 2, s rising as the next channel nears its clip).
Binned previews keep each channel the mean of its block and mark blocks with a
clipped sample in a mask (develop_binned_masked, reconstruct_masked), instead
of taking the block maximum: one clipped specular sample no longer becomes a
green speckle at 500 px (HUAWEI PRA-LX1, WAS-LX1A). The denoised twin uses the
same mask. Camera-colour fitting keeps the old binning.
RENDER_CACHE_VERSION 26 (adjust at merge).
Rows below the split row use the existing lossy Huffman path; from the split
row on every pixel is a (bits - 4)-bit word: a complete prefix code for the
difference category followed by the top bits of the T.81 extra bits, dequantised
to the centre of the kept bin, with the usual two-left predictor and an
unclamped running value. The rule was derived by black-box analysis of the 11
CC0 samples (all decode to the same samples as the analysis decoder).
The decoder checks that the fixed-rate section ends within the last byte of the
strip (whole rows plus 0..7 padding bits) and otherwise keeps refusing the file
(preview only), so a file that does not follow the rule is never decoded wrongly.
RENDER_CACHE_VERSION 28 (main is 22, open PRs use 23-27): adjust at merge.
A DNG can state a white level the sensor never reaches (an iPhone 15 Pro Max Bayer DNG states 4095 but clips at 3567), so clipped highlights were not recognised and white balance tinted them magenta. On decode, samples that form a plateau clearly below the white level, with nothing above it, are raised to the white level. RENDER_CACHE_VERSION 25 (needs adjusting at merge).
Sony's Dynamic Range Optimizer (factory default Auto) brightens the darker
regions of the camera JPEG, not the raw. The tone curve fitted to that JPEG
carried the brightening into the raw's starting look: on 8 ILCE-7CR photos
with DRO on, LightCraft rendered +6.4 L* from Lightroom's default (median
dE76 7.5), against -0.2 L* (3.6) for the 16 with DRO off.
- lightcraft_raw::embedded_preview_dynamic_range_optimized reads Sony
maker-note tag 0xb025 (every documented value: 0 off; 1-3, 8-12 and
16-23, i.e. up to Lv8, on). It was present on all 264 Sony raws checked
(98 bodies); 0xb04f only says on/off and is missing on older bodies.
- With DRO on, the look keeps the colour fitted to the JPEG and its tone
curve is lowered, knot by knot, to Sony's curve without DRO wherever that
is darker (assets/camera-tone/sony-dro-off.json, pooled from 12 CC0
raw.pixls.us ARWs of 10 interchangeable-lens bodies; sources and
derivation in the file and assets/ATTRIBUTION.md). The second-last knot
is lowered only as far as keeps the shoulder past the last knot at or
below the fitted curve, so the result is never brighter anywhere.
Compacts that map raw values darker keep their own curve.
- DRO off, unknown and other makers: unchanged (byte-identical renders on
71 + 35 photos). Render cache version 23.
DRO-on photos now land at +0.8 L* from Lightroom (median dE76 3.3), band by
band like the DRO-off ones. Docs: camera-preview-colour.md, parity row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A raw without colour matrices of its own now gets DNG ColorMatrix1/2 and
ForwardMatrix1/2 for its model when the model is one of the 52 cameras
in rawtoaces-data (Academy Software Foundation, Apache-2.0): 22 Canon,
11 Nikon, 11 Sony, 3 Fujifilm and 5 others, plus the other names its
alias list gives (Rebel T3i and Kiss X5 for the EOS 600D; bodies it
lists as sharing a sensor). tools/spectral_camera_table.py fits, under
Standard Illuminant A and D65, a white-preserving 3x3 to the measured
spectral sensitivities over rawtoaces-data's 190 training reflectances
and writes crates/raw/src/spectral_table.rs. No number comes from Adobe
or from a raw decoder. lightcraft_raw::spectral looks a camera up by
Exif make and model; the DNG colour model does the rest.
camera_preview::starting_colour takes the first available of: the
file's own matrices, a camera profile for the model (fitted to the
JPEG for tone), the spectral matrices, the file-local JPEG fit, the
neutral fallback. A profiled model whose profile fit fails takes its
spectral matrices. White balance stays relative to the as-shot look for
every raw without matrices of its own, so Photo::relative_wb and the
controls are unchanged; calibrate still pools covered models.
Tests: one per precedence step (synthetic), the table's lookup, aliases
and DNG shape, and a corpus test of the order on ILCE-7M4 (profile),
EOS 5D Mark III and D5100 (spectral), EOS 7D (own fit) and a DNG. The
ILCE-7RM2 and GFX100S corpus tests now expect the spectral matrices.
Attribution in assets/ATTRIBUTION.md and NOTICE. RENDER_CACHE_VERSION 23.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lossless JPEG scans whose SOS gives predictor selection value 8 (which T.81 leaves undefined) are read as Hasselblad codes them (ljpeg::decode_hasselblad): 32-bit little-endian words read from their most significant bit, no byte stuffing; each row in pairs of samples, both Huffman categories before both differences; category 16 is followed by 16 ignored bits and means -32768; each sample is the previous sample of its column parity in the row plus the difference, modulo 2^16, both starting at 32768 on every row. Only the layout measured is accepted (one component, 16 bits, no point transform or restarts, even width); other layouts and selection values above 8 stay Unsupported. The rule was measured on 14 CC0 files with the ljpeg-explorer tool. RENDER_CACHE_VERSION 23.
Renders a raw's DNG ProfileGainTableMap (Apple ProRAW's local tone mapping)
when the photo's Profile option "Camera local tone mapping" is on
(`profile.camera_local_tone`, off by default: Lightroom Classic renders
ProRAW without it). Built on #427's parser; supersedes the decode-time
applier of the first version of #273.
- raw/gaintable: GainTableMap::gain / evaluator (GainEval) per DNG 1.7.1:
bilinear between tables with edges replicated, input clamp(weights .
(R,G,B,min,max))^gamma in linear ProPhoto after the baseline exposure,
index input x MapPointsN clamped to the last point. Saturating cell
math; maps whose sizes don't match their gains aren't evaluated.
SourcePlacement: where a developed source sits in the active area.
- pipeline/local_tone: the gain per output pixel in the scene-linear
stage, before the white balance, mapped back through the frame (crop,
user orientation, warp) and the EXIF orientation. lin_needs_cpu routes
the stage to the CPU when it's on (CPU and GPU match); lin_key includes
it. SourceInfo::local_tone carries the map from the decoder.
- develop: Profile::camera_local_tone (omitted from JSON when off, so
existing settings and their hashes are unchanged); Profile row checkbox
when the source has a map; translations.
- RENDER_CACHE_VERSION 17. Docs: parity row and What's New as an opt-in.
Off renders pixel-identical to main on two iPhone 15 Pro ProRAWs (12 and
48 MP). On adds ~60-75 ms to the scene-linear stage at 1536x2048 on a
2-core VM, the same for both files (cost follows output pixels).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VFqow2P1EtDxt6nRjjQoTA
Samsung SRW raws whose raw IFD (private compression 32769 or 32770) stores the samples uncompressed are decoded from the sensor data: 12-bit samples packed two in three bytes (bit order told apart from the samples; NX5, NX10, NX11 and NX20, NX200, NX210, NX1000, NX1100) and 16-bit little-endian words (EX1, WB2000). CFA layout from the raw IFD's CFAPattern; framing, white balance, black and saturation levels from the maker note (value offsets relative to the note; the vectors are stored offset by its EncryptionKey); a one-direction Exif picture size crops to the centre of the framing; black from dark rows or columns when the file states none; saturation plateau below the stated level wins. Compressed codings (32770 with fewer bytes, 32772, 32773) stay preview only with that reason. srw is imported from folders; SRW takes the file-local camera look. RENDER_CACHE_VERSION 17.
Compression 32773 is accepted in the PEF reader only when every strip has exactly the packed size of its rows at the declared bit depth, and is then read as uncompressed MSB-first samples. Any other 32773 size stays unsupported. Uncompressed PEFs take their colour filter from the Exif CFAPattern as seen from the image area origin. A Samsung-branded TIFF whose maker note has the Pentax layout is probed as PEF. RENDER_CACHE_VERSION 16 to 17.
One config-dir helper (engine::config::config_dir; camera_profiles re-exports it) with both
the face-models and denoise-models defaults; hosts set up both model folders; Settings has
Faces and AI Denoise tabs; both command sets registered; both model dialogs; locale entries
from both; one [features] table per app manifest; Cargo.lock regenerated by cargo; the
DecodedSource cache cost counts both segmentation mattes and the denoised twin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TIFF raws that describe a full-resolution CFA IFD with DNG's tags (levels, crop, AsShotNeutral) but carry no DNG version tag are read by the DNG reader (RawFormat::CfaTiff, dng::decode_as). RGGB when the file states no CFAPattern. Lossless JPEG scans with a predictor selection value above 7 are rejected with that reason and stay preview-only. .3fr and .fff are imported from folders. RENDER_CACHE_VERSION 17.
Recover the unpublished denoise work against current upstream. Preserve later opt-in model setup, shared Rust HTTPS downloads, pending-photo resume and an Amount-independent switch. Keep original RAWs unchanged and share the current color finish between ordinary and denoised sources.
Bound public tiling inputs, refuse corrupt-cache exports, and retain synthetic/real-model coverage. Include two Windows path spelling fixes in existing tests. No model weights, face code, media or Tract runtime are included.
Validation: cargo xtask ci (all seven gates); installed RawNIND CPU/GPU agreement 1.38e-6; native CPU preview/export on a public 24 MP Sony RAW, unchanged original SHA-256.
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Generated with Codex.
ColorData::baseline_sharpness holds the file's BaselineSharpness (raw IFD
or IFD 0; zero, negative, non-finite or above 16 is ignored) and the DNG
writer writes it back, so a conversion or smart preview keeps it. Merge
outputs leave it unset. Rendering doesn't use it yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read DNG 1.6 semantic masks (SemanticName, SemanticInstanceID,
MaskSubArea; any coding the tile/strip reader decodes, e.g. ProRAW's
lossy-JPEG matte) with lightcraft_raw::semantic_masks, map them onto the
developed image, and carry Apple's sky / portrait / skin / hair / teeth /
glasses mattes in SourceInfo::mattes (SourceInfo is no longer Copy).
Sky, Subject, Background and People components sample them through
orientation, warp (Warp::frame, as geometry.rs) and crop like the image,
so preview = export, instead of the heuristics; files without mattes
render as before. Malformed mask IFDs are skipped, and only the first
MAX_MASKS mask IFDs are decoded, so a file full of them costs bounded
work. The render cache version is bumped once, in the white-balance commit.
The corpus test reads corpus/raw/dng-apple-iphone12pro-proraw.dng, the
name `cargo xtask corpus` downloads it under.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parse ProfileGainTableMap (DNG 1.6) and ProfileGainTableMap2 (DNG 1.7):
both versions, DNG 1.7 precedence (version 2 in IFD 0, then the raw IFD;
version 1 in the raw IFD, then IFD 0), u8/u16/f16/f32 gains, gamma.
Malformed tags (wrong size, zero or huge grids, non-finite or negative
gains, bad spacing or gamma) are ignored, never allocated for.
Byte order: the specification gives this tag no byte order of its own,
unlike opcode lists, so the payload is read in the file's byte order with
a big-endian fallback; a wrong-order read can't pass the size checks.
The module docs quote the DNG 1.6 "Byte Order" rule.
The map is kept in ColorData, written back by the DNG writer (version 1,
or version 2 when its gamma isn't 1) and dropped from merge outputs, whose
geometry no longer matches the capture's active area. It is not applied
to the render: Lightroom Classic renders Apple ProRAW without it.
Supersedes the parser of #273.
The CC0 iPhone 12 Pro ProRAW from raw.pixls.us joins the corpus.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compression 52546 tiles and strips decode through jxl-oxide (new
default-on `jxl` feature of lightcraft-raw): bare codestreams and
ISO-BMFF containers, 1 or 3 planes, 8-16-bit integers scaled back
exactly from the codestream's own bit depth, 16-bit floats. Edge tiles
may be coded at their nominal or in-image size. Headers are checked
against the IFD in header-only probing too. A single-chunk JPEG XL
preview IFD (RGB or grey) is offered as the embedded preview.
The decoder's memory is bounded by the tile: 16 bytes per sample (three
times that for a lossy grey tile, which the decoder expands to three
channels) plus 64 KiB; jxl-oxide 0.12 was measured at <= 10.3 bytes per
sample for lossless and <= 13.5 per colour sample for lossy tiles. A
codestream needing more (e.g. extra channels) is an error.
Lossless tiles are checked bit-exact on synthetic CFA and LinearRaw
files (zune-jpegxl as a test-only encoder); lossy tiles are unverified
on a real file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Widens the preview-only fallback. A recognised raw whose decode fails now falls back
to its embedded JPEG for any decode error (unsupported, corrupt, limit, TIFF
structure), not only an unsupported variant, and the preview search covers more
containers:
- Canon CRW, Minolta MRW and Sigma X3F are recognised by their magic (RawFormat
Crw, Mrw, X3f) instead of failing as "RawOther files are not supported yet".
- A TIFF whose IFD0 has no ImageWidth/ImageLength but a JPEG pointer or SubIFDs
(Kodak KDC) is a raw container, not a malformed image. TIFF compression 99
(Leaf MOS) counts as a raw compression.
- When no tag points to a preview, a recognised raw we cannot decode is scanned
for the largest complete three-component DCT JPEG. A preview whose first byte
was overwritten (Minolta, Epson) is restored. A one-component JPEG is skipped
(a PowerShot CRW stores its mosaic that way).
Files with no JPEG at all still fail with "without an embedded preview".
Phase One / Leaf IIQ, Canon EOS-1D / 1Ds and Kodak DCS TIFFs keep their raw data in
a private block. IFD0 is only a 160-640 px RGB copy and no IFD is marked as raw, so
they opened as an ordinary TIFF of that size (a 150 MP file rendered 296x220) with
no preview-only label.
probe() now recognises such a file from its structure (OtherTiff): a picture at
least 4x larger than IFD0 in both directions is declared by a SubIFDs child, or by
Exif PixelXDimension/PixelYDimension together with enough bytes outside every IFD's
strips and tiles to hold it at one bit per pixel. decode() reports the sizes as the
reason. A raw we cannot decode now falls back to its embedded JPEG, else to the TIFF
reader's view of its first image, so it imports preview-only with that reason.
RENDER_CACHE_VERSION 12: these files render differently.
Decode lossless CRX and version 0x100/0x200 C-RAW in portable Rust,
including sensor geometry, timed white balance and black/white levels.
Verify all sensor samples against six independent M50/R100/R8 references.
Any CR3 decode error falls back to the embedded JPEG, as CR3 files did
before this decoder. The full sensor buffer is allocated only after the
first plane decodes, and plane buffers grow row by row. Tile planes
decode in parallel with a windowed bit reader (24 MP: 424 -> 89 ms
lossless, 519 -> 138 ms C-RAW on an M4).
Add guarded CR3 camera-JPEG colour fitting with independent framing
validation, relative white balance and cache invalidation.
Validation: cargo xtask ci.
Split out of #240 without the compressed decoder (crates/raw/src/vendor/orfc.rs
is not part of this change; compressed ORF stays preview-only).
- The layout is read from Exif CFAPattern (0xa302, repeat counts in either
byte order) when it describes a 2x2 Bayer cell. BGGR and GBRG were not
reachable before.
- Without a usable tag the green diagonal is still found from the samples,
now from totals of 32x32 blocks instead of single-pixel differences, so
fine texture no longer outweighs a small red/blue difference. The old
version indexed the samples directly; this one reads them with get().
- probe_info no longer unpacks 12-bit packed ORFs that state their layout
(XZ-2: under 0.1 ms; it ran the same unpack as a decode before, which
takes 11 ms). 16-bit files still read their samples to find the bit depth.
Tests: stated layouts in both byte orders, unusable tags falling back to the
samples, the texture case, hostile areas for the fallback, probe_info equal
to decode; a corpus test comparing the E-1, E-400 and XZ-2 mosaics with
their embedded JPEGs under every layout.
No user-visible change yet for the bodies whose layout the old guess got
wrong (E-M1 and E-620 are BGGR): they write compressed ORF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decode every raw encoding Panasonic and Leica bodies write, from the DMC-LX1 (2005) to the DC-S1RM2 (2025):
compressed formats 4 (most bodies, 2008-2022) and 6 (S1 / S1R / S1H / S5, GH5M2), the prefix-coded strips of
format 8 (GH6 and later), packed formats 2, 5 and 7, and the 16-bit words of the 2005-2007 .RAW files. Until now
only format 5 decoded; every other Panasonic file opened as preview only.
Clean-room: no raw-decoder source was read. Tag names come from tag documentation; every encoding was recovered
by black-box analysis of 178 CC0 files from raw.pixls.us (118 bodies), and the module docs note the measurement
that settled each rule. Format-8 tags of unknown meaning must carry the values every sample has, otherwise the file
is reported as unsupported rather than guessed at.
Levels and geometry: formats 4 and 6 store samples 15 above the recorded black level (files without one use 15);
white is the active area's saturation plateau (the linearity limits are wrong on extended low-ISO shots, the GF1
and the Digilux 3); the in-camera aspect ratio is the default crop; mapped-out defect pixels are filled from
same-colour neighbours; the CFA pattern applies at sample (0, 0).
.rwl and .raw files are imported (engine, MCP folder expansion, file dialog, Linux MIME types). One file per
encoding joins the never-panic proptest; 11 CC0 files join `cargo xtask corpus`, and corpus_panasonic_encodings
checks each encoding's bit depth, black level, crop and CFA layout against the camera's own JPEG.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtQmZ6peZXGn3wiqeowV2v
Lightroom-converted DNGs carry their camera profile's ProfileHueSatMapData1/2
and ProfileLookTableData (and sometimes ProfileToneCurve). We used only the
colour matrices, so such DNGs rendered flat and muted.
- raw: new `profile` module reads the tags (DNG spec ch. 6: value-major,
hue-middle, saturation-minor tables, wrap-around hue, tri-linear lookups,
sRGB value encoding) from the raw IFD or IFD 0; malformed tags are ignored.
- engine: the raw loader applies the hue/sat map (interpolated by the same
illuminant weight as the matrices) before BaselineExposure and the look
table after it, in linear ProPhoto; a ProfileToneCurve becomes the source's
camera tone. Without one the standard base tone is kept.
- dngwrite keeps the look with the data (conversions, smart previews, merges).
- Render cache version 7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decode Nikon Huffman-compressed NEF (compression 34713): lossless and lossy
(type 1 and 2) at 12 and 14 bits, with the maker-note 0x96 linearization
curve, predictor seeds and either byte order. Clean-room: 0x96 layout and
encoding-table sizes from prose write-ups (Clevy, Claff); the fixed Huffman
tables were recovered by black-box analysis of CC0 raw.pixls.us samples
(compressed vs uncompressed shots of the same scene) and verified on ~45
files from ~35 bodies. Lossy-after-split files stay preview only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brandon Thomas