Quantile matching fits the held-out pixels slightly worse than the conditional-median fit it replaced. On 7 of 1023 archive files (compacts, bridge cameras, early Micro Four Thirds) the look passed the acceptance gates only with the old curve, and the quantile curve lost it. The search now runs with the quantile fit first (all pixels, away from edges, with and without the camera profile) and, only when no look passes, runs again with the earlier conditional-median plus isotonic fit, restored as fit_tone_conditional. The gates are unchanged, so no file loses its look relative to the previous fit, and the quantile curve is used wherever it passes. Tests: the flat-step test stays; a synthetic photo whose JPEG carries per-pixel noise misses the gates with the quantile fit and passes with the fallback, and the order of the two is checked.
The per-photo look fitted its tone curve as the camera JPEG's median per
scene-luminance bin followed by isotonic pooling. On lens-corrected, sharpened
JPEGs of busy scenes neighbouring bins disagree by more than the gap between
them, so pooling produced flat steps: 13 of 31 segments on a CC0 Z 6II sample,
flat segments in 9 of 13 fitted Nikon Z / ILCE-7 samples. Each step maps a
range of scene tones to one output tone, which renders as posterised patches
with hard edges (issue #475, also #211, #454), and the conditional medians
flatten both ends of the curve.
Quantile matching (sorted scene quantiles against sorted JPEG quantiles) is
increasing by construction and insensitive to misregistration. On the same 13
files no fit has a flat segment, all still pass the gates, held-out RMS within
0.005 of before. Regression test: a misregistered synthetic JPEG.
Samsung SRW raws whose raw IFD (private compression 32769 or 32770) stores the samples uncompressed are decoded from the sensor data: 12-bit samples packed two in three bytes (bit order told apart from the samples; NX5, NX10, NX11 and NX20, NX200, NX210, NX1000, NX1100) and 16-bit little-endian words (EX1, WB2000). CFA layout from the raw IFD's CFAPattern; framing, white balance, black and saturation levels from the maker note (value offsets relative to the note; the vectors are stored offset by its EncryptionKey); a one-direction Exif picture size crops to the centre of the framing; black from dark rows or columns when the file states none; saturation plateau below the stated level wins. Compressed codings (32770 with fewer bytes, 32772, 32773) stay preview only with that reason. srw is imported from folders; SRW takes the file-local camera look. RENDER_CACHE_VERSION 17.
CR2 and PEF decode without a camera colour matrix, like ARW, NEF, RW2,
RAF and CR3, but camera_preview::file_local_look (and the catalog's
relative white balance) left them out, so they rendered with the neutral
fallback matrix: the camera preview showed first, then a flat,
desaturated render replaced it. They now use the same guarded fit and
relative WB; a fit that fails the gates keeps the fallback, so a file can
only stay as it was or move closer to its camera JPEG. Not measured on
the corpus here (raw.pixls.us is blocked from this environment).
RENDER_CACHE_VERSION 15 re-renders cached CR2/PEF thumbnails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01912WaK6Rmew3wxZuPyX7E8
Found running cargo xtask ci with the corpus on Windows. The two changes
landed on the same day: the Fujifilm test (#282) expected the compressed
X-T20 file to keep the neutral fallback because its fit failed the gates
on all pixels (held-out RMS 0.114 against 0.10). The retry away from
edges (#263, issue #232) fits it (0.056), so the fit is accepted at both
sizes. Expect true.
Decode lossless CRX and version 0x100/0x200 C-RAW in portable Rust,
including sensor geometry, timed white balance and black/white levels.
Verify all sensor samples against six independent M50/R100/R8 references.
Any CR3 decode error falls back to the embedded JPEG, as CR3 files did
before this decoder. The full sensor buffer is allocated only after the
first plane decodes, and plane buffers grow row by row. Tile planes
decode in parallel with a windowed bit reader (24 MP: 424 -> 89 ms
lossless, 519 -> 138 ms C-RAW on an M4).
Add guarded CR3 camera-JPEG colour fitting with independent framing
validation, relative white balance and cache invalidation.
Validation: cargo xtask ci.
Sony A7R II camera JPEGs are distortion-corrected by default while the raw is not. On the public
CC0 ILCE-7RM2 samples the JPEG is up to 15 px of 1440 off, and the misaligned window frames alone
pushed the held-out RMS to 0.108-0.115 (gate 0.10): the photo opened in the grey neutral fallback,
13 L* too dark. The same shot with distortion correction off fits at 0.059.
When the fit on all pixels fails the gates, try it once more, with the same gates, on pixels whose
3x3 neighbourhood in either proxy spans a luminance ratio of at most 3. Fits that already pass are
unchanged (29 other corpus ARW/NEF/RW2 files render pixel-identical). ILCE-7RM2 ΔE76 vs the camera
JPEG (registered) 16.6-16.9 -> 3.7-3.8; corpus GX80 19.1 -> 7.9. RENDER_CACHE_VERSION 11 -> 12.
RW2 / RWL / RAW join ARW and NEF: the starting look (matrix, hue/saturation table, tone and chroma curves) is
fitted to the file's own embedded JPEG behind the same gates, with white balance relative to the as-shot look.
Panasonic previews always show the whole active area while the default crop is the in-camera aspect ratio, so when
the preview matches the active area instead of the crop, the reference is cropped to the default crop before
fitting (corpus_rw2_with_an_in_camera_crop_gets_a_camera_look).
On 174 CC0 samples with a preview 140 fits are accepted (93 of the 114 bodies with a preview), median ΔE against
the camera JPEG 5.0 with matching lightness; most of the others are compacts and kit zooms whose camera JPEG is
distortion-corrected (tag 0x0119 isn't used yet). `lightcraft-cli calibrate` keeps collecting ARW and NEF only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtQmZ6peZXGn3wiqeowV2v
Conflicts with issue #150 (NEF colour from the camera JPEG): the per-file
look, its hue/saturation/value table and chroma curve, and camera
profiles now apply to every `file_local_look` format (ARW, NEF, NRW);
`calibrate` also collects NEF/NRW. Render cache version 11 (main 9,
branch 10).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One photo can show too little of a colour for its own fit: a lime shirt
covering a few dozen proxy pixels next to foliage at the same hue stayed
yellow (hue 68 deg vs the JPEG's 82 deg).
- `lightcraft-cli calibrate [--max N] [--out DIR] FOLDERS...` pools the
colour pairs of many ARWs per model (each against its embedded JPEG,
192 px proxies, <= 4000 pairs per photo) and fits one matrix from
white-balanced camera RGB plus a hue/saturation/value table.
- Profiles live in $LIGHTCRAFT_CAMERA_PROFILES or
<config>/camera-profiles/<model>.json; validated on load, damaged files
ignored with a warning, part of the render cache keys.
- Photos of a profiled model take its colour and fit only their own tone
and chroma curves; the acceptance gates still apply.
- The table fit buckets samples by hue and runs nodes in parallel; the
app's config_dir moves to the engine.
Profile from 300 Feb-Aug ILCE-7M4 photos (2 min); 29 held-out September
photos: mean dE vs camera JPEG 3.54 -> 3.10, saturated 6.41 -> 5.20;
the shirt's hue 68 -> 82 deg (JPEG 81.5).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A sunny ILCE-7M4 waterfall shot rendered yellow: sunlit rock came out
cream, +4 b* in the brightest fifth of the picture. Camera JPEGs make
shadows up to 1.4x more colourful and bleach highlights to 0.1-0.3x
(a per-channel tone curve), which a luminance-only curve can't do.
- CameraTone gains an 8-node chroma-by-display-luminance curve, fitted
per file (bright pixels included) and kept only when it lowers the
held-out error; applied after the tone map on CPU and GPU, keeping
luminance. Old smart-preview headers load with an identity curve.
- The hue/saturation table gets an sRGB-encoded value axis: dark yellows
had to turn toward orange while bright yellow-greens turned toward
green. It is applied after the baseline exposure, as it is fitted.
- Matrix-only and matrix+table looks are completed (tone, chroma) and
compared in gamma-encoded values; acceptance gates stay linear.
29 local ARWs vs their camera JPEGs: mean dE 4.23 -> 3.54, every file
closer, saturated areas 7.65 -> 6.41. Render cache v10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A global 3x3 matrix can't follow the camera's hue-dependent rendering:
on an ILCE-7M4 photo of a lime shirt among grey rocks the best matrix
rendered the shirt olive (16 deg hue error), since the few saturated
pixels carry little weight and a matrix fitted to them breaks the rest.
After the matrix, fit a 72x5 hue/saturation table (DNG ProfileHueSatMap
layout, linear ProPhoto) to the residual hue shifts and saturation
ratios: kernel-weighted per node, shrunk to identity where samples are
sparse, identity at saturation 0. It preserves luminance (the tone curve
owns it) and is kept only when it lowers the held-out error.
28 local ILCE-7M4 ARWs: mean dE vs camera JPEG 4.74 -> 4.27, none worse
overall; the shirt's hue error drops to 6-8 deg. Render cache v9.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LightCraft has no colour matrix for Nikon cameras. Because of this, NEF
files showed colours that were too gray and too green.
- Colour: LightCraft now calculates a colour matrix and a tone curve from
the JPEG preview in each NEF file. Sony ARW files use the same
procedure and the same quality checks. White balance is relative to
the as-shot look, as for ARW files.
- One function, camera_preview::file_local_look, selects the formats for
the fit and for the relative white balance. Photo::relative_wb in the
catalog uses the same formats.
- Black level: Nikon stores the black level (maker note 0x003d) in
14-bit units, also in 12-bit files. The decoder read it as a 12-bit
value. Because of this, 12-bit files from the D750, D780, D850, D7500
and Z 50 were too dark. Now the decoder divides the value by 4 for
12-bit files.
- The corpus gets the D7500 12-bit and 14-bit lossless samples. A test
compares their levels. The render cache version increases to 9.
Results on 13 NEF files that LightCraft can decode: LightCraft accepted
the fit for all 13 files. The mean colour difference from the camera
JPEG (CIE76 dE) decreased from 13-46 to 2.9-7.8. One file with mixed
light improved by a small quantity only (12.7 to 12.0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 0.055 cap rejected half of the public Sony samples tried (ILCE-6400,
ILCE-6700, ILCE-7M4) although their fits were 1.5-3x closer to the
camera JPEG than the fallback and visibly better (mean dE vs the embedded
preview 19-26 -> 5-10). Unrelated previews stay rejected (synthetic test
lands at 0.115).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brandon Thomas