Canon's SensorInfo left border is not where light starts: on every corpus
body 10-36 columns left of it already hold image (EOS 6D: masked columns
0-70, border 84), and the black level was averaged over all of them. The
6D read 2591 instead of 2047 (4 % of the range), so CR2s rendered dark with
crushed shadows; 40D +141, 5DS R +131, 5D Mark II +73. The masked run is
now measured from the data: columns whose mean stays within 1/256 of the
range of the leftmost columns', minus a two-column margin.
Against Lightroom Classic (EOS 6D still life, 2000 px, ΔE00): base 7.63 ->
5.23 (p95 24.7 -> 11.3); mean over 86 edit variants abs 9.13 -> 6.19,
effect 3.09 -> 2.75.
RENDER_CACHE_VERSION 23.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ORF decoder supplies the as-shot white balance (ImageProcessing
WB_RBLevels) and no colour matrix, like ARW, NEF, RW2, RAF, CR2/CR3, PEF
and SRW, but ORF was left out of camera_preview::file_local_look and the
catalog's relative_wb, so it rendered with the neutral fallback matrix:
flat, grey and dark. It now gets the same guarded fit to its camera JPEG
and white balance relative to the as-shot look. Corpus (held-out RMS
against the fallback, all accepted): E-1 0.126 -> 0.014, E-400 0.087 ->
0.022, XZ-2 0.061 -> 0.023; compressed ORFs still open from their preview.
Against Lightroom Classic (CC0 E-1 kit, 2000 px, ΔE00): unedited render
11.25 -> 8.30; mean over 86 variants abs 12.10 -> 9.48, effect 3.44 ->
2.80. Every other kit's unedited render is byte-identical.
RENDER_CACHE_VERSION 23.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sensor-clipped pixels are rebuilt at raw decode. The tone map rolls them off to
white at Neutral, so any colour they carry shows only when Highlights, a negative
Exposure or Whites darkens them. They took colour from the nearest unclipped pixel
(dark foliage, lens fringes along branches, pixels demosaiced from clipped samples),
left two clipped channels at their white-balance-scaled clip levels (red or pink),
and were neutral only in camera RGB, not after the colour model (a look fitted to
the camera's JPEG, a camera profile).
- highlight::reconstruct_with: chromaticity only from bright unclipped content away
from edges and from the clip, through a normalised push-pull pyramid; a neutral
estimate blended in where reliable content is scarce at every scale up to a
quarter of the long edge, applied only as far as the pixel lies past the nearest
near-clip unclipped pixels (so a surface still runs into clipping without a
step); clipped pairs keep the estimate's ratio at their clip floors; partly
clipped pixels fade into the clipped neutral on their way from the first channel
clipping to the last; fully clipped pixels become the colour model's neutral.
Pixels without a clipped channel are left exactly as they were.
- engine: the decode colour step as CameraColourModel, with clipped_white, which
keeps the camera's neutral for nearly singular or implausible models.
- Tests in raw (continuity along a ramp through every clip, also as lines 1 to 16
rows thick with no reliable colour, on canvases 32 to 1024 rows), engine
(pipeline renders at Highlights -100, #523's edit and Exposure -2 through the
first clip, over the same sweep; a CC0 ILCE-7RM4 corpus file added to
RAW_SAMPLES) and the GPU equivalence suite.
- Render cache version 27.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
High Efficiency NEFs (HE / HE★, Z 8, Z 9, Z 6III, Z f) keep compression
34713 but carry a JPEG XS-based codestream (SOC + CAP, FF10 FF50) and no
maker-note 0x0096 table. They fell through to "Nikon compressed NEF
without a linearization table", so the Preview only notice gave Z f users
no hint why their raw wasn't decoded or that Lossless compressed NEFs work.
Detect the codestream before asking for the table and report "Nikon High
Efficiency NEF" with the workaround. Add a synthetic regression test, CC0
Z f HE + Lossless samples to the corpus (raw.pixls.us) with a corpus test,
and list HE NEF among the format gaps in parity.md and ROADMAP.md.
No decoder: the format has no public specification and the existing
decoders (LibRaw, rawler) are LGPL and derived from Nikon's binaries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011bbmJTxYJyYqKfNH8uXRC6
C5060WZ, C7070WZ, SP-510UZ, SP-550UZ, SP-565UZ and SP-570UZ files say
uncompressed 12-bit with many strips whose byte counts sum to exactly
width x height x 1.5. main refused them because the 12-bit branch wanted a
single strip, so they fell to "Olympus compressed ORF" and opened
preview-only. The samples are packed MSB-first, the strips hold two fields
(even output rows, then odd ones) with one gap at the seam, and the last row
of the second field sits right after the last strip. Accepted only when all of
that holds exactly. CFA comes from the Exif tag, white balance from maker-note
entries 0x1017/0x1018, black 0. RENDER_CACHE_VERSION 27 (adjust at merge).
A DNG can state a white level the sensor never reaches (an iPhone 15 Pro Max Bayer DNG states 4095 but clips at 3567), so clipped highlights were not recognised and white balance tinted them magenta. On decode, samples that form a plateau clearly below the white level, with nothing above it, are raised to the white level. RENDER_CACHE_VERSION 25 (needs adjusting at merge).
Lightroom Classic 15.6 imports an XMP packet's channel curves only when it holds
ToneCurvePV2012 and all three of ToneCurvePV2012Red/Green/Blue; any partial set
(master + one or two channels, channels without the master) imports without
channel curves. #416 read them whenever the master curve was present.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #535 (items 4 and 5).
- Pre-2017 bodies without Sony crop tags shot in the camera's 16:9 mode now get
the 16:9 crop of their camera JPEG: width from FullImageSize, anchored left as
before (drops the ILCE-7SM2's 32 padding columns); height from FullImageSize
(ILCE-7SM2) or from an Exif image size of the same width (DSLR-A580, whose
FullImageSize still says 3:2); centred vertically at an even offset, as
measured by registering each camera JPEG on its raw. The aspect now matches
the JPEG, so the camera-look fit runs: ILCE-7SM2 dE 15.2 -> 4.8,
DSLR-A580 dE 26.7 -> 3.9 (vs main 88ea1a8).
- The DSC-R1's SR2 (maker-note FileFormat 1 0 0 0) stores encrypted samples and
decoded to noise; it is now Unsupported in full and header mode and imports
preview-only with that reason.
- Only these 3 of 140 Sony decodes change; render cache version 26 (23-25 are
taken by #568, #542, #575).
- Tests: crop helper (3:2 unchanged, both 16:9 sources, even offsets, hostile
sizes), synthetic Sony file with FileFormat SR2 vs ARW, corpus test of the
two crops and the SR2 refusal; three samples added to the corpus list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #535 (item 1). Stacked on #542 (SR2 black levels); supersedes its
positional keystream tables.
- White balance is WB_RGGBLevels (0x7313), the gains the camera applied:
the raw IFD's plain copy when present, else the copy in the encrypted
SR2SubIFD, which every ARW from the DSLR-A200 on carries (identical to
the plain one on all 57 CC0 files that have both). The maker note's
Tag2010 gains stay only as a last resort.
- 14 bodies had no white balance and rendered green whenever the camera
look was rejected (A450 ... A700, NEX-3/5/C3, SLT-A33/A35/A55,
ILCE-6001, UMC-R10C). On others Tag2010 missed the applied gains by up
to 100 % (ILCE-3500 at 5600 K: R -31 / B +101 % against the camera
JPEG's neutral pixels, now +4 / +0 %; ILCE-7S on Shade -9 / +15 ->
-1 / +1 %; SLT-A57 -25 / +26 -> -6 / -3 %).
- New vendor/sr2.rs decrypts the whole SR2SubIFD, found black-box:
exiftool -v5's decrypted dump as known plaintext gives the keystream
(key 11 22 33 44, the same at every position in 126 CC0 ARWs from 97
bodies, 15 layouts); an exhaustive search over lag pairs found
k[n] = k[n-63] ^ k[n-127] on 32-bit words, so the first 127 words
generate the rest, matching all 126 files (5.2 MB of keystream).
- The SR2 black level (0x7310) is now read through the same decryptor,
replacing #542's head and per-position keystream tables: one SR2
decryption path. Identical results on all 131 Sony files (every
value #542 decoded; the 51 it could not are the files' plain 0x7310).
Unit tests pin the keystream against answers recorded without the
generator: every byte #542 and #148 had recovered, the SHA-256 of the
whole 62112-byte keystream recovered from the CC0 DSLR-A700 sample, and
the encrypted WB_RGGBLevels of four CC0 files (one per layout) with the
levels ExifTool decrypts.
- Sweep (4f350c4 -> this): gains equal ExifTool's decrypted
WB_RGGBLevels on all 100 decoded CC0 CFA files; on 83 files with
>= 200 JPEG-neutral pixels, within 10 % of the camera's applied gains
71 -> 83, over 10 % 5 -> 0. Accepted fits unchanged (max 0.15 dE);
forced fallback median dE 18.50 -> 18.06, neutral-area cast on the 22
changed files 8.8 -> 1.2.
- Corpus test for five CC0 samples (A500, A33, A700, ILCE-3500 5600 K,
ILCE-7S Shade; three added to RAW_SAMPLES); the pre-2017 test's gains
move to WB_RGGBLevels (RX100 III, RX100, ILCE-7RM2: within 2.6 % of
the old Tag2010 values in Auto). Synthetic tests for decryption,
directory parsing, source precedence, black level through the full
DNGPrivateData chain, and hostile input; #542's layout tests (incl.
the A700 trap) now run through the decryptor.
- Render cache version 25.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #535 (items 2 and 3).
- SR2SubIFD black level: decrypt the directory's first entry (always the
black level, 0x7310) to find where its value sits, and decrypt it with the
keystream known for that position (1638, 2166, 2418, 2510). The fixed read at
2510 gave 354-365 on the DSLR-A450/A500/A550 and 975 on the DSLR-A700, where
ExifTool and the data floor say 512 (A700 render: dE 20.1 -> 8.8,
dL* -9.8 -> 0.0). Keystreams recovered black-box (plain 0x7310 and exiftool -v3
as known plaintext), consistent over 115 ARWs from 97 bodies, 15 layouts.
- Downsized lossless M/S ARWs (linear YCbCr): black 1024 and white 16895, i.e.
the recorded 512 plus 512, measured against same-scene full-size codings on
12 files from 4 bodies (offset 1015-1052, scale 0.98-1.02). The 512 pedestal
lifted shadows by ~3 % of white; M/S renders now match their full-size twins
(7RM5 M dE 6.7 -> 3.1).
- Only these 16 of 140 Sony files change; render cache version 24.
- Tests: synthetic SR2 blocks for every known layout (incl. the A700 trap),
YCbCr black/white; corpus test of black levels against the data floor;
A500 and A700 samples added to the corpus list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brandon Thomas