scan_for_jpeg walks every JPEG-like start in a container it can't parse;
each walk runs through the marker segments that follow (up to 4096) and
their entropy data. A crafted CRW/MRW/X3F (or unsupported TIFF raw) of
fake starts one after another makes every walk cover most of the file:
quadratic, about a minute for 40 MB, on the import's probe thread. The
walks now report how far they read and share a budget of eight times the
file's size (at least 1 MiB); real files' candidates fail within a few
bytes or are the preview itself. Test: 200,000 fake starts, 0.15 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01912WaK6Rmew3wxZuPyX7E8
Widens the preview-only fallback. A recognised raw whose decode fails now falls back
to its embedded JPEG for any decode error (unsupported, corrupt, limit, TIFF
structure), not only an unsupported variant, and the preview search covers more
containers:
- Canon CRW, Minolta MRW and Sigma X3F are recognised by their magic (RawFormat
Crw, Mrw, X3f) instead of failing as "RawOther files are not supported yet".
- A TIFF whose IFD0 has no ImageWidth/ImageLength but a JPEG pointer or SubIFDs
(Kodak KDC) is a raw container, not a malformed image. TIFF compression 99
(Leaf MOS) counts as a raw compression.
- When no tag points to a preview, a recognised raw we cannot decode is scanned
for the largest complete three-component DCT JPEG. A preview whose first byte
was overwritten (Minolta, Epson) is restored. A one-component JPEG is skipped
(a PowerShot CRW stores its mosaic that way).
Files with no JPEG at all still fail with "without an embedded preview".
Phase One / Leaf IIQ, Canon EOS-1D / 1Ds and Kodak DCS TIFFs keep their raw data in
a private block. IFD0 is only a 160-640 px RGB copy and no IFD is marked as raw, so
they opened as an ordinary TIFF of that size (a 150 MP file rendered 296x220) with
no preview-only label.
probe() now recognises such a file from its structure (OtherTiff): a picture at
least 4x larger than IFD0 in both directions is declared by a SubIFDs child, or by
Exif PixelXDimension/PixelYDimension together with enough bytes outside every IFD's
strips and tiles to hold it at one bit per pixel. decode() reports the sizes as the
reason. A raw we cannot decode now falls back to its embedded JPEG, else to the TIFF
reader's view of its first image, so it imports preview-only with that reason.
RENDER_CACHE_VERSION 12: these files render differently.
Decode lossless CRX and version 0x100/0x200 C-RAW in portable Rust,
including sensor geometry, timed white balance and black/white levels.
Verify all sensor samples against six independent M50/R100/R8 references.
Any CR3 decode error falls back to the embedded JPEG, as CR3 files did
before this decoder. The full sensor buffer is allocated only after the
first plane decodes, and plane buffers grow row by row. Tile planes
decode in parallel with a windowed bit reader (24 MP: 424 -> 89 ms
lossless, 519 -> 138 ms C-RAW on an M4).
Add guarded CR3 camera-JPEG colour fitting with independent framing
validation, relative white balance and cache invalidation.
Validation: cargo xtask ci.
Maker note tag 0x4001 is UNDEFINED bytes (ColorData versions -3 and -4) on
PowerShot and EOS M bodies. It was read as one number per byte, so no plausible
as-shot levels were found and the white balance was 1, 1, 1 (green cast, a* about
-13 on neutrals). Byte and undefined data is now paired into 16-bit words in the
maker note's byte order before the existing probe (word 71). No per-model table.
RENDER_CACHE_VERSION 11 -> 12.
Maker note 0x0096 of the D780, D6, Z 5, Z 6, Z 7 and Z 7_2 starts 0x44 0x40
and holds 257 points, but its first interval rises by 2^bits / 1024 where the
0x44 0x20 tables rise by 2^bits / 256: the curve is the identity in the dark
part and covers 2^bits / 4 codes, not 2^bits. The decoder spaced the points
2^bits / 256 codes apart for both, so these files decoded to a fraction of
their values, far below the black level.
parse_table now takes the span from the version (0x20: all codes, 0x40: a
quarter). Other versions decode to identical samples. A known-answer test
built from hand-coded table bytes and bit strings fails without the change.
Bumps RENDER_CACHE_VERSION to 12.
Split out of #240 without the compressed decoder (crates/raw/src/vendor/orfc.rs
is not part of this change; compressed ORF stays preview-only).
- The layout is read from Exif CFAPattern (0xa302, repeat counts in either
byte order) when it describes a 2x2 Bayer cell. BGGR and GBRG were not
reachable before.
- Without a usable tag the green diagonal is still found from the samples,
now from totals of 32x32 blocks instead of single-pixel differences, so
fine texture no longer outweighs a small red/blue difference. The old
version indexed the samples directly; this one reads them with get().
- probe_info no longer unpacks 12-bit packed ORFs that state their layout
(XZ-2: under 0.1 ms; it ran the same unpack as a decode before, which
takes 11 ms). 16-bit files still read their samples to find the bit depth.
Tests: stated layouts in both byte orders, unusable tags falling back to the
samples, the texture case, hostile areas for the fallback, probe_info equal
to decode; a corpus test comparing the E-1, E-400 and XZ-2 mosaics with
their embedded JPEGs under every layout.
No user-visible change yet for the bodies whose layout the old guess got
wrong (E-M1 and E-620 are BGGR): they write compressed ORF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The RW2 reader reads PanasonicRaw tag 0x0119 (16 LE words) and expresses the
camera's own correction as the equivalent DNG WarpRectilinear in OpcodeList3,
so embedded_lens() turns it into the existing EmbeddedLens/EmbeddedWarp and
"Enable Profile Corrections" applies it like a DNG's warp (no new warp path;
Convert to DNG keeps it). Rule, from our own measurements against the
embedded camera JPEGs (clean-room; ExifTool tag names only):
corrected = rho*(1 + (w8 rho^2 + w4 rho^4 + w11 rho^6)/32768), rho = r/w12
(w12 = active-area half diagonal), output = corrected / (1 + w5/32768),
on when (w7 & 0xf) == 1. Chosen on 38 GX85 photos, held out on 35 photos
from 12 other bodies / 9 other lenses; corrected renders line up with the
camera JPEG within 0.75 px at 1440 px at the corners (up to 13 % of the
half diagonal before). Implausible values leave the photo uncorrected.
The loader now removes lens opcodes before the RW2 camera-look fit (its
binned proxy needs an empty OpcodeList3); Opcode::is_lens_correction()
replaces the duplicated matches. RENDER_CACHE_VERSION 11 -> 12.
Raw format 8 codes a cell top-left, bottom-left, top-right, bottom-right.
decode_strip placed the values in row order, which put the two green
samples of every RGGB cell on each other's sites: flat areas were right,
but every edge got a one-pixel colour zipper. Measured on a G9 II and the
corpus GH6, S5 II and S9: the Gb plane sat at (+0.5, -0.5) half-res cells
from Gr instead of the Bayer (-0.5, +0.5); with this change all four match.
The test encoder used the same order, so the round trip passed either way;
a new known-answer test hand-codes one cell and checks where it lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NEF (and other vendor raws without a colour matrix) carry their as-shot
white balance as vendor multipliers, not as a DNG AsShotNeutral. The DNG
writer only copied AsShotNeutral/AsShotWhiteXY, so Convert to DNG and
Copy as DNG produced files with no white balance at all, which develop
dark and green. Write the reciprocal of the vendor multipliers as
AsShotNeutral when the source has no DNG white of its own.
Decode every raw encoding Panasonic and Leica bodies write, from the DMC-LX1 (2005) to the DC-S1RM2 (2025):
compressed formats 4 (most bodies, 2008-2022) and 6 (S1 / S1R / S1H / S5, GH5M2), the prefix-coded strips of
format 8 (GH6 and later), packed formats 2, 5 and 7, and the 16-bit words of the 2005-2007 .RAW files. Until now
only format 5 decoded; every other Panasonic file opened as preview only.
Clean-room: no raw-decoder source was read. Tag names come from tag documentation; every encoding was recovered
by black-box analysis of 178 CC0 files from raw.pixls.us (118 bodies), and the module docs note the measurement
that settled each rule. Format-8 tags of unknown meaning must carry the values every sample has, otherwise the file
is reported as unsupported rather than guessed at.
Levels and geometry: formats 4 and 6 store samples 15 above the recorded black level (files without one use 15);
white is the active area's saturation plateau (the linearity limits are wrong on extended low-ISO shots, the GF1
and the Digilux 3); the in-camera aspect ratio is the default crop; mapped-out defect pixels are filled from
same-colour neighbours; the CFA pattern applies at sample (0, 0).
.rwl and .raw files are imported (engine, MCP folder expansion, file dialog, Linux MIME types). One file per
encoding joins the never-panic proptest; 11 CC0 files join `cargo xtask corpus`, and corpus_panasonic_encodings
checks each encoding's bit depth, black level, crop and CFA layout against the camera's own JPEG.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtQmZ6peZXGn3wiqeowV2v
CR3 files imported as preview-only with the 1620x1080 PRVW JPEG and no metadata
(no camera, capture time, exposure, lens or GPS): nothing read the container.
- crates/meta/src/cr3.rs: bounded ISO base media box parser for CR3 (Canon uuid
box with CMT1-4 and THMB, tracks with their sample entries, CMP1 coding header
and sample location, XMP uuid box). Every size/offset checked; depth and box
count bounded.
- Metadata: CMT1 (IFD0), CMT2 (Exif) and CMT4 (GPS) merged into one TIFF Exif
block, so CR3 reads like any TIFF-based raw; XMP from its uuid box.
- Preview: the full-size JPEG track (6960x4640 on an EOS R6 Mark III) before
PRVW/THMB; imports report the full image size.
- Tests: synthetic CR3 (boxes, tracks, merged Exif), truncation/corruption sweep,
JPEG track preferred over PRVW with fallback.
Raw (CRX) data is still not decoded; CR3 stays preview-only.
LightCraft has no colour matrix for Nikon cameras. Because of this, NEF
files showed colours that were too gray and too green.
- Colour: LightCraft now calculates a colour matrix and a tone curve from
the JPEG preview in each NEF file. Sony ARW files use the same
procedure and the same quality checks. White balance is relative to
the as-shot look, as for ARW files.
- One function, camera_preview::file_local_look, selects the formats for
the fit and for the relative white balance. Photo::relative_wb in the
catalog uses the same formats.
- Black level: Nikon stores the black level (maker note 0x003d) in
14-bit units, also in 12-bit files. The decoder read it as a 12-bit
value. Because of this, 12-bit files from the D750, D780, D850, D7500
and Z 50 were too dark. Now the decoder divides the value by 4 for
12-bit files.
- The corpus gets the D7500 12-bit and 14-bit lossless samples. A test
compares their levels. The render cache version increases to 9.
Results on 13 NEF files that LightCraft can decode: LightCraft accepted
the fit for all 13 files. The mean colour difference from the camera
JPEG (CIE76 dE) decreased from 13-46 to 2.9-7.8. One file with mixed
light improved by a small quantity only (12.7 to 12.0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sony bodies from before ~2017 (RX100–RX100M5, RX10, NEX, SLT, ILCE-6000,
A7/A7II/A7RII, ...) write no plain 0x7313/0x7310/0x74c7 tags in the raw IFD,
so the decoder used unit WB multipliers (strong green cast, e.g. the CC0
DSC-RX100M3 sample), black 512 (800 on 1-inch sensors), and kept padding
columns at the right edge.
- White balance: WB_RGBLevels from the maker note's enciphered Tag2010
block (layouts per ExifTool's Tag2010a..i tables; byte substitution
p^3 mod 249 checked black-box on 15 CC0 files: SonyISO matches Exif ISO,
ILCE-7M3 levels match its plain 0x7313).
- Black level: recovered from the encrypted SR2SubIFD by known-plaintext
analysis (constant key in every file; RX100M6/ILCE-7M3 plain 0x7310 as
plaintext; 8 keystream bytes at offset 2510, plausibility-checked).
- Crop: DNG DefaultCrop or the maker note's FullImageSize.
- "12-bit uncompressed" ARWs hold 14-bit-scale data: black/white follow it.
- Render cache version bumped; corpus manifest + corpus test, unit tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lightroom-converted DNGs carry their camera profile's ProfileHueSatMapData1/2
and ProfileLookTableData (and sometimes ProfileToneCurve). We used only the
colour matrices, so such DNGs rendered flat and muted.
- raw: new `profile` module reads the tags (DNG spec ch. 6: value-major,
hue-middle, saturation-minor tables, wrap-around hue, tri-linear lookups,
sRGB value encoding) from the raw IFD or IFD 0; malformed tags are ignored.
- engine: the raw loader applies the hue/sat map (interpolated by the same
illuminant weight as the matrices) before BaselineExposure and the look
table after it, in linear ProPhoto; a ProfileToneCurve becomes the source's
camera tone. Without one the standard base tone is kept.
- dngwrite keeps the look with the data (conversions, smart previews, merges).
- Render cache version 7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Bayer layout of Canon CR2 files differs by model (GBRG on the 7D, 50D,
60D, 550D, 600D, 1200D, 1300D, 5D Mark II, 1D Mark IV; RGGB on the 40D,
5D Mark III, 6D, 5DS R and others). Read it from raw IFD tag 0xc5e0
(CR2CFAPattern: 1 RGGB, 2 BGGR, 3 GBRG, 4 GRBG), anchored at the full
decoded sensor; fall back to RGGB when the tag is missing. No model table.
Checked on 35 decoded CC0 raw.pixls.us CR2s (35 models): the tag agrees
with the mosaic's green-diagonal statistic on every file; the SensorInfo
border parity does not predict the layout. Corpus manifest gains the
reporter's samples; a corpus test asserts each layout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decode Nikon Huffman-compressed NEF (compression 34713): lossless and lossy
(type 1 and 2) at 12 and 14 bits, with the maker-note 0x96 linearization
curve, predictor seeds and either byte order. Clean-room: 0x96 layout and
encoding-table sizes from prose write-ups (Clevy, Claff); the fixed Huffman
tables were recovered by black-box analysis of CC0 raw.pixls.us samples
(compressed vs uncompressed shots of the same scene) and verified on ~45
files from ~35 bodies. Lossy-after-split files stay preview only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ljpeg: a scan naming an undefined Huffman table is an error, not an expect.
- ARW2/RW2 block reads use get() + try_from instead of slice + expect.
- Vendor fallback CFAs use Cfa::bayer_static (literal names, tested)
instead of Cfa::bayer(..).expect("static").
- dngwrite matches (data, compression) instead of a let-else unreachable!.
- lightcraft-raw and lightcraft-tiff now #![deny] unwrap/expect/panic/
unimplemented/todo/unreachable; clippy.toml (same as the steering PR)
keeps them allowed in tests.
- tests/vendor_robust.rs: synthetic NEF, ARW, PEF, SRW, ORF, RW2, CR2 and
RAF files, truncated at every length and randomly mutated, must decode
to an error, never panic (no new crash found).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rust 1.99 (stable since 2026-09-28) adds clippy::chunks_exact_to_as_chunks and widens
clippy::question_mark, which failed `cargo xtask ci` on main. Same behaviour: the
remainder that chunks_exact skipped is the remainder as_chunks returns separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tiffraw: Mode::Header parses the first LJ92 frame header (Compression 7),
so probe_info reports Unsupported for subsampled frames (Sony lossless
M/S) as the full decode does, instead of claiming the file decodes.
- corpus: CC0 ILCE-7M4 lossless L/M/S samples from raw.pixls.us in
`cargo xtask corpus`; M/S listed as known unsupported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DNG compression 34892: each tile / strip is a baseline JPEG of 8-bit samples (decoded with
zune-jpeg, pure Rust), then the usual linearization table, levels and crop apply. Per the DNG
specification. Lossy DNGs (Smart Previews, "lossy compression" DNG exports, many phone DNGs)
now open as raw files instead of falling back to their embedded preview. Unit test with
synthetic JPEG tiles (edge tile cut short); the corpus sample decodes (5760×3840 in 144 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sony's lossless-compressed raw (Compression 7, e.g. ILCE-7M4) stores
512×512 tiles as LJ92 frames of 256×256 with four components: each
frame sample is one 2×2 CFA cell (top-left, top-right, bottom-left,
bottom-right). The generic TIFF path reads a frame as a row-major sample
stream (DNG's convention), which interleaved each tile's left and right
halves row by row: vertical bands, scrambled CFA, purple cast.
arw.rs now detects the layout from the first tile's frame header and
places the cells itself; other lossless-JPEG layouts keep the generic
path. Component order verified on a real ILCE-7M4 file (green
differences to the diagonal neighbour cell match those within a cell).
RENDER_CACHE_VERSION bumped so cached thumbnails of affected files are
re-rendered.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCLtGBctp1CxX2MdCuKCrm
A clipped pixel is never valid at full resolution, so its diffused chromaticity was always the
bilinear sample of the half-resolution pyramid level; the full-resolution chroma field (288 MB
at 24 MP) and its fill pass were wasted work. The pyramid now starts at half resolution straight
from the image. Bit-identical to the previous implementation (kept as a test reference over odd
sizes and clipping densities). 24 MP: 319 -> 145 ms isolated; real CR2/ARW loads 0.8-1.1 s ->
0.3-0.5 s under load. Runs on every full-resolution raw load (1:1 loupe, export).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- memory::budget (default min(25% RAM, 1.5 GiB); LIGHTCRAFT_MEMORY_MB, app.memoryBudget):
half for engine caches (sources + rendered previews, LRU across them via a global
use tick; the photo on screen stays), a quarter for decodes in flight (WorkGate:
thumbnails, prefetch and import probes wait; loupe/export loads never wait), an
eighth for the GPU free pool.
- GPU: best-fit buffer reuse (<= 25% slack), buffers released outside a render are
pooled at once (no stranded retired lists), pool trimmed after 3 s idle.
- Decodes drop file bytes and raw samples early, colour/crop/orient in place.
- macOS: apps install malloc_zone_pressure_relief as the release hook (after each
decode, import and idle trim): freed large blocks no longer stay resident.
14 raws + loupe + 12 steps (M4 Pro): peak RSS 1.9-2.3 GB -> 0.83-0.99 GB, footprint
2.5-2.6 GB -> 0.91-0.96 GB, heap peak 930 -> 656 MB; stepping median ~13 ms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decoders take a Mode: Header reads everything but the samples (DNG, CR2, ARW,
RAF, RW2; PEF when it has crop tags). NEF/ORF (and PEF without crop tags)
derive geometry/CFA from the samples and still unpack them. decode() stays
bit-exact (debug-hash of every corpus/raw decode identical before/after); the
corpus test asserts probe_info == decode().info() for all 37 files.
Import of 14 corpus raws: 3.4 s -> 1.4-2.0 s; peak RSS through import
1.41-1.49 GB -> 1.12-1.22 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lightcraft-raw: RawImage::develop_binned(k) — k×k superpixel binning of the CFA
(Bayer k even, X-Trans k multiple of 3), black/white normalised row by row, default
crop only, clipping preserved for highlight reconstruction. 2560 px previews of a
24 MP raw bin 2×, thumbnails 4–8×; exports / 1:1 still demosaic (AHD) at full size.
- highlight::reconstruct and its pyramid fill are rayon-parallel (bit-exact).
- X-Trans directional demosaic: precomputed per-phase neighbour offsets and no
reflection in the interior, no per-pixel allocation (bit-exact, tested against the
reference formulation): 2.9 s → 0.75 s CPU at 24 MP.
- files::load_bytes runs on a rayon worker so its short parallel loops don't each
wait for the pool to wake (dominant cost on a loaded machine); LIGHTCRAFT_PROFILE
prints per-stage source timings; files::bin_factor picks the block size.
- render_bench ONLY=source: per-file decode / normalise / demosaic / binning / load.
- Fix: u64 overflow on hostile IFD sizes in DNG/ARW/NEF raw-IFD selection (proptest).
load_bytes(2560) CPU time, ARW a7 III 24 MP: 2347 → 240 ms; X-T20: 3120 → 1220 ms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- embedded_preview: CR3 PRVW/THMB boxes (plus RAF, RW2, ORF, PEF from the
format commits); stops at the first valid box of each kind
- engine: raw variants that report Unsupported are probed and shown from
their embedded JPEG (orientation from the JPEG or the raw), not an error
- crates/raw/tests/corpus.rs: every corpus/raw file is recognised, has a
JPEG preview (DNG optional) and decodes or is a known-unsupported
variant; prints decode timings
- xtask corpus --download: 37 CC0 raw.pixls.us samples (matched by SHA-256)
- ROADMAP/crate docs: format coverage, NEF Huffman / RW2 format 4 /
compressed ORF+RAF gaps, neutral colour-matrix fallback
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16-bit LE words (12-bit-in-top-bits detected and shifted), 12-bit rows
of LE 32-bit words read MSB-first (black-box), maker-note
ImageProcessing crop, BlackLevel2, WB_RBLevels for old- and new-style
notes, CFA phase from the data (GRBG/RGGB), CameraSettings preview.
Compressed ORF reports Unsupported (documented gap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Huffman table from maker note 0x0220 (codes/lengths, black-box layout),
T.81 difference categories, same-colour horizontal prediction with
per-row-parity vertical seeds; BlackPoint, WB levels, maker-note image
area or dark-column trimming. Unit round-trip test with our own table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Black-box derived layout: 0x4000-byte chunks stored rotated by 0x1ff8,
16-byte LE blocks read LSB-first (10x12-bit or 9x14-bit samples).
Sensor borders, CFA code, per-channel black, linearity limit, WB levels,
ISO; JpgFromRaw preview via a generic undefined-tag JPEG candidate.
The quantised raw format 4 reports Unsupported (documented gap).
Adds a rawdump debug example.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RAF header/record parsing, raw IFD (0xf000) with 16-bit, 12-bit LSB and
32-bit-word MSB packings found by black-box analysis of CC0 samples,
X-Trans layout (reversed record, anchored at 0,0), crop, black levels,
WB_GRB levels, metadata and preview from the embedded JPEG. Compressed
RAF reports Unsupported. Colour uses the documented neutral fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude