mirror of
https://github.com/storytold/lightcraft.git
synced 2026-10-30 21:18:11 -06:00
M11.1: decode Fujifilm lossless and lossy compressed RAF
This commit is contained in:
sebastian pfluegelmeier
@@ -288,7 +288,7 @@ LightCraft is young and moving fast. **Where we honestly stand** (details in the
|
||||
older-Canon raws on one machine.
|
||||
- **The biggest gaps:**
|
||||
- **camera colour calibration:** raws other than DNG develop with a neutral colour matrix today, so colour is muted;
|
||||
- **CR3 and compressed Fujifilm / Olympus raws:** these open as embedded previews only;
|
||||
- **CR3 and compressed Olympus raws:** these open as embedded previews only. Fujifilm lossless/lossy compressed RAF now decodes sensor data; [verification and existing-library reload instructions](docs/raf-compression.md);
|
||||
- **AI masks and denoise:** subject and sky selection are classical heuristics;
|
||||
- **HDR, video and the Classic Print / Book / Map modules.**
|
||||
- **What's next:** see [where we're going](ROADMAP.md#where-were-going).
|
||||
@@ -303,7 +303,7 @@ LightCraft is young and moving fast. **Where we honestly stand** (details in the
|
||||
| Profiles (Color, Neutral, Vivid, Landscape, Portrait, Monochrome: our own looks), presets, versions, history, copy/paste/sync settings | ✅ |
|
||||
| Camera colour: DNG files use their own matrices | ✅ · our own calibration for other raws ⬜ (top priority; ARW, NEF and RW2 start from a look fitted to their own JPEG, other raws from a neutral fallback) |
|
||||
| Native macOS menu bar (generated from the command registry), control channel + every widget addressable, headless UI snapshots | ✅ |
|
||||
| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed, Bayer + X-Trans), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format, DMC-LX1 to DC-S1RM2), Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | ✅ · CR3, compressed RAF/ORF decode ⬜ |
|
||||
| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed + lossless/lossy compressed, Bayer + X-Trans), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format, DMC-LX1 to DC-S1RM2), Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | 🟡 · CR3, compressed ORF decode ⬜ |
|
||||
| Detail: sharpening, luminance + colour noise reduction | ✅ · AI Denoise, Super Resolution ⬜ |
|
||||
| Remove / Heal / Clone spots (auto source), Visualize Spots, Red Eye and Pet Eye (auto pupil detection, catchlight) | ✅ · content-aware fill, spot pin editing 🚧 |
|
||||
| Export: JPEG / PNG / TIFF / WebP / AVIF / DNG / original, sizing, file-size limit, output sharpening, naming templates, batch, metadata policy, text or image watermark | ✅ · HDR export ⬜ |
|
||||
|
||||
+12
-10
@@ -6,11 +6,11 @@ projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revi
|
||||
|
||||
## Where we stand
|
||||
|
||||
*Honest assessment, 2026-10-05. Agents: read this before picking work. The checklist in
|
||||
*Honest assessment, 2026-10-08. Agents: read this before picking work. The checklist in
|
||||
[`docs/parity.md`](docs/parity.md) counts features that **exist**; this section is about whether a photographer can
|
||||
**switch** from Lightroom. Update it when a gap below closes.*
|
||||
|
||||
**In one line:** the checklist says **79%** (P0 98.5%, P1 95.9%, P2 39%), but measured by whether a working
|
||||
**In one line:** the checklist says **79.7%** (P0 98.5%, P1 95.6%, P2 41.2%), but measured by whether a working
|
||||
photographer could replace Lightroom without noticing, we are at roughly **60–70%**. The remaining gap is mostly
|
||||
**quality of results and camera coverage**, not missing buttons.
|
||||
|
||||
@@ -23,7 +23,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|
||||
| Dimension | Estimate | What's true today | Biggest gaps |
|
||||
|---|---:|---|---|
|
||||
| **Feature checklist** | 79% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
|
||||
| **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) |
|
||||
| **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans), uncompressed ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) |
|
||||
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF and RW2 have a guarded per-file embedded-JPEG colour estimate (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
|
||||
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
|
||||
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
|
||||
@@ -37,7 +37,8 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|
||||
|---|---:|---|
|
||||
| JPEG / DNG shooter, single machine | ~85% | Fidelity polish, AI masks |
|
||||
| Nikon / Sony / Panasonic / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW, NEF and RW2 preview estimates are only a starting point); Panasonic compacts and kit zooms also miss their embedded distortion correction |
|
||||
| Canon CR3 / Fujifilm / Olympus shooter | ~35% | Their raws open as embedded previews only (CR3: the full-size JPEG with full metadata) |
|
||||
| Fujifilm raw shooter | ~65% | Uncompressed and lossless/lossy compressed RAF decoded; camera colour calibration remains missing. Verified on 13 bodies, including 14/16-bit GFX; see `docs/raf-compression.md` |
|
||||
| Canon CR3 / Olympus compressed-raw shooter | ~35% | Their raws open as embedded previews only (CR3: the full-size JPEG with full metadata) |
|
||||
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
|
||||
| Relies on AI (masks, denoise) | ~25% | Object / Describe masks via optional SAM 3 (download not yet hosted); no AI denoise |
|
||||
|
||||
@@ -48,7 +49,7 @@ Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/par
|
||||
1. **Camera colour calibration of our own** (LR-PROF-CAMERACOLOR, P0): fit each camera to its own embedded JPEG, use
|
||||
matrices the files carry themselves, then chart shots. Sony ARW, Nikon NEF and Panasonic RW2 have the file-local fit
|
||||
(matrix + tone curve from their own JPEG); generalise it to the other makes' raws (PEF, ORF…), then validate fidelity.
|
||||
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, NEF
|
||||
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed ORF, NEF
|
||||
lossy-after-split, sRAW. Decided 2026-10-05: write our own decoders from prose descriptions (never decoder source,
|
||||
no LGPL dependency); compressed NEF (#86) is the template.
|
||||
3. **Verified camera coverage** (LR-IMP-CAMERA-COVERAGE, P0): a CC0 sample per model in the corpus, each decoded and
|
||||
@@ -84,7 +85,7 @@ hardening (#78), copyright metadata (#51), GPX geotagging (#60), import tag help
|
||||
| M8 | Heal / Remove | content-aware remove (PatchMatch), heal, clone, brush spots, visualize spots, red/pet eye | 6–10 | 🚧 (heal, clone, auto source, visualize spots, red/pet eye ✅; PatchMatch remove ⬜) |
|
||||
| M9 | Presets, profiles, versions, sync | preset browser + amount, create/import presets, profile browser, versions, history, copy/paste/sync settings | 5–8 | ✅ |
|
||||
| M10 | Export & share | export dialog (JPEG/PNG/TIFF/DNG/AVIF/JXL/original), sizing, sharpening, metadata, watermark, naming, batch jobs, XMP sidecars, HDR export | 6–10 | 🚧 (all formats incl. DNG/original, sizing, presets, background jobs ✅; JXL encode, HDR export ⬜) |
|
||||
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 (every raw format), PEF, ORF uncompressed ✅; **camera colour calibration** 🚧 (guarded ARW, NEF and RW2 preview fitting; measured database still missing), CR3, compressed ORF/RAF ⬜) |
|
||||
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed and lossless/lossy compressed, RW2 (every raw format), PEF, ORF uncompressed ✅; **camera colour calibration** 🚧 (guarded ARW, NEF and RW2 preview fitting; measured database still missing), CR3, compressed ORF ⬜) |
|
||||
| M12 | AI & smart features | subject/sky/background/people/object masks, semantic search, faces/People (permissively licensed models, pure-Rust inference) | 20–40 | ⬜ |
|
||||
| M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ |
|
||||
| M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ |
|
||||
@@ -118,7 +119,7 @@ trip, slideshow, auto import…) in ≈ 12 h on 2026-10-02; four to six parallel
|
||||
| Work package | Tracker rows | Agent-hours | Risk |
|
||||
|---|---|---:|---|
|
||||
| Remaining P0/P1 UI and library features (folder rename/move, keyword painter, people view…) | ≈ 8 | 5–10 | low |
|
||||
| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
|
||||
| Raw codecs: CR3 (CRX), compressed ORF, NEF lossy-after-split, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
|
||||
| Lens-profile database of our own (calibration targets, fitting, data) | LR-EDIT-OPTICS-PROFILE | 15–30 | data collection |
|
||||
| Video: playback, trim, edits, export (pure-Rust decode, ideally shared with FilmCraft) | R. Video | 20–40 | medium |
|
||||
| AI: subject / sky / background / people / object masks, object-aware remove, AI denoise, super resolution, lens blur, people & faces, natural-language search, culling | ≈ 30 | 80–150 | **high** — permissively licensed weights, pure-Rust inference, maybe training |
|
||||
@@ -167,8 +168,8 @@ private 3:2 & 4:3 examples verified through native rendering & full-resolution 1
|
||||
|
||||
Decoded (CC0 corpus from raw.pixls.us, `cargo xtask corpus --download`, `crates/raw/tests/corpus.rs`): DNG (uncompressed,
|
||||
LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92; as-shot white balance and black level of pre-2017 bodies from the enciphered
|
||||
maker-note `Tag2010` and the encrypted `SR2SubIFD`, both recovered by black-box analysis, `crates/raw/src/vendor/arw.rs`), NEF/NRW uncompressed and Huffman-compressed (lossless, lossy type 1/2, 12/14-bit), RAF uncompressed (Bayer and
|
||||
X-Trans), RW2 / Leica RWL / Panasonic RAW in every raw format (compressed formats 4 and 6, the prefix-coded strips of format 8,
|
||||
maker-note `Tag2010` and the encrypted `SR2SubIFD`, both recovered by black-box analysis, `crates/raw/src/vendor/arw.rs`), NEF/NRW uncompressed and Huffman-compressed (lossless, lossy type 1/2, 12/14-bit), RAF uncompressed and lossless/lossy compressed (Bayer and
|
||||
X-Trans, 14/16-bit real-file verification), RW2 / Leica RWL / Panasonic RAW in every raw format (compressed formats 4 and 6, the prefix-coded strips of format 8,
|
||||
packed formats 2, 5 and 7, the 16-bit words of the 2005–2007 bodies; all recovered by black-box analysis of 178 CC0 files from
|
||||
118 bodies, `crates/raw/src/vendor/rw2.rs`), PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every
|
||||
supported container also yields its embedded JPEG preview (CR3 too), and the engine shows that preview for raw variants
|
||||
@@ -179,7 +180,7 @@ Not decoded yet — preview only (no permissively licensed description; black-bo
|
||||
files): rows above the split decode with the regular lossy table; from the split row on a different code is used
|
||||
that our black-box analysis has not recovered yet (none of the four regular tables fits, also not with byte
|
||||
alignment or reset predictors at the split row). The other Nikon Huffman variants are decoded (`crates/raw/src/vendor/nefc.rs` documents the analysis).
|
||||
- **Olympus compressed ORF**, **Fujifilm compressed RAF**, **Canon CR3/CRX** (M11.1), **Canon sRAW/mRAW**, lossy DNG.
|
||||
- **Olympus compressed ORF**, **Canon CR3/CRX** (M11.1), **Canon sRAW/mRAW**, lossy DNG.
|
||||
|
||||
**Camera colour matrices:** ARW, NEF and RW2 files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
|
||||
`matrix_is_fallback`) with the file's as-shot white-balance multipliers. Clean sources to evaluate next: manufacturer
|
||||
@@ -190,6 +191,7 @@ own chart-based calibration (M11.4). Adobe matrices are never used.
|
||||
- 2026-09-30: roadmap created; M0 in progress; research docs (Lightroom reference, Rust imaging ecosystem) complete.
|
||||
- 2026-09-30 (later): app running with the full Lightroom-style UI; pipeline v0; DNG/CR2/ARW; README showcase. ≈ 8 h elapsed.
|
||||
- 2026-10-02: parity estimate added (xtask parity prints weighted completion); milestone statuses refreshed.
|
||||
- 2026-10-08: Fujifilm lossless/lossy compressed RAF: pure-Rust striped decoder; full sensor arrays verified against a black-box reference on 17 CC0 files and 7 supplied X-H2S / X-T4 files (13 camera bodies, 14/16-bit Bayer and X-Trans). Camera colour calibration remains a separate gap.
|
||||
- 2026-10-01: RAW II formats: RAF (uncompressed Bayer + X-Trans), RW2 (packed), PEF (incl. Huffman), ORF (uncompressed); embedded previews for every container incl. CR3; raw corpus test with 37 CC0 samples.
|
||||
- 2026-10-03 – 10-05: community PRs (#42–#51, #60) and 25+ issues landed: compressed NEF, Move import, folder templates,
|
||||
Local roots and cleanup, 85k-photo grid and catalog performance, background compaction, failed-save errors, GPU
|
||||
|
||||
@@ -13,10 +13,10 @@
|
||||
//!
|
||||
//! Formats: DNG (uncompressed, lossless JPEG, lossy JPEG (Smart Previews), Deflate incl. floating point, tiled/stripped, CFA and LinearRaw),
|
||||
//! Canon CR2, Nikon NEF/NRW (uncompressed, Huffman lossless / lossy compressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
|
||||
//! and X-Trans), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format: compressed 4 and 6, the prefix-coded strips of 8,
|
||||
//! and X-Trans, lossless and lossy compressed), Panasonic RW2 / Leica RWL / Panasonic RAW (every raw format: compressed 4 and 6, the prefix-coded strips of 8,
|
||||
//! packed 2/5/7, the 16-bit words of the oldest bodies), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed).
|
||||
//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon "lossy after split" NEF,
|
||||
//! compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
|
||||
//! compressed ORF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
|
||||
//! (public specifications, tag-name documentation, black-box analysis of CC0 samples) and gaps. Non-DNG files carry no
|
||||
//! colour matrix: [`color`] falls back to a documented neutral model. The decoders never panic on malformed input.
|
||||
#![forbid(unsafe_code)]
|
||||
|
||||
Vendored
+1
@@ -7,6 +7,7 @@ pub mod nefc;
|
||||
pub mod orf;
|
||||
pub mod pef;
|
||||
pub mod raf;
|
||||
mod rafc;
|
||||
pub mod rw2;
|
||||
|
||||
use crate::{BlackLevel, Rect};
|
||||
|
||||
Vendored
+55
-25
@@ -1,4 +1,4 @@
|
||||
//! Fujifilm RAF — uncompressed Bayer and X-Trans.
|
||||
//! Fujifilm RAF — uncompressed, lossless compressed and lossy compressed Bayer and X-Trans.
|
||||
//!
|
||||
//! Sources: the ExifTool FujiFilm tag-name documentation (RAF header record tags `0x0100` RawImageFullSize,
|
||||
//! `0x0110` RawImageCropTopLeft, `0x0111` RawImageCroppedSize, `0x0131` XTransLayout; raw-IFD tags `0xf001`
|
||||
@@ -17,7 +17,7 @@
|
||||
//! is anchored at raw pixel (0, 0) — verified on X-Trans I and III samples by minimising the difference
|
||||
//! between horizontally / vertically adjacent "green" samples over all 72 orientations/phases, and by colour
|
||||
//! renders. Bayer bodies (X-A series) use RGGB at (0, 0), verified by colour renders.
|
||||
//! - Fujifilm's compressed RAF variants are reported as unsupported (their embedded preview still works).
|
||||
//! - Striped predictive compression is decoded by `rafc`; see `docs/raf-compression.md`.
|
||||
|
||||
use super::white_from_data;
|
||||
use crate::unpack::{unpack_lsb, unpack_msb};
|
||||
@@ -118,35 +118,45 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
|
||||
let h = header(bytes)?;
|
||||
let raw = h.raw.ok_or_else(|| RawError::Corrupt("RAF without raw data".into()))?;
|
||||
let ifd = raw_ifd(raw)?;
|
||||
let get = |t: u16| ifd.u64(t).map(|v| v as usize);
|
||||
let (w, hgt) = (get(0xf001).unwrap_or(0), get(0xf002).unwrap_or(0));
|
||||
let bits = get(0xf003).unwrap_or(0) as u32;
|
||||
let (off, len) = (get(0xf007).unwrap_or(0), get(0xf008).unwrap_or(0));
|
||||
let get = |t: u16| ifd.u64(t).map(|v| usize::try_from(v).map_err(|_| RawError::Limit("RAF IFD value too large"))).transpose();
|
||||
let (w, hgt) = (get(0xf001)?.unwrap_or(0), get(0xf002)?.unwrap_or(0));
|
||||
let bits = ifd.u64(0xf003).and_then(|v| u32::try_from(v).ok()).unwrap_or(0);
|
||||
let (off, len) = (get(0xf007)?.unwrap_or(0), get(0xf008)?.unwrap_or(0));
|
||||
if w == 0 || hgt == 0 || !(8..=16).contains(&bits) {
|
||||
return Err(RawError::Corrupt(format!("RAF raw IFD: {w}x{hgt}, {bits} bits")));
|
||||
}
|
||||
let n = w.checked_mul(hgt).filter(|n| *n <= crate::MAX_SAMPLES).ok_or(RawError::Limit("image too large"))?;
|
||||
// samples of any depth may be stored in 16-bit words (e.g. 14-bit X-T20); otherwise packed at `bits`
|
||||
let store = if len as u64 >= (n as u64) * 2 { 16 } else { bits };
|
||||
let packed_row = if store == 16 { w * 2 } else { (w * store as usize).div_ceil(8) };
|
||||
let stride = match get(0xf005).unwrap_or(0) {
|
||||
0 => packed_row,
|
||||
words => (words * 4).max(packed_row),
|
||||
};
|
||||
if (len as u64) < (stride as u64) * (hgt as u64 - 1) + packed_row as u64 {
|
||||
return Err(RawError::Unsupported(format!("Fujifilm compressed RAF ({len} bytes for {w}x{hgt} {bits}-bit)")));
|
||||
}
|
||||
let src = raw.get(off..off.saturating_add(len).min(raw.len())).ok_or_else(|| RawError::Corrupt("RAF strip outside file".into()))?;
|
||||
let src = raw
|
||||
.get(off..off.checked_add(len).ok_or_else(|| RawError::Corrupt("RAF strip size overflow".into()))?)
|
||||
.ok_or_else(|| RawError::Corrupt("RAF strip outside file".into()))?;
|
||||
let cfa = cfa(&h);
|
||||
let compression = be32(bytes, 108).unwrap_or(0);
|
||||
let compressed = matches!(compression, 2 | 3);
|
||||
let mut data = Vec::new();
|
||||
if mode == Mode::Full {
|
||||
data = vec![0u16; n];
|
||||
data.par_chunks_mut(w).enumerate().for_each(|(y, row)| {
|
||||
let s = src.get(y * stride..(y * stride + packed_row).min(src.len())).unwrap_or(&[]);
|
||||
unpack_row(s, store, row);
|
||||
});
|
||||
if compressed {
|
||||
data = super::rafc::decode(src, w, hgt, bits, cfa.width == 6, mode)?;
|
||||
} else {
|
||||
// Samples may use 16-bit words (e.g. 14-bit X-T20); otherwise they are packed at `bits`.
|
||||
let store = if len as u64 >= (n as u64) * 2 { 16 } else { bits };
|
||||
let packed_row = w.checked_mul(store as usize).map(|n| n.div_ceil(8)).ok_or(RawError::Limit("RAF row too large"))?;
|
||||
let stride = match get(0xf005)?.unwrap_or(0) {
|
||||
0 => packed_row,
|
||||
words => words.checked_mul(4).ok_or(RawError::Limit("RAF stride too large"))?.max(packed_row),
|
||||
};
|
||||
let required = stride.checked_mul(hgt - 1).and_then(|n| n.checked_add(packed_row)).ok_or(RawError::Limit("RAF strip too large"))?;
|
||||
if len < required {
|
||||
return Err(RawError::Unsupported(format!("Fujifilm RAF packing ({len} bytes for {w}x{hgt} {bits}-bit)")));
|
||||
}
|
||||
if mode == Mode::Full {
|
||||
data = vec![0u16; n];
|
||||
data.par_chunks_mut(w).enumerate().for_each(|(y, row)| {
|
||||
// `required <= src.len()` proves these row offsets fit, including the final row.
|
||||
let s = src.get(y * stride..y * stride + packed_row).unwrap_or(&[]);
|
||||
unpack_row(s, store, row);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
let cfa = cfa(&h);
|
||||
let active = match (pair(&h, CROP_TOP_LEFT), pair(&h, CROPPED_SIZE)) {
|
||||
(Some((top, left)), Some((ch, cw))) if ch > 0 && cw > 0 => Rect::new(left, top, cw, ch).clipped(w, hgt),
|
||||
_ => Rect::new(0, 0, w, hgt),
|
||||
@@ -194,7 +204,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
pub(super) mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Build a minimal RAF: header, records, raw block (LE TIFF whose IFD0 points to a 0xf000 raw IFD).
|
||||
@@ -247,6 +257,10 @@ mod tests {
|
||||
dir.extend_from_slice(&recs);
|
||||
let mut out = b"FUJIFILMCCD-RAW 0201FF000000TEST".to_vec();
|
||||
out.resize(108, 0);
|
||||
if strip.starts_with(b"IS") {
|
||||
let compression = if strip.get(2) == Some(&0) { 3u32 } else { 2u32 };
|
||||
out.extend_from_slice(&compression.to_be_bytes());
|
||||
}
|
||||
let jpeg_off = 160u32;
|
||||
let dir_off = jpeg_off + jpeg.len() as u32;
|
||||
let raw_off = dir_off + dir.len() as u32;
|
||||
@@ -310,8 +324,24 @@ mod tests {
|
||||
}
|
||||
let bytes = raf(w, h, 16, pack(&[0; 96], 16), None, b"");
|
||||
assert_eq!(crate::decode(&bytes).unwrap().cfa.unwrap().name(), "RGGB");
|
||||
let mut px = vec![0; 96];
|
||||
px[0] = 0x5349;
|
||||
let mut bytes = raf(w, h, 16, pack(&px, 16), None, b"");
|
||||
bytes[108..112].copy_from_slice(&0u32.to_be_bytes());
|
||||
assert_eq!(crate::decode(&bytes).unwrap().data, RawData::U16(px));
|
||||
// too little data for the size: compressed
|
||||
let bytes = raf(w, h, 14, vec![0; 20], None, b"");
|
||||
assert!(matches!(crate::decode(&bytes), Err(RawError::Unsupported(_))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overflowing_row_stride_is_an_error() {
|
||||
let mut bytes = raf(16, 6, 16, vec![0; 192], None, &[]);
|
||||
let off = be32(&bytes, 100).unwrap();
|
||||
// Replace WB with a LONG8 stride, pointing to the existing out-of-line WB storage.
|
||||
bytes[off + 100..off + 104].copy_from_slice(&[0x05, 0xf0, 16, 0]);
|
||||
bytes[off + 104..off + 108].copy_from_slice(&1u32.to_le_bytes());
|
||||
bytes[off + 116..off + 124].copy_from_slice(&u64::MAX.to_le_bytes());
|
||||
assert!(matches!(crate::decode(&bytes), Err(RawError::Limit(_))));
|
||||
}
|
||||
}
|
||||
|
||||
Vendored
+477
@@ -0,0 +1,477 @@
|
||||
//! Fujifilm's striped predictive RAW compression.
|
||||
//!
|
||||
//! Clean-room implementation: Fabian's prose description of lossless RAF compression
|
||||
//! (https://capnfabs.net/posts/fuji-raf-compression-algorithm/) supplied the broad structure;
|
||||
//! headers, colour-vector placement, predictors, contexts and adaptation were recovered
|
||||
//! from sample bytes and black-box comparisons of decoded sensor arrays. No other RAW
|
||||
//! decoder's source was consulted. See docs/raf-compression.md for verification details.
|
||||
|
||||
use crate::{Mode, RawError, Result};
|
||||
use rayon::prelude::*;
|
||||
|
||||
const BLOCK: usize = 768;
|
||||
const XTRANS: [[usize; 6]; 6] =
|
||||
[[1, 1, 0, 1, 1, 2], [1, 1, 2, 1, 1, 0], [2, 0, 1, 0, 2, 1], [1, 1, 2, 1, 1, 0], [1, 1, 0, 1, 1, 2], [0, 2, 1, 2, 0, 1]];
|
||||
const MASKS: [[u8; 2]; 6] = [[10, 15], [15, 10], [14, 10], [15, 11], [11, 15], [10, 14]];
|
||||
|
||||
fn corrupt(why: &str) -> RawError {
|
||||
RawError::Corrupt(format!("Fujifilm compressed RAF: {why}"))
|
||||
}
|
||||
|
||||
fn be16(src: &[u8], at: usize) -> Result<usize> {
|
||||
let s = src.get(at..at.saturating_add(2)).ok_or_else(|| corrupt("truncated header"))?;
|
||||
let [a, b] = s else { return Err(corrupt("truncated header")) };
|
||||
Ok(u16::from_be_bytes([*a, *b]) as usize)
|
||||
}
|
||||
|
||||
struct Stripe<'a> {
|
||||
bytes: &'a [u8],
|
||||
quantizers: Option<&'a [u8]>,
|
||||
width: usize,
|
||||
}
|
||||
|
||||
pub(super) fn decode(src: &[u8], width: usize, height: usize, bits: u32, xtrans: bool, mode: Mode) -> Result<Vec<u16>> {
|
||||
let h = src.get(..16).ok_or_else(|| corrupt("truncated header"))?;
|
||||
if !h.starts_with(b"IS") {
|
||||
return Err(corrupt("missing compression signature"));
|
||||
}
|
||||
let flag = h.get(2).copied().ok_or_else(|| corrupt("truncated header"))?;
|
||||
if flag > 1 {
|
||||
return Err(corrupt("invalid compression flag"));
|
||||
}
|
||||
let sensor = h.get(3).copied().ok_or_else(|| corrupt("truncated header"))?;
|
||||
if sensor != if xtrans { 16 } else { 0 } {
|
||||
return Err(corrupt("compression sensor type disagrees with CFA"));
|
||||
}
|
||||
if !matches!(bits, 12 | 14 | 16) {
|
||||
return Err(RawError::Unsupported(format!("Fujifilm compressed RAF at {bits} bits")));
|
||||
}
|
||||
let count = h.get(13).copied().ok_or_else(|| corrupt("truncated header"))? as usize;
|
||||
if h.get(4).copied() != Some(bits as u8)
|
||||
|| be16(h, 5)? != height
|
||||
|| be16(h, 9)? != width
|
||||
|| count == 0
|
||||
|| count != width.div_ceil(BLOCK)
|
||||
|| be16(h, 7)? != count * BLOCK
|
||||
|| be16(h, 11)? != BLOCK
|
||||
|| height == 0
|
||||
|| !height.is_multiple_of(6)
|
||||
|| be16(h, 14)? != height / 6
|
||||
{
|
||||
return Err(corrupt("inconsistent compression dimensions"));
|
||||
}
|
||||
let n = width.checked_mul(height).filter(|&n| n <= crate::MAX_SAMPLES).ok_or(RawError::Limit("image too large"))?;
|
||||
let table_end = 16 + count * 4;
|
||||
let mut offset = table_end.div_ceil(16) * 16;
|
||||
let quantizer_start = offset;
|
||||
let quantizer_stride = (height / 6).div_ceil(16) * 16;
|
||||
if flag == 0 {
|
||||
offset += quantizer_stride * count;
|
||||
if src.get(quantizer_start..offset).is_none() {
|
||||
return Err(corrupt("truncated quantizer tables"));
|
||||
}
|
||||
}
|
||||
let mut stripes = Vec::with_capacity(count);
|
||||
for i in 0..count {
|
||||
let entry = src.get(16 + i * 4..20 + i * 4).ok_or_else(|| corrupt("truncated stripe size table"))?;
|
||||
let [a, b, c, d] = entry else { return Err(corrupt("truncated stripe size table")) };
|
||||
let size = u32::from_be_bytes([*a, *b, *c, *d]) as usize;
|
||||
let end = offset.checked_add(size).ok_or_else(|| corrupt("stripe size overflow"))?;
|
||||
let bytes = src.get(offset..end).filter(|s| !s.is_empty()).ok_or_else(|| corrupt("stripe outside file"))?;
|
||||
let quantizers = if flag == 0 {
|
||||
Some(
|
||||
src.get(quantizer_start + i * quantizer_stride..quantizer_start + i * quantizer_stride + height / 6)
|
||||
.ok_or_else(|| corrupt("truncated quantizer table"))?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
stripes.push(Stripe { bytes, quantizers, width: (width - i * BLOCK).min(BLOCK) });
|
||||
offset = end;
|
||||
}
|
||||
if mode == Mode::Header {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let decoded: Vec<Vec<u16>> = stripes.par_iter().map(|s| decode_stripe(s, height, bits, xtrans)).collect::<Result<_>>()?;
|
||||
let mut output = vec![0; n];
|
||||
for (i, (stripe, data)) in stripes.iter().zip(&decoded).enumerate() {
|
||||
for (row, samples) in output.chunks_exact_mut(width).zip(data.chunks_exact(stripe.width)) {
|
||||
row.get_mut(i * BLOCK..i * BLOCK + stripe.width).ok_or_else(|| corrupt("invalid stripe placement"))?.copy_from_slice(samples);
|
||||
}
|
||||
}
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
struct Bits<'a> {
|
||||
src: &'a [u8],
|
||||
pos: usize,
|
||||
acc: u64,
|
||||
available: u32,
|
||||
}
|
||||
|
||||
impl<'a> Bits<'a> {
|
||||
fn new(src: &'a [u8]) -> Self {
|
||||
Self { src, pos: 0, acc: 0, available: 0 }
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn get(&mut self, n: u32) -> Result<u32> {
|
||||
while self.available < n {
|
||||
let b = self.src.get(self.pos).copied().ok_or_else(|| corrupt("truncated entropy stream"))?;
|
||||
self.pos += 1;
|
||||
self.acc = (self.acc << 8) | u64::from(b);
|
||||
self.available += 8;
|
||||
}
|
||||
self.available -= n;
|
||||
Ok(((self.acc >> self.available) & ((1u64 << n) - 1)) as u32)
|
||||
}
|
||||
|
||||
fn residual(&mut self, k: u32, bits: u32, range: u32, limit: u32) -> Result<i32> {
|
||||
let mut zeros = 0;
|
||||
while self.get(1)? == 0 {
|
||||
zeros += 1;
|
||||
if zeros > limit {
|
||||
return Err(corrupt("invalid residual escape"));
|
||||
}
|
||||
}
|
||||
let code = if zeros == limit { self.get(bits)? + 1 } else { (zeros << k) | self.get(k)? };
|
||||
if code > range {
|
||||
return Err(corrupt(&format!(
|
||||
"residual {code} outside sample range {range} (k={k}, prefix={zeros}, bits={bits}, position={})",
|
||||
self.pos * 8 - self.available as usize
|
||||
)));
|
||||
}
|
||||
Ok(if code & 1 == 0 { (code / 2) as i32 } else { -((code / 2) as i32) - 1 })
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
struct State {
|
||||
sum: u32,
|
||||
count: u32,
|
||||
}
|
||||
|
||||
impl State {
|
||||
fn k(self) -> u32 {
|
||||
let mut k = 0;
|
||||
while self.count << k < self.sum {
|
||||
k += 1;
|
||||
}
|
||||
k
|
||||
}
|
||||
|
||||
fn update(&mut self, delta: i32) {
|
||||
self.sum += delta.unsigned_abs();
|
||||
if self.count == 64 {
|
||||
self.sum >>= 1;
|
||||
self.count >>= 1;
|
||||
}
|
||||
self.count += 1;
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn quant(v: i32, near: u32) -> i32 {
|
||||
let t = [18 + 3 * near, 67 + 5 * near, 276 + 7 * near];
|
||||
let a = v.unsigned_abs();
|
||||
let bucket = if a <= near {
|
||||
0
|
||||
} else if a < t[0] {
|
||||
1
|
||||
} else if a < t[1] {
|
||||
2
|
||||
} else if a < t[2] {
|
||||
3
|
||||
} else {
|
||||
4
|
||||
};
|
||||
if v < 0 { -bucket } else { bucket }
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn even_prediction(nw: i32, n: i32, ne: i32, nn: i32) -> i32 {
|
||||
let (a, b, c) = ((nw - n).abs(), (ne - n).abs(), (nn - n).abs());
|
||||
let pair = if a > b && a > c {
|
||||
ne + nn
|
||||
} else if b > a && b > c {
|
||||
nw + nn
|
||||
} else {
|
||||
nw + ne
|
||||
};
|
||||
(pair + 2 * n) >> 2
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn sample(rows: &[Vec<i32>], y: usize, x: usize) -> Result<i32> {
|
||||
rows.get(y).and_then(|r| r.get(x)).copied().ok_or_else(|| corrupt("invalid predictor position"))
|
||||
}
|
||||
|
||||
fn decode_stripe(stripe: &Stripe<'_>, height: usize, bits: u32, xtrans: bool) -> Result<Vec<u16>> {
|
||||
let size = if xtrans { BLOCK * 2 / 3 } else { BLOCK / 2 };
|
||||
let mut previous = vec![vec![vec![0; size + 2]; 2]; 3];
|
||||
let max = (1u32 << bits) - 1;
|
||||
let range = |near: u32| (max + 2 * near) / (2 * near + 1) + 1;
|
||||
let seed = |near: u32| State { sum: ((range(near) + 32) / 64).max(2), count: 1 };
|
||||
let mut states = [[[seed(0); 41]; 2]; 3];
|
||||
// Smooth regions retain finer precision and independent state across block quantizer changes.
|
||||
let mut fine = [[[[seed(0); 5]; 2]; 3], [[[seed(1); 5]; 2]; 3], [[[seed(2); 5]; 2]; 3]];
|
||||
let mut last_near = None;
|
||||
let mut reader = Bits::new(stripe.bytes);
|
||||
let n = height.checked_mul(stripe.width).filter(|&n| n <= crate::MAX_SAMPLES).ok_or(RawError::Limit("image too large"))?;
|
||||
let mut output = Vec::with_capacity(n);
|
||||
for group in 0..height / 6 {
|
||||
let near = match stripe.quantizers {
|
||||
Some(q) => u32::from(*q.get(group).ok_or_else(|| corrupt("truncated quantizer table"))?),
|
||||
None => 0,
|
||||
};
|
||||
if last_near != Some(near) {
|
||||
states = [[[seed(near); 41]; 2]; 3];
|
||||
last_near = Some(near);
|
||||
}
|
||||
let mut rows = previous;
|
||||
for pair in 0..6 {
|
||||
let colours = if pair % 2 == 0 { [0, 1] } else { [1, 2] };
|
||||
for &colour in &colours {
|
||||
let lines = rows.get_mut(colour).ok_or_else(|| corrupt("invalid colour vector"))?;
|
||||
let mut row = vec![0; size + 2];
|
||||
let prev = lines.len() - 1;
|
||||
let left = sample(lines, prev, 1)?;
|
||||
let right = sample(lines, prev, size)?;
|
||||
if let Some(v) = row.first_mut() {
|
||||
*v = left;
|
||||
}
|
||||
if let Some(v) = row.last_mut() {
|
||||
*v = right;
|
||||
}
|
||||
lines.push(row);
|
||||
}
|
||||
for even in (0..size + 8).step_by(2) {
|
||||
for (odd, position) in [(0, Some(even)), (1, even.checked_sub(7))] {
|
||||
let Some(pos) = position.filter(|&x| x < size) else { continue };
|
||||
let x = pos + 1;
|
||||
for (which, &colour) in colours.iter().enumerate() {
|
||||
let lines = rows.get_mut(colour).ok_or_else(|| corrupt("invalid colour vector"))?;
|
||||
let y = lines.len() - 1;
|
||||
let (nw, north, ne, nn) =
|
||||
(sample(lines, y - 1, x - 1)?, sample(lines, y - 1, x)?, sample(lines, y - 1, x + 1)?, sample(lines, y - 2, x)?);
|
||||
let west = sample(lines, y, x - 1)?;
|
||||
let east = sample(lines, y, x + 1)?;
|
||||
let (predict, d1, d2) = if odd == 0 {
|
||||
(even_prediction(nw, north, ne, nn), north - nn, nw - north)
|
||||
} else {
|
||||
let p = if (nw.min(ne)..=nw.max(ne)).contains(&north) { (west + east) >> 1 } else { (west + east + 2 * north) >> 2 };
|
||||
(p, north - nw, nw - west)
|
||||
};
|
||||
let real = !xtrans || MASKS.get(pair).and_then(|m| m.get(which)).is_some_and(|m| (m >> (pos % 4)) & 1 != 0);
|
||||
let value = if real {
|
||||
let contrast = d1.unsigned_abs() + d2.unsigned_abs();
|
||||
let precision = match contrast {
|
||||
0..=5 => 0,
|
||||
6 => near.min(1),
|
||||
7 => near.min(2),
|
||||
_ => near,
|
||||
};
|
||||
let (context, state) = if precision < near {
|
||||
let sign = |v: i32| if v.unsigned_abs() <= precision { 0 } else { v.signum() };
|
||||
let context = 3 * sign(d1) + sign(d2);
|
||||
let state = fine
|
||||
.get_mut(precision as usize)
|
||||
.and_then(|f| f.get_mut(pair % 3))
|
||||
.and_then(|s| s.get_mut(odd))
|
||||
.and_then(|s| s.get_mut(context.unsigned_abs() as usize))
|
||||
.ok_or_else(|| corrupt("invalid fine coding context"))?;
|
||||
(context, state)
|
||||
} else {
|
||||
let context = 9 * quant(d1, near) + quant(d2, near);
|
||||
let state = states
|
||||
.get_mut(pair % 3)
|
||||
.and_then(|s| s.get_mut(odd))
|
||||
.and_then(|s| s.get_mut(context.unsigned_abs() as usize))
|
||||
.ok_or_else(|| corrupt("invalid coding context"))?;
|
||||
(context, state)
|
||||
};
|
||||
let coded_range = range(precision);
|
||||
let coded_bits = 32 - (coded_range - 1).leading_zeros();
|
||||
let mut delta = reader.residual(state.k(), coded_bits, coded_range, 4 * bits - coded_bits - 1)?;
|
||||
state.update(delta);
|
||||
if context < 0 {
|
||||
delta = -delta;
|
||||
}
|
||||
let step = (2 * precision + 1) as i32;
|
||||
let mut value = predict + delta * step;
|
||||
if value < -(precision as i32) {
|
||||
value += coded_range as i32 * step;
|
||||
} else if value > (max + precision) as i32 {
|
||||
value -= coded_range as i32 * step;
|
||||
}
|
||||
value.clamp(0, max as i32)
|
||||
} else {
|
||||
predict
|
||||
};
|
||||
*lines.get_mut(y).and_then(|r| r.get_mut(x)).ok_or_else(|| corrupt("invalid predictor position"))? = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for y in 0..6 {
|
||||
for x in 0..stripe.width {
|
||||
let colour = if xtrans {
|
||||
XTRANS.get(y).and_then(|r| r.get(x % 6)).copied().ok_or_else(|| corrupt("invalid CFA position"))?
|
||||
} else {
|
||||
match (y % 2, x % 2) {
|
||||
(0, 0) => 0,
|
||||
(1, 1) => 2,
|
||||
_ => 1,
|
||||
}
|
||||
};
|
||||
let row = 2 + if colour == 1 { y } else { y / 2 };
|
||||
let col = 1 + if xtrans { 2 * (x / 3) + (x % 3).min(1) } else { x / 2 };
|
||||
output.push(sample(rows.get(colour).ok_or_else(|| corrupt("invalid colour vector"))?, row, col)? as u16);
|
||||
}
|
||||
}
|
||||
previous = rows.into_iter().map(|mut r| r.split_off(r.len() - 2)).collect();
|
||||
}
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use proptest::prelude::*;
|
||||
|
||||
/// Procedural zero residuals: no camera media or encoder dependency.
|
||||
fn zero_stream(bits: u32, xtrans: bool, quantizers: Option<&[u8]>) -> Vec<u8> {
|
||||
let groups = quantizers.map_or(3, <[u8]>::len);
|
||||
let mut main = [[(1u32 << (bits - 6), 1u32); 2]; 3];
|
||||
let mut fine = main;
|
||||
let mut previous = None;
|
||||
let mut stream = Vec::new();
|
||||
for group in 0..groups {
|
||||
let q = quantizers.map_or(0, |q| q[group]);
|
||||
if previous != Some(q) {
|
||||
main = [[(1u32 << (bits - 6), 1u32); 2]; 3];
|
||||
previous = Some(q);
|
||||
}
|
||||
for pair in 0..6 {
|
||||
let width = if xtrans { 512 } else { 384 };
|
||||
for even in (0usize..width + 8).step_by(2) {
|
||||
for (parity, column) in [(0, Some(even)), (1, even.checked_sub(7))] {
|
||||
let Some(column) = column.filter(|&x| x < width) else { continue };
|
||||
for colour in 0..2 {
|
||||
if xtrans && (MASKS[pair][colour] >> (column % 4)) & 1 == 0 {
|
||||
continue;
|
||||
}
|
||||
let (sum, count) = &mut if q == 0 { &mut main } else { &mut fine }[pair % 3][parity];
|
||||
let k = 32 - (sum.saturating_sub(1) / *count).leading_zeros();
|
||||
stream.push(true);
|
||||
stream.extend(std::iter::repeat_n(false, k as usize));
|
||||
if *count == 64 {
|
||||
*sum /= 2;
|
||||
*count /= 2;
|
||||
}
|
||||
*count += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
stream.chunks(8).map(|chunk| chunk.iter().enumerate().fold(0, |b, (i, bit)| b | (u8::from(*bit) << (7 - i)))).collect()
|
||||
}
|
||||
|
||||
fn compressed(bits: u32, xtrans: bool, quantizers: Option<&[u8]>) -> Vec<u8> {
|
||||
let groups = quantizers.map_or(3, <[u8]>::len);
|
||||
let stream = zero_stream(bits, xtrans, quantizers);
|
||||
// Two stripes; the second is mostly padding, which must never enter the output.
|
||||
let mut out = b"IS".to_vec();
|
||||
out.extend_from_slice(&[u8::from(quantizers.is_none()), if xtrans { 16 } else { 0 }, bits as u8]);
|
||||
for v in [groups as u16 * 6, 1536, 780, 768] {
|
||||
out.extend_from_slice(&v.to_be_bytes());
|
||||
}
|
||||
out.push(2);
|
||||
out.extend_from_slice(&(groups as u16).to_be_bytes());
|
||||
for _ in 0..2 {
|
||||
out.extend_from_slice(&(stream.len() as u32).to_be_bytes());
|
||||
}
|
||||
out.resize(32, 0);
|
||||
if let Some(q) = quantizers {
|
||||
for _ in 0..2 {
|
||||
out.extend_from_slice(q);
|
||||
out.resize(out.len() + q.len().div_ceil(16) * 16 - q.len(), 0);
|
||||
}
|
||||
}
|
||||
out.extend_from_slice(&stream);
|
||||
out.extend_from_slice(&stream);
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lossless_and_lossy_bayer_and_xtrans_stripes() {
|
||||
for bits in [12, 14, 16] {
|
||||
for xtrans in [false, true] {
|
||||
for q in [None, Some([2, 0, 5].as_slice())] {
|
||||
let src = compressed(bits, xtrans, q);
|
||||
let out = decode(&src, 780, 18, bits, xtrans, Mode::Full).unwrap();
|
||||
assert_eq!(out, vec![0; 780 * 18]);
|
||||
assert!(decode(&src, 780, 18, bits, xtrans, Mode::Header).unwrap().is_empty());
|
||||
let layout = xtrans.then(|| {
|
||||
let mut l = [0; 36];
|
||||
for (to, from) in l.iter_mut().zip(XTRANS.iter().flatten().rev()) {
|
||||
*to = *from as u8;
|
||||
}
|
||||
l
|
||||
});
|
||||
let raf = super::super::raf::tests::raf(780, 18, bits, src, layout, &[]);
|
||||
let image = crate::decode(&raf).unwrap();
|
||||
assert_eq!(crate::probe_info(&raf).unwrap(), image.info());
|
||||
assert_eq!(image.data, crate::RawData::U16(out));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_headers_and_truncated_entropy_are_errors() {
|
||||
let valid = compressed(14, true, Some(&[1, 2, 0]));
|
||||
for len in (0..96).chain([valid.len() / 2, valid.len() - 1]) {
|
||||
assert!(decode(&valid[..len], 780, 18, 14, true, Mode::Full).is_err(), "length {len}");
|
||||
}
|
||||
for at in [2, 3, 4, 5, 7, 9, 11, 13, 14, 16] {
|
||||
let mut src = valid.clone();
|
||||
src[at] ^= 0xff;
|
||||
assert!(decode(&src, 780, 18, 14, true, Mode::Full).is_err(), "header byte {at}");
|
||||
}
|
||||
assert!(decode(&valid, usize::MAX, usize::MAX, 14, true, Mode::Full).is_err());
|
||||
let mut reader = Bits::new(&[0; 64]);
|
||||
assert!(reader.residual(0, 14, 16384, 41).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn residual_sign_wrap_and_escape() {
|
||||
// Ordinary codes 0, 1, 2, 3 with k=2; then the escape for -8192.
|
||||
let mut stream = vec![true, false, false, true, false, true, true, true, false, true, true, true];
|
||||
stream.extend(std::iter::repeat_n(false, 41));
|
||||
stream.push(true);
|
||||
let escaped = 16382u32;
|
||||
stream.extend((0..14).rev().map(|i| (escaped >> i) & 1 != 0));
|
||||
let bytes: Vec<u8> = stream.chunks(8).map(|chunk| chunk.iter().enumerate().fold(0, |b, (i, bit)| b | (u8::from(*bit) << (7 - i)))).collect();
|
||||
let mut reader = Bits::new(&bytes);
|
||||
for delta in [0, -1, 1, -2, -8192] {
|
||||
assert_eq!(reader.residual(2, 14, 16384, 41).unwrap(), delta);
|
||||
}
|
||||
assert_eq!(even_prediction(1112, 858, 1104, 1152), 983);
|
||||
}
|
||||
|
||||
proptest! {
|
||||
#![proptest_config(ProptestConfig::with_cases(256))]
|
||||
#[test]
|
||||
fn malformed_streams_never_panic(data in prop::collection::vec(any::<u8>(), 0..2048), quantizer in any::<u8>(), bits in prop::sample::select(vec![12,14,16]), xtrans in any::<bool>()) {
|
||||
let mut src = compressed(bits, xtrans, Some(&[quantizer]));
|
||||
let start = 64;
|
||||
src.truncate(start);
|
||||
for i in [16, 20] { src[i..i+4].copy_from_slice(&(data.len() as u32).to_be_bytes()); }
|
||||
src.extend_from_slice(&data);
|
||||
src.extend_from_slice(&data);
|
||||
let _ = decode(&src, 780, 6, bits, xtrans, Mode::Full);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -16,7 +16,6 @@ fn corpus_root() -> PathBuf {
|
||||
/// Variants known not to decode yet (see the crate docs): matched against the lower-case file name.
|
||||
const KNOWN_UNSUPPORTED: &[&str] = &[
|
||||
"cr3-", // CR3 / CRX (M11.1)
|
||||
"raf-fuji-xt20-compressed", // Fujifilm compressed RAF
|
||||
"orf-olympus-em", // Olympus compressed ORF
|
||||
"sraw", // Canon sRAW / mRAW
|
||||
];
|
||||
@@ -427,3 +426,44 @@ fn corpus_panasonic_encodings() {
|
||||
}
|
||||
eprintln!("Panasonic encodings checked on {seen} files");
|
||||
}
|
||||
|
||||
/// Reference sensor sums and position-weighted sums from black-box decoding, independent
|
||||
/// of this implementation. These cover complete images, padding, stripe boundaries and
|
||||
/// lossy block quantizer changes; the source files are CC0, fetched by `xtask corpus`.
|
||||
#[test]
|
||||
fn corpus_fujifilm_compressed_samples() {
|
||||
let cases: &[(&str, usize, usize, u64, u64)] = &[
|
||||
("raf-fuji-gfx100-3773.raf", 11808, 8754, 483668733090, 6453395759968151080),
|
||||
("raf-fuji-gfx100-3775.raf", 11808, 8754, 120434414850, 6144091676930008551),
|
||||
("raf-fuji-gfx100rf-8091.raf", 11808, 8754, 509005112862, 6771145666297463071),
|
||||
("raf-fuji-gfx100s-4495.raf", 11808, 8754, 941987648006, 4811558780180901389),
|
||||
("raf-fuji-gfx100s-4503.raf", 11808, 8754, 939580201129, 4821879272388485929),
|
||||
("raf-fuji-gfx50s-1435.raf", 9216, 6210, 79545293042, 2565169091480702981),
|
||||
("raf-fuji-xe5-8509.raf", 7872, 5196, 76435253154, 1299804440011842144),
|
||||
("raf-fuji-xh2-6001.raf", 7872, 5196, 83480264024, 1799924693528487099),
|
||||
("raf-fuji-xh2-6002.raf", 7872, 5196, 72476569659, 1547275125734054244),
|
||||
("raf-fuji-xm5-7748.raf", 6336, 4182, 93243605328, 986712869515471030),
|
||||
("raf-fuji-xt2-865.raf", 6048, 4038, 44212684244, 541983062567959797),
|
||||
("raf-fuji-xt20-compressed.raf", 6048, 4038, 72690241711, 882852288583469317),
|
||||
("raf-fuji-xt4-3914.raf", 6384, 4182, 45943146703, 599112804573868025),
|
||||
("raf-fuji-xt4-3918.raf", 6384, 4182, 46401490361, 605690093061448074),
|
||||
("raf-fuji-xt5-6122.raf", 7872, 5196, 109559716516, 1653376659872066392),
|
||||
("raf-fuji-xt5-6123.raf", 7872, 5196, 109489917369, 1673725237502600190),
|
||||
("raf-fuji-xt50-7807.raf", 7872, 5196, 76453904143, 1262190863834170778),
|
||||
];
|
||||
let dir = corpus_root().join("raw");
|
||||
let mut seen = 0;
|
||||
for &(name, width, height, sum, weighted) in cases {
|
||||
let Ok(bytes) = std::fs::read(dir.join(name)) else { continue };
|
||||
let img = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
|
||||
assert_eq!((img.width, img.height), (width, height), "{name}");
|
||||
let lightcraft_raw::RawData::U16(data) = img.data else { panic!("{name}: float data") };
|
||||
let actual = data
|
||||
.iter()
|
||||
.enumerate()
|
||||
.fold((0u64, 0u64), |(s, w), (i, &v)| (s.wrapping_add(u64::from(v)), w.wrapping_add((i as u64 + 1).wrapping_mul(u64::from(v)))));
|
||||
assert_eq!(actual, (sum, weighted), "{name}: sensor samples differ from reference");
|
||||
seen += 1;
|
||||
}
|
||||
eprintln!("verified {seen} Fujifilm compressed sensor arrays");
|
||||
}
|
||||
|
||||
+3
-3
@@ -60,7 +60,7 @@ that decide whether a photographer can switch (see the honest assessment in [ROA
|
||||
Take the first one nobody is working on.
|
||||
|
||||
1. **LR-PROF-CAMERACOLOR** (P0): our own camera colour calibration. Sony ARW, Nikon NEF and Panasonic RW2 now get a guarded file-local fit to their own embedded JPEG (colour matrix + hue/saturation/value table + tone and chroma curves, relative WB; per-model profiles pooled from many photos via `lightcraft-cli calibrate` for ARW and NEF; ARW: 7 of 10 public samples accepted, mean ΔE vs the camera JPEG 17–26 → 3–10; NEF: 13 of 13 decodable samples from 6 bodies accepted, ΔE 13–46 → 3–8, one mixed-light scene 13 → 12; RW2: 140 of 174 public samples accepted, from 93 of the 114 bodies with a preview, median ΔE 5.0); measured calibration and fidelity remain missing. Other non-DNG raws and rejected fits still use a neutral matrix. Expand validated preview fitting and use matrices the files carry themselves; never Adobe data.
|
||||
2. **LR-IMP-FORMATS** (P0): **CR3** first (every Canon body since ~2018), then compressed RAF / ORF, Nikon
|
||||
2. **LR-IMP-FORMATS** (P0): **CR3** first (every Canon body since ~2018), then compressed ORF, Nikon
|
||||
"lossy after split" NEF, Canon sRAW; HEIC/AVIF decode. Clean-room, from prose descriptions only (see
|
||||
`crates/raw/src/vendor/nefc.rs` for how compressed NEF was done). Until decoded, such photos are `preview_only`.
|
||||
3. **LR-IMP-CAMERA-COVERAGE** (P0): per-model verification; grow the CC0 corpus and fix per-model bugs (like the CR2
|
||||
@@ -124,8 +124,8 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
|
||||
| LR-IMP-PROFILES | Import profiles & presets | P1 | ✅ | `cmd:file.importPresets`, `cmd:preset.import`, `cmd:profile.import`, `crates/engine/src/preset_import.rs`, `crates/engine/src/preset_luminar.rs` | presets: .lcpreset, XMP, classic .lrtemplate, photos carrying edits ("DNG presets"), Luminar looks (.lmp, .mplumpack collections; sliders with a counterpart), .zip bundles, folders (folder → group), drag & drop; masks carried over; unmapped settings reported; profiles: .cube 3D LUTs; Adobe profile formats deliberately unsupported |
|
||||
| LR-IMP-LOCAL | Work on files in place | P0 | ✅ | `cmd:library.browse`, `cmd:photo.addToLibrary`, `cmd:library.import` (mode add), `cmd:local.addRoot`, `cmd:local.hide`, `cmd:local.restoreHidden`, `crates/engine/src/cmd/browse.rs`, `crates/ui-egui/src/panels/left.rs` (`local_section`), `crates/ui-egui/src/panels/grid.rs` (`folder_header`) | Local: browse Pictures / Desktop / Downloads / Home or any folder without adding it (breadcrumb, Include subfolders, Add N to My Photos); browsed photos stay out of All Photos, albums and counts; folders picked with Browse Folder… (or Keep in Local / Add to Local on a folder row) stay listed across restarts, and browsing a folder inside a listed one highlights it inside that tree (opened down to it, siblings reachable) instead of adding a root; edits go to XMP sidecars; importing promotes them; right-click a top-level location → Remove from Local hides the shortcut only (nothing on disk or in the catalog changes), saved with the UI state, and “Show N hidden locations” (or Browse Folder… on it) brings it back; a folder is one location however its path is spelled (`/` or `\`, trailing separator, `.`/`..`, drive-letter case; `folder_key` in `crates/catalog/src/query.rs`); browsed photos nobody changed are forgotten once their folder has not been browsed for 30 days (Settings → Performance → Local folders, 0 = never; checked when the library opens, or `cmd:library.forgetLocal` with `dryRun`; files and sidecars stay, browsing again brings them back; see `crates/catalog/src/local.rs` for what counts as a change) |
|
||||
| LR-IMP-SIDECAR-SPLIT | Separate XMP sidecar variants | P2 | ⬜ | `cmd:library.xmpPreferences` | sidecar naming option exists (stem/full), no split sidecars |
|
||||
| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW (pre-2017 bodies: white balance and black level from enciphered maker-note / `SR2SubIFD` data, issue #148; downsized Sony lossless YCbCr 4:2:0 / 4:2:2 tiles decoded to linear RGB, verified on private ILCE-7M4 3:2 & 4:3 examples including 16-bit full-resolution export), NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF, RW2 / Leica RWL / Panasonic RAW in every raw format (compressed 4 and 6, the prefix-coded strips of format 8 from the GH6 on, packed 2/5/7, the 16-bit words of the 2005–2007 bodies; black levels, in-camera aspect crops and mapped-out defect pixels; 178 CC0 files from 118 bodies checked, `crates/raw/src/vendor/rw2.rs`), PEF, ORF. 12-bit NEF black level: maker note `0x003d` is in 14-bit units (D750, D780, D850, D7500, Z 50 checked). CR3 opens from its full-size embedded JPEG with full metadata (Exif, GPS, XMP from the `CMT1`/`CMT2`/`CMT4` boxes, `crates/meta/src/cr3.rs`) but its raw data is not decoded yet. Missing: CR3 raw decoding (CRX), compressed RAF/ORF, NEFs labelled compressed (34713) but stored uncompressed without a `0x0096` table (Z 6 packed 14-bit, D850 12-bit uncompressed), NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
|
||||
| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~55 CC0 corpus files (plus ~60 NEFs checked by hand and 178 Panasonic / Leica RW2, RWL and RAW files from 118 bodies, checked against each file's own JPEG) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies). Grow the CC0 corpus per model and test each sample decodes with plausible colour |
|
||||
| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW (pre-2017 bodies: white balance and black level from enciphered maker-note / `SR2SubIFD` data, issue #148; downsized Sony lossless YCbCr 4:2:0 / 4:2:2 tiles decoded to linear RGB, verified on private ILCE-7M4 3:2 & 4:3 examples including 16-bit full-resolution export), NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans; `crates/raw/src/vendor/rafc.rs`, full sensor verification on 13 bodies, `docs/raf-compression.md`), RW2 / Leica RWL / Panasonic RAW in every raw format (compressed 4 and 6, the prefix-coded strips of format 8 from the GH6 on, packed 2/5/7, the 16-bit words of the 2005–2007 bodies; black levels, in-camera aspect crops and mapped-out defect pixels; 178 CC0 files from 118 bodies checked, `crates/raw/src/vendor/rw2.rs`), PEF, ORF. 12-bit NEF black level: maker note `0x003d` is in 14-bit units (D750, D780, D850, D7500, Z 50 checked). CR3 opens from its full-size embedded JPEG with full metadata (Exif, GPS, XMP from the `CMT1`/`CMT2`/`CMT4` boxes, `crates/meta/src/cr3.rs`) but its raw data is not decoded yet. Missing: CR3 raw decoding (CRX), compressed ORF, NEFs labelled compressed (34713) but stored uncompressed without a `0x0096` table (Z 6 packed 14-bit, D850 12-bit uncompressed), NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
|
||||
| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~55 CC0 corpus files (plus ~60 NEFs checked by hand and 178 Panasonic / Leica RW2, RWL and RAW files from 118 bodies, checked against each file's own JPEG) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies). Compressed RAF sensor arrays verified on 13 Fujifilm bodies (`docs/raf-compression.md`). Grow the CC0 corpus per model and test each sample decodes with plausible colour |
|
||||
| LR-IMP-CULL-AT-IMPORT | Culling analysis at import | P2 | 🟡 | `cmd:photo.analyze` | run Assisted Culling on the imported photos (they're selected after an import); not automatic |
|
||||
| LR-IMP-MOVE | Move on import [Classic] | P1 | ✅ | `cmd:library.import` (`mode` move), `crates/engine/src/import_move.rs`, `crates/engine/src/tests_import_move.rs`, `crates/ui-egui/src/import.rs` | Transfer ▸ Move with Copy's destination, folders (day / month / one folder / custom template), rename template and example destination; XMP sidecars (both namings) move along; each source is removed only after its destination is written (hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate, unchecked files keep their sources, taken names get -1, -2…, never overwritten; files already in the destination/library are added in place; sources that can't be removed (read-only card) are kept and reported (`kept`). Undo removes the photos from the library but leaves the moved files at the destination. No Copy as DNG while moving |
|
||||
| LR-IMP-DNG-CONVERT | Convert to DNG on import [Classic] | P2 | ✅ | `cmd:library.import` (`dng`), `crates/ui-egui/src/import.rs` | copy imports: Raw files ▸ Copy as DNG (lossless; the card is untouched); the raw copy is removed only once its DNG is verified and on disk (issue #106) |
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
# Fujifilm compressed RAF
|
||||
|
||||
LightCraft decodes uncompressed, lossless compressed and lossy compressed RAF sensor data in pure Rust. The shared RAW decoder serves import, desktop previews, editing, CLI rendering, web and export. These files no longer enter the embedded-JPEG-only fallback. After restarting with the updated build, existing imported preview-only photos need **Photo → Reload from Disk** to refresh their status; edits are retained.
|
||||
|
||||
Compression support does not supply camera colour calibration. Fujifilm currently uses the existing neutral camera-colour fallback; rendering can differ from the camera JPEG and Lightroom.
|
||||
|
||||
## Format and implementation
|
||||
|
||||
`crates/raw/src/vendor/raf.rs` reads the RAF header, raw TIFF IFD, crop, CFA, black level and white balance. The big-endian compression field at byte 108 distinguishes uncompressed (0), lossless compressed (2) and lossy compressed (3); testing only the strip's first two bytes would misclassify an uncompressed 16-bit sample of `0x5349`.
|
||||
|
||||
`crates/raw/src/vendor/rafc.rs` handles the compression stream:
|
||||
|
||||
- A 16-byte `IS` header carries compression mode, sensor type (Bayer or X-Trans), bit depth, dimensions, padded width, 768-pixel stripe width, stripe count and six-row group count. It must agree with the TIFF dimensions and CFA.
|
||||
- A big-endian size table, padded to 16 bytes, locates independent entropy streams. Lossy files add a table of one quantizer byte per six-row group for each stripe, each table padded to 16 bytes.
|
||||
- Each six-row group is collated into three red, six green and three blue vectors. Bayer vectors have 384 samples; X-Trans vectors have 512 slots, including interpolation slots that consume no bits. The last stripe's padding is decoded for predictor state but excluded from the output.
|
||||
- Colour-vector pairs alternate R/G and G/B. Even columns lead odd columns by seven positions. Two preceding lines of each colour and the preceding line's edge values provide prediction context.
|
||||
- Signed prediction errors use adaptive Rice coding: a unary prefix followed by a context-dependent number of bits, with a bounded escape for large errors. Three pair phases have separate even/odd statistics. The 81 signed gradient contexts fold into 41 states.
|
||||
- Lossy quantizers change the error range and reconstruction step. Low-gradient sites retain finer precision (0, 1 or 2), with five folded contexts per precision and statistics that survive main-quantizer changes. The main context statistics reset when its quantizer changes. Their initial sum is `max(2, (range + 32) / 64)`; using 16 as the minimum breaks more heavily quantized files.
|
||||
|
||||
Stripes decode in parallel using the existing Rayon dependency. Offsets, sizes, dimensions and entropy reads are checked; truncated data, impossible codes and inconsistent headers return errors. Header-only probing validates the tables and stripe ranges without decompressing the image.
|
||||
|
||||
## Verification
|
||||
|
||||
Complete sensor arrays were compared sample by sample with an external decoder used only as a black-box oracle (rawpy 0.27.1). No decoder source, Adobe data, camera matrices or profiles were used. This tool is not a product or test dependency.
|
||||
|
||||
| Coverage | Files |
|
||||
|---|---|
|
||||
| Supplied X-H2S | DSCF4192 uncompressed; DSCF4193 lossless; DSCF4194 lossy |
|
||||
| Supplied X-T4 | _DSF2355 / _DSF2356 uncompressed; _DSF2357 lossless; _DSF2358 lossy |
|
||||
| CC0 X-Trans | X-T2, X-T20, X-T4, X-H2, X-T5, X-M5, X-T50, X-E5 |
|
||||
| CC0 Bayer | GFX 50S (14-bit), GFX 100 (14/16-bit), GFX100S (16-bit lossless/lossy), GFX100RF (16-bit lossy) |
|
||||
|
||||
All 24 files matched the reference sensor arrays exactly. This covers 13 camera bodies and both older and current compression variants. Synthetic tests also cover 12-bit streams; a real compressed 12-bit camera file has not been verified. Support is selected from the file's layout, without a camera-name allowlist. Other bodies using this layout should decode, but this is not a claim that every Fujifilm model has been tested. Older FinePix layouts without the supported raw TIFF IFD remain unsupported.
|
||||
|
||||
The 17 CC0 files live in the gitignored corpus, with published SHA-256 and licence records. `cargo xtask corpus --download` fetches them. `crates/raw/tests/corpus.rs` records reference sensor sums and position-weighted sums as regression checks; the full-array comparisons were performed locally. No supplied media or large binary fixtures are committed.
|
||||
|
||||
Unconditional procedural tests cover Bayer/X-Trans, 12/14/16 bits, partial final stripes, quantizer changes, residual signs and escapes, malformed headers and truncated/random entropy streams. A regression also checks overflowing uncompressed row strides.
|
||||
|
||||
Local release decode measurements: about 0.16–0.24 seconds for 24–40 MP X-Trans files, 0.29 seconds for 50 MP Bayer, and 0.5–0.6 seconds for 100 MP Bayer. These measure sensor decoding, not the full develop/export pipeline.
|
||||
|
||||
App verification imported all four supplied compressed files, confirmed `previewOnly: null`, applied +2 EV exposure edits and inspected headless UI screenshots. CLI JPEG exports at a 1600-pixel long edge succeeded for all four files (0.58–0.76 seconds each, including process startup).
|
||||
|
||||
## Sources
|
||||
|
||||
- [Fabian's original prose description of lossless RAF compression](https://capnfabs.net/posts/fuji-raf-compression-algorithm/): independent stripes, colour vectors, iteration order, predictors' neighbourhoods, Rice coding and gradient adaptation. Its linked implementations were not consulted.
|
||||
- [ExifTool Fujifilm tag-name documentation](https://exiftool.sourceforge.net/TagNames/FujiFilm.html): RAF header and raw-IFD metadata fields.
|
||||
- [Fujifilm's compression-mode explanation](https://digitalcamera-support-en.fujifilm.com/digitalcameraengpcdetail?aid=000008311&wd=compressed+raw).
|
||||
- [raw.pixls.us](https://raw.pixls.us/): CC0 camera samples, published checksums and licence metadata.
|
||||
- [rawpy's documented `raw_image` API](https://letmaik.github.io/rawpy/api/rawpy.RawPy.html): the complete sensor array includes margins, allowing comparisons before demosaic or colour processing.
|
||||
|
||||
Exact byte layouts, colour-vector masks, prediction formulas, adaptation parameters and lossy precision selection were established through sample analysis and black-box comparisons.
|
||||
@@ -393,6 +393,41 @@ const RAW_SAMPLES: &[(&str, &str)] = &[
|
||||
("raw-panasonic-fz50.raw", "https://raw.pixls.us/getfile.php/2234/nice/Panasonic%20-%20DMC-FZ50%20-%204:3.RAW"),
|
||||
("raw-panasonic-fz8.raw", "https://raw.pixls.us/getfile.php/2282/nice/Panasonic%20-%20DMC-FZ8%20-%204:3.RAW"),
|
||||
("rw2-panasonic-fz1000m2-4x3.rw2", "https://raw.pixls.us/getfile.php/4706/nice/Panasonic%20-%20DC-FZ10002%20-%204:3.RW2"),
|
||||
// Fujifilm predictive compression: older X-Trans, 40 MP X-Trans, 14/16-bit GFX; both compression modes.
|
||||
(
|
||||
"raf-fuji-gfx100-3773.raf",
|
||||
"https://raw.pixls.us/getfile.php/3773/nice/Fujifilm%20-%20GFX%20100%20-%2016bit%2016bit%20compressed%20%284:3%29.RAF",
|
||||
),
|
||||
(
|
||||
"raf-fuji-gfx100-3775.raf",
|
||||
"https://raw.pixls.us/getfile.php/3775/nice/Fujifilm%20-%20GFX%20100%20-%2014bit%2014bit%20compressed%20%284:3%29.RAF",
|
||||
),
|
||||
("raf-fuji-gfx100rf-8091.raf", "https://raw.pixls.us/getfile.php/8091/nice/Fujifilm%20-%20GFX100RF%20-%2016bit%20compressed%20%284:3%29.RAF"),
|
||||
(
|
||||
"raf-fuji-gfx100s-4495.raf",
|
||||
"https://raw.pixls.us/getfile.php/4495/nice/Fujifilm%20-%20GFX100S%20-%2016bit%2016bit%20lossless%20compressed%20%284:3%29.RAF",
|
||||
),
|
||||
(
|
||||
"raf-fuji-gfx100s-4503.raf",
|
||||
"https://raw.pixls.us/getfile.php/4503/nice/Fujifilm%20-%20GFX100S%20-%2016bit%2016bit%20compressed%20%284:3%29.RAF",
|
||||
),
|
||||
(
|
||||
"raf-fuji-gfx50s-1435.raf",
|
||||
"https://raw.pixls.us/getfile.php/1435/nice/Fujifilm%20-%20GFX%2050S%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF",
|
||||
),
|
||||
("raf-fuji-xe5-8509.raf", "https://raw.pixls.us/getfile.php/8509/nice/Fujifilm%20-%20X-E5%20-%2014bit%20lossy%20compressed%20%283:2%29.RAF"),
|
||||
("raf-fuji-xh2-6001.raf", "https://raw.pixls.us/getfile.php/6001/nice/Fujifilm%20-%20X-H2%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
("raf-fuji-xh2-6002.raf", "https://raw.pixls.us/getfile.php/6002/nice/Fujifilm%20-%20X-H2%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
("raf-fuji-xm5-7748.raf", "https://raw.pixls.us/getfile.php/7748/nice/Fujifilm%20-%20X-M5%20-%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
("raf-fuji-xt2-865.raf", "https://raw.pixls.us/getfile.php/865/nice/Fujifilm%20-%20X-T2%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
(
|
||||
"raf-fuji-xt4-3914.raf",
|
||||
"https://raw.pixls.us/getfile.php/3914/nice/Fujifilm%20-%20X-T4%20-%2014bit%2014bit%20lossless%20compressed%20%283:2%29.RAF",
|
||||
),
|
||||
("raf-fuji-xt4-3918.raf", "https://raw.pixls.us/getfile.php/3918/nice/Fujifilm%20-%20X-T4%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
("raf-fuji-xt5-6122.raf", "https://raw.pixls.us/getfile.php/6122/nice/Fujifilm%20-%20X-T5%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
("raf-fuji-xt5-6123.raf", "https://raw.pixls.us/getfile.php/6123/nice/Fujifilm%20-%20X-T5%20-%2014bit%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
("raf-fuji-xt50-7807.raf", "https://raw.pixls.us/getfile.php/7807/nice/Fujifilm%20-%20X-T50%20-%2014bit%20compressed%20%283:2%29.RAF"),
|
||||
("rw2-panasonic-g9-b.rw2", "https://raw.pixls.us/getfile.php/2348/nice/Panasonic%20-%20DC-G9%20-%204:3.RW2"),
|
||||
("rw2-panasonic-g9.rw2", "https://raw.pixls.us/getfile.php/2585/nice/Panasonic%20-%20DC-G9%20-%204:3.RW2"),
|
||||
("rw2-panasonic-gh1.rw2", "https://raw.pixls.us/getfile.php/1323/nice/Panasonic%20-%20DMC-GH1%20-%204:3.RW2"),
|
||||
|
||||
Reference in New Issue
Block a user