LR-IMP-PREVIEW-READ: a preview reads the raw's header, not the raw

A grid thumbnail and a loupe preview of an unedited raw show the camera's embedded JPEG, which
sits near the start of the file — for a 24 MP Fuji RAF, 5.6 MB of a 56 MB file (measured:
DSCF9492.RAF). The loader read the file whole and threw the sensor data away, so every preview
paid for 50 MB nobody looked at. On a local disk that is merely wasteful; on a network share,
where photographers keep libraries, it is the wait: the same preview took 10 s over a GVFS/SMB
Wi-Fi share against 0.96 s after this change.

- `lightcraft_raw::bounded_preview(bytes)` locates the preview from the header alone and reports
  its byte range; `embedded_preview_from_range` extracts it, byte for byte what `embedded_preview`
  returns for the whole file (the pointer is read from the RAF header's fixed u32s at 84/88).
- `engine::files::preview_window` reads a 4 KiB prefix, then — because the container reader clamps
  a slice to the buffer it was given, so a short prefix reports a plausible but truncated 3948
  bytes — re-reads exactly the extent the header declares, and takes the window out of the prefix
  rather than reading those bytes twice. `LIGHTCRAFT_PROFILE=1` prints what each preview cost.
- Formats whose preview only the whole file reveals (CR3 boxes, TIFF pointer tags past the prefix,
  CRW/MRW/X3F scans), a lying pointer, a short read and a file that changes underneath us all fall
  back to the whole-file path: a damaged header never costs the user their preview.
- Tests: the header range matches `embedded_preview` byte for byte, with a clamped prefix, a padded
  extent, a late pointer, a lying pointer and a non-JPEG window covered; on a synthetic RAF with a
  200 KB tail the quick path and the whole-file path produce identical pixels and the read stays
  under half the file; `crates/engine/tests/preview_read.rs` (opt-in, `#[ignore]`) measures a real
  file on a share. No raw fixture is committed.
- `docs/raw-preview-reads.md` records the mechanism and the numbers.

Co-Authored-By: DeepSeek V3.2 <noreply@deepseek.com>
This commit is contained in:
goujicheng
2026-10-10 22:16:04 +08:00
co-authored by DeepSeek V3.2
parent b70ae1741e
commit ab438b05ca
7 changed files with 457 additions and 8 deletions
+1
View File
@@ -307,6 +307,7 @@ LightCraft is young and moving fast. **Where we honestly stand** (details in the
- **camera colour calibration:** Sony, Nikon, Panasonic, Fujifilm and Canon CR3 raws have guarded estimates from their camera JPEGs, with built-in ILCE-7CR, ILCE-7M4, X-H2S and X-T4 profiles; measured calibration is missing, and other raws or rejected fits retain a neutral matrix;
- **compressed Olympus raws and unsupported CR3 variants:** these use embedded JPEG previews when present. Fujifilm lossless/lossy compressed RAF now decodes sensor data; [verification and existing-library reload instructions](docs/raf-compression.md);
- **AI masks and denoise:** subject and sky selection are classical heuristics;
- **network libraries:** a preview reads the raw's header and its embedded JPEG only (56.5 MB → 5.9 MB, 10 s → 1 s on the test share), but the *developed* picture still needs the sensor data over the wire; see [docs/raw-preview-reads.md](docs/raw-preview-reads.md);
- **on-screen HDR display, video and the Classic Print / Book / Map modules.**
- **What's next:** see [where we're going](ROADMAP.md#where-were-going).
+179 -2
View File
@@ -605,17 +605,144 @@ fn fit_box(max_edge: usize) -> lightcraft_codecs::DecodeOptions {
/// Filesystem-backed embedded-preview hook (native).
///
/// Reads the whole file to find the preview, so it holds the memory gate for twice the file's size (the file and the
/// decoded preview): background work (the face scan) waits for room, interactive work is counted and never waits.
/// The quick path reads the file's header and its embedded preview and nothing else (see
/// [`read_preview_window`]); a format whose preview the header cannot point at falls back to the
/// whole-file path, which is what every format used to cost. The memory gate still counts the whole
/// file, which is what the decode of one can hold.
pub fn fs_preview_loader() -> PreviewLoader {
Arc::new(|path: &str, max_edge: usize| {
let len = std::fs::metadata(path).map(|m| usize::try_from(m.len()).unwrap_or(usize::MAX)).unwrap_or(0);
let gate = crate::memory::work_gate();
let _permit = if crate::memory::is_background() { gate.acquire(len.saturating_mul(2)) } else { gate.acquire_urgent(len.saturating_mul(2)) };
if let Some(image) = preview_window(path).ok().flatten().and_then(|(_, window, _)| quick_image_srgb(&window, max_edge)) {
return Some(image);
}
embedded_preview_srgb(&std::fs::read(path).ok()?, max_edge)
})
}
/// What [`preview_window`] read from a file, so a caller can see what a preview cost.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct PreviewRead {
/// Bytes read.
pub read: u64,
/// The file's size.
pub total: u64,
/// Whether the preview came from a bounded read (`false`: the whole file was read, which is what
/// every preview cost before the header path existed).
pub header_path: bool,
}
impl PreviewRead {
/// The share of the file that was read, `0.0`..=`1.0`.
pub fn fraction(self) -> f64 {
if self.total == 0 {
return 0.0;
}
(self.read as f64 / self.total as f64).clamp(0.0, 1.0)
}
}
/// The file's header and its embedded preview, read without the rest of the file: `(header bytes,
/// preview bytes, what was read)`. `None` when the header points at no preview this build can find,
/// which is the caller's cue to read the file whole.
///
/// Why it exists: a 56 MB Fuji RAF keeps its JPEG in the first 5.6 MB (measured: `DSCF9492.RAF`),
/// so reading the file whole to extract it costs ten times the bytes — and on a network share ten
/// times the wait (measured on that file: 2.5 s local against 25.6 s over SMB). `embedded_preview`
/// still needs the whole container, so the header comes back with the window; for a preview taken
/// out of a container that carries no preview colour space of its own (Nikon's maker note) the
/// whole-file fallback stays in charge.
pub fn preview_window(path: &str) -> Result<Option<(Vec<u8>, Vec<u8>, PreviewRead)>, String> {
use std::io::{Read, Seek, SeekFrom};
let mut file = std::fs::File::open(path).map_err(|e| format!("{path}: {e}"))?;
let total = file.metadata().map_err(|e| format!("{path}: {e}"))?.len();
let mut prefix = read_prefix(&mut file, path, total)?;
// What the header itself declares, before the container reader is asked: it clamps a slice to
// the buffer it was given, so a prefix that stops inside the declared preview yields a
// self-consistently short range that looks trustworthy (measured on a real 56 MB DSCF9492.RAF:
// a 4 KiB prefix reports 3948 bytes where the file holds 5.6 MB). Reading the declared extent
// first costs nothing — it lives in the first bytes of every format covered here.
if let Some(extent) = raf_preview_extent(&prefix)
&& extent > prefix.len() as u64
{
prefix = read_prefix_at(&mut file, path, extent.min(total))?;
}
let Some(range) = lightcraft_raw::bounded_preview(&prefix) else { return Ok(None) };
// A preview at the file's end may have been padded past it: clamp rather than fail.
let end = range.offset.saturating_add(range.window_len).min(total);
let len = usize::try_from(end.saturating_sub(range.offset)).map_err(|_| format!("{path}: preview too large"))?;
// When the prefix already covers the range (the usual case: the prefix was grown to the declared
// extent), take the window out of it. Reading those bytes twice would double a 5.6 MB preview.
let prefix_len = prefix.len() as u64;
let from_prefix = usize::try_from(range.offset).ok().and_then(|start| prefix.get(start..start.checked_add(len)?)).map(<[u8]>::to_vec);
let (window, bytes_read) = match from_prefix {
Some(window) => (window, prefix_len),
None => {
let mut window = vec![0u8; len];
file.seek(SeekFrom::Start(range.offset)).map_err(|e| format!("{path}: {e}"))?;
file.read_exact(&mut window).map_err(|e| format!("{path}: {e}"))?;
(window, prefix_len + len as u64)
}
};
// A lying pointer must not cost the user their preview: hand the caller the whole-file fallback.
if lightcraft_raw::embedded_preview_from_range(&window, range).is_none() {
return Ok(None);
}
let read = PreviewRead { read: bytes_read, total, header_path: true };
// On a network share the bytes are the wait, so say what this preview actually cost. A window
// taken from the prefix cost nothing beyond the prefix itself.
if lightcraft_pipeline::profiling() {
eprintln!("[profile] preview {path}: read {} of {} bytes (header {}, window {})", read.read, total, prefix_len, window.len());
}
Ok(Some((prefix, window, read)))
}
/// The file's first [`lightcraft_raw::PREVIEW_PREFIX`] bytes (or the whole file when it is shorter),
/// which is what every format [`lightcraft_raw::bounded_preview`] covers keeps its pointer in.
fn read_prefix(file: &mut std::fs::File, path: &str, total: u64) -> Result<Vec<u8>, String> {
let len = lightcraft_raw::PREVIEW_PREFIX.min(usize::try_from(total).unwrap_or(usize::MAX));
read_prefix_at(file, path, len as u64)
}
/// The file's first `len` bytes, read from the start.
fn read_prefix_at(file: &mut std::fs::File, path: &str, len: u64) -> Result<Vec<u8>, String> {
use std::io::{Read, Seek, SeekFrom};
let len = usize::try_from(len).map_err(|_| format!("{path}: file too large"))?;
let mut prefix = vec![0u8; len];
file.seek(SeekFrom::Start(0)).map_err(|e| format!("{path}: {e}"))?;
file.read_exact(&mut prefix).map_err(|e| format!("{path}: {e}"))?;
Ok(prefix)
}
/// The end of a Fuji RAF's preview block, from the two big-endian u32s every RAF header carries at
/// 84 (offset) and 88 (length) — the same fields `lightcraft_raw`'s RAF reader uses, read here
/// without the rest of the file so the prefix can be made big enough first.
fn raf_preview_extent(prefix: &[u8]) -> Option<u64> {
if !prefix.starts_with(b"FUJIFILMCCD-RAW") {
return None;
}
let u32_at = |at: usize| prefix.get(at..at + 4).map(|s| u32::from_be_bytes([s[0], s[1], s[2], s[3]])).map(u64::from);
let (offset, len) = (u32_at(84)?, u32_at(88)?);
(offset > 0 && len > 0).then(|| offset.checked_add(len)).flatten()
}
/// The display-ready preview of a raw whose embedded JPEG was already located: the counterpart of
/// [`embedded_preview_srgb`] for [`read_preview_window`], which does not re-read the container.
/// `None` when the bytes are not a decodable image.
fn quick_image_srgb(window: &[u8], max_edge: usize) -> Option<lightcraft_raster::Rgba8> {
let mut d = lightcraft_codecs::decode(window, fit_box(max_edge)).ok()?;
if d.image.width.max(d.image.height) > max_edge {
d.image = fit(&d.image, max_edge, max_edge, Filter::Box);
d.alpha = None;
}
// The window is the JPEG itself, so its own EXIF carries the orientation; `decode` reports it.
Some(d.to_srgb8().oriented(Orientation::from_exif(d.orientation)))
}
/// The filesystem-backed [`PairLoader`]: the file is read and decoded once, and developed as usual and from the photo's
/// denoised picture. It holds two pictures at once, so it counts for more at the memory gate than a plain load.
pub fn fs_pair_loader() -> PairLoader {
@@ -689,6 +816,56 @@ mod tests {
use super::*;
use lightcraft_codecs::{ChromaSubsampling, EncodeImage, EncodeMeta, Samples, encode_jpeg};
/// A RAF whose preview pointer is in its header, then a large raw tail the reader must not need.
fn raf_with_embedded_jpeg(jpeg: &[u8], tail: usize) -> Vec<u8> {
let head = 160;
let mut f = vec![0u8; head];
f.get_mut(..16).expect("head").copy_from_slice(b"FUJIFILMCCD-RAW ");
f.get_mut(84..88).expect("jpeg offset").copy_from_slice(&(head as u32).to_be_bytes());
f.get_mut(88..92).expect("jpeg length").copy_from_slice(&(jpeg.len() as u32).to_be_bytes());
f.extend_from_slice(jpeg);
f.extend(std::iter::repeat_n(0x33u8, tail));
f
}
/// A raw's preview is read from its header and its JPEG only. Every preview used to cost a
/// whole-file read, which on a 56 MB RAF over a network share is ten times the bytes and ten
/// times the wait (measured: 2.5 s local against 25.6 s over SMB, DSCF9492.RAF). The pixels the
/// quick path produces must equal the whole-file path's, exactly.
#[test]
fn preview_loading_reads_only_the_header_and_the_jpeg() {
let (w, h) = (48u32, 32u32);
let rgb: Vec<u8> = (0..w * h * 3).map(|i| (i * 13 % 251) as u8).collect();
let img = EncodeImage::new(w, h, 3, Samples::U8(&rgb));
let jpeg = encode_jpeg(&img, 90, ChromaSubsampling::S420, &EncodeMeta::default()).unwrap();
let file = raf_with_embedded_jpeg(&jpeg, 200_000);
let dir = crate::tests_xmp::temp_dir("preview-window");
let path = dir.join("dscf0001.raf");
std::fs::write(&path, &file).unwrap();
let path = path.to_string_lossy().into_owned();
let max_edge = 512;
let (header, window, read) = preview_window(&path).expect("the file reads").expect("it has a preview");
assert_eq!(header.len(), lightcraft_raw::PREVIEW_PREFIX, "the header prefix is bounded");
assert!(read.header_path, "the header path is what answered");
assert_eq!(read.total, file.len() as u64);
assert!(read.fraction() < 0.5, "the read is a fraction of the file: {read:?}");
assert!(window.len() < file.len() / 2, "the raw tail must not be read: {} of {} bytes", window.len(), file.len());
// The quick path and the whole-file path agree byte for byte.
let quick = quick_image_srgb(&window, max_edge).expect("the window holds a decodable preview");
let whole = embedded_preview_srgb(&file, max_edge).expect("the whole file decodes too");
assert_eq!((quick.width, quick.height), (whole.width, whole.height), "same preview size");
assert_eq!(quick.data, whole.data, "byte for byte");
// A file with no locatable preview declines (the caller then reads it whole), a missing file errors.
let plain = dir.join("plain.jpg");
std::fs::write(&plain, &jpeg).unwrap();
assert!(preview_window(&plain.to_string_lossy()).expect("a plain file reads").is_none());
assert!(preview_window(&dir.join("nope.raf").to_string_lossy()).is_err());
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #367: probes read headers, not pixels, and report what the decode-based probe did:
/// oriented dimensions (EXIF orientation 6 swaps them), metadata, the error for a truncated
/// file or one that isn't an image.
+42
View File
@@ -0,0 +1,42 @@
//! Opt-in test of the embedded-preview read path on a **real file the user has** — typically one on a
//! network share, which is where reading only the header instead of the whole raw matters:
//!
//! LC_PREVIEW_RAW="/run/user/1000/gvfs/smb-share:.../DSCF0001.RAF" \
//! cargo test --release -p lightcraft-engine --test preview_read -- --ignored --nocapture
//!
//! It reports what the preview cost and fails when the read is not the small part of the file the
//! header path promises. Skipped without `LC_PREVIEW_RAW` (there is no committed raw fixture: media
//! is never committed, see AGENTS.md).
use std::time::Instant;
#[test]
#[ignore = "needs LC_PREVIEW_RAW pointing at a real raw file"]
fn a_preview_reads_only_the_header_and_the_preview() {
let Ok(path) = std::env::var("LC_PREVIEW_RAW") else {
eprintln!("skipped: set LC_PREVIEW_RAW=<path to a raw file>");
return;
};
let file_len = std::fs::metadata(&path).expect("the file exists").len();
assert!(file_len > 0, "empty file");
let started = Instant::now();
let window =
lightcraft_engine::files::preview_window(&path).expect("the file reads").expect("it carries an embedded preview the header can point at");
let elapsed = started.elapsed();
let (_, preview, read) = window;
let frac = read.fraction();
println!(
"preview of {path}\n read {} of {} bytes ({:.1} %) in {:.2} s{}",
read.read,
read.total,
frac * 100.0,
elapsed.as_secs_f64(),
if read.header_path { " [header path]" } else { " [whole file]" }
);
assert!(read.header_path, "the header path must answer for a raw whose pointer is in its header");
assert_eq!(read.total, file_len, "the report covers the whole file");
assert!(frac < 0.5, "a preview must not cost most of the file: read {} of {} bytes", read.read, read.total);
assert!(preview.starts_with(&[0xff, 0xd8]), "the preview bytes are a JPEG: {:?}", preview.get(..4));
}
+4 -1
View File
@@ -52,7 +52,10 @@ pub use lightcraft_geom::Orientation;
pub use lightcraft_meta::Metadata;
pub use lightcraft_raster::Rgb32f;
pub use opcodes::{Opcode, OpcodeLists};
pub use preview::{PreviewColorSpace, embedded_preview, embedded_preview_color_space, embedded_preview_dynamic_range_optimized};
pub use preview::{
PREVIEW_PREFIX, PreviewColorSpace, PreviewRange, bounded_preview, embedded_preview, embedded_preview_color_space,
embedded_preview_dynamic_range_optimized, embedded_preview_from_range,
};
pub use semantic::{SemanticMask, semantic_masks};
use lightcraft_color::Xy;
+163
View File
@@ -167,6 +167,63 @@ pub fn embedded_preview(bytes: &[u8]) -> Option<Vec<u8>> {
/// walking the marker segments and the entropy-coded data. `None` when it is not a complete, displayable
/// (baseline or progressive) JPEG, so a stray `D8 FF` in other data is rejected. `*reach` is left at
/// the furthest byte looked at (what the walk cost; see [`scan_for_jpeg`]).
/// Where a preview lives in a file, so a caller with a seekable file can read just that part of it.
///
/// Why it exists: a 56 MB Fuji RAF keeps its JPEG in the first 5.6 MB (measured: DSCF9492.RAF at
/// 5.6 MB), so a grid thumbnail or a loupe preview has no reason to pull the remaining 50 MB —
/// which on a network share is the difference between a moment and half a minute. A caller that has
/// only bytes (`wasm`, tests) keeps using [`embedded_preview`].
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct PreviewRange {
/// Byte offset of the preview in the file.
pub offset: u64,
/// Bytes to read at [`Self::offset`]: the container's declared preview extent, which may carry
/// padding past the preview's end-of-image marker.
pub window_len: u64,
}
/// The preview's range in the file, when the file's header alone locates it.
///
/// `bytes` is the file's first [`PREVIEW_PREFIX`] bytes (more is fine). `None` means "read the file
/// whole and call [`embedded_preview`]": the formats not covered here (a TIFF-based raw's pointer
/// tag can sit in an IFD the prefix cuts short; CR3 scans box by box) and files whose pointer is
/// damaged.
///
/// The caller reads `[range.offset, range.offset + range.window_len)` and passes those bytes to
/// [`embedded_preview_from_range`], which returns exactly what [`embedded_preview`] returns for the
/// whole file.
///
/// Pass a prefix that actually covers the pointer's range (read [`Self::offset`] +
/// [`Self::window_len`] bytes, or start with [`PREVIEW_PREFIX`] and grow when it is not enough):
/// [`crate::vendor::raf::header`] clamps a slice to the buffer it was given, so a prefix that stops
/// inside the declared extent yields a shorter `window_len` than the file really holds.
pub fn bounded_preview(bytes: &[u8]) -> Option<PreviewRange> {
if !bytes.starts_with(b"FUJIFILMCCD-RAW") {
return None;
}
let jpeg = crate::vendor::raf::header(bytes).ok()?.jpeg?;
let offset = (jpeg.as_ptr() as usize).checked_sub(bytes.as_ptr() as usize)?;
if offset.checked_add(jpeg.len())? > bytes.len() {
return None;
}
Some(PreviewRange { offset: offset as u64, window_len: jpeg.len() as u64 })
}
/// The preview inside `window`, which holds the file's bytes at `[range.offset, range.offset +
/// range.window_len)`: the counterpart of [`bounded_preview`]. Padding past the end-of-image marker
/// is dropped, exactly as [`embedded_preview`] drops it.
pub fn embedded_preview_from_range(window: &[u8], range: PreviewRange) -> Option<Vec<u8>> {
let preview = window.get(..usize::try_from(range.window_len).ok()?)?;
if !is_dct_jpeg(preview) {
return None;
}
Some(trim_eoi(preview).to_vec())
}
/// How many bytes of a raw file are enough for [`bounded_preview`]: every covered format keeps its
/// preview pointer in a fixed header, well inside this.
pub const PREVIEW_PREFIX: usize = 4096;
fn jpeg_extent(b: &[u8], soi: usize, reach: &mut usize) -> Option<(usize, u8)> {
let mut i = soi + 2;
let r = walk_jpeg(b, &mut i);
@@ -322,6 +379,112 @@ mod tests {
j
}
/// A minimal RAF: the fixed header (JPEG offset/length at 84/88), the JPEG, then a raw tail the
/// caller must not have to read. `jpeg_len` is written as declared, so a test can claim more than
/// the JPEG holds the way a padded container does.
fn raf_fixture(jpeg: &[u8], gap: usize, preamble: usize, declared_len: Option<usize>) -> Vec<u8> {
let head = 160 + preamble;
let jpeg_off = head;
let mut f = vec![0u8; head];
f[..16].copy_from_slice(b"FUJIFILMCCD-RAW ");
f[84..88].copy_from_slice(&(jpeg_off as u32).to_be_bytes());
f[88..92].copy_from_slice(&(declared_len.unwrap_or(jpeg.len()) as u32).to_be_bytes());
// records directory and raw block are absent (offset 0), which `header` reads as "none"
f.extend_from_slice(jpeg);
f.extend(std::iter::repeat_n(0x33u8, gap));
f
}
/// A RAF's preview pointer sits at a fixed offset, so a header-sized prefix locates it and the
/// caller reads just that range instead of the whole file — the whole point on a network share.
#[test]
fn bounded_preview_locates_a_rafs_jpeg_from_the_header_alone() {
let jpeg = fake_jpeg(5000);
let file = raf_fixture(&jpeg, 40_000, 0, None);
// A prefix that covers the pointer's range: header + JPEG, and nothing of the tail.
let prefix_len = 8192.min(file.len());
let prefix = &file[..prefix_len];
let range = bounded_preview(prefix).expect("a RAF's preview is found from its header");
assert_eq!(range.offset, 160);
assert_eq!(range.window_len, jpeg.len() as u64);
assert!(range.offset + range.window_len <= prefix_len as u64, "the prefix covers the whole range: {range:?} of {prefix_len}");
assert!((range.offset + range.window_len) < file.len() as u64, "the range stops short of the tail: {range:?} of {}", file.len());
// What the caller reads, and what it holds afterwards: identical to the whole-file answer.
let window = prefix.get(range.offset as usize..(range.offset + range.window_len) as usize).expect("the range lies inside the prefix");
let bounded = embedded_preview_from_range(window, range).expect("the range holds the preview");
assert_eq!(Some(bounded.clone()), embedded_preview(&file));
assert!(bounded.ends_with(&[0xff, 0xd9]), "the preview ends at its EOI marker");
// A prefix that stops inside the declared extent is clamped by the container reader. The
// caller then reads a short window; what comes back is the whole JPEG it holds (the EOI
// marker is inside), never bytes past the window.
let short = &file[..2000];
let clipped = bounded_preview(short).expect("a pointer before the cut is still found");
assert!(clipped.window_len < jpeg.len() as u64, "clamped: {clipped:?}");
let window = short.get(clipped.offset as usize..).expect("inside");
let preview = embedded_preview_from_range(window, clipped).expect("the bytes are a JPEG start");
// No end-of-image marker inside the window: the bytes come back as they are, exactly as the
// whole-file path returns an unclosed JPEG. The decoder refuses it later; nothing is invented.
assert_eq!(preview, window, "an unclosed JPEG is handed on unchanged, never padded");
assert!(preview.len() <= window.len(), "never past the window");
}
/// The preview pointer may sit past the prefix (a long maker-note record before it), and a
/// container may declare padding after the JPEG: both stay correct, and the padding is dropped.
#[test]
fn bounded_preview_handles_a_late_pointer_and_padding() {
let jpeg = fake_jpeg(300);
let late = raf_fixture(&jpeg, 0, 9000, None);
let range = bounded_preview(&late).expect("the whole file is always a valid prefix");
assert_eq!(range.offset, (160 + 9000) as u64);
let padded = raf_fixture(&jpeg, 500, 0, Some(jpeg.len() + 500));
let range = bounded_preview(&padded).expect("a padded extent is still located");
assert_eq!(range.window_len, (jpeg.len() + 500) as u64);
let window = padded.get(range.offset as usize..(range.offset + range.window_len) as usize).expect("the window is inside the file");
let preview = embedded_preview_from_range(window, range).expect("the padding is dropped");
assert_eq!(preview, embedded_preview(&padded).expect("the whole-file answer agrees"));
assert!(preview.ends_with(&[0xff, 0xd9]));
}
/// Without a usable header or pointer the caller must read the file whole: the answer is `None`,
/// never a wrong range.
#[test]
fn bounded_preview_declines_what_the_header_cannot_answer() {
let jpeg = fake_jpeg(64);
let file = raf_fixture(&jpeg, 0, 0, None);
assert_eq!(bounded_preview(&jpeg), None, "not a raw at all");
assert_eq!(bounded_preview(b""), None);
if let Some(short) = file.get(..64) {
assert_eq!(bounded_preview(short), None, "a header cut short has not shown the pointer yet");
}
// a pointer that claims more than the file holds: `header` clamps the slice to the buffer,
// so the range never runs past the end (no out-of-bounds read can follow from it)
let mut lying = raf_fixture(&jpeg, 4096, 0, None);
lying[88..92].copy_from_slice(&(u32::MAX / 2).to_be_bytes());
let clamped = bounded_preview(&lying).expect("a pointer inside the file is still usable");
assert!(clamped.offset + clamped.window_len <= lying.len() as u64, "clamped to the file: {clamped:?}");
// a range the window cannot satisfy yields nothing
assert_eq!(embedded_preview_from_range(&jpeg, PreviewRange { offset: 0, window_len: u64::MAX }), None);
// a TIFF-based raw: its pointer tag lives in an IFD this prefix may cut short
let mut tiff = b"II*\0\x08\0\0\0".to_vec();
tiff.extend_from_slice(&[0u8; 64]);
assert_eq!(bounded_preview(&tiff), None);
}
/// A range that does not hold a JPEG yields nothing, rather than a truncated image handed on.
#[test]
fn a_range_that_does_not_hold_a_jpeg_yields_nothing() {
let jpeg = fake_jpeg(200);
let range = PreviewRange { offset: 0, window_len: jpeg.len() as u64 };
assert_eq!(embedded_preview_from_range(&jpeg, range).as_deref(), Some(&jpeg[..]));
assert_eq!(embedded_preview_from_range(&jpeg[..10], range), None, "window shorter than the range claims");
let not_jpeg = vec![0u8; jpeg.len()];
assert_eq!(embedded_preview_from_range(&not_jpeg, range), None, "not a JPEG");
assert_eq!(embedded_preview_from_range(&[], PreviewRange { offset: 0, window_len: 0 }), None, "empty window");
}
#[test]
fn picks_largest_dct_jpeg() {
let small = fake_jpeg(10);
+6 -5
View File
File diff suppressed because one or more lines are too long
+62
View File
@@ -0,0 +1,62 @@
# Reading a raw's preview without reading the raw
A grid thumbnail and a loupe preview of an unedited raw both show the camera's embedded JPEG, not a
decode of the sensor data. That JPEG lives near the *start* of the file, but the rest of the file is
the sensor data — for a 24 MP Fuji RAF, **50 of 56 MB**. Until this change the preview loader called
`std::fs::read` on the whole file and threw the sensor data away, so every preview paid for bytes
nobody looked at.
That is free on a local SSD and expensive on a network share, which is where photographers keep
libraries. Measured on one real file (`DSCF9492.RAF`, 56.5 MB) over a GVFS/SMB share on Wi-Fi:
| | read | time |
|---|---|---|
| whole file | 59 277 632 B (100 %) | ~10 s |
| header + preview | **5 857 257 B (9.9 %)** | **0.96 s** |
## How it works
1. `read_prefix` reads the first `PREVIEW_PREFIX` (4 KiB); every format whose preview is locatable
from the header keeps its pointer well inside that.
2. For Fuji RAF the header's two big-endian `u32`s at **84** (offset) and **88** (length) are the
same fields `lightcraft_raw`'s RAF reader uses. When they declare more than the prefix holds, the
prefix is re-read once at exactly that extent: `lightcraft_raw::bounded_preview` reports the range
its buffer covers (`slice` clamps to `b.len()`), so a short prefix yields a *plausible but
truncated* range — a real 4 KiB prefix reports 3 948 bytes where the file holds 5.6 MB.
3. `embedded_preview_from_range` checks the window really starts with a JPEG SOI and trims it at the
last EOI marker, exactly as `embedded_preview` trims the whole-file answer.
4. When the prefix already covers the range — the usual case, since it was grown to it — the window
is a slice of the prefix. Reading it again would double a 5.6 MB preview.
Formats whose preview pointer is not in the header (CR3 scans boxes; a TIFF pointer tag can sit in
an IFD the prefix cuts short; CRW/MRW/X3F are found by scanning) keep the whole-file path, so
nothing regresses: `preview_window` returns `None` and the loader reads the file as before. A pointer
that lies, a short read or a file that changes underneath us ends in the same fallback — the user's
preview is never lost to a damaged header.
`LIGHTCRAFT_PROFILE=1` prints what each preview cost:
```
[profile] preview …/DSCF9492.RAF: read 5857257 of 59277632 bytes (header 5857257, window 5857109)
```
## Tests
- `crates/raw/src/preview.rs`: the header-only range matches `embedded_preview` byte for byte; a
clamped prefix, a padded extent, a late pointer, a lying pointer and a non-JPEG window are all
covered.
- `crates/engine/src/files.rs`: on a synthetic RAF with a 200 KB tail, the quick path and the
whole-file path produce identical pixels, and the read stays under half the file.
- `crates/engine/tests/preview_read.rs` (opt-in, `#[ignore]`): the same measurement on a **real**
file, typically on a share:
```sh
LC_PREVIEW_RAW="/mnt/nas/…/DSCF0001.RAF" \
cargo test --release -p lightcraft-engine --test preview_read -- --ignored --nocapture
```
No raw fixture is committed (media never is, see `AGENTS.md`).
A network share is still slower than a local disk — the preview itself is 5.6 MB — and the sensor
data still has to come over the wire for the *developed* picture, which is a separate path
(`fs_pair_loader`, the full decode). This change removes the preview's share of it, nothing more.