From ed59d78e837ea806662694340a83508e6f9ccb44 Mon Sep 17 00:00:00 2001 From: Exalted5373 <247056922+Exalted5373@users.noreply.github.com> Date: Thu, 8 Oct 2026 15:04:19 -0700 Subject: [PATCH] LR-MASK-SKY: AI masks use DNG semantic mattes (iPhone ProRAW sky matte) Read DNG 1.6 semantic masks (SemanticName, SemanticInstanceID, MaskSubArea; any coding the tile/strip reader decodes, e.g. ProRAW's lossy-JPEG matte) with lightcraft_raw::semantic_masks, map them onto the developed image, and carry Apple's sky / portrait / skin / hair / teeth / glasses mattes in SourceInfo::mattes (SourceInfo is no longer Copy). Sky, Subject, Background and People components sample them through orientation, warp (Warp::frame, as geometry.rs) and crop like the image, so preview = export, instead of the heuristics; files without mattes render as before. Malformed mask IFDs are skipped, and only the first MAX_MASKS mask IFDs are decoded, so a file full of them costs bounded work. The render cache version is bumped once, in the white-balance commit. The corpus test reads corpus/raw/dng-apple-iphone12pro-proraw.dng, the name `cargo xtask corpus` downloads it under. Co-Authored-By: Claude Opus 5.5 --- ROADMAP.md | 2 +- crates/engine/src/files.rs | 105 +++++++++++++- crates/engine/src/media.rs | 22 +-- crates/gpu/src/render.rs | 4 +- crates/pipeline/src/lib.rs | 14 +- crates/pipeline/src/local.rs | 13 +- crates/pipeline/src/masks.rs | 264 +++++++++++++++++++++++++++++++---- crates/raw/src/lib.rs | 5 +- crates/raw/src/semantic.rs | 136 ++++++++++++++++++ crates/raw/tests/corpus.rs | 27 ++++ crates/raw/tests/dng.rs | 127 +++++++++++++++++ crates/tiff/src/tags.rs | 8 ++ docs/parity.md | 8 +- docs/whats-new.md | 3 + 14 files changed, 686 insertions(+), 52 deletions(-) create mode 100644 crates/raw/src/semantic.rs diff --git a/ROADMAP.md b/ROADMAP.md index 26acc1765..2ad5884ea 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78). | **Feature checklist** | 79.7% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P0: full-resolution zoom detail; P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim | | **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, CR3 lossless Bayer and version 0x100/0x200 C-RAW (M50/R100/R8 pixel-exact corpus), ARW, NEF (uncompressed + Huffman lossless/lossy), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans), uncompressed ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) | | **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF, RW2, RAF, CR3, CR2 and PEF have guarded camera-JPEG colour estimates, with bundled ILCE-7M4, X-H2S and X-T4 profiles (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) | -| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) | +| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics, or the photo's own segmentation mattes when it ships them (DNG semantic masks, e.g. iPhone ProRAW's sky matte) | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) | | **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering | | **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ | | **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export | diff --git a/crates/engine/src/files.rs b/crates/engine/src/files.rs index bad30e149..8e14c68f7 100644 --- a/crates/engine/src/files.rs +++ b/crates/engine/src/files.rs @@ -245,7 +245,8 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result< raw.opcodes.list3.retain(|op| !op.is_lens_correction()); let xy = lightcraft_raw::color::as_shot_white_xy(&raw); let t = lightcraft_raw::color::camera_transform(&raw, xy); - let camera_look = crate::camera_preview::fit_preview(&raw, &bytes, &t); + // the source's segmentation mattes (DNG semantic masks), read while the preview is fitted + let (camera_look, mattes) = rayon::join(|| crate::camera_preview::fit_preview(&raw, &bytes, &t), || dng_mattes(&bytes, &raw)); drop(bytes); // Previews and thumbnails bin the mosaic straight to (about) the size they need; only // larger levels (exports, 1:1) demosaic the whole sensor. @@ -308,7 +309,7 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result< let relative = crate::camera_preview::file_local_look(raw.format) && t.matrix_is_fallback; let camera_tone = camera_look.as_ref().map(|p| p.tone).or_else(|| raw.color.profile.tone_curve.as_ref().and_then(dng_tone_curve)); let (temp, tint) = if relative { (6500.0, 0.0) } else { (temp.round(), tint.round()) }; - return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone })); + return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone, mattes })); } let d = lightcraft_codecs::decode(&bytes, fit_box(max_edge)).map_err(|e| e.to_string())?; drop(bytes); @@ -317,6 +318,42 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result< Ok((img.into_oriented(Orientation::from_exif(d.orientation)), SourceInfo::default())) } +/// The semantic masks of a DNG that AI masks understand, over the developed image (default crop, +/// oriented like it). Person mattes without a confidently selected pixel are left out (the +/// Subject mask then falls back to its heuristic); a sky matte counts even when empty (no sky). +fn dng_mattes(bytes: &[u8], raw: &lightcraft_raw::RawImage) -> Option> { + use lightcraft_pipeline::masks::{MatteKind, Mattes}; + if raw.format != lightcraft_raw::RawFormat::Dng { + return None; + } + let mut mattes = Mattes::default(); + for m in lightcraft_raw::semantic_masks(bytes) { + let Some(kind) = matte_kind(&m.name) else { continue }; + let Some(img) = m.developed(raw.active_area, raw.crop) else { continue }; + if kind != MatteKind::Sky && !img.data.iter().any(|&v| v >= 128) { + continue; + } + mattes.push(kind, img.into_oriented(raw.orientation)); + } + (!mattes.is_empty()).then(|| Arc::new(mattes)) +} + +/// What a semantic mask selects, by its `SemanticName`: Apple's (iPhone ProRAW) are named after +/// their auxiliary image types, `urn:com:apple:photo::aux:`. Other names: `None`. +fn matte_kind(name: &str) -> Option { + use lightcraft_pipeline::masks::MatteKind::*; + let kind = name.strip_prefix("urn:com:apple:photo:")?.rsplit(':').next()?; + Some(match kind { + "semanticskymatte" => Sky, + "semanticskinmatte" => Skin, + "semantichairmatte" => Hair, + "semanticteethmatte" => Teeth, + "semanticglassesmatte" => Glasses, + "portraiteffectsmatte" => Person, + _ => return None, + }) +} + /// A DNG `ProfileToneCurve` (linear in, linear out, 1.0 = white after exposure compensation) as the /// finish stage's camera tone curve: 32 knots, log-spaced over 12 stops below white; above white /// the camera tone's shoulder continues it. @@ -699,6 +736,70 @@ mod tests { assert!((mean(&dim) / mean(&before) - 0.5).abs() < 0.02, "{} vs {}", mean(&dim), mean(&before)); } + /// iPhone ProRAW-style semantic masks drive the Sky mask: Apple's sky matte (here the right + /// half of the sensor, i.e. the bottom of the photo once turned 90° clockwise — where the sky + /// heuristic would never look) is used; a matte with an unknown name is ignored. + #[test] + fn dng_sky_matte_drives_the_sky_mask() { + use lightcraft_develop::{DevelopSettings, LocalAdjustments, Mask, MaskComponent, MaskOp, MaskShape}; + use lightcraft_pipeline::{RenderRequest, render}; + use lightcraft_tiff::tags::{self as t, photometric}; + use lightcraft_tiff::{ByteOrder, IfdBuilder, ImageData, TiffWriter, Value}; + let (w, h) = (16u32, 8u32); + let image = |ifd: &mut IfdBuilder, w: u32, h: u32, cpp: u16, bits: u16, data: Vec| { + ifd.set(t::IMAGE_WIDTH, Value::Long(vec![w])); + ifd.set(t::IMAGE_LENGTH, Value::Long(vec![h])); + ifd.set(t::BITS_PER_SAMPLE, Value::Short(vec![bits; cpp as usize])); + ifd.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![cpp])); + ifd.set(t::COMPRESSION, Value::Short(vec![1])); + ifd.set_image(ImageData::Strips { rows_per_strip: h, strips: vec![data] }); + }; + let mut raw = IfdBuilder::new(); + raw.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![0])); + raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::LINEAR_RAW])); + image(&mut raw, w, h, 3, 16, 12000u16.to_le_bytes().repeat((w * h * 3) as usize)); + let matte = |name: &str, right: bool| { + let mut m = IfdBuilder::new(); + m.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![t::SUBFILE_SEMANTIC_MASK])); + m.set(t::PHOTOMETRIC, Value::Short(vec![photometric::MASK])); + m.set(t::SEMANTIC_NAME, Value::Ascii(name.into())); + image(&mut m, w / 2, h / 2, 1, 8, (0..w / 2 * h / 2).map(|i| if (i % (w / 2) >= w / 4) == right { 255 } else { 0 }).collect()); + m + }; + let mut ifd0 = IfdBuilder::new(); + ifd0.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![1])); + ifd0.set(t::DNG_VERSION, Value::Byte(vec![1, 6, 0, 0])); + ifd0.set(t::MAKE, Value::Ascii("Apple".into())); + ifd0.set(t::ORIENTATION, Value::Short(vec![6])); + ifd0.set(t::COLOR_MATRIX_1, Value::SRational(vec![(1, 1), (0, 1), (0, 1), (0, 1), (1, 1), (0, 1), (0, 1), (0, 1), (1, 1)])); + ifd0.set(t::CALIBRATION_ILLUMINANT_1, Value::Short(vec![21])); + ifd0.set(t::PHOTOMETRIC, Value::Short(vec![photometric::RGB])); + image(&mut ifd0, 4, 2, 3, 8, vec![128; 24]); + ifd0.add_sub_ifd(raw); + ifd0.add_sub_ifd(matte("urn:com:apple:photo:2020:aux:semanticskymatte", true)); + ifd0.add_sub_ifd(matte("urn:com:apple:photo:2020:aux:semanticsomethingelse", false)); + let bytes = TiffWriter::new(ByteOrder::Little, false).write(&[ifd0]).unwrap(); + + let (img, info) = load_bytes(&bytes, 64).unwrap(); + assert_eq!((img.width, img.height), (8, 16)); + assert!(info.mattes.is_some()); + let sky = Mask { + components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: false, shape: MaskShape::Sky }], + adjust: LocalAdjustments { exposure: -3.0, ..Default::default() }, + ..Default::default() + }; + let s = DevelopSettings { masks: vec![sky], ..Default::default() }; + let req = RenderRequest::fit(8, 16); + let green = |s: &DevelopSettings, info: &SourceInfo, y: usize| render(&img, info, s, &req).image.get(4, y)[1] as i32; + let plain = DevelopSettings::default(); + assert_eq!(green(&s, &info, 2), green(&plain, &info, 2), "no sky at the top"); + assert!(green(&s, &info, 13) + 40 < green(&plain, &info, 13), "the sky at the bottom is darkened"); + // without the matte the heuristic looks at the top of the frame instead + let heuristic = SourceInfo { mattes: None, ..info.clone() }; + assert!(green(&s, &heuristic, 2) < green(&plain, &heuristic, 2)); + assert_eq!(green(&s, &heuristic, 13), green(&plain, &heuristic, 13)); + } + /// Apple ProRAW: like Lightroom Classic, the default render ignores the file's /// `ProfileGainTableMap` (Apple's local tone mapping); a map that would double every pixel /// changes nothing. diff --git a/crates/engine/src/media.rs b/crates/engine/src/media.rs index 1ed89e473..56ad62d4a 100644 --- a/crates/engine/src/media.rs +++ b/crates/engine/src/media.rs @@ -143,7 +143,7 @@ impl DecodedSource { /// What to render these pixels against: the decoder's facts, else `header` (the catalog's) /// with any stored camera tone curve. pub fn info_or(&self, header: SourceInfo) -> SourceInfo { - self.info.unwrap_or(SourceInfo { camera_tone: self.camera_tone.or(header.camera_tone), ..header }) + self.info.clone().unwrap_or_else(|| SourceInfo { camera_tone: self.camera_tone.or(header.camera_tone), ..header }) } } @@ -253,8 +253,9 @@ fn rendered_budget(share: usize) -> usize { RENDERED_MEM_BYTES.min(share / 4) } -fn source_bytes(img: &Rgb32f) -> usize { - img.data.len() * 12 + 64 + std::mem::size_of::() +fn source_bytes(s: &DecodedSource) -> usize { + let mattes = s.info.as_ref().and_then(|i| i.mattes.as_ref()).map_or(0, |m| m.bytes()); + s.image.data.len() * 12 + 64 + std::mem::size_of::() + mattes } impl MediaCache { @@ -319,8 +320,8 @@ impl MediaCache { let Some((tick, which)) = candidates.into_iter().flatten().min() else { break }; let freed = match which { 0 => self.thumbs.pop_oldest(), - 1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1.image)), - 2 => self.full.take().map(|e| source_bytes(&e.1.image)), + 1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1)), + 2 => self.full.take().map(|e| source_bytes(&e.1)), _ => self.rendered.evict_oldest(), }; match freed { @@ -372,7 +373,7 @@ impl MediaCache { let tick = lightcraft_preview::next_tick(); match level { SourceLevel::Thumb => { - let cost = source_bytes(&img.image); + let cost = source_bytes(&img); self.thumbs.insert(id, img, cost); } SourceLevel::Preview => { @@ -415,8 +416,8 @@ impl MediaCache { /// Decoded sources held: (thumbnail level, preview level, full size). pub fn usage(&self) -> (crate::memory::Usage, crate::memory::Usage, crate::memory::Usage) { use crate::memory::Usage; - let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1.image)).sum()); - let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1.image))).unwrap_or_default(); + let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1)).sum()); + let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1))).unwrap_or_default(); (Usage::new(self.thumbs.len(), self.thumbs.cost()), previews, full) } @@ -1210,20 +1211,21 @@ mod tests { let mut s = crate::Session::with_demo(); let id = s.active().unwrap(); let mut job = s.render_job(id, 64, 64, false, true).unwrap(); + let loaded_info = info.clone(); job.source = SourceRef::File { path: "synthetic.arw".into(), max_edge: 64, loader: Some(Arc::new(move |_, _| { let mut image = Rgb32f::new(64, 64); image.data.fill([0.1; 3]); - Ok((image, info)) + Ok((image, loaded_info.clone())) })), fallback: None, }; job.info = SourceInfo::default(); // Header facts cannot override decoder facts. job.settings = Arc::new(DevelopSettings::default()); let r = job.clone().run(); - assert_eq!(r.loaded.as_ref().unwrap().info, Some(info)); + assert_eq!(r.loaded.as_ref().unwrap().info.as_ref(), Some(&info)); let expected = lightcraft_pipeline::render(&r.loaded.as_ref().unwrap().image, &info, &job.settings, &job.request); assert_eq!(r.rendered.as_ref().unwrap().image.data, expected.image.data); s.accept(&r); diff --git a/crates/gpu/src/render.rs b/crates/gpu/src/render.rs index 60ffd8065..7e7ac8a3e 100644 --- a/crates/gpu/src/render.rs +++ b/crates/gpu/src/render.rs @@ -717,7 +717,7 @@ pub fn render( Err(m) => { let img = host.lin.get_or_insert_with(|| Arc::new(cx.read_rgb(&lin, w, h))).clone(); let l = host.log_l.get_or_insert_with(|| Arc::new(cx.read_plane(&prep.log_l, w, h))).clone(); - lightcraft_pipeline::masks::evaluate_one(m, &plan.frame, w, h, &img, &l, s.light.exposure as f32) + lightcraft_pipeline::masks::evaluate_one(m, &plan.frame, w, h, &img, &l, s.light.exposure as f32, plan.mattes.as_deref()) } }); lightcraft_pipeline::visualize::apply(&mut image, req.overlay, &plan, overlay_mask.as_ref()); @@ -1018,7 +1018,7 @@ fn masks(cx: &mut Cx<'_>, lin: &Buf, prep: &Prep, plan: &Plan<'_>, host: &mut Ho // no kernel: evaluate on the CPU let img = host.lin.get_or_insert_with(|| Arc::new(cx.read_rgb(lin, w, h))).clone(); let l = host.log_l.get_or_insert_with(|| Arc::new(cx.read_plane(&prep.log_l, w, h))).clone(); - let mut v = lightcraft_pipeline::masks::shape_alpha(&comp.shape, frame, w, h, &img, &l, ev); + let mut v = lightcraft_pipeline::masks::shape_alpha(&comp.shape, frame, w, h, &img, &l, ev, plan.mattes.as_deref()); if comp.invert { v.data.iter_mut().for_each(|x| *x = 1.0 - *x); } diff --git a/crates/pipeline/src/lib.rs b/crates/pipeline/src/lib.rs index 057fb42fa..76e80ed22 100644 --- a/crates/pipeline/src/lib.rs +++ b/crates/pipeline/src/lib.rs @@ -55,7 +55,7 @@ use lightcraft_raster::{Histogram, Plane, Rgb32f, Rgba8, par_rows}; pub use tone::ToneMap; /// Facts about the source the settings are interpreted against. -#[derive(Clone, Copy, Debug, PartialEq)] +#[derive(Clone, Debug, PartialEq)] pub struct SourceInfo { /// Lens corrections embedded in the file (DNG opcodes), relative to the EXIF-oriented source. pub lens: Option, @@ -67,11 +67,13 @@ pub struct SourceInfo { /// No measured camera illuminant: WB adjustments are relative to the camera's rendered look. pub relative_wb: bool, pub camera_tone: Option, + /// Segmentation mattes stored in the file (DNG semantic masks): AI masks use them. + pub mattes: Option>, } impl Default for SourceInfo { fn default() -> Self { - Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None, relative_wb: false, camera_tone: None } + Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None, relative_wb: false, camera_tone: None, mattes: None } } } @@ -332,6 +334,8 @@ pub struct Plan<'a> { /// [`spots::window_for_reads`]); this is the requested window inside the rendered one, which /// the result is cut to. pub keep: Option, + /// The source's segmentation mattes ([`SourceInfo::mattes`]). + pub mattes: Option>, } /// Long edge of the small render a windowed render estimates the whole frame's airlight from. @@ -392,7 +396,7 @@ pub fn plan<'a>(src: &Rgb32f, info: &SourceInfo, s: &'a DevelopSettings, req: &R [d.nr_luminance, d.nr_detail, d.nr_color, d.nr_color_detail, d.nr_color_smoothness].map(f64::to_bits), src_long, )); - Plan { settings, frame, w, h, px_per_long, src_long, geo, lin_key, eyes, fixed_air, keep } + Plan { settings, frame, w, h, px_per_long, src_long, geo, lin_key, eyes, fixed_air, keep, mattes: info.mattes.clone() } } /// Whether the scene-linear stage needs work only the CPU does (defringe, spot removal). @@ -474,7 +478,7 @@ fn render_impl(src: Src<'_>, info: &SourceInfo, s: &DevelopSettings, req: &Rende Some(p) if p.key == lin_key => p, _ => local::Planes { key: lin_key, ..Default::default() }, }; - let mut prep = local::prepare(lin.clone(), s, frame, px_per_long, req.quality, &mut planes); + let mut prep = local::prepare(lin.clone(), s, frame, px_per_long, req.quality, &mut planes, plan.mattes.as_deref()); if let Some(a) = plan.fixed_air { prep.air = a; } @@ -543,7 +547,7 @@ fn overlay_alpha(o: Overlay, plan: &Plan<'_>, prep: &Prepared) -> Option return Some(e.alpha.clone()); } let ev = plan.settings.light.exposure as f32; - Some(masks::evaluate_one(m, &plan.frame, plan.w, plan.h, &prep.img, &prep.log_l, ev)) + Some(masks::evaluate_one(m, &plan.frame, plan.w, plan.h, &prep.img, &prep.log_l, ev, plan.mattes.as_deref())) } /// Convenience: render a before/after pair side by side is up to the UI; this renders "before" diff --git a/crates/pipeline/src/local.rs b/crates/pipeline/src/local.rs index 61ec84820..898e69cfd 100644 --- a/crates/pipeline/src/local.rs +++ b/crates/pipeline/src/local.rs @@ -279,7 +279,16 @@ pub fn plane_sigmas(s: &DevelopSettings, px_per_long: f64, q: Quality) -> PlaneS /// The spatial planes the per-pixel stage needs for `img` (white-balanced, before exposure), /// reusing whatever `planes` already holds for it. Missing planes are computed side by side (each /// one alone scales poorly: the guided filters work on small subsampled grids). -pub(crate) fn prepare(img: Arc, s: &DevelopSettings, frame: &Frame, px_per_long: f64, q: Quality, planes: &mut Planes) -> Prepared { +#[allow(clippy::too_many_arguments)] +pub(crate) fn prepare( + img: Arc, + s: &DevelopSettings, + frame: &Frame, + px_per_long: f64, + q: Quality, + planes: &mut Planes, + mattes: Option<&masks::Mattes>, +) -> Prepared { let log_l = planes.log_l.get_or_insert_with(|| Arc::new(timed("log_l", || img.map(log_lum)))).clone(); let PlaneSigmas { base: base_sigma, clarity: clarity_sigma, texture: texture_sigma, dark: dark_sigma, chroma: chroma_sigma } = plane_sigmas(s, px_per_long, q); @@ -324,7 +333,7 @@ pub(crate) fn prepare(img: Arc, s: &DevelopSettings, frame: &Frame, px_p None => (None, 1.0), }; let ev = s.light.exposure as f32; - let masks = timed("masks", || masks::evaluate(&s.masks, frame, img.width, img.height, &img, &log_l, ev)); + let masks = timed("masks", || masks::evaluate(&s.masks, frame, img.width, img.height, &img, &log_l, ev, mattes)); Prepared { img, log_l, base, clarity_blur, texture_blur, dark, chroma_blur, air, masks, px_per_long } } diff --git a/crates/pipeline/src/masks.rs b/crates/pipeline/src/masks.rs index aae261c5a..4777c8782 100644 --- a/crates/pipeline/src/masks.rs +++ b/crates/pipeline/src/masks.rs @@ -2,14 +2,96 @@ //! //! Shapes are defined in normalized oriented-image coordinates; sizes are in "long-edge units". //! Components combine with Add (max), Subtract (a·(1−c)) and Intersect (a·c). +//! +//! AI shapes (Sky, Subject, Background, People) use the segmentation [`Mattes`] the source carries +//! (DNG semantic masks, e.g. iPhone ProRAW's) when it has a matching one, else a heuristic. use lightcraft_develop::{BrushStroke, LocalAdjustments, Mask, MaskOp, MaskShape, SegMask}; -use lightcraft_geom::Point; -use lightcraft_raster::{Plane, Rgb32f}; +use lightcraft_geom::{Orientation, Point}; +use lightcraft_raster::{Image, Plane, Rgb32f}; use crate::for_rows; use crate::geometry::Frame; +/// What an embedded segmentation matte selects. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum MatteKind { + Sky, + /// A whole person (all people in the photo). + Person, + Skin, + Hair, + Teeth, + Glasses, +} + +/// Segmentation mattes that came with the source: 0..255 alpha planes, each covering the whole +/// (EXIF-oriented) source image at its own resolution. Several mattes of one kind (e.g. one per +/// person) are combined with max. +#[derive(Clone, Default, PartialEq)] +pub struct Mattes { + mattes: Vec<(MatteKind, Image)>, +} + +impl std::fmt::Debug for Mattes { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_list().entries(self.mattes.iter().map(|(k, m)| (k, m.width, m.height))).finish() + } +} + +impl Mattes { + /// Add a matte (empty ones are ignored). + pub fn push(&mut self, kind: MatteKind, matte: Image) { + if matte.width > 0 && matte.height > 0 && matte.data.len() == matte.width * matte.height { + self.mattes.push((kind, matte)); + } + } + + pub fn is_empty(&self) -> bool { + self.mattes.is_empty() + } + + /// Memory held (bytes). + pub fn bytes(&self) -> usize { + self.mattes.iter().map(|(_, m)| m.data.len()).sum() + } + + fn of(&self, kinds: &[MatteKind]) -> Vec<&Image> { + self.mattes.iter().filter(|(k, _)| kinds.contains(k)).map(|(_, m)| m).collect() + } + + /// The mattes a shape uses instead of its heuristic (empty: none fits). + fn for_shape(&self, shape: &MaskShape) -> Vec<&Image> { + use MatteKind::*; + // the whole person: a person matte, else the union of the parts + let person = || { + let p = self.of(&[Person]); + if p.is_empty() { self.of(&[Skin, Hair, Teeth, Glasses]) } else { p } + }; + match shape { + MaskShape::Sky => self.of(&[Sky]), + MaskShape::Subject => person(), + MaskShape::People { parts, .. } => { + let wanted: Vec = parts + .iter() + .filter_map(|p| { + let p = p.to_ascii_lowercase(); + [("skin", Skin), ("hair", Hair), ("teeth", Teeth), ("glasses", Glasses)] + .into_iter() + .find(|(n, _)| p.contains(n)) + .map(|(_, k)| k) + }) + .collect(); + let whole = + parts.is_empty() || parts.iter().any(|p| matches!(p.to_ascii_lowercase().as_str(), "person" | "entire person" | "entireperson")); + // parts without a matte (lips, clothes, …) keep the heuristic + if whole { person() } else { self.of(&wanted) } + } + _ => Vec::new(), + } + } +} + pub struct Evaluated { /// The mask's id ([`Mask::id`]). pub id: u32, @@ -23,22 +105,24 @@ fn smooth(e0: f32, e1: f32, x: f32) -> f32 { t * t * (3.0 - 2.0 * t) } -/// The visible masks with components, evaluated in order. -pub fn evaluate(masks: &[Mask], frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Vec { +/// The visible masks with components, evaluated in order (`mattes`: the source's, see [`Mattes`]). +#[allow(clippy::too_many_arguments)] +pub fn evaluate(masks: &[Mask], frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Vec { masks .iter() .filter(|m| m.visible && !m.components.is_empty()) - .map(|m| Evaluated { id: m.id, alpha: evaluate_one(m, frame, w, h, img, log_l, ev), adjust: m.adjust }) + .map(|m| Evaluated { id: m.id, alpha: evaluate_one(m, frame, w, h, img, log_l, ev, mattes), adjust: m.adjust }) .collect() } /// The alpha plane of mask `m` (whether visible or not): its components combined, inverted and /// scaled by its amount. -pub fn evaluate_one(m: &Mask, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Plane { +#[allow(clippy::too_many_arguments)] +pub fn evaluate_one(m: &Mask, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Plane { let mut alpha = Plane::new(w, h); let mut first = true; for comp in &m.components { - let mut c = shape_alpha(&comp.shape, frame, w, h, img, log_l, ev); + let mut c = shape_alpha(&comp.shape, frame, w, h, img, log_l, ev, mattes); if comp.invert { c.data.iter_mut().for_each(|v| *v = 1.0 - *v); } @@ -88,7 +172,13 @@ fn for_each_pos(frame: &Frame, w: usize, h: usize, out: &mut Plane, f: impl Fn(P }); } -pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Plane { +/// The alpha plane of one component shape. AI shapes use the source's matching `mattes`, if any. +#[allow(clippy::too_many_arguments)] +pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Plane { + let fitting = mattes.map(|m| m.for_shape(shape)).unwrap_or_default(); + if !fitting.is_empty() { + return matte_alpha(&fitting, frame, w, h); + } let mut out = Plane::new(w, h); let to_long = |p: Point| frame.norm_to_long(p); // `img`/`log_l` are before exposure: range masks select on the exposed values. @@ -180,7 +270,7 @@ pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &R smooth_plane(&mut out, 0.015 * frame_px(frame, w)); } MaskShape::Background => { - let mut s = shape_alpha(&MaskShape::Subject, frame, w, h, img, log_l, ev); + let mut s = shape_alpha(&MaskShape::Subject, frame, w, h, img, log_l, ev, mattes); s.data.iter_mut().for_each(|v| *v = 1.0 - *v); out = s; } @@ -252,6 +342,71 @@ fn smooth_plane(p: &mut Plane, sigma: f32) { *p = lightcraft_raster::blur::gaussian(p, sigma.max(0.5)); } +/// The union (max) of `mattes` (0..255, over the EXIF-oriented source) at output resolution, +/// sampled like the image: user orientation, lens / perspective warp, crop and flips; area-averaged +/// first when the output is much smaller than the matte, bilinear otherwise. +fn matte_alpha(mattes: &[&Image], frame: &Frame, w: usize, h: usize) -> Plane { + let mut out = Plane::new(w, h); + let o2t = frame.out_to_oriented(w, h); + let ppl = frame.px_per_long(w).max(1e-9); + for m in mattes { + let oriented = + if frame.orient == Orientation::Normal { std::borrow::Cow::Borrowed(*m) } else { std::borrow::Cow::Owned(m.oriented(frame.orient)) }; + // matte px per output px + let k = oriented.width as f64 / frame.ow * frame.ow.max(frame.oh) / ppl; + let base = box_down(&oriented, if k >= 2.0 { (k.floor() as usize).min(64) } else { 1 }); + let (sx, sy) = ((base.width as f64 / frame.ow) as f32, (base.height as f64 / frame.oh) as f32); + for_rows(&mut out.data, w, |y, row| { + for (x, v) in row.iter_mut().enumerate() { + let s = match &frame.warp { + Some(wp) => { + let Some((_, s)) = wp.frame(&o2t, x, y) else { + continue; + }; + s + } + None => o2t.apply(Point::new(x as f64 + 0.5, y as f64 + 0.5)), + }; + *v = v.max(bilinear(&base, s.x as f32 * sx, s.y as f32 * sy) / 255.0); + } + }); + } + out +} + +/// `m` area-averaged by `f × f` (0..255 values kept); `f == 1` converts only. +fn box_down(m: &Image, f: usize) -> Plane { + let f = f.max(1); + let (bw, bh) = (m.width.div_ceil(f).max(1), m.height.div_ceil(f).max(1)); + let mut out = Plane::new(bw, bh); + for_rows(&mut out.data, bw, |by, row| { + let rows = by * f..((by + 1) * f).min(m.height); + for (bx, v) in row.iter_mut().enumerate() { + let cols = bx * f..((bx + 1) * f).min(m.width); + let (mut sum, mut n) = (0u32, 0u32); + for y in rows.clone() { + for x in cols.clone() { + sum += m.data.get(y * m.width + x).copied().unwrap_or(0) as u32; + n += 1; + } + } + *v = sum as f32 / n.max(1) as f32; + } + }); + out +} + +/// Bilinear sample at continuous pixel coordinates (pixel centres at +0.5), edges extended. +fn bilinear(p: &Plane, x: f32, y: f32) -> f32 { + let (fx, fy) = (x - 0.5, y - 0.5); + let (x0, y0) = (fx.floor(), fy.floor()); + let (tx, ty) = (fx - x0, fy - y0); + let (x0, y0) = (x0 as isize, y0 as isize); + let a = p.get_clamped(x0, y0) + (p.get_clamped(x0 + 1, y0) - p.get_clamped(x0, y0)) * tx; + let b = p.get_clamped(x0, y0 + 1) + (p.get_clamped(x0 + 1, y0 + 1) - p.get_clamped(x0, y0 + 1)) * tx; + a + (b - a) * ty +} + /// A rough display mapping for colour picking (so samples taken on screen match). pub(crate) fn tonemap_for_select(c: [f32; 3]) -> [f32; 3] { c.map(|v| v / (1.0 + v)) @@ -436,7 +591,7 @@ mod tests { let f = frame(100, 50); let img = Rgb32f::new(100, 50); let l = Plane::new(100, 50); - let a = shape_alpha(&MaskShape::Linear { start: Point::new(0.0, 0.0), end: Point::new(1.0, 0.0) }, &f, 100, 50, &img, &l, 0.0); + let a = shape_alpha(&MaskShape::Linear { start: Point::new(0.0, 0.0), end: Point::new(1.0, 0.0) }, &f, 100, 50, &img, &l, 0.0, None); assert!(a.get(1, 25) > 0.99 && a.get(98, 25) < 0.01); assert!((a.get(50, 25) - 0.5).abs() < 0.05); } @@ -452,7 +607,7 @@ mod tests { MaskShape::Prompt { text: "left".into(), seg: Some(seg.clone()), detail: vec![], edge: 0.0 }, ] { for (w, h) in [(40, 20), (400, 200)] { - let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0); + let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None); assert!(a.get(1, h / 2) > 0.99 && a.get(w - 2, h / 2) < 0.01, "{w}×{h}"); if w >= 400 { let soft = (0..w).filter(|x| (0.05..0.95).contains(&a.get(*x, h / 2))).count(); @@ -465,7 +620,7 @@ mod tests { d.crop.geometry.rect = lightcraft_geom::Rect::from_center(Point::new(0.75, 0.5), 0.5, 1.0); let f = Frame::new(40, 20, &d, true); let shape = MaskShape::Prompt { text: "left".into(), seg: Some(seg), detail: vec![], edge: 0.0 }; - let a = shape_alpha(&shape, &f, 20, 20, &Rgb32f::new(20, 20), &Plane::new(20, 20), 0.0); + let a = shape_alpha(&shape, &f, 20, 20, &Rgb32f::new(20, 20), &Plane::new(20, 20), 0.0, None); assert!(a.data.iter().all(|v| *v < 0.05)); } @@ -478,7 +633,7 @@ mod tests { let patch = SegMask::from_logits_in(side, &l, [0.5, 0.0, 1.0, 1.0]); let shape = MaskShape::Object { hint: vec![Point::new(0.6, 0.5)], exclude: vec![], seg: Some(coarse), detail: vec![patch], edge: 0.0 }; let (w, h) = (200, 100); - let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0); + let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None); assert!(a.get(20, 50) < 0.01, "outside the patch: the coarse mask"); assert!(a.get(120, 50) > 0.99, "inside the patch, its selected half"); assert!(a.get(190, 50) < 0.01, "inside the patch, its unselected half"); @@ -494,7 +649,7 @@ mod tests { let shape = MaskShape::Object { hint: vec![Point::new(0.2, 0.5)], exclude: vec![], seg: None, detail: vec![patch; 20_000], edge: 0.0 }; let (w, h) = (64, 32); let t = std::time::Instant::now(); - let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0); + let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None); assert!(a.get(5, 16) > 0.99 && a.get(60, 16) < 0.01); assert!(t.elapsed() < std::time::Duration::from_secs(20), "{:?}", t.elapsed()); } @@ -507,7 +662,7 @@ mod tests { let (w, h) = (400, 40); let soft_px = |edge: f64| { let shape = MaskShape::Prompt { text: "x".into(), seg: Some(seg.clone()), detail: vec![], edge }; - let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0); + let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None); assert!(a.get(2, h / 2) > 0.9 && a.get(w - 3, h / 2) < 0.1, "still the same selection at {edge}"); (0..w).filter(|x| (0.1..0.9).contains(&a.get(*x, h / 2))).count() }; @@ -524,7 +679,7 @@ mod tests { MaskShape::Prompt { text: "sky".into(), seg: None, detail: vec![], edge: 0.0 }, MaskShape::Prompt { text: "sky".into(), seg: Some(SegMask { side: 4, data: "damaged!".into(), rect: None }), detail: vec![], edge: 0.0 }, ] { - assert!(shape_alpha(&shape, &f, 30, 30, &img, &l, 0.0).data.iter().all(|v| *v == 0.0)); + assert!(shape_alpha(&shape, &f, 30, 30, &img, &l, 0.0, None).data.iter().all(|v| *v == 0.0)); } } @@ -534,10 +689,10 @@ mod tests { let img = Rgb32f::new(100, 100); let l = Plane::new(100, 100); let shape = MaskShape::Radial { center: Point::new(0.5, 0.5), rx: 0.2, ry: 0.2, angle: 0.0, feather: 20.0, invert: false }; - let a = shape_alpha(&shape, &f, 100, 100, &img, &l, 0.0); + let a = shape_alpha(&shape, &f, 100, 100, &img, &l, 0.0, None); assert!(a.get(50, 50) > 0.99 && a.get(5, 5) < 0.01); let m = Mask { components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: true, shape }], ..Default::default() }; - let e = evaluate(&[m], &f, 100, 100, &img, &l, 0.0); + let e = evaluate(&[m], &f, 100, 100, &img, &l, 0.0, None); assert!(e[0].alpha.get(50, 50) < 0.01); } @@ -552,7 +707,7 @@ mod tests { components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: false, shape: MaskShape::Brush { strokes: vec![stroke, erase] } }], ..Default::default() }; - let e = evaluate(&[m], &f, 200, 100, &img, &l, 0.0); + let e = evaluate(&[m], &f, 200, 100, &img, &l, 0.0, None); let a = &e[0].alpha; assert!(a.get(40, 50) > 0.9, "{}", a.get(40, 50)); assert!(a.get(100, 50) < 0.05, "erased centre {}", a.get(100, 50)); @@ -574,8 +729,8 @@ mod tests { ..Default::default() }; let comp = |auto| MaskShape::Brush { strokes: vec![stroke(auto)] }; - let plain = shape_alpha(&comp(false), &f, w, h, &img, &l, 0.0); - let auto = shape_alpha(&comp(true), &f, w, h, &img, &l, 0.0); + let plain = shape_alpha(&comp(false), &f, w, h, &img, &l, 0.0, None); + let auto = shape_alpha(&comp(true), &f, w, h, &img, &l, 0.0, None); // without Auto Mask the brush spills over the edge; with it, it stays on the dark side assert!(plain.get(105, 50) > 0.9, "{}", plain.get(105, 50)); assert!(auto.get(105, 50) < 0.05, "spill {}", auto.get(105, 50)); @@ -585,8 +740,8 @@ mod tests { // on a flat area Auto Mask paints like the plain brush let flat = Rgb32f::from_fn(w, h, |_, _| [0.2; 3]); let fl = flat.map(crate::local::log_lum); - let a = shape_alpha(&comp(true), &f, w, h, &flat, &fl, 0.0); - let b = shape_alpha(&comp(false), &f, w, h, &flat, &fl, 0.0); + let a = shape_alpha(&comp(true), &f, w, h, &flat, &fl, 0.0, None); + let b = shape_alpha(&comp(false), &f, w, h, &flat, &fl, 0.0, None); assert!((a.get(94, 50) - b.get(94, 50)).abs() < 0.02, "{} vs {}", a.get(94, 50), b.get(94, 50)); } @@ -609,7 +764,7 @@ mod tests { }; let refined = Mask { refine: 100.0, ..soft.clone() }; let step = |m: &Mask| { - let a = evaluate_one(m, &f, w, h, &img, &log_l, 0.0); + let a = evaluate_one(m, &f, w, h, &img, &log_l, 0.0, None); a.data[40 * w + 63] - a.data[40 * w + 56] }; let (plain, sharp) = (step(&soft), step(&refined)); @@ -640,7 +795,7 @@ mod tests { } let alpha = |refine| { let shape = MaskShape::ColorRange { samples: vec![sample], refine }; - let a = shape_alpha(&shape, &f, w, h, &img, &log_l, 0.0); + let a = shape_alpha(&shape, &f, w, h, &img, &log_l, 0.0, None); (a.get(30, 20), a.get(90, 20)) }; for refine in [0.0, 50.0, 85.0] { @@ -653,4 +808,63 @@ mod tests { let (l, r) = alpha(100.0); assert!(l > 0.95 && r < 0.3, "refine 100: {l} vs {r}"); } + + /// A half-resolution matte selecting the left `frac` of a `w × h` source. + fn left_matte(w: usize, h: usize, frac: f64) -> Image { + Image::from_fn(w / 2, h / 2, |x, _| if (x as f64 + 0.5) < frac * (w / 2) as f64 { 255 } else { 0 }) + } + + /// The Sky mask follows the source's sky matte where it has one (here: the bottom-left of a + /// uniformly bright blue photo, where the heuristic sees no sky, and not its top right, where + /// it does), through user orientation, at any output size; without one it keeps the heuristic. + #[test] + fn sky_uses_the_sources_matte() { + let (w, h) = (200, 100); + let img = Rgb32f::filled(w, h, [0.5, 0.7, 1.4]); + let l = img.map(crate::local::log_lum); + let mut mattes = Mattes::default(); + mattes.push(MatteKind::Sky, left_matte(w, h, 0.5)); + let f = frame(w, h); + let heuristic = shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, None); + let matte = shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, Some(&mattes)); + assert!(heuristic.get(150, 5) > 0.9 && heuristic.get(25, 90) < 0.1, "{} {}", heuristic.get(150, 5), heuristic.get(25, 90)); + assert!(matte.get(150, 5) < 0.01 && matte.get(25, 90) > 0.99, "{} {}", matte.get(150, 5), matte.get(25, 90)); + // mattes of other kinds leave the heuristic alone + let mut hair = Mattes::default(); + hair.push(MatteKind::Hair, left_matte(w, h, 0.5)); + assert_eq!(shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, Some(&hair)), heuristic); + // rotated 90° clockwise, the source's left half is the top half; a preview and a larger + // render agree + let s = lightcraft_develop::DevelopSettings { orientation: Orientation::Rotate90, ..Default::default() }; + let f = Frame::new(w, h, &s, true); + for (ow, oh) in [(100, 200), (25, 50)] { + let small = Rgb32f::filled(ow, oh, [0.5, 0.7, 1.4]); + let a = shape_alpha(&MaskShape::Sky, &f, ow, oh, &small, &small.map(crate::local::log_lum), 0.0, Some(&mattes)); + let at = |x: f64, y: f64| a.get((x * ow as f64) as usize, (y * oh as f64) as usize); + assert!(at(0.5, 0.2) > 0.99 && at(0.5, 0.8) < 0.01, "{ow}×{oh}: {} {}", at(0.5, 0.2), at(0.5, 0.8)); + } + } + + /// Subject, Background and People use person mattes: the whole person is the union of the + /// parts; a People part without a matte keeps the heuristic. + #[test] + fn people_and_subject_use_person_mattes() { + let (w, h) = (200, 100); + let img = Rgb32f::filled(w, h, [0.2; 3]); + let l = img.map(crate::local::log_lum); + let mut mattes = Mattes::default(); + mattes.push(MatteKind::Skin, left_matte(w, h, 0.25)); + let right: Image = Image::from_fn(w / 2, h / 2, |x, _| if x >= 75 { 255 } else { 0 }); + mattes.push(MatteKind::Hair, right); + let f = frame(w, h); + let eval = |shape: &MaskShape| shape_alpha(shape, &f, w, h, &img, &l, 0.0, Some(&mattes)); + let subject = eval(&MaskShape::Subject); + assert!(subject.get(10, 50) > 0.99 && subject.get(190, 50) > 0.99 && subject.get(100, 50) < 0.01); + let background = eval(&MaskShape::Background); + assert!(background.get(100, 50) > 0.99 && background.get(10, 50) < 0.01); + let hair = eval(&MaskShape::People { person: 0, parts: vec!["Hair".into()] }); + assert!(hair.get(190, 50) > 0.99 && hair.get(10, 50) < 0.01); + let lips = MaskShape::People { person: 0, parts: vec!["Lips".into()] }; + assert_eq!(eval(&lips), shape_alpha(&lips, &f, w, h, &img, &l, 0.0, None)); + } } diff --git a/crates/raw/src/lib.rs b/crates/raw/src/lib.rs index b9bc029ad..82f1bdf8d 100644 --- a/crates/raw/src/lib.rs +++ b/crates/raw/src/lib.rs @@ -2,7 +2,8 @@ //! //! - [`probe`] recognises raw containers; [`decode`] turns a file into a [`RawImage`] (sensor data + everything //! needed to render it: CFA, black/white levels, active area, default crop, orientation, DNG colour tags, -//! opcode lists, [`Metadata`]); [`embedded_preview`] returns the largest embedded JPEG (or DNG 1.7 JPEG XL) preview. +//! opcode lists, [`Metadata`]); [`embedded_preview`] returns the largest embedded JPEG (or DNG 1.7 JPEG XL) preview; +//! [`semantic_masks`] reads a DNG's semantic masks (segmentation mattes, e.g. iPhone ProRAW's sky matte). //! - [`RawImage::normalized`] subtracts black, scales white to 1.0 and crops to the active area (applying DNG //! `OpcodeList1`/`OpcodeList2`); [`demosaic`] turns CFA data into camera-RGB [`Rgb32f`]; //! [`RawImage::develop`] does all of it plus `OpcodeList3` and the default crop; [`RawImage::develop_binned`] @@ -36,6 +37,7 @@ pub mod ljpeg; pub mod opcodes; mod preview; pub mod profile; +pub mod semantic; mod tiffraw; mod unpack; mod vendor; @@ -48,6 +50,7 @@ pub use lightcraft_meta::Metadata; pub use lightcraft_raster::Rgb32f; pub use opcodes::{Opcode, OpcodeLists}; pub use preview::{PreviewColorSpace, embedded_preview, embedded_preview_color_space}; +pub use semantic::{SemanticMask, semantic_masks}; use lightcraft_color::Xy; use lightcraft_tiff::{Tiff, TiffError}; diff --git a/crates/raw/src/semantic.rs b/crates/raw/src/semantic.rs new file mode 100644 index 000000000..e9af87c08 --- /dev/null +++ b/crates/raw/src/semantic.rs @@ -0,0 +1,136 @@ +//! DNG semantic masks (DNG 1.6+, chapter "Semantic Masks"): single-channel segmentation mattes +//! stored in their own IFDs (NewSubFileType 10004.H, PhotometricInterpretation 52527) with a +//! `SemanticName`, an optional `SemanticInstanceID` and an optional `MaskSubArea`. iPhone ProRAW +//! stores e.g. a sky matte this way. +//! +//! [`semantic_masks`] reads them separately from [`crate::decode`] (only callers that use them +//! pay for them); malformed or unsupported mask IFDs are skipped. + +use crate::Rect; +use crate::tiffraw::{Packing, read_image}; +use crate::{RawData, Result}; +use lightcraft_raster::Image; +use lightcraft_tiff::Tiff; +use lightcraft_tiff::tags::{self as t, photometric}; + +/// At most this many mask IFDs are read from one file. +pub const MAX_MASKS: usize = 16; +/// Masks with more samples than this are skipped (a mask is a low-resolution matte; this is +/// 8192 × 4096). +pub const MAX_MASK_SAMPLES: usize = 1 << 25; + +/// Where a cropped mask sits in the uncropped mask (DNG `MaskSubArea`). The uncropped mask +/// (`full_width × full_height`) spans the main image's active area. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct MaskSubArea { + pub top: u32, + pub left: u32, + pub full_width: u32, + pub full_height: u32, +} + +/// One semantic mask: `width × height` values, 0 = not part of the attribute, 65535 = fully part. +#[derive(Clone, Debug, PartialEq)] +pub struct SemanticMask { + /// `SemanticName`, e.g. `urn:com:apple:photo:2020:aux:semanticskymatte`. + pub name: String, + pub instance_id: Option, + pub width: usize, + pub height: usize, + /// `None`: the mask covers the whole active area. + pub sub_area: Option, + pub data: Vec, +} + +/// The semantic masks of a DNG, in file order (none for other files). Only the first +/// [`MAX_MASKS`] mask IFDs are looked at, malformed ones included, so a file full of mask IFDs +/// costs at most that many decodes. +pub fn semantic_masks(bytes: &[u8]) -> Vec { + let Ok(tiff) = Tiff::parse(bytes) else { return Vec::new() }; + tiff.all_ifds() + .into_iter() + .filter(|i| i.u16(t::PHOTOMETRIC) == Some(photometric::MASK) || i.u32(t::NEW_SUBFILE_TYPE) == Some(t::SUBFILE_SEMANTIC_MASK)) + .take(MAX_MASKS) + .filter_map(|ifd| read_mask(bytes, &tiff, ifd).ok().flatten()) + .collect() +} + +fn read_mask(bytes: &[u8], tiff: &Tiff, ifd: &lightcraft_tiff::Ifd) -> Result> { + let Some(name) = ifd.string(t::SEMANTIC_NAME).filter(|n| !n.is_empty()) else { return Ok(None) }; + let info = ifd.image()?; + let (w, h) = (info.width as usize, info.height as usize); + if info.samples_per_pixel != 1 || w.checked_mul(h).is_none_or(|n| n > MAX_MASK_SAMPLES) { + return Ok(None); + } + let bits = info.bits() as u32; + let data: Vec = match read_image(bytes, &info, tiff.order, Packing::Msb)? { + RawData::U16(v) if (1..=16).contains(&bits) => { + let max = ((1u32 << bits) - 1) as f32; + if bits == 16 { v } else { v.into_iter().map(|s| (s as f32 / max * 65535.0).round().min(65535.0) as u16).collect() } + } + RawData::F32(v) => v.into_iter().map(|s| if s.is_finite() { (s.clamp(0.0, 1.0) * 65535.0).round() as u16 } else { 0 }).collect(), + RawData::U16(_) => return Ok(None), + }; + if data.len() != w * h { + return Ok(None); + } + // the tag is ignored when the stored mask does not fit inside the uncropped one + let sub_area = match ifd.u64s(t::MASK_SUB_AREA).as_deref() { + Some(&[top, left, fw, fh]) => { + let fits = fw > 0 && fh > 0 && left.saturating_add(w as u64) <= fw && top.saturating_add(h as u64) <= fh && fw.max(fh) <= u32::MAX as u64; + fits.then_some(MaskSubArea { top: top as u32, left: left as u32, full_width: fw as u32, full_height: fh as u32 }) + } + _ => None, + }; + Ok(Some(SemanticMask { name, instance_id: ifd.string(t::SEMANTIC_INSTANCE_ID).filter(|s| !s.is_empty()), width: w, height: h, sub_area, data })) +} + +impl SemanticMask { + /// The mask value (0..1) at a continuous position of the uncropped mask (pixel centres at + /// +0.5); 0 outside the stored (cropped) mask. + pub fn value_at(&self, x: f64, y: f64) -> f32 { + let (left, top) = self.sub_area.map_or((0.0, 0.0), |a| (a.left as f64, a.top as f64)); + let (x, y) = (x - left, y - top); + if self.width == 0 || self.height == 0 || !(x >= 0.0 && y >= 0.0 && x <= self.width as f64 && y <= self.height as f64) { + return 0.0; + } + let (fx, fy) = ((x - 0.5).max(0.0), (y - 0.5).max(0.0)); + let (x0, y0) = (fx.floor() as usize, fy.floor() as usize); + let (tx, ty) = ((fx - x0 as f64) as f32, (fy - y0 as f64) as f32); + let at = |x: usize, y: usize| { + let (x, y) = (x.min(self.width - 1), y.min(self.height - 1)); + self.data.get(y * self.width + x).copied().unwrap_or(0) as f32 / 65535.0 + }; + let top = at(x0, y0) + (at(x0 + 1, y0) - at(x0, y0)) * tx; + let bottom = at(x0, y0 + 1) + (at(x0 + 1, y0 + 1) - at(x0, y0 + 1)) * tx; + top + (bottom - top) * ty + } + + /// The mask over the developed image ([`crate::RawImage::develop`]: the default `crop` of the + /// `active_area`, before orientation), at the mask's own resolution, as 0..255 alpha. `None` + /// for an empty crop. + pub fn developed(&self, active_area: Rect, crop: Rect) -> Option> { + let c = crop.clipped(active_area.width, active_area.height); + if c.width == 0 || c.height == 0 || self.width == 0 || self.height == 0 { + return None; + } + let (fw, fh) = self.sub_area.map_or((self.width, self.height), |a| (a.full_width as usize, a.full_height as usize)); + // uncropped-mask px per active-area px + let (sx, sy) = (fw as f64 / active_area.width as f64, fh as f64 / active_area.height as f64); + let ow = ((c.width as f64 * sx).round() as usize).clamp(1, self.width.max(fw)); + let oh = ((c.height as f64 * sy).round() as usize).clamp(1, self.height.max(fh)); + if ow.checked_mul(oh).is_none_or(|n| n > MAX_MASK_SAMPLES) { + return None; + } + let (kx, ky) = (c.width as f64 / ow as f64, c.height as f64 / oh as f64); + let mut out = Image::new(ow, oh); + lightcraft_raster::par_rows(&mut out.data, ow, |y, row| { + let ay = c.y as f64 + (y as f64 + 0.5) * ky; + for (x, v) in row.iter_mut().enumerate() { + let ax = c.x as f64 + (x as f64 + 0.5) * kx; + *v = (self.value_at(ax * sx, ay * sy) * 255.0).round() as u8; + } + }); + Some(out) + } +} diff --git a/crates/raw/tests/corpus.rs b/crates/raw/tests/corpus.rs index 683381c40..53193f1f1 100644 --- a/crates/raw/tests/corpus.rs +++ b/crates/raw/tests/corpus.rs @@ -300,6 +300,33 @@ fn corpus_dngs_carry_profile_looks() { eprintln!("{seen} DNGs with profile looks checked"); } +/// iPhone ProRAW (`dng-apple-iphone12pro-proraw.dng`, iPhone 12 Pro, CC0 from raw.pixls.us, see +/// `cargo xtask corpus`) carries a lossy-JPEG sky matte at half resolution as a DNG semantic mask: +/// it decodes, and it is a sky. The matte is stored like the raw (landscape; the photo is shown +/// rotated 90° clockwise), so the sky is on its left and the lake and shore on its right. +#[test] +fn corpus_proraw_sky_matte() { + let path = corpus_root().join("raw/dng-apple-iphone12pro-proraw.dng"); + let Ok(bytes) = std::fs::read(&path) else { + eprintln!("skip: {} absent", path.display()); + return; + }; + let t0 = Instant::now(); + let masks = lightcraft_raw::semantic_masks(&bytes); + eprintln!("semantic masks read in {:.1} ms", t0.elapsed().as_secs_f64() * 1e3); + let names: Vec<&str> = masks.iter().map(|m| m.name.as_str()).collect(); + assert_eq!(names, ["urn:com:apple:photo:2020:aux:semanticskymatte"]); + let m = &masks[0]; + assert_eq!((m.width, m.height, m.sub_area), (2016, 1512, None)); + let cols = |x0: usize, x1: usize| { + let sum: f64 = (0..m.height).flat_map(|y| m.data[y * m.width + x0..y * m.width + x1].iter()).map(|&v| v as f64 / 65535.0).sum(); + sum / ((x1 - x0) * m.height) as f64 + }; + let (left, right) = (cols(0, 100), cols(1916, 2016)); + eprintln!("sky matte: left columns {left:.3}, right columns {right:.3}"); + assert!(left > 0.5 && right < 0.02, "left {left}, right {right}"); +} + /// Issue #148: Sony ARWs from before ~2017 carry no plain white-balance, black-level or crop tags in the raw IFD. /// White balance comes from the maker note's enciphered `Tag2010`, the black level from the encrypted `SR2SubIFD` /// and the crop from `FullImageSize`; without them the RX100 III opened bright green. Expected values: the black diff --git a/crates/raw/tests/dng.rs b/crates/raw/tests/dng.rs index 9cbbb0ea3..02d4f82a3 100644 --- a/crates/raw/tests/dng.rs +++ b/crates/raw/tests/dng.rs @@ -824,3 +824,130 @@ fn gain_table_map_byte_orders() { assert_eq!(img.color.profile.gain_table_map.map(|m| m.gains), read.then(|| vec![1.0, 2.0]), "{data:?} payload, {file:?} file"); } } + +/// A semantic mask IFD (8-bit, uncompressed) named `name`. +fn mask_ifd(w: usize, h: usize, name: &str, px: Vec) -> IfdBuilder { + let mut m = IfdBuilder::new(); + m.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![t::SUBFILE_SEMANTIC_MASK])); + m.set(t::IMAGE_WIDTH, Value::Long(vec![w as u32])); + m.set(t::IMAGE_LENGTH, Value::Long(vec![h as u32])); + m.set(t::BITS_PER_SAMPLE, Value::Short(vec![8])); + m.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![1])); + m.set(t::PHOTOMETRIC, Value::Short(vec![photometric::MASK])); + m.set(t::COMPRESSION, Value::Short(vec![1])); + if !name.is_empty() { + m.set(t::SEMANTIC_NAME, Value::Ascii(name.into())); + } + m.set_image(ImageData::Strips { rows_per_strip: h as u32, strips: vec![px] }); + m +} + +/// Semantic masks: a cropped mask is placed by its `MaskSubArea` (zero outside it), mapped onto the +/// developed image (default crop of the active area), and doesn't disturb the raw decode. +#[test] +fn semantic_masks_are_read_and_placed() { + use lightcraft_raw::semantic::MaskSubArea; + for order in [ByteOrder::Little, ByteOrder::Big] { + let bytes = profile_dng(order, |ifd0| { + // a 4 × 2 crop at (top 1, left 2) of an 8 × 4 mask over the 8 × 6 image + let mut sky = mask_ifd(4, 2, "sky", vec![10, 20, 30, 40, 50, 60, 70, 255]); + sky.set(t::MASK_SUB_AREA, Value::Long(vec![1, 2, 8, 4])); + ifd0.add_sub_ifd(sky); + let mut skin = mask_ifd(2, 2, "skin", vec![0, 255, 255, 0]); + skin.set(t::SEMANTIC_INSTANCE_ID, Value::Ascii("person_a".into())); + ifd0.add_sub_ifd(skin); + }); + let raw = decode(&bytes).unwrap(); + assert_eq!((raw.width, raw.height, raw.cpp), (8, 6, 3), "the masks are not the raw image"); + let masks = lightcraft_raw::semantic_masks(&bytes); + assert_eq!(masks.len(), 2); + let (sky, skin) = (&masks[0], &masks[1]); + assert_eq!((sky.name.as_str(), sky.instance_id.as_deref(), sky.width, sky.height), ("sky", None, 4, 2)); + assert_eq!(sky.sub_area, Some(MaskSubArea { top: 1, left: 2, full_width: 8, full_height: 4 })); + assert_eq!((sky.data[0], sky.data[7]), (10 * 257, 65535)); + assert_eq!((skin.name.as_str(), skin.instance_id.as_deref(), skin.sub_area), ("skin", Some("person_a"), None)); + // positions are in uncropped-mask pixels: zero outside the stored crop + assert!((sky.value_at(2.5, 1.5) - 10.0 / 255.0).abs() < 1e-4); + assert!((sky.value_at(5.5, 2.5) - 1.0).abs() < 1e-4); + assert_eq!((sky.value_at(1.5, 1.5), sky.value_at(2.5, 0.5), sky.value_at(6.5, 1.5)), (0.0, 0.0, 0.0)); + // over the whole image: the uncropped mask's 8 × 4 grid + let full = sky.developed(Rect::new(0, 0, 8, 6), Rect::new(0, 0, 8, 6)).unwrap(); + assert_eq!((full.width, full.height), (8, 4)); + assert_eq!((full.get(2, 1), full.get(5, 2), full.get(1, 1), full.get(6, 2), full.get(2, 0)), (10, 255, 0, 0, 0)); + // a default crop keeping columns 2..6 of the image: the stored crop fills it + let cropped = sky.developed(Rect::new(0, 0, 8, 6), Rect::new(2, 0, 4, 6)).unwrap(); + assert_eq!((cropped.width, cropped.height), (4, 4)); + assert_eq!((cropped.get(0, 1), cropped.get(3, 2), cropped.get(0, 0)), (10, 255, 0)); + } +} + +/// Malformed semantic masks are skipped (never a panic), and so is the `MaskSubArea` of a mask it +/// doesn't fit; the raw image still decodes. +#[test] +fn malformed_semantic_masks_are_skipped() { + let bytes = profile_dng(ByteOrder::Little, |ifd0| { + ifd0.add_sub_ifd(mask_ifd(2, 2, "", vec![1, 2, 3, 4])); // no SemanticName + let mut rgb = mask_ifd(2, 2, "rgb", vec![0; 12]); + rgb.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![3])); + rgb.set(t::BITS_PER_SAMPLE, Value::Short(vec![8, 8, 8])); + ifd0.add_sub_ifd(rgb); + let mut huge = mask_ifd(2, 2, "huge", vec![0; 4]); + huge.set(t::IMAGE_WIDTH, Value::Long(vec![100_000])); + huge.set(t::IMAGE_LENGTH, Value::Long(vec![100_000])); + ifd0.add_sub_ifd(huge); + let mut tiny = mask_ifd(2, 2, "tiny data", vec![0; 4]); + tiny.set(t::IMAGE_WIDTH, Value::Long(vec![4000])); + tiny.set(t::IMAGE_LENGTH, Value::Long(vec![3000])); + ifd0.add_sub_ifd(tiny); + let mut jxl = mask_ifd(2, 2, "unknown coding", vec![0; 4]); + jxl.set(t::COMPRESSION, Value::Short(vec![60_000])); + ifd0.add_sub_ifd(jxl); + let mut zero = mask_ifd(2, 2, "zero size", vec![0; 4]); + zero.set(t::IMAGE_WIDTH, Value::Long(vec![0])); + ifd0.add_sub_ifd(zero); + let mut short = mask_ifd(4, 4, "short strip", vec![200; 3]); + short.set(t::MASK_SUB_AREA, Value::Long(vec![u32::MAX, u32::MAX, u32::MAX, u32::MAX])); + ifd0.add_sub_ifd(short); + let mut outside = mask_ifd(2, 2, "sub area too small", vec![255; 4]); + outside.set(t::MASK_SUB_AREA, Value::Long(vec![3, 0, 2, 4])); + ifd0.add_sub_ifd(outside); + }); + assert_eq!(decode(&bytes).unwrap().cpp, 3); + let masks = lightcraft_raw::semantic_masks(&bytes); + let names: Vec<&str> = masks.iter().map(|m| m.name.as_str()).collect(); + assert_eq!(names, ["short strip", "sub area too small"]); + assert!(masks.iter().all(|m| m.sub_area.is_none()), "sub areas the mask doesn't fit are ignored"); + for m in &masks { + let d = m.developed(Rect::new(0, 0, 8, 6), Rect::new(0, 0, 8, 6)).unwrap(); + assert_eq!((d.width, d.height), (m.width, m.height)); + assert!(m.developed(Rect::new(0, 0, 8, 6), Rect::new(8, 6, 0, 0)).is_none()); + } + // not a TIFF at all + assert!(lightcraft_raw::semantic_masks(b"not a dng").is_empty()); +} + +/// Only the first `MAX_MASKS` mask IFDs are decoded, malformed ones included, so a file full of +/// mask IFDs costs a bounded number of decodes: here the first `MAX_MASKS` can't be decoded, and +/// the valid masks after them are never looked at. +#[test] +fn semantic_mask_ifds_past_the_cap_are_not_decoded() { + use lightcraft_raw::semantic::MAX_MASKS; + let with = |undecodable: usize, valid: usize| { + profile_dng(ByteOrder::Little, |ifd0| { + for i in 0..undecodable { + let mut m = mask_ifd(2, 2, &format!("undecodable {i}"), vec![0; 4]); + m.set(t::COMPRESSION, Value::Short(vec![60_000])); + ifd0.add_sub_ifd(m); + } + for i in 0..valid { + ifd0.add_sub_ifd(mask_ifd(2, 2, &format!("valid {i}"), vec![255; 4])); + } + }) + }; + let bytes = with(MAX_MASKS, 3 * MAX_MASKS); + assert_eq!(decode(&bytes).unwrap().cpp, 3); + assert!(lightcraft_raw::semantic_masks(&bytes).is_empty()); + // one fewer undecodable mask leaves room for exactly one valid one + let names: Vec = lightcraft_raw::semantic_masks(&with(MAX_MASKS - 1, 3 * MAX_MASKS)).into_iter().map(|m| m.name).collect(); + assert_eq!(names, ["valid 0"]); +} diff --git a/crates/tiff/src/tags.rs b/crates/tiff/src/tags.rs index acdc76697..5771316a5 100644 --- a/crates/tiff/src/tags.rs +++ b/crates/tiff/src/tags.rs @@ -180,6 +180,12 @@ pub const FORWARD_MATRIX_3: u16 = 52532; pub const ILLUMINANT_DATA_1: u16 = 52533; pub const ILLUMINANT_DATA_2: u16 = 52534; pub const ILLUMINANT_DATA_3: u16 = 52535; +// DNG 1.6 semantic masks (spec chapter 4, "Semantic Masks") +pub const SEMANTIC_NAME: u16 = 52526; +pub const SEMANTIC_INSTANCE_ID: u16 = 52528; +pub const MASK_SUB_AREA: u16 = 52536; +/// NewSubFileType of a semantic mask IFD. +pub const SUBFILE_SEMANTIC_MASK: u32 = 0x10004; pub const PROFILE_GAIN_TABLE_MAP_2: u16 = 52544; // --- Compression codes (TIFF 6.0, DNG, and widely used registrations) @@ -208,4 +214,6 @@ pub mod photometric { pub const YCBCR: u16 = 6; pub const CFA: u16 = 32803; pub const LINEAR_RAW: u16 = 34892; + /// DNG 1.6 semantic mask. + pub const MASK: u16 = 52527; } diff --git a/docs/parity.md b/docs/parity.md index 628cd8629..8e6c6f5a1 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -341,11 +341,11 @@ under the UI command that wraps it (a no-op error); the UI command now wins. | Id | Feature | Tier | Status | Evidence | Missing / notes | |---|---|---|---|---|---| | LR-MASK-PANEL | Masks panel | P0 | ✅ | `cmd:panel.masking`, `cmd:mask.add`, `cmd:mask.select`, `cmd:mask.rename`, `cmd:mask.duplicate` (`invert`), `cmd:mask.move`, `cmd:mask.visible`, `cmd:mask.delete`, `crates/ui-egui/src/panels/masking.rs` | list with per-mask show/hide eye, double-click rename, right-click menu (duplicate, duplicate and invert, invert, hide, move up/down, rename, delete); fits any panel width (create tiles go to three columns, mask actions wrap, long component names are cut short); no drag-to-reorder | -| LR-MASK-SUBJECT | Select subject | P2 | 🟡 | `cmd:mask.add` (`subject`), `crates/pipeline/src/masks.rs` | saliency heuristic, no segmentation model | -| LR-MASK-SKY | Select sky | P2 | 🟡 | `cmd:mask.add` (`sky`) | heuristic | -| LR-MASK-BACKGROUND | Select background | P2 | 🟡 | `cmd:mask.add` (`background`) | inverse of the subject heuristic | +| LR-MASK-SUBJECT | Select subject | P2 | 🟡 | `cmd:mask.add` (`subject`), `crates/pipeline/src/masks.rs` | saliency heuristic, no segmentation model; a photo that carries person mattes (DNG semantic masks: iPhone ProRAW portrait / skin / hair / teeth / glasses mattes) uses them instead (synthetic files only: no corpus file carries person mattes) | +| LR-MASK-SKY | Select sky | P2 | 🟡 | `cmd:mask.add` (`sky`), `crates/pipeline/src/masks.rs` (`Mattes`), `crates/raw/src/semantic.rs` | heuristic; a photo that carries a sky matte (DNG semantic mask, e.g. iPhone ProRAW) uses it instead, through orientation / crop / lens corrections, preview = export (checked on one CC0 iPhone 12 Pro ProRAW and synthetic files) | +| LR-MASK-BACKGROUND | Select background | P2 | 🟡 | `cmd:mask.add` (`background`) | inverse of the subject mask (heuristic, or the photo's person mattes) | | LR-MASK-OBJECTS | Object selection | P2 | 🟡 | `cmd:mask.add` (`object`, `prompt`), `cmd:mask.objectPoint`, `cmd:mask.refineDetail`, `cmd:segment.prepare`, `cmd:segment.model.status`, `cmd:segment.model.download`, `cmd:segment.model.cancel`, `crates/segment`, `crates/engine/src/segment/mod.rs` | SAM 3 in pure Rust (candle; Metal on macOS, CPU elsewhere): Object tile → click to include, ⌥-click to leave out; Describe tile → a text prompt selects every instance ("sky", "the red car"); both also as Add/Subtract/Intersect components; + / − under the selected mask; comma lists (`car, road`); a zoomed-in detail pass for 5–10× finer edges on small objects; per-selection Edge (hard ↔ soft). The model runs on its own worker thread (the UI never waits; panics become errors), is unloaded after 10 min idle, and is optional: the segmentation is stored with the mask (288² logits), so renders and exports never need it. Weights are never bundled (SAM License): without them the app offers a consented background download (mirrors, resume, timeouts, SHA-256) — **but no default download location is configured yet** (users set `LIGHTCRAFT_SAM3_MIRRORS` or install by hand: the dialog names the three files, shows the model folder and links docs/ai-masks.md). Not verified against Lightroom's Select Object; no brush/box object mode; the first click on a photo waits for its analysis (~4 s on an M4 Pro, much longer on CPU) | -| LR-MASK-PEOPLE | People parts | P2 | ⬜ | | | +| LR-MASK-PEOPLE | People parts | P2 | ⬜ | | shape exists (no UI): uses the photo's person / skin / hair / teeth / glasses mattes when it carries them (DNG semantic masks), else the subject heuristic | | LR-MASK-LANDSCAPE | Landscape classes | P2 | ⬜ | | shape exists, evaluates empty | | LR-MASK-BRUSH | Brush mask | P0 | ✅ | `cmd:tool.brush`, `cmd:mask.brushStroke` (`autoMask`), `crates/pipeline/src/masks.rs` | size/feather/flow/density/erase; Auto Mask: dabs weighted by similarity to the colour under the dab centre, refined by a guided filter on luminance (CPU + GPU); no A/B brushes, no pressure | | LR-MASK-LINEAR | Linear gradient | P0 | ✅ | `cmd:tool.linear`, `cmd:mask.update` | | diff --git a/docs/whats-new.md b/docs/whats-new.md index f6361b048..ca5758bf4 100644 --- a/docs/whats-new.md +++ b/docs/whats-new.md @@ -22,6 +22,9 @@ embedded JPEG. - Apple ProRAW's gain table map (its local tone mapping, `ProfileGainTableMap`) is now read and kept when a photo is exported or converted to DNG. It is not applied to the render: Lightroom Classic renders ProRAW without it. +- iPhone ProRAW and other DNGs that carry their own segmentation mattes (DNG semantic masks) use them for the Select + Sky, Subject and Background masks instead of our heuristics, so the sky is selected where the camera found it + (checked on one CC0 iPhone 12 Pro ProRAW). Photos without mattes are unchanged. - Panasonic and Leica raws (RW2, RWL) are now corrected for lens distortion the way the camera corrects its own JPEG (issue #256): the correction the camera records in the file is applied under Lens Corrections ▸ Enable Profile Corrections, on by default for newly imported photos, with the same framing as the camera's JPEG. At 12 mm the