mirror of
https://github.com/storytold/lightcraft.git
synced 2026-11-02 14:39:41 -06:00
LR-MASK-SKY: AI masks use DNG semantic mattes (iPhone ProRAW sky matte)
Read DNG 1.6 semantic masks (SemanticName, SemanticInstanceID, MaskSubArea; any coding the tile/strip reader decodes, e.g. ProRAW's lossy-JPEG matte) with lightcraft_raw::semantic_masks, map them onto the developed image, and carry Apple's sky / portrait / skin / hair / teeth / glasses mattes in SourceInfo::mattes (SourceInfo is no longer Copy). Sky, Subject, Background and People components sample them through orientation, warp (Warp::frame, as geometry.rs) and crop like the image, so preview = export, instead of the heuristics; files without mattes render as before. Malformed mask IFDs are skipped, and only the first MAX_MASKS mask IFDs are decoded, so a file full of them costs bounded work. The render cache version is bumped once, in the white-balance commit. The corpus test reads corpus/raw/dng-apple-iphone12pro-proraw.dng, the name `cargo xtask corpus` downloads it under. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Exalted5373
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
a28dc4dbdf
commit
ed59d78e83
+1
-1
@@ -25,7 +25,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|
||||
| **Feature checklist** | 79.7% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P0: full-resolution zoom detail; P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
|
||||
| **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, CR3 lossless Bayer and version 0x100/0x200 C-RAW (M50/R100/R8 pixel-exact corpus), ARW, NEF (uncompressed + Huffman lossless/lossy), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans), uncompressed ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) |
|
||||
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF, RW2, RAF, CR3, CR2 and PEF have guarded camera-JPEG colour estimates, with bundled ILCE-7M4, X-H2S and X-T4 profiles (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
|
||||
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
|
||||
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics, or the photo's own segmentation mattes when it ships them (DNG semantic masks, e.g. iPhone ProRAW's sky matte) | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
|
||||
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
|
||||
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
|
||||
| **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export |
|
||||
|
||||
+103
-2
@@ -245,7 +245,8 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
|
||||
raw.opcodes.list3.retain(|op| !op.is_lens_correction());
|
||||
let xy = lightcraft_raw::color::as_shot_white_xy(&raw);
|
||||
let t = lightcraft_raw::color::camera_transform(&raw, xy);
|
||||
let camera_look = crate::camera_preview::fit_preview(&raw, &bytes, &t);
|
||||
// the source's segmentation mattes (DNG semantic masks), read while the preview is fitted
|
||||
let (camera_look, mattes) = rayon::join(|| crate::camera_preview::fit_preview(&raw, &bytes, &t), || dng_mattes(&bytes, &raw));
|
||||
drop(bytes);
|
||||
// Previews and thumbnails bin the mosaic straight to (about) the size they need; only
|
||||
// larger levels (exports, 1:1) demosaic the whole sensor.
|
||||
@@ -308,7 +309,7 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
|
||||
let relative = crate::camera_preview::file_local_look(raw.format) && t.matrix_is_fallback;
|
||||
let camera_tone = camera_look.as_ref().map(|p| p.tone).or_else(|| raw.color.profile.tone_curve.as_ref().and_then(dng_tone_curve));
|
||||
let (temp, tint) = if relative { (6500.0, 0.0) } else { (temp.round(), tint.round()) };
|
||||
return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone }));
|
||||
return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone, mattes }));
|
||||
}
|
||||
let d = lightcraft_codecs::decode(&bytes, fit_box(max_edge)).map_err(|e| e.to_string())?;
|
||||
drop(bytes);
|
||||
@@ -317,6 +318,42 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
|
||||
Ok((img.into_oriented(Orientation::from_exif(d.orientation)), SourceInfo::default()))
|
||||
}
|
||||
|
||||
/// The semantic masks of a DNG that AI masks understand, over the developed image (default crop,
|
||||
/// oriented like it). Person mattes without a confidently selected pixel are left out (the
|
||||
/// Subject mask then falls back to its heuristic); a sky matte counts even when empty (no sky).
|
||||
fn dng_mattes(bytes: &[u8], raw: &lightcraft_raw::RawImage) -> Option<Arc<lightcraft_pipeline::masks::Mattes>> {
|
||||
use lightcraft_pipeline::masks::{MatteKind, Mattes};
|
||||
if raw.format != lightcraft_raw::RawFormat::Dng {
|
||||
return None;
|
||||
}
|
||||
let mut mattes = Mattes::default();
|
||||
for m in lightcraft_raw::semantic_masks(bytes) {
|
||||
let Some(kind) = matte_kind(&m.name) else { continue };
|
||||
let Some(img) = m.developed(raw.active_area, raw.crop) else { continue };
|
||||
if kind != MatteKind::Sky && !img.data.iter().any(|&v| v >= 128) {
|
||||
continue;
|
||||
}
|
||||
mattes.push(kind, img.into_oriented(raw.orientation));
|
||||
}
|
||||
(!mattes.is_empty()).then(|| Arc::new(mattes))
|
||||
}
|
||||
|
||||
/// What a semantic mask selects, by its `SemanticName`: Apple's (iPhone ProRAW) are named after
|
||||
/// their auxiliary image types, `urn:com:apple:photo:<year>:aux:<type>`. Other names: `None`.
|
||||
fn matte_kind(name: &str) -> Option<lightcraft_pipeline::masks::MatteKind> {
|
||||
use lightcraft_pipeline::masks::MatteKind::*;
|
||||
let kind = name.strip_prefix("urn:com:apple:photo:")?.rsplit(':').next()?;
|
||||
Some(match kind {
|
||||
"semanticskymatte" => Sky,
|
||||
"semanticskinmatte" => Skin,
|
||||
"semantichairmatte" => Hair,
|
||||
"semanticteethmatte" => Teeth,
|
||||
"semanticglassesmatte" => Glasses,
|
||||
"portraiteffectsmatte" => Person,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
/// A DNG `ProfileToneCurve` (linear in, linear out, 1.0 = white after exposure compensation) as the
|
||||
/// finish stage's camera tone curve: 32 knots, log-spaced over 12 stops below white; above white
|
||||
/// the camera tone's shoulder continues it.
|
||||
@@ -699,6 +736,70 @@ mod tests {
|
||||
assert!((mean(&dim) / mean(&before) - 0.5).abs() < 0.02, "{} vs {}", mean(&dim), mean(&before));
|
||||
}
|
||||
|
||||
/// iPhone ProRAW-style semantic masks drive the Sky mask: Apple's sky matte (here the right
|
||||
/// half of the sensor, i.e. the bottom of the photo once turned 90° clockwise — where the sky
|
||||
/// heuristic would never look) is used; a matte with an unknown name is ignored.
|
||||
#[test]
|
||||
fn dng_sky_matte_drives_the_sky_mask() {
|
||||
use lightcraft_develop::{DevelopSettings, LocalAdjustments, Mask, MaskComponent, MaskOp, MaskShape};
|
||||
use lightcraft_pipeline::{RenderRequest, render};
|
||||
use lightcraft_tiff::tags::{self as t, photometric};
|
||||
use lightcraft_tiff::{ByteOrder, IfdBuilder, ImageData, TiffWriter, Value};
|
||||
let (w, h) = (16u32, 8u32);
|
||||
let image = |ifd: &mut IfdBuilder, w: u32, h: u32, cpp: u16, bits: u16, data: Vec<u8>| {
|
||||
ifd.set(t::IMAGE_WIDTH, Value::Long(vec![w]));
|
||||
ifd.set(t::IMAGE_LENGTH, Value::Long(vec![h]));
|
||||
ifd.set(t::BITS_PER_SAMPLE, Value::Short(vec![bits; cpp as usize]));
|
||||
ifd.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![cpp]));
|
||||
ifd.set(t::COMPRESSION, Value::Short(vec![1]));
|
||||
ifd.set_image(ImageData::Strips { rows_per_strip: h, strips: vec![data] });
|
||||
};
|
||||
let mut raw = IfdBuilder::new();
|
||||
raw.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![0]));
|
||||
raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::LINEAR_RAW]));
|
||||
image(&mut raw, w, h, 3, 16, 12000u16.to_le_bytes().repeat((w * h * 3) as usize));
|
||||
let matte = |name: &str, right: bool| {
|
||||
let mut m = IfdBuilder::new();
|
||||
m.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![t::SUBFILE_SEMANTIC_MASK]));
|
||||
m.set(t::PHOTOMETRIC, Value::Short(vec![photometric::MASK]));
|
||||
m.set(t::SEMANTIC_NAME, Value::Ascii(name.into()));
|
||||
image(&mut m, w / 2, h / 2, 1, 8, (0..w / 2 * h / 2).map(|i| if (i % (w / 2) >= w / 4) == right { 255 } else { 0 }).collect());
|
||||
m
|
||||
};
|
||||
let mut ifd0 = IfdBuilder::new();
|
||||
ifd0.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![1]));
|
||||
ifd0.set(t::DNG_VERSION, Value::Byte(vec![1, 6, 0, 0]));
|
||||
ifd0.set(t::MAKE, Value::Ascii("Apple".into()));
|
||||
ifd0.set(t::ORIENTATION, Value::Short(vec![6]));
|
||||
ifd0.set(t::COLOR_MATRIX_1, Value::SRational(vec![(1, 1), (0, 1), (0, 1), (0, 1), (1, 1), (0, 1), (0, 1), (0, 1), (1, 1)]));
|
||||
ifd0.set(t::CALIBRATION_ILLUMINANT_1, Value::Short(vec![21]));
|
||||
ifd0.set(t::PHOTOMETRIC, Value::Short(vec![photometric::RGB]));
|
||||
image(&mut ifd0, 4, 2, 3, 8, vec![128; 24]);
|
||||
ifd0.add_sub_ifd(raw);
|
||||
ifd0.add_sub_ifd(matte("urn:com:apple:photo:2020:aux:semanticskymatte", true));
|
||||
ifd0.add_sub_ifd(matte("urn:com:apple:photo:2020:aux:semanticsomethingelse", false));
|
||||
let bytes = TiffWriter::new(ByteOrder::Little, false).write(&[ifd0]).unwrap();
|
||||
|
||||
let (img, info) = load_bytes(&bytes, 64).unwrap();
|
||||
assert_eq!((img.width, img.height), (8, 16));
|
||||
assert!(info.mattes.is_some());
|
||||
let sky = Mask {
|
||||
components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: false, shape: MaskShape::Sky }],
|
||||
adjust: LocalAdjustments { exposure: -3.0, ..Default::default() },
|
||||
..Default::default()
|
||||
};
|
||||
let s = DevelopSettings { masks: vec![sky], ..Default::default() };
|
||||
let req = RenderRequest::fit(8, 16);
|
||||
let green = |s: &DevelopSettings, info: &SourceInfo, y: usize| render(&img, info, s, &req).image.get(4, y)[1] as i32;
|
||||
let plain = DevelopSettings::default();
|
||||
assert_eq!(green(&s, &info, 2), green(&plain, &info, 2), "no sky at the top");
|
||||
assert!(green(&s, &info, 13) + 40 < green(&plain, &info, 13), "the sky at the bottom is darkened");
|
||||
// without the matte the heuristic looks at the top of the frame instead
|
||||
let heuristic = SourceInfo { mattes: None, ..info.clone() };
|
||||
assert!(green(&s, &heuristic, 2) < green(&plain, &heuristic, 2));
|
||||
assert_eq!(green(&s, &heuristic, 13), green(&plain, &heuristic, 13));
|
||||
}
|
||||
|
||||
/// Apple ProRAW: like Lightroom Classic, the default render ignores the file's
|
||||
/// `ProfileGainTableMap` (Apple's local tone mapping); a map that would double every pixel
|
||||
/// changes nothing.
|
||||
|
||||
+12
-10
@@ -143,7 +143,7 @@ impl DecodedSource {
|
||||
/// What to render these pixels against: the decoder's facts, else `header` (the catalog's)
|
||||
/// with any stored camera tone curve.
|
||||
pub fn info_or(&self, header: SourceInfo) -> SourceInfo {
|
||||
self.info.unwrap_or(SourceInfo { camera_tone: self.camera_tone.or(header.camera_tone), ..header })
|
||||
self.info.clone().unwrap_or_else(|| SourceInfo { camera_tone: self.camera_tone.or(header.camera_tone), ..header })
|
||||
}
|
||||
}
|
||||
|
||||
@@ -253,8 +253,9 @@ fn rendered_budget(share: usize) -> usize {
|
||||
RENDERED_MEM_BYTES.min(share / 4)
|
||||
}
|
||||
|
||||
fn source_bytes(img: &Rgb32f) -> usize {
|
||||
img.data.len() * 12 + 64 + std::mem::size_of::<SourceInfo>()
|
||||
fn source_bytes(s: &DecodedSource) -> usize {
|
||||
let mattes = s.info.as_ref().and_then(|i| i.mattes.as_ref()).map_or(0, |m| m.bytes());
|
||||
s.image.data.len() * 12 + 64 + std::mem::size_of::<SourceInfo>() + mattes
|
||||
}
|
||||
|
||||
impl MediaCache {
|
||||
@@ -319,8 +320,8 @@ impl MediaCache {
|
||||
let Some((tick, which)) = candidates.into_iter().flatten().min() else { break };
|
||||
let freed = match which {
|
||||
0 => self.thumbs.pop_oldest(),
|
||||
1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1.image)),
|
||||
2 => self.full.take().map(|e| source_bytes(&e.1.image)),
|
||||
1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1)),
|
||||
2 => self.full.take().map(|e| source_bytes(&e.1)),
|
||||
_ => self.rendered.evict_oldest(),
|
||||
};
|
||||
match freed {
|
||||
@@ -372,7 +373,7 @@ impl MediaCache {
|
||||
let tick = lightcraft_preview::next_tick();
|
||||
match level {
|
||||
SourceLevel::Thumb => {
|
||||
let cost = source_bytes(&img.image);
|
||||
let cost = source_bytes(&img);
|
||||
self.thumbs.insert(id, img, cost);
|
||||
}
|
||||
SourceLevel::Preview => {
|
||||
@@ -415,8 +416,8 @@ impl MediaCache {
|
||||
/// Decoded sources held: (thumbnail level, preview level, full size).
|
||||
pub fn usage(&self) -> (crate::memory::Usage, crate::memory::Usage, crate::memory::Usage) {
|
||||
use crate::memory::Usage;
|
||||
let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1.image)).sum());
|
||||
let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1.image))).unwrap_or_default();
|
||||
let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1)).sum());
|
||||
let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1))).unwrap_or_default();
|
||||
(Usage::new(self.thumbs.len(), self.thumbs.cost()), previews, full)
|
||||
}
|
||||
|
||||
@@ -1210,20 +1211,21 @@ mod tests {
|
||||
let mut s = crate::Session::with_demo();
|
||||
let id = s.active().unwrap();
|
||||
let mut job = s.render_job(id, 64, 64, false, true).unwrap();
|
||||
let loaded_info = info.clone();
|
||||
job.source = SourceRef::File {
|
||||
path: "synthetic.arw".into(),
|
||||
max_edge: 64,
|
||||
loader: Some(Arc::new(move |_, _| {
|
||||
let mut image = Rgb32f::new(64, 64);
|
||||
image.data.fill([0.1; 3]);
|
||||
Ok((image, info))
|
||||
Ok((image, loaded_info.clone()))
|
||||
})),
|
||||
fallback: None,
|
||||
};
|
||||
job.info = SourceInfo::default(); // Header facts cannot override decoder facts.
|
||||
job.settings = Arc::new(DevelopSettings::default());
|
||||
let r = job.clone().run();
|
||||
assert_eq!(r.loaded.as_ref().unwrap().info, Some(info));
|
||||
assert_eq!(r.loaded.as_ref().unwrap().info.as_ref(), Some(&info));
|
||||
let expected = lightcraft_pipeline::render(&r.loaded.as_ref().unwrap().image, &info, &job.settings, &job.request);
|
||||
assert_eq!(r.rendered.as_ref().unwrap().image.data, expected.image.data);
|
||||
s.accept(&r);
|
||||
|
||||
@@ -717,7 +717,7 @@ pub fn render(
|
||||
Err(m) => {
|
||||
let img = host.lin.get_or_insert_with(|| Arc::new(cx.read_rgb(&lin, w, h))).clone();
|
||||
let l = host.log_l.get_or_insert_with(|| Arc::new(cx.read_plane(&prep.log_l, w, h))).clone();
|
||||
lightcraft_pipeline::masks::evaluate_one(m, &plan.frame, w, h, &img, &l, s.light.exposure as f32)
|
||||
lightcraft_pipeline::masks::evaluate_one(m, &plan.frame, w, h, &img, &l, s.light.exposure as f32, plan.mattes.as_deref())
|
||||
}
|
||||
});
|
||||
lightcraft_pipeline::visualize::apply(&mut image, req.overlay, &plan, overlay_mask.as_ref());
|
||||
@@ -1018,7 +1018,7 @@ fn masks(cx: &mut Cx<'_>, lin: &Buf, prep: &Prep, plan: &Plan<'_>, host: &mut Ho
|
||||
// no kernel: evaluate on the CPU
|
||||
let img = host.lin.get_or_insert_with(|| Arc::new(cx.read_rgb(lin, w, h))).clone();
|
||||
let l = host.log_l.get_or_insert_with(|| Arc::new(cx.read_plane(&prep.log_l, w, h))).clone();
|
||||
let mut v = lightcraft_pipeline::masks::shape_alpha(&comp.shape, frame, w, h, &img, &l, ev);
|
||||
let mut v = lightcraft_pipeline::masks::shape_alpha(&comp.shape, frame, w, h, &img, &l, ev, plan.mattes.as_deref());
|
||||
if comp.invert {
|
||||
v.data.iter_mut().for_each(|x| *x = 1.0 - *x);
|
||||
}
|
||||
|
||||
@@ -55,7 +55,7 @@ use lightcraft_raster::{Histogram, Plane, Rgb32f, Rgba8, par_rows};
|
||||
pub use tone::ToneMap;
|
||||
|
||||
/// Facts about the source the settings are interpreted against.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct SourceInfo {
|
||||
/// Lens corrections embedded in the file (DNG opcodes), relative to the EXIF-oriented source.
|
||||
pub lens: Option<lightcraft_develop::EmbeddedLens>,
|
||||
@@ -67,11 +67,13 @@ pub struct SourceInfo {
|
||||
/// No measured camera illuminant: WB adjustments are relative to the camera's rendered look.
|
||||
pub relative_wb: bool,
|
||||
pub camera_tone: Option<tone::CameraTone>,
|
||||
/// Segmentation mattes stored in the file (DNG semantic masks): AI masks use them.
|
||||
pub mattes: Option<Arc<masks::Mattes>>,
|
||||
}
|
||||
|
||||
impl Default for SourceInfo {
|
||||
fn default() -> Self {
|
||||
Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None, relative_wb: false, camera_tone: None }
|
||||
Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None, relative_wb: false, camera_tone: None, mattes: None }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -332,6 +334,8 @@ pub struct Plan<'a> {
|
||||
/// [`spots::window_for_reads`]); this is the requested window inside the rendered one, which
|
||||
/// the result is cut to.
|
||||
pub keep: Option<PixelWindow>,
|
||||
/// The source's segmentation mattes ([`SourceInfo::mattes`]).
|
||||
pub mattes: Option<Arc<masks::Mattes>>,
|
||||
}
|
||||
|
||||
/// Long edge of the small render a windowed render estimates the whole frame's airlight from.
|
||||
@@ -392,7 +396,7 @@ pub fn plan<'a>(src: &Rgb32f, info: &SourceInfo, s: &'a DevelopSettings, req: &R
|
||||
[d.nr_luminance, d.nr_detail, d.nr_color, d.nr_color_detail, d.nr_color_smoothness].map(f64::to_bits),
|
||||
src_long,
|
||||
));
|
||||
Plan { settings, frame, w, h, px_per_long, src_long, geo, lin_key, eyes, fixed_air, keep }
|
||||
Plan { settings, frame, w, h, px_per_long, src_long, geo, lin_key, eyes, fixed_air, keep, mattes: info.mattes.clone() }
|
||||
}
|
||||
|
||||
/// Whether the scene-linear stage needs work only the CPU does (defringe, spot removal).
|
||||
@@ -474,7 +478,7 @@ fn render_impl(src: Src<'_>, info: &SourceInfo, s: &DevelopSettings, req: &Rende
|
||||
Some(p) if p.key == lin_key => p,
|
||||
_ => local::Planes { key: lin_key, ..Default::default() },
|
||||
};
|
||||
let mut prep = local::prepare(lin.clone(), s, frame, px_per_long, req.quality, &mut planes);
|
||||
let mut prep = local::prepare(lin.clone(), s, frame, px_per_long, req.quality, &mut planes, plan.mattes.as_deref());
|
||||
if let Some(a) = plan.fixed_air {
|
||||
prep.air = a;
|
||||
}
|
||||
@@ -543,7 +547,7 @@ fn overlay_alpha(o: Overlay, plan: &Plan<'_>, prep: &Prepared) -> Option<Plane>
|
||||
return Some(e.alpha.clone());
|
||||
}
|
||||
let ev = plan.settings.light.exposure as f32;
|
||||
Some(masks::evaluate_one(m, &plan.frame, plan.w, plan.h, &prep.img, &prep.log_l, ev))
|
||||
Some(masks::evaluate_one(m, &plan.frame, plan.w, plan.h, &prep.img, &prep.log_l, ev, plan.mattes.as_deref()))
|
||||
}
|
||||
|
||||
/// Convenience: render a before/after pair side by side is up to the UI; this renders "before"
|
||||
|
||||
@@ -279,7 +279,16 @@ pub fn plane_sigmas(s: &DevelopSettings, px_per_long: f64, q: Quality) -> PlaneS
|
||||
/// The spatial planes the per-pixel stage needs for `img` (white-balanced, before exposure),
|
||||
/// reusing whatever `planes` already holds for it. Missing planes are computed side by side (each
|
||||
/// one alone scales poorly: the guided filters work on small subsampled grids).
|
||||
pub(crate) fn prepare(img: Arc<Rgb32f>, s: &DevelopSettings, frame: &Frame, px_per_long: f64, q: Quality, planes: &mut Planes) -> Prepared {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(crate) fn prepare(
|
||||
img: Arc<Rgb32f>,
|
||||
s: &DevelopSettings,
|
||||
frame: &Frame,
|
||||
px_per_long: f64,
|
||||
q: Quality,
|
||||
planes: &mut Planes,
|
||||
mattes: Option<&masks::Mattes>,
|
||||
) -> Prepared {
|
||||
let log_l = planes.log_l.get_or_insert_with(|| Arc::new(timed("log_l", || img.map(log_lum)))).clone();
|
||||
let PlaneSigmas { base: base_sigma, clarity: clarity_sigma, texture: texture_sigma, dark: dark_sigma, chroma: chroma_sigma } =
|
||||
plane_sigmas(s, px_per_long, q);
|
||||
@@ -324,7 +333,7 @@ pub(crate) fn prepare(img: Arc<Rgb32f>, s: &DevelopSettings, frame: &Frame, px_p
|
||||
None => (None, 1.0),
|
||||
};
|
||||
let ev = s.light.exposure as f32;
|
||||
let masks = timed("masks", || masks::evaluate(&s.masks, frame, img.width, img.height, &img, &log_l, ev));
|
||||
let masks = timed("masks", || masks::evaluate(&s.masks, frame, img.width, img.height, &img, &log_l, ev, mattes));
|
||||
Prepared { img, log_l, base, clarity_blur, texture_blur, dark, chroma_blur, air, masks, px_per_long }
|
||||
}
|
||||
|
||||
|
||||
+239
-25
@@ -2,14 +2,96 @@
|
||||
//!
|
||||
//! Shapes are defined in normalized oriented-image coordinates; sizes are in "long-edge units".
|
||||
//! Components combine with Add (max), Subtract (a·(1−c)) and Intersect (a·c).
|
||||
//!
|
||||
//! AI shapes (Sky, Subject, Background, People) use the segmentation [`Mattes`] the source carries
|
||||
//! (DNG semantic masks, e.g. iPhone ProRAW's) when it has a matching one, else a heuristic.
|
||||
|
||||
use lightcraft_develop::{BrushStroke, LocalAdjustments, Mask, MaskOp, MaskShape, SegMask};
|
||||
use lightcraft_geom::Point;
|
||||
use lightcraft_raster::{Plane, Rgb32f};
|
||||
use lightcraft_geom::{Orientation, Point};
|
||||
use lightcraft_raster::{Image, Plane, Rgb32f};
|
||||
|
||||
use crate::for_rows;
|
||||
use crate::geometry::Frame;
|
||||
|
||||
/// What an embedded segmentation matte selects.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum MatteKind {
|
||||
Sky,
|
||||
/// A whole person (all people in the photo).
|
||||
Person,
|
||||
Skin,
|
||||
Hair,
|
||||
Teeth,
|
||||
Glasses,
|
||||
}
|
||||
|
||||
/// Segmentation mattes that came with the source: 0..255 alpha planes, each covering the whole
|
||||
/// (EXIF-oriented) source image at its own resolution. Several mattes of one kind (e.g. one per
|
||||
/// person) are combined with max.
|
||||
#[derive(Clone, Default, PartialEq)]
|
||||
pub struct Mattes {
|
||||
mattes: Vec<(MatteKind, Image<u8>)>,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for Mattes {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_list().entries(self.mattes.iter().map(|(k, m)| (k, m.width, m.height))).finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl Mattes {
|
||||
/// Add a matte (empty ones are ignored).
|
||||
pub fn push(&mut self, kind: MatteKind, matte: Image<u8>) {
|
||||
if matte.width > 0 && matte.height > 0 && matte.data.len() == matte.width * matte.height {
|
||||
self.mattes.push((kind, matte));
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.mattes.is_empty()
|
||||
}
|
||||
|
||||
/// Memory held (bytes).
|
||||
pub fn bytes(&self) -> usize {
|
||||
self.mattes.iter().map(|(_, m)| m.data.len()).sum()
|
||||
}
|
||||
|
||||
fn of(&self, kinds: &[MatteKind]) -> Vec<&Image<u8>> {
|
||||
self.mattes.iter().filter(|(k, _)| kinds.contains(k)).map(|(_, m)| m).collect()
|
||||
}
|
||||
|
||||
/// The mattes a shape uses instead of its heuristic (empty: none fits).
|
||||
fn for_shape(&self, shape: &MaskShape) -> Vec<&Image<u8>> {
|
||||
use MatteKind::*;
|
||||
// the whole person: a person matte, else the union of the parts
|
||||
let person = || {
|
||||
let p = self.of(&[Person]);
|
||||
if p.is_empty() { self.of(&[Skin, Hair, Teeth, Glasses]) } else { p }
|
||||
};
|
||||
match shape {
|
||||
MaskShape::Sky => self.of(&[Sky]),
|
||||
MaskShape::Subject => person(),
|
||||
MaskShape::People { parts, .. } => {
|
||||
let wanted: Vec<MatteKind> = parts
|
||||
.iter()
|
||||
.filter_map(|p| {
|
||||
let p = p.to_ascii_lowercase();
|
||||
[("skin", Skin), ("hair", Hair), ("teeth", Teeth), ("glasses", Glasses)]
|
||||
.into_iter()
|
||||
.find(|(n, _)| p.contains(n))
|
||||
.map(|(_, k)| k)
|
||||
})
|
||||
.collect();
|
||||
let whole =
|
||||
parts.is_empty() || parts.iter().any(|p| matches!(p.to_ascii_lowercase().as_str(), "person" | "entire person" | "entireperson"));
|
||||
// parts without a matte (lips, clothes, …) keep the heuristic
|
||||
if whole { person() } else { self.of(&wanted) }
|
||||
}
|
||||
_ => Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct Evaluated {
|
||||
/// The mask's id ([`Mask::id`]).
|
||||
pub id: u32,
|
||||
@@ -23,22 +105,24 @@ fn smooth(e0: f32, e1: f32, x: f32) -> f32 {
|
||||
t * t * (3.0 - 2.0 * t)
|
||||
}
|
||||
|
||||
/// The visible masks with components, evaluated in order.
|
||||
pub fn evaluate(masks: &[Mask], frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Vec<Evaluated> {
|
||||
/// The visible masks with components, evaluated in order (`mattes`: the source's, see [`Mattes`]).
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn evaluate(masks: &[Mask], frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Vec<Evaluated> {
|
||||
masks
|
||||
.iter()
|
||||
.filter(|m| m.visible && !m.components.is_empty())
|
||||
.map(|m| Evaluated { id: m.id, alpha: evaluate_one(m, frame, w, h, img, log_l, ev), adjust: m.adjust })
|
||||
.map(|m| Evaluated { id: m.id, alpha: evaluate_one(m, frame, w, h, img, log_l, ev, mattes), adjust: m.adjust })
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The alpha plane of mask `m` (whether visible or not): its components combined, inverted and
|
||||
/// scaled by its amount.
|
||||
pub fn evaluate_one(m: &Mask, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Plane {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn evaluate_one(m: &Mask, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Plane {
|
||||
let mut alpha = Plane::new(w, h);
|
||||
let mut first = true;
|
||||
for comp in &m.components {
|
||||
let mut c = shape_alpha(&comp.shape, frame, w, h, img, log_l, ev);
|
||||
let mut c = shape_alpha(&comp.shape, frame, w, h, img, log_l, ev, mattes);
|
||||
if comp.invert {
|
||||
c.data.iter_mut().for_each(|v| *v = 1.0 - *v);
|
||||
}
|
||||
@@ -88,7 +172,13 @@ fn for_each_pos(frame: &Frame, w: usize, h: usize, out: &mut Plane, f: impl Fn(P
|
||||
});
|
||||
}
|
||||
|
||||
pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Plane {
|
||||
/// The alpha plane of one component shape. AI shapes use the source's matching `mattes`, if any.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Plane {
|
||||
let fitting = mattes.map(|m| m.for_shape(shape)).unwrap_or_default();
|
||||
if !fitting.is_empty() {
|
||||
return matte_alpha(&fitting, frame, w, h);
|
||||
}
|
||||
let mut out = Plane::new(w, h);
|
||||
let to_long = |p: Point| frame.norm_to_long(p);
|
||||
// `img`/`log_l` are before exposure: range masks select on the exposed values.
|
||||
@@ -180,7 +270,7 @@ pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &R
|
||||
smooth_plane(&mut out, 0.015 * frame_px(frame, w));
|
||||
}
|
||||
MaskShape::Background => {
|
||||
let mut s = shape_alpha(&MaskShape::Subject, frame, w, h, img, log_l, ev);
|
||||
let mut s = shape_alpha(&MaskShape::Subject, frame, w, h, img, log_l, ev, mattes);
|
||||
s.data.iter_mut().for_each(|v| *v = 1.0 - *v);
|
||||
out = s;
|
||||
}
|
||||
@@ -252,6 +342,71 @@ fn smooth_plane(p: &mut Plane, sigma: f32) {
|
||||
*p = lightcraft_raster::blur::gaussian(p, sigma.max(0.5));
|
||||
}
|
||||
|
||||
/// The union (max) of `mattes` (0..255, over the EXIF-oriented source) at output resolution,
|
||||
/// sampled like the image: user orientation, lens / perspective warp, crop and flips; area-averaged
|
||||
/// first when the output is much smaller than the matte, bilinear otherwise.
|
||||
fn matte_alpha(mattes: &[&Image<u8>], frame: &Frame, w: usize, h: usize) -> Plane {
|
||||
let mut out = Plane::new(w, h);
|
||||
let o2t = frame.out_to_oriented(w, h);
|
||||
let ppl = frame.px_per_long(w).max(1e-9);
|
||||
for m in mattes {
|
||||
let oriented =
|
||||
if frame.orient == Orientation::Normal { std::borrow::Cow::Borrowed(*m) } else { std::borrow::Cow::Owned(m.oriented(frame.orient)) };
|
||||
// matte px per output px
|
||||
let k = oriented.width as f64 / frame.ow * frame.ow.max(frame.oh) / ppl;
|
||||
let base = box_down(&oriented, if k >= 2.0 { (k.floor() as usize).min(64) } else { 1 });
|
||||
let (sx, sy) = ((base.width as f64 / frame.ow) as f32, (base.height as f64 / frame.oh) as f32);
|
||||
for_rows(&mut out.data, w, |y, row| {
|
||||
for (x, v) in row.iter_mut().enumerate() {
|
||||
let s = match &frame.warp {
|
||||
Some(wp) => {
|
||||
let Some((_, s)) = wp.frame(&o2t, x, y) else {
|
||||
continue;
|
||||
};
|
||||
s
|
||||
}
|
||||
None => o2t.apply(Point::new(x as f64 + 0.5, y as f64 + 0.5)),
|
||||
};
|
||||
*v = v.max(bilinear(&base, s.x as f32 * sx, s.y as f32 * sy) / 255.0);
|
||||
}
|
||||
});
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// `m` area-averaged by `f × f` (0..255 values kept); `f == 1` converts only.
|
||||
fn box_down(m: &Image<u8>, f: usize) -> Plane {
|
||||
let f = f.max(1);
|
||||
let (bw, bh) = (m.width.div_ceil(f).max(1), m.height.div_ceil(f).max(1));
|
||||
let mut out = Plane::new(bw, bh);
|
||||
for_rows(&mut out.data, bw, |by, row| {
|
||||
let rows = by * f..((by + 1) * f).min(m.height);
|
||||
for (bx, v) in row.iter_mut().enumerate() {
|
||||
let cols = bx * f..((bx + 1) * f).min(m.width);
|
||||
let (mut sum, mut n) = (0u32, 0u32);
|
||||
for y in rows.clone() {
|
||||
for x in cols.clone() {
|
||||
sum += m.data.get(y * m.width + x).copied().unwrap_or(0) as u32;
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
*v = sum as f32 / n.max(1) as f32;
|
||||
}
|
||||
});
|
||||
out
|
||||
}
|
||||
|
||||
/// Bilinear sample at continuous pixel coordinates (pixel centres at +0.5), edges extended.
|
||||
fn bilinear(p: &Plane, x: f32, y: f32) -> f32 {
|
||||
let (fx, fy) = (x - 0.5, y - 0.5);
|
||||
let (x0, y0) = (fx.floor(), fy.floor());
|
||||
let (tx, ty) = (fx - x0, fy - y0);
|
||||
let (x0, y0) = (x0 as isize, y0 as isize);
|
||||
let a = p.get_clamped(x0, y0) + (p.get_clamped(x0 + 1, y0) - p.get_clamped(x0, y0)) * tx;
|
||||
let b = p.get_clamped(x0, y0 + 1) + (p.get_clamped(x0 + 1, y0 + 1) - p.get_clamped(x0, y0 + 1)) * tx;
|
||||
a + (b - a) * ty
|
||||
}
|
||||
|
||||
/// A rough display mapping for colour picking (so samples taken on screen match).
|
||||
pub(crate) fn tonemap_for_select(c: [f32; 3]) -> [f32; 3] {
|
||||
c.map(|v| v / (1.0 + v))
|
||||
@@ -436,7 +591,7 @@ mod tests {
|
||||
let f = frame(100, 50);
|
||||
let img = Rgb32f::new(100, 50);
|
||||
let l = Plane::new(100, 50);
|
||||
let a = shape_alpha(&MaskShape::Linear { start: Point::new(0.0, 0.0), end: Point::new(1.0, 0.0) }, &f, 100, 50, &img, &l, 0.0);
|
||||
let a = shape_alpha(&MaskShape::Linear { start: Point::new(0.0, 0.0), end: Point::new(1.0, 0.0) }, &f, 100, 50, &img, &l, 0.0, None);
|
||||
assert!(a.get(1, 25) > 0.99 && a.get(98, 25) < 0.01);
|
||||
assert!((a.get(50, 25) - 0.5).abs() < 0.05);
|
||||
}
|
||||
@@ -452,7 +607,7 @@ mod tests {
|
||||
MaskShape::Prompt { text: "left".into(), seg: Some(seg.clone()), detail: vec![], edge: 0.0 },
|
||||
] {
|
||||
for (w, h) in [(40, 20), (400, 200)] {
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
|
||||
assert!(a.get(1, h / 2) > 0.99 && a.get(w - 2, h / 2) < 0.01, "{w}×{h}");
|
||||
if w >= 400 {
|
||||
let soft = (0..w).filter(|x| (0.05..0.95).contains(&a.get(*x, h / 2))).count();
|
||||
@@ -465,7 +620,7 @@ mod tests {
|
||||
d.crop.geometry.rect = lightcraft_geom::Rect::from_center(Point::new(0.75, 0.5), 0.5, 1.0);
|
||||
let f = Frame::new(40, 20, &d, true);
|
||||
let shape = MaskShape::Prompt { text: "left".into(), seg: Some(seg), detail: vec![], edge: 0.0 };
|
||||
let a = shape_alpha(&shape, &f, 20, 20, &Rgb32f::new(20, 20), &Plane::new(20, 20), 0.0);
|
||||
let a = shape_alpha(&shape, &f, 20, 20, &Rgb32f::new(20, 20), &Plane::new(20, 20), 0.0, None);
|
||||
assert!(a.data.iter().all(|v| *v < 0.05));
|
||||
}
|
||||
|
||||
@@ -478,7 +633,7 @@ mod tests {
|
||||
let patch = SegMask::from_logits_in(side, &l, [0.5, 0.0, 1.0, 1.0]);
|
||||
let shape = MaskShape::Object { hint: vec![Point::new(0.6, 0.5)], exclude: vec![], seg: Some(coarse), detail: vec![patch], edge: 0.0 };
|
||||
let (w, h) = (200, 100);
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
|
||||
assert!(a.get(20, 50) < 0.01, "outside the patch: the coarse mask");
|
||||
assert!(a.get(120, 50) > 0.99, "inside the patch, its selected half");
|
||||
assert!(a.get(190, 50) < 0.01, "inside the patch, its unselected half");
|
||||
@@ -494,7 +649,7 @@ mod tests {
|
||||
let shape = MaskShape::Object { hint: vec![Point::new(0.2, 0.5)], exclude: vec![], seg: None, detail: vec![patch; 20_000], edge: 0.0 };
|
||||
let (w, h) = (64, 32);
|
||||
let t = std::time::Instant::now();
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
|
||||
assert!(a.get(5, 16) > 0.99 && a.get(60, 16) < 0.01);
|
||||
assert!(t.elapsed() < std::time::Duration::from_secs(20), "{:?}", t.elapsed());
|
||||
}
|
||||
@@ -507,7 +662,7 @@ mod tests {
|
||||
let (w, h) = (400, 40);
|
||||
let soft_px = |edge: f64| {
|
||||
let shape = MaskShape::Prompt { text: "x".into(), seg: Some(seg.clone()), detail: vec![], edge };
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
|
||||
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
|
||||
assert!(a.get(2, h / 2) > 0.9 && a.get(w - 3, h / 2) < 0.1, "still the same selection at {edge}");
|
||||
(0..w).filter(|x| (0.1..0.9).contains(&a.get(*x, h / 2))).count()
|
||||
};
|
||||
@@ -524,7 +679,7 @@ mod tests {
|
||||
MaskShape::Prompt { text: "sky".into(), seg: None, detail: vec![], edge: 0.0 },
|
||||
MaskShape::Prompt { text: "sky".into(), seg: Some(SegMask { side: 4, data: "damaged!".into(), rect: None }), detail: vec![], edge: 0.0 },
|
||||
] {
|
||||
assert!(shape_alpha(&shape, &f, 30, 30, &img, &l, 0.0).data.iter().all(|v| *v == 0.0));
|
||||
assert!(shape_alpha(&shape, &f, 30, 30, &img, &l, 0.0, None).data.iter().all(|v| *v == 0.0));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -534,10 +689,10 @@ mod tests {
|
||||
let img = Rgb32f::new(100, 100);
|
||||
let l = Plane::new(100, 100);
|
||||
let shape = MaskShape::Radial { center: Point::new(0.5, 0.5), rx: 0.2, ry: 0.2, angle: 0.0, feather: 20.0, invert: false };
|
||||
let a = shape_alpha(&shape, &f, 100, 100, &img, &l, 0.0);
|
||||
let a = shape_alpha(&shape, &f, 100, 100, &img, &l, 0.0, None);
|
||||
assert!(a.get(50, 50) > 0.99 && a.get(5, 5) < 0.01);
|
||||
let m = Mask { components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: true, shape }], ..Default::default() };
|
||||
let e = evaluate(&[m], &f, 100, 100, &img, &l, 0.0);
|
||||
let e = evaluate(&[m], &f, 100, 100, &img, &l, 0.0, None);
|
||||
assert!(e[0].alpha.get(50, 50) < 0.01);
|
||||
}
|
||||
|
||||
@@ -552,7 +707,7 @@ mod tests {
|
||||
components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: false, shape: MaskShape::Brush { strokes: vec![stroke, erase] } }],
|
||||
..Default::default()
|
||||
};
|
||||
let e = evaluate(&[m], &f, 200, 100, &img, &l, 0.0);
|
||||
let e = evaluate(&[m], &f, 200, 100, &img, &l, 0.0, None);
|
||||
let a = &e[0].alpha;
|
||||
assert!(a.get(40, 50) > 0.9, "{}", a.get(40, 50));
|
||||
assert!(a.get(100, 50) < 0.05, "erased centre {}", a.get(100, 50));
|
||||
@@ -574,8 +729,8 @@ mod tests {
|
||||
..Default::default()
|
||||
};
|
||||
let comp = |auto| MaskShape::Brush { strokes: vec![stroke(auto)] };
|
||||
let plain = shape_alpha(&comp(false), &f, w, h, &img, &l, 0.0);
|
||||
let auto = shape_alpha(&comp(true), &f, w, h, &img, &l, 0.0);
|
||||
let plain = shape_alpha(&comp(false), &f, w, h, &img, &l, 0.0, None);
|
||||
let auto = shape_alpha(&comp(true), &f, w, h, &img, &l, 0.0, None);
|
||||
// without Auto Mask the brush spills over the edge; with it, it stays on the dark side
|
||||
assert!(plain.get(105, 50) > 0.9, "{}", plain.get(105, 50));
|
||||
assert!(auto.get(105, 50) < 0.05, "spill {}", auto.get(105, 50));
|
||||
@@ -585,8 +740,8 @@ mod tests {
|
||||
// on a flat area Auto Mask paints like the plain brush
|
||||
let flat = Rgb32f::from_fn(w, h, |_, _| [0.2; 3]);
|
||||
let fl = flat.map(crate::local::log_lum);
|
||||
let a = shape_alpha(&comp(true), &f, w, h, &flat, &fl, 0.0);
|
||||
let b = shape_alpha(&comp(false), &f, w, h, &flat, &fl, 0.0);
|
||||
let a = shape_alpha(&comp(true), &f, w, h, &flat, &fl, 0.0, None);
|
||||
let b = shape_alpha(&comp(false), &f, w, h, &flat, &fl, 0.0, None);
|
||||
assert!((a.get(94, 50) - b.get(94, 50)).abs() < 0.02, "{} vs {}", a.get(94, 50), b.get(94, 50));
|
||||
}
|
||||
|
||||
@@ -609,7 +764,7 @@ mod tests {
|
||||
};
|
||||
let refined = Mask { refine: 100.0, ..soft.clone() };
|
||||
let step = |m: &Mask| {
|
||||
let a = evaluate_one(m, &f, w, h, &img, &log_l, 0.0);
|
||||
let a = evaluate_one(m, &f, w, h, &img, &log_l, 0.0, None);
|
||||
a.data[40 * w + 63] - a.data[40 * w + 56]
|
||||
};
|
||||
let (plain, sharp) = (step(&soft), step(&refined));
|
||||
@@ -640,7 +795,7 @@ mod tests {
|
||||
}
|
||||
let alpha = |refine| {
|
||||
let shape = MaskShape::ColorRange { samples: vec![sample], refine };
|
||||
let a = shape_alpha(&shape, &f, w, h, &img, &log_l, 0.0);
|
||||
let a = shape_alpha(&shape, &f, w, h, &img, &log_l, 0.0, None);
|
||||
(a.get(30, 20), a.get(90, 20))
|
||||
};
|
||||
for refine in [0.0, 50.0, 85.0] {
|
||||
@@ -653,4 +808,63 @@ mod tests {
|
||||
let (l, r) = alpha(100.0);
|
||||
assert!(l > 0.95 && r < 0.3, "refine 100: {l} vs {r}");
|
||||
}
|
||||
|
||||
/// A half-resolution matte selecting the left `frac` of a `w × h` source.
|
||||
fn left_matte(w: usize, h: usize, frac: f64) -> Image<u8> {
|
||||
Image::from_fn(w / 2, h / 2, |x, _| if (x as f64 + 0.5) < frac * (w / 2) as f64 { 255 } else { 0 })
|
||||
}
|
||||
|
||||
/// The Sky mask follows the source's sky matte where it has one (here: the bottom-left of a
|
||||
/// uniformly bright blue photo, where the heuristic sees no sky, and not its top right, where
|
||||
/// it does), through user orientation, at any output size; without one it keeps the heuristic.
|
||||
#[test]
|
||||
fn sky_uses_the_sources_matte() {
|
||||
let (w, h) = (200, 100);
|
||||
let img = Rgb32f::filled(w, h, [0.5, 0.7, 1.4]);
|
||||
let l = img.map(crate::local::log_lum);
|
||||
let mut mattes = Mattes::default();
|
||||
mattes.push(MatteKind::Sky, left_matte(w, h, 0.5));
|
||||
let f = frame(w, h);
|
||||
let heuristic = shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, None);
|
||||
let matte = shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, Some(&mattes));
|
||||
assert!(heuristic.get(150, 5) > 0.9 && heuristic.get(25, 90) < 0.1, "{} {}", heuristic.get(150, 5), heuristic.get(25, 90));
|
||||
assert!(matte.get(150, 5) < 0.01 && matte.get(25, 90) > 0.99, "{} {}", matte.get(150, 5), matte.get(25, 90));
|
||||
// mattes of other kinds leave the heuristic alone
|
||||
let mut hair = Mattes::default();
|
||||
hair.push(MatteKind::Hair, left_matte(w, h, 0.5));
|
||||
assert_eq!(shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, Some(&hair)), heuristic);
|
||||
// rotated 90° clockwise, the source's left half is the top half; a preview and a larger
|
||||
// render agree
|
||||
let s = lightcraft_develop::DevelopSettings { orientation: Orientation::Rotate90, ..Default::default() };
|
||||
let f = Frame::new(w, h, &s, true);
|
||||
for (ow, oh) in [(100, 200), (25, 50)] {
|
||||
let small = Rgb32f::filled(ow, oh, [0.5, 0.7, 1.4]);
|
||||
let a = shape_alpha(&MaskShape::Sky, &f, ow, oh, &small, &small.map(crate::local::log_lum), 0.0, Some(&mattes));
|
||||
let at = |x: f64, y: f64| a.get((x * ow as f64) as usize, (y * oh as f64) as usize);
|
||||
assert!(at(0.5, 0.2) > 0.99 && at(0.5, 0.8) < 0.01, "{ow}×{oh}: {} {}", at(0.5, 0.2), at(0.5, 0.8));
|
||||
}
|
||||
}
|
||||
|
||||
/// Subject, Background and People use person mattes: the whole person is the union of the
|
||||
/// parts; a People part without a matte keeps the heuristic.
|
||||
#[test]
|
||||
fn people_and_subject_use_person_mattes() {
|
||||
let (w, h) = (200, 100);
|
||||
let img = Rgb32f::filled(w, h, [0.2; 3]);
|
||||
let l = img.map(crate::local::log_lum);
|
||||
let mut mattes = Mattes::default();
|
||||
mattes.push(MatteKind::Skin, left_matte(w, h, 0.25));
|
||||
let right: Image<u8> = Image::from_fn(w / 2, h / 2, |x, _| if x >= 75 { 255 } else { 0 });
|
||||
mattes.push(MatteKind::Hair, right);
|
||||
let f = frame(w, h);
|
||||
let eval = |shape: &MaskShape| shape_alpha(shape, &f, w, h, &img, &l, 0.0, Some(&mattes));
|
||||
let subject = eval(&MaskShape::Subject);
|
||||
assert!(subject.get(10, 50) > 0.99 && subject.get(190, 50) > 0.99 && subject.get(100, 50) < 0.01);
|
||||
let background = eval(&MaskShape::Background);
|
||||
assert!(background.get(100, 50) > 0.99 && background.get(10, 50) < 0.01);
|
||||
let hair = eval(&MaskShape::People { person: 0, parts: vec!["Hair".into()] });
|
||||
assert!(hair.get(190, 50) > 0.99 && hair.get(10, 50) < 0.01);
|
||||
let lips = MaskShape::People { person: 0, parts: vec!["Lips".into()] };
|
||||
assert_eq!(eval(&lips), shape_alpha(&lips, &f, w, h, &img, &l, 0.0, None));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
//!
|
||||
//! - [`probe`] recognises raw containers; [`decode`] turns a file into a [`RawImage`] (sensor data + everything
|
||||
//! needed to render it: CFA, black/white levels, active area, default crop, orientation, DNG colour tags,
|
||||
//! opcode lists, [`Metadata`]); [`embedded_preview`] returns the largest embedded JPEG (or DNG 1.7 JPEG XL) preview.
|
||||
//! opcode lists, [`Metadata`]); [`embedded_preview`] returns the largest embedded JPEG (or DNG 1.7 JPEG XL) preview;
|
||||
//! [`semantic_masks`] reads a DNG's semantic masks (segmentation mattes, e.g. iPhone ProRAW's sky matte).
|
||||
//! - [`RawImage::normalized`] subtracts black, scales white to 1.0 and crops to the active area (applying DNG
|
||||
//! `OpcodeList1`/`OpcodeList2`); [`demosaic`] turns CFA data into camera-RGB [`Rgb32f`];
|
||||
//! [`RawImage::develop`] does all of it plus `OpcodeList3` and the default crop; [`RawImage::develop_binned`]
|
||||
@@ -36,6 +37,7 @@ pub mod ljpeg;
|
||||
pub mod opcodes;
|
||||
mod preview;
|
||||
pub mod profile;
|
||||
pub mod semantic;
|
||||
mod tiffraw;
|
||||
mod unpack;
|
||||
mod vendor;
|
||||
@@ -48,6 +50,7 @@ pub use lightcraft_meta::Metadata;
|
||||
pub use lightcraft_raster::Rgb32f;
|
||||
pub use opcodes::{Opcode, OpcodeLists};
|
||||
pub use preview::{PreviewColorSpace, embedded_preview, embedded_preview_color_space};
|
||||
pub use semantic::{SemanticMask, semantic_masks};
|
||||
|
||||
use lightcraft_color::Xy;
|
||||
use lightcraft_tiff::{Tiff, TiffError};
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
//! DNG semantic masks (DNG 1.6+, chapter "Semantic Masks"): single-channel segmentation mattes
|
||||
//! stored in their own IFDs (NewSubFileType 10004.H, PhotometricInterpretation 52527) with a
|
||||
//! `SemanticName`, an optional `SemanticInstanceID` and an optional `MaskSubArea`. iPhone ProRAW
|
||||
//! stores e.g. a sky matte this way.
|
||||
//!
|
||||
//! [`semantic_masks`] reads them separately from [`crate::decode`] (only callers that use them
|
||||
//! pay for them); malformed or unsupported mask IFDs are skipped.
|
||||
|
||||
use crate::Rect;
|
||||
use crate::tiffraw::{Packing, read_image};
|
||||
use crate::{RawData, Result};
|
||||
use lightcraft_raster::Image;
|
||||
use lightcraft_tiff::Tiff;
|
||||
use lightcraft_tiff::tags::{self as t, photometric};
|
||||
|
||||
/// At most this many mask IFDs are read from one file.
|
||||
pub const MAX_MASKS: usize = 16;
|
||||
/// Masks with more samples than this are skipped (a mask is a low-resolution matte; this is
|
||||
/// 8192 × 4096).
|
||||
pub const MAX_MASK_SAMPLES: usize = 1 << 25;
|
||||
|
||||
/// Where a cropped mask sits in the uncropped mask (DNG `MaskSubArea`). The uncropped mask
|
||||
/// (`full_width × full_height`) spans the main image's active area.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct MaskSubArea {
|
||||
pub top: u32,
|
||||
pub left: u32,
|
||||
pub full_width: u32,
|
||||
pub full_height: u32,
|
||||
}
|
||||
|
||||
/// One semantic mask: `width × height` values, 0 = not part of the attribute, 65535 = fully part.
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct SemanticMask {
|
||||
/// `SemanticName`, e.g. `urn:com:apple:photo:2020:aux:semanticskymatte`.
|
||||
pub name: String,
|
||||
pub instance_id: Option<String>,
|
||||
pub width: usize,
|
||||
pub height: usize,
|
||||
/// `None`: the mask covers the whole active area.
|
||||
pub sub_area: Option<MaskSubArea>,
|
||||
pub data: Vec<u16>,
|
||||
}
|
||||
|
||||
/// The semantic masks of a DNG, in file order (none for other files). Only the first
|
||||
/// [`MAX_MASKS`] mask IFDs are looked at, malformed ones included, so a file full of mask IFDs
|
||||
/// costs at most that many decodes.
|
||||
pub fn semantic_masks(bytes: &[u8]) -> Vec<SemanticMask> {
|
||||
let Ok(tiff) = Tiff::parse(bytes) else { return Vec::new() };
|
||||
tiff.all_ifds()
|
||||
.into_iter()
|
||||
.filter(|i| i.u16(t::PHOTOMETRIC) == Some(photometric::MASK) || i.u32(t::NEW_SUBFILE_TYPE) == Some(t::SUBFILE_SEMANTIC_MASK))
|
||||
.take(MAX_MASKS)
|
||||
.filter_map(|ifd| read_mask(bytes, &tiff, ifd).ok().flatten())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn read_mask(bytes: &[u8], tiff: &Tiff, ifd: &lightcraft_tiff::Ifd) -> Result<Option<SemanticMask>> {
|
||||
let Some(name) = ifd.string(t::SEMANTIC_NAME).filter(|n| !n.is_empty()) else { return Ok(None) };
|
||||
let info = ifd.image()?;
|
||||
let (w, h) = (info.width as usize, info.height as usize);
|
||||
if info.samples_per_pixel != 1 || w.checked_mul(h).is_none_or(|n| n > MAX_MASK_SAMPLES) {
|
||||
return Ok(None);
|
||||
}
|
||||
let bits = info.bits() as u32;
|
||||
let data: Vec<u16> = match read_image(bytes, &info, tiff.order, Packing::Msb)? {
|
||||
RawData::U16(v) if (1..=16).contains(&bits) => {
|
||||
let max = ((1u32 << bits) - 1) as f32;
|
||||
if bits == 16 { v } else { v.into_iter().map(|s| (s as f32 / max * 65535.0).round().min(65535.0) as u16).collect() }
|
||||
}
|
||||
RawData::F32(v) => v.into_iter().map(|s| if s.is_finite() { (s.clamp(0.0, 1.0) * 65535.0).round() as u16 } else { 0 }).collect(),
|
||||
RawData::U16(_) => return Ok(None),
|
||||
};
|
||||
if data.len() != w * h {
|
||||
return Ok(None);
|
||||
}
|
||||
// the tag is ignored when the stored mask does not fit inside the uncropped one
|
||||
let sub_area = match ifd.u64s(t::MASK_SUB_AREA).as_deref() {
|
||||
Some(&[top, left, fw, fh]) => {
|
||||
let fits = fw > 0 && fh > 0 && left.saturating_add(w as u64) <= fw && top.saturating_add(h as u64) <= fh && fw.max(fh) <= u32::MAX as u64;
|
||||
fits.then_some(MaskSubArea { top: top as u32, left: left as u32, full_width: fw as u32, full_height: fh as u32 })
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
Ok(Some(SemanticMask { name, instance_id: ifd.string(t::SEMANTIC_INSTANCE_ID).filter(|s| !s.is_empty()), width: w, height: h, sub_area, data }))
|
||||
}
|
||||
|
||||
impl SemanticMask {
|
||||
/// The mask value (0..1) at a continuous position of the uncropped mask (pixel centres at
|
||||
/// +0.5); 0 outside the stored (cropped) mask.
|
||||
pub fn value_at(&self, x: f64, y: f64) -> f32 {
|
||||
let (left, top) = self.sub_area.map_or((0.0, 0.0), |a| (a.left as f64, a.top as f64));
|
||||
let (x, y) = (x - left, y - top);
|
||||
if self.width == 0 || self.height == 0 || !(x >= 0.0 && y >= 0.0 && x <= self.width as f64 && y <= self.height as f64) {
|
||||
return 0.0;
|
||||
}
|
||||
let (fx, fy) = ((x - 0.5).max(0.0), (y - 0.5).max(0.0));
|
||||
let (x0, y0) = (fx.floor() as usize, fy.floor() as usize);
|
||||
let (tx, ty) = ((fx - x0 as f64) as f32, (fy - y0 as f64) as f32);
|
||||
let at = |x: usize, y: usize| {
|
||||
let (x, y) = (x.min(self.width - 1), y.min(self.height - 1));
|
||||
self.data.get(y * self.width + x).copied().unwrap_or(0) as f32 / 65535.0
|
||||
};
|
||||
let top = at(x0, y0) + (at(x0 + 1, y0) - at(x0, y0)) * tx;
|
||||
let bottom = at(x0, y0 + 1) + (at(x0 + 1, y0 + 1) - at(x0, y0 + 1)) * tx;
|
||||
top + (bottom - top) * ty
|
||||
}
|
||||
|
||||
/// The mask over the developed image ([`crate::RawImage::develop`]: the default `crop` of the
|
||||
/// `active_area`, before orientation), at the mask's own resolution, as 0..255 alpha. `None`
|
||||
/// for an empty crop.
|
||||
pub fn developed(&self, active_area: Rect, crop: Rect) -> Option<Image<u8>> {
|
||||
let c = crop.clipped(active_area.width, active_area.height);
|
||||
if c.width == 0 || c.height == 0 || self.width == 0 || self.height == 0 {
|
||||
return None;
|
||||
}
|
||||
let (fw, fh) = self.sub_area.map_or((self.width, self.height), |a| (a.full_width as usize, a.full_height as usize));
|
||||
// uncropped-mask px per active-area px
|
||||
let (sx, sy) = (fw as f64 / active_area.width as f64, fh as f64 / active_area.height as f64);
|
||||
let ow = ((c.width as f64 * sx).round() as usize).clamp(1, self.width.max(fw));
|
||||
let oh = ((c.height as f64 * sy).round() as usize).clamp(1, self.height.max(fh));
|
||||
if ow.checked_mul(oh).is_none_or(|n| n > MAX_MASK_SAMPLES) {
|
||||
return None;
|
||||
}
|
||||
let (kx, ky) = (c.width as f64 / ow as f64, c.height as f64 / oh as f64);
|
||||
let mut out = Image::new(ow, oh);
|
||||
lightcraft_raster::par_rows(&mut out.data, ow, |y, row| {
|
||||
let ay = c.y as f64 + (y as f64 + 0.5) * ky;
|
||||
for (x, v) in row.iter_mut().enumerate() {
|
||||
let ax = c.x as f64 + (x as f64 + 0.5) * kx;
|
||||
*v = (self.value_at(ax * sx, ay * sy) * 255.0).round() as u8;
|
||||
}
|
||||
});
|
||||
Some(out)
|
||||
}
|
||||
}
|
||||
@@ -300,6 +300,33 @@ fn corpus_dngs_carry_profile_looks() {
|
||||
eprintln!("{seen} DNGs with profile looks checked");
|
||||
}
|
||||
|
||||
/// iPhone ProRAW (`dng-apple-iphone12pro-proraw.dng`, iPhone 12 Pro, CC0 from raw.pixls.us, see
|
||||
/// `cargo xtask corpus`) carries a lossy-JPEG sky matte at half resolution as a DNG semantic mask:
|
||||
/// it decodes, and it is a sky. The matte is stored like the raw (landscape; the photo is shown
|
||||
/// rotated 90° clockwise), so the sky is on its left and the lake and shore on its right.
|
||||
#[test]
|
||||
fn corpus_proraw_sky_matte() {
|
||||
let path = corpus_root().join("raw/dng-apple-iphone12pro-proraw.dng");
|
||||
let Ok(bytes) = std::fs::read(&path) else {
|
||||
eprintln!("skip: {} absent", path.display());
|
||||
return;
|
||||
};
|
||||
let t0 = Instant::now();
|
||||
let masks = lightcraft_raw::semantic_masks(&bytes);
|
||||
eprintln!("semantic masks read in {:.1} ms", t0.elapsed().as_secs_f64() * 1e3);
|
||||
let names: Vec<&str> = masks.iter().map(|m| m.name.as_str()).collect();
|
||||
assert_eq!(names, ["urn:com:apple:photo:2020:aux:semanticskymatte"]);
|
||||
let m = &masks[0];
|
||||
assert_eq!((m.width, m.height, m.sub_area), (2016, 1512, None));
|
||||
let cols = |x0: usize, x1: usize| {
|
||||
let sum: f64 = (0..m.height).flat_map(|y| m.data[y * m.width + x0..y * m.width + x1].iter()).map(|&v| v as f64 / 65535.0).sum();
|
||||
sum / ((x1 - x0) * m.height) as f64
|
||||
};
|
||||
let (left, right) = (cols(0, 100), cols(1916, 2016));
|
||||
eprintln!("sky matte: left columns {left:.3}, right columns {right:.3}");
|
||||
assert!(left > 0.5 && right < 0.02, "left {left}, right {right}");
|
||||
}
|
||||
|
||||
/// Issue #148: Sony ARWs from before ~2017 carry no plain white-balance, black-level or crop tags in the raw IFD.
|
||||
/// White balance comes from the maker note's enciphered `Tag2010`, the black level from the encrypted `SR2SubIFD`
|
||||
/// and the crop from `FullImageSize`; without them the RX100 III opened bright green. Expected values: the black
|
||||
|
||||
@@ -824,3 +824,130 @@ fn gain_table_map_byte_orders() {
|
||||
assert_eq!(img.color.profile.gain_table_map.map(|m| m.gains), read.then(|| vec![1.0, 2.0]), "{data:?} payload, {file:?} file");
|
||||
}
|
||||
}
|
||||
|
||||
/// A semantic mask IFD (8-bit, uncompressed) named `name`.
|
||||
fn mask_ifd(w: usize, h: usize, name: &str, px: Vec<u8>) -> IfdBuilder {
|
||||
let mut m = IfdBuilder::new();
|
||||
m.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![t::SUBFILE_SEMANTIC_MASK]));
|
||||
m.set(t::IMAGE_WIDTH, Value::Long(vec![w as u32]));
|
||||
m.set(t::IMAGE_LENGTH, Value::Long(vec![h as u32]));
|
||||
m.set(t::BITS_PER_SAMPLE, Value::Short(vec![8]));
|
||||
m.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![1]));
|
||||
m.set(t::PHOTOMETRIC, Value::Short(vec![photometric::MASK]));
|
||||
m.set(t::COMPRESSION, Value::Short(vec![1]));
|
||||
if !name.is_empty() {
|
||||
m.set(t::SEMANTIC_NAME, Value::Ascii(name.into()));
|
||||
}
|
||||
m.set_image(ImageData::Strips { rows_per_strip: h as u32, strips: vec![px] });
|
||||
m
|
||||
}
|
||||
|
||||
/// Semantic masks: a cropped mask is placed by its `MaskSubArea` (zero outside it), mapped onto the
|
||||
/// developed image (default crop of the active area), and doesn't disturb the raw decode.
|
||||
#[test]
|
||||
fn semantic_masks_are_read_and_placed() {
|
||||
use lightcraft_raw::semantic::MaskSubArea;
|
||||
for order in [ByteOrder::Little, ByteOrder::Big] {
|
||||
let bytes = profile_dng(order, |ifd0| {
|
||||
// a 4 × 2 crop at (top 1, left 2) of an 8 × 4 mask over the 8 × 6 image
|
||||
let mut sky = mask_ifd(4, 2, "sky", vec![10, 20, 30, 40, 50, 60, 70, 255]);
|
||||
sky.set(t::MASK_SUB_AREA, Value::Long(vec![1, 2, 8, 4]));
|
||||
ifd0.add_sub_ifd(sky);
|
||||
let mut skin = mask_ifd(2, 2, "skin", vec![0, 255, 255, 0]);
|
||||
skin.set(t::SEMANTIC_INSTANCE_ID, Value::Ascii("person_a".into()));
|
||||
ifd0.add_sub_ifd(skin);
|
||||
});
|
||||
let raw = decode(&bytes).unwrap();
|
||||
assert_eq!((raw.width, raw.height, raw.cpp), (8, 6, 3), "the masks are not the raw image");
|
||||
let masks = lightcraft_raw::semantic_masks(&bytes);
|
||||
assert_eq!(masks.len(), 2);
|
||||
let (sky, skin) = (&masks[0], &masks[1]);
|
||||
assert_eq!((sky.name.as_str(), sky.instance_id.as_deref(), sky.width, sky.height), ("sky", None, 4, 2));
|
||||
assert_eq!(sky.sub_area, Some(MaskSubArea { top: 1, left: 2, full_width: 8, full_height: 4 }));
|
||||
assert_eq!((sky.data[0], sky.data[7]), (10 * 257, 65535));
|
||||
assert_eq!((skin.name.as_str(), skin.instance_id.as_deref(), skin.sub_area), ("skin", Some("person_a"), None));
|
||||
// positions are in uncropped-mask pixels: zero outside the stored crop
|
||||
assert!((sky.value_at(2.5, 1.5) - 10.0 / 255.0).abs() < 1e-4);
|
||||
assert!((sky.value_at(5.5, 2.5) - 1.0).abs() < 1e-4);
|
||||
assert_eq!((sky.value_at(1.5, 1.5), sky.value_at(2.5, 0.5), sky.value_at(6.5, 1.5)), (0.0, 0.0, 0.0));
|
||||
// over the whole image: the uncropped mask's 8 × 4 grid
|
||||
let full = sky.developed(Rect::new(0, 0, 8, 6), Rect::new(0, 0, 8, 6)).unwrap();
|
||||
assert_eq!((full.width, full.height), (8, 4));
|
||||
assert_eq!((full.get(2, 1), full.get(5, 2), full.get(1, 1), full.get(6, 2), full.get(2, 0)), (10, 255, 0, 0, 0));
|
||||
// a default crop keeping columns 2..6 of the image: the stored crop fills it
|
||||
let cropped = sky.developed(Rect::new(0, 0, 8, 6), Rect::new(2, 0, 4, 6)).unwrap();
|
||||
assert_eq!((cropped.width, cropped.height), (4, 4));
|
||||
assert_eq!((cropped.get(0, 1), cropped.get(3, 2), cropped.get(0, 0)), (10, 255, 0));
|
||||
}
|
||||
}
|
||||
|
||||
/// Malformed semantic masks are skipped (never a panic), and so is the `MaskSubArea` of a mask it
|
||||
/// doesn't fit; the raw image still decodes.
|
||||
#[test]
|
||||
fn malformed_semantic_masks_are_skipped() {
|
||||
let bytes = profile_dng(ByteOrder::Little, |ifd0| {
|
||||
ifd0.add_sub_ifd(mask_ifd(2, 2, "", vec![1, 2, 3, 4])); // no SemanticName
|
||||
let mut rgb = mask_ifd(2, 2, "rgb", vec![0; 12]);
|
||||
rgb.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![3]));
|
||||
rgb.set(t::BITS_PER_SAMPLE, Value::Short(vec![8, 8, 8]));
|
||||
ifd0.add_sub_ifd(rgb);
|
||||
let mut huge = mask_ifd(2, 2, "huge", vec![0; 4]);
|
||||
huge.set(t::IMAGE_WIDTH, Value::Long(vec![100_000]));
|
||||
huge.set(t::IMAGE_LENGTH, Value::Long(vec![100_000]));
|
||||
ifd0.add_sub_ifd(huge);
|
||||
let mut tiny = mask_ifd(2, 2, "tiny data", vec![0; 4]);
|
||||
tiny.set(t::IMAGE_WIDTH, Value::Long(vec![4000]));
|
||||
tiny.set(t::IMAGE_LENGTH, Value::Long(vec![3000]));
|
||||
ifd0.add_sub_ifd(tiny);
|
||||
let mut jxl = mask_ifd(2, 2, "unknown coding", vec![0; 4]);
|
||||
jxl.set(t::COMPRESSION, Value::Short(vec![60_000]));
|
||||
ifd0.add_sub_ifd(jxl);
|
||||
let mut zero = mask_ifd(2, 2, "zero size", vec![0; 4]);
|
||||
zero.set(t::IMAGE_WIDTH, Value::Long(vec![0]));
|
||||
ifd0.add_sub_ifd(zero);
|
||||
let mut short = mask_ifd(4, 4, "short strip", vec![200; 3]);
|
||||
short.set(t::MASK_SUB_AREA, Value::Long(vec![u32::MAX, u32::MAX, u32::MAX, u32::MAX]));
|
||||
ifd0.add_sub_ifd(short);
|
||||
let mut outside = mask_ifd(2, 2, "sub area too small", vec![255; 4]);
|
||||
outside.set(t::MASK_SUB_AREA, Value::Long(vec![3, 0, 2, 4]));
|
||||
ifd0.add_sub_ifd(outside);
|
||||
});
|
||||
assert_eq!(decode(&bytes).unwrap().cpp, 3);
|
||||
let masks = lightcraft_raw::semantic_masks(&bytes);
|
||||
let names: Vec<&str> = masks.iter().map(|m| m.name.as_str()).collect();
|
||||
assert_eq!(names, ["short strip", "sub area too small"]);
|
||||
assert!(masks.iter().all(|m| m.sub_area.is_none()), "sub areas the mask doesn't fit are ignored");
|
||||
for m in &masks {
|
||||
let d = m.developed(Rect::new(0, 0, 8, 6), Rect::new(0, 0, 8, 6)).unwrap();
|
||||
assert_eq!((d.width, d.height), (m.width, m.height));
|
||||
assert!(m.developed(Rect::new(0, 0, 8, 6), Rect::new(8, 6, 0, 0)).is_none());
|
||||
}
|
||||
// not a TIFF at all
|
||||
assert!(lightcraft_raw::semantic_masks(b"not a dng").is_empty());
|
||||
}
|
||||
|
||||
/// Only the first `MAX_MASKS` mask IFDs are decoded, malformed ones included, so a file full of
|
||||
/// mask IFDs costs a bounded number of decodes: here the first `MAX_MASKS` can't be decoded, and
|
||||
/// the valid masks after them are never looked at.
|
||||
#[test]
|
||||
fn semantic_mask_ifds_past_the_cap_are_not_decoded() {
|
||||
use lightcraft_raw::semantic::MAX_MASKS;
|
||||
let with = |undecodable: usize, valid: usize| {
|
||||
profile_dng(ByteOrder::Little, |ifd0| {
|
||||
for i in 0..undecodable {
|
||||
let mut m = mask_ifd(2, 2, &format!("undecodable {i}"), vec![0; 4]);
|
||||
m.set(t::COMPRESSION, Value::Short(vec![60_000]));
|
||||
ifd0.add_sub_ifd(m);
|
||||
}
|
||||
for i in 0..valid {
|
||||
ifd0.add_sub_ifd(mask_ifd(2, 2, &format!("valid {i}"), vec![255; 4]));
|
||||
}
|
||||
})
|
||||
};
|
||||
let bytes = with(MAX_MASKS, 3 * MAX_MASKS);
|
||||
assert_eq!(decode(&bytes).unwrap().cpp, 3);
|
||||
assert!(lightcraft_raw::semantic_masks(&bytes).is_empty());
|
||||
// one fewer undecodable mask leaves room for exactly one valid one
|
||||
let names: Vec<String> = lightcraft_raw::semantic_masks(&with(MAX_MASKS - 1, 3 * MAX_MASKS)).into_iter().map(|m| m.name).collect();
|
||||
assert_eq!(names, ["valid 0"]);
|
||||
}
|
||||
|
||||
@@ -180,6 +180,12 @@ pub const FORWARD_MATRIX_3: u16 = 52532;
|
||||
pub const ILLUMINANT_DATA_1: u16 = 52533;
|
||||
pub const ILLUMINANT_DATA_2: u16 = 52534;
|
||||
pub const ILLUMINANT_DATA_3: u16 = 52535;
|
||||
// DNG 1.6 semantic masks (spec chapter 4, "Semantic Masks")
|
||||
pub const SEMANTIC_NAME: u16 = 52526;
|
||||
pub const SEMANTIC_INSTANCE_ID: u16 = 52528;
|
||||
pub const MASK_SUB_AREA: u16 = 52536;
|
||||
/// NewSubFileType of a semantic mask IFD.
|
||||
pub const SUBFILE_SEMANTIC_MASK: u32 = 0x10004;
|
||||
pub const PROFILE_GAIN_TABLE_MAP_2: u16 = 52544;
|
||||
|
||||
// --- Compression codes (TIFF 6.0, DNG, and widely used registrations)
|
||||
@@ -208,4 +214,6 @@ pub mod photometric {
|
||||
pub const YCBCR: u16 = 6;
|
||||
pub const CFA: u16 = 32803;
|
||||
pub const LINEAR_RAW: u16 = 34892;
|
||||
/// DNG 1.6 semantic mask.
|
||||
pub const MASK: u16 = 52527;
|
||||
}
|
||||
|
||||
+4
-4
@@ -341,11 +341,11 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
|
||||
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|
||||
|---|---|---|---|---|---|
|
||||
| LR-MASK-PANEL | Masks panel | P0 | ✅ | `cmd:panel.masking`, `cmd:mask.add`, `cmd:mask.select`, `cmd:mask.rename`, `cmd:mask.duplicate` (`invert`), `cmd:mask.move`, `cmd:mask.visible`, `cmd:mask.delete`, `crates/ui-egui/src/panels/masking.rs` | list with per-mask show/hide eye, double-click rename, right-click menu (duplicate, duplicate and invert, invert, hide, move up/down, rename, delete); fits any panel width (create tiles go to three columns, mask actions wrap, long component names are cut short); no drag-to-reorder |
|
||||
| LR-MASK-SUBJECT | Select subject | P2 | 🟡 | `cmd:mask.add` (`subject`), `crates/pipeline/src/masks.rs` | saliency heuristic, no segmentation model |
|
||||
| LR-MASK-SKY | Select sky | P2 | 🟡 | `cmd:mask.add` (`sky`) | heuristic |
|
||||
| LR-MASK-BACKGROUND | Select background | P2 | 🟡 | `cmd:mask.add` (`background`) | inverse of the subject heuristic |
|
||||
| LR-MASK-SUBJECT | Select subject | P2 | 🟡 | `cmd:mask.add` (`subject`), `crates/pipeline/src/masks.rs` | saliency heuristic, no segmentation model; a photo that carries person mattes (DNG semantic masks: iPhone ProRAW portrait / skin / hair / teeth / glasses mattes) uses them instead (synthetic files only: no corpus file carries person mattes) |
|
||||
| LR-MASK-SKY | Select sky | P2 | 🟡 | `cmd:mask.add` (`sky`), `crates/pipeline/src/masks.rs` (`Mattes`), `crates/raw/src/semantic.rs` | heuristic; a photo that carries a sky matte (DNG semantic mask, e.g. iPhone ProRAW) uses it instead, through orientation / crop / lens corrections, preview = export (checked on one CC0 iPhone 12 Pro ProRAW and synthetic files) |
|
||||
| LR-MASK-BACKGROUND | Select background | P2 | 🟡 | `cmd:mask.add` (`background`) | inverse of the subject mask (heuristic, or the photo's person mattes) |
|
||||
| LR-MASK-OBJECTS | Object selection | P2 | 🟡 | `cmd:mask.add` (`object`, `prompt`), `cmd:mask.objectPoint`, `cmd:mask.refineDetail`, `cmd:segment.prepare`, `cmd:segment.model.status`, `cmd:segment.model.download`, `cmd:segment.model.cancel`, `crates/segment`, `crates/engine/src/segment/mod.rs` | SAM 3 in pure Rust (candle; Metal on macOS, CPU elsewhere): Object tile → click to include, ⌥-click to leave out; Describe tile → a text prompt selects every instance ("sky", "the red car"); both also as Add/Subtract/Intersect components; + / − under the selected mask; comma lists (`car, road`); a zoomed-in detail pass for 5–10× finer edges on small objects; per-selection Edge (hard ↔ soft). The model runs on its own worker thread (the UI never waits; panics become errors), is unloaded after 10 min idle, and is optional: the segmentation is stored with the mask (288² logits), so renders and exports never need it. Weights are never bundled (SAM License): without them the app offers a consented background download (mirrors, resume, timeouts, SHA-256) — **but no default download location is configured yet** (users set `LIGHTCRAFT_SAM3_MIRRORS` or install by hand: the dialog names the three files, shows the model folder and links docs/ai-masks.md). Not verified against Lightroom's Select Object; no brush/box object mode; the first click on a photo waits for its analysis (~4 s on an M4 Pro, much longer on CPU) |
|
||||
| LR-MASK-PEOPLE | People parts | P2 | ⬜ | | |
|
||||
| LR-MASK-PEOPLE | People parts | P2 | ⬜ | | shape exists (no UI): uses the photo's person / skin / hair / teeth / glasses mattes when it carries them (DNG semantic masks), else the subject heuristic |
|
||||
| LR-MASK-LANDSCAPE | Landscape classes | P2 | ⬜ | | shape exists, evaluates empty |
|
||||
| LR-MASK-BRUSH | Brush mask | P0 | ✅ | `cmd:tool.brush`, `cmd:mask.brushStroke` (`autoMask`), `crates/pipeline/src/masks.rs` | size/feather/flow/density/erase; Auto Mask: dabs weighted by similarity to the colour under the dab centre, refined by a guided filter on luminance (CPU + GPU); no A/B brushes, no pressure |
|
||||
| LR-MASK-LINEAR | Linear gradient | P0 | ✅ | `cmd:tool.linear`, `cmd:mask.update` | |
|
||||
|
||||
@@ -22,6 +22,9 @@
|
||||
embedded JPEG.
|
||||
- Apple ProRAW's gain table map (its local tone mapping, `ProfileGainTableMap`) is now read and kept when a photo is
|
||||
exported or converted to DNG. It is not applied to the render: Lightroom Classic renders ProRAW without it.
|
||||
- iPhone ProRAW and other DNGs that carry their own segmentation mattes (DNG semantic masks) use them for the Select
|
||||
Sky, Subject and Background masks instead of our heuristics, so the sky is selected where the camera found it
|
||||
(checked on one CC0 iPhone 12 Pro ProRAW). Photos without mattes are unchanged.
|
||||
- Panasonic and Leica raws (RW2, RWL) are now corrected for lens distortion the way the camera corrects its own JPEG
|
||||
(issue #256): the correction the camera records in the file is applied under Lens Corrections ▸ Enable Profile
|
||||
Corrections, on by default for newly imported photos, with the same framing as the camera's JPEG. At 12 mm the
|
||||
|
||||
Reference in New Issue
Block a user