Image Trace: Overlapping layers reach only under areas that hold their own color

With Method Overlapping, each color layer covered every pixel of the colors
above it in the stack. A small dot or square on a background was then traced
once in its own color and again, at the same bounds, in every color between
the background and it, and with Shapes on each copy became its own live shape.
Those layers also ended on edges they never touched and showed through the
anti-aliased rim there.

Each layer now leaves out every 4-connected part of its stacked area that holds
no pixel of its own color. This changes the result of many Overlapping traces,
the photo presets included: fewer paths (High Fidelity Photo 121 to 69.6 on its
own quality inputs, Low Fidelity Photo 37.1 to 26.1) and no rim, around a
shape, of a color that sits apart from it. One existing test that counted the
extra copies now expects one layer per area, and another that compared layer
pixel counts now checks that the bottom layer covers the whole image.

crates/trace/src/contour.rs

@@ -48,7 +48,7 @@ fn step(d: u8) -> (i32, i32) {
 }
 
 /// 4-connected component labels (`u32::MAX` outside) and per-component pixel counts.
-fn label(mask: &[bool], w: usize, h: usize) -> (Vec<u32>, Vec<usize>) {
+pub(crate) fn label(mask: &[bool], w: usize, h: usize) -> (Vec<u32>, Vec<usize>) {
     let mut comp = vec![u32::MAX; w * h];
     let mut sizes = Vec::new();
     // (Indices as `u32`: a mask has at most 64 million pixels, and this stack can hold them all.)

crates/trace/src/lib.rs

@@ -40,7 +40,8 @@
 //! parts become stroked outlines.
 //!
 //! Colour layers are traced either *abutting* (each colour's own area; shapes share edges) or
-//! *overlapping* (stacked: each layer also covers every layer above it, so no hairline gaps).
+//! *overlapping* (stacked: each layer also covers the layers above it within the areas that hold
+//! its own color, so no hairline gaps).
 #![forbid(unsafe_code)]
 
 mod centerline;
@@ -234,7 +235,8 @@ pub enum Method {
     /// Each colour traced on its own; neighbouring shapes share their edge.
     #[default]
     Abutting,
-    /// Stacked: every layer extends under the layers above it.
+    /// Stacked: every layer extends under the layers above it, within the areas that hold its own
+    /// color.
     Overlapping,
 }
 
@@ -603,10 +605,11 @@ fn trace_paths(img: &Raster, params: &TraceParams, max_anchors: usize) -> Result
     order.sort_by(|a, b| counts[*b].cmp(&counts[*a]).then(a.cmp(b)));
     if params.method == Method::Overlapping {
         // Stacked layers go by lightness instead, from the largest colour's end of the scale: each
-        // layer reaches under all those above it, so where two colours meet, every layer between
-        // them in the stack ends on that same edge and shows through its anti-aliasing. Neighbours
-        // in a picture are mostly neighbours in tone, and in this order they are neighbours in the
-        // stack too (by area, an unrelated colour's edge would shine through as a hairline).
+        // layer reaches under those above it in the areas that hold its own color, so where two
+        // colours meet, every layer between them in the stack that has a pixel in that area ends
+        // on that same edge and shows through its anti-aliasing. Neighbours in a picture are
+        // mostly neighbours in tone, and in this order they are neighbours in the stack too (by
+        // area, an unrelated colour's edge would shine through as a hairline).
         let light = |i: usize| q.palette.get(i).map_or(0, |c| u32::from(c[0]) * 299 + u32::from(c[1]) * 587 + u32::from(c[2]) * 114);
         let first = order.first().copied();
         order.sort_by(|a, b| light(*a).cmp(&light(*b)).then(a.cmp(b)));
@@ -639,6 +642,8 @@ fn trace_paths(img: &Raster, params: &TraceParams, max_anchors: usize) -> Result
     // A layer takes about ten bytes a pixel of its box while it is traced (its mask, the labels
     // of its components, the stack that fills them, its edges): as many layers at a time as fit
     // in half a gigabyte, so that a large image is not traced at the cost of gigabytes.
+    // (Overlapping labels a layer's mask once more before it is traced, with its labels and stack
+    // freed before the trace starts.)
     let largest = job.boxes.iter().flatten().map(|b| (b.2 - b.0) * (b.3 - b.1)).max().unwrap_or(1).max(1);
     let at_a_time = ((512 << 20) / (largest * 10)).max(1);
     for (paths, &(_, ci)) in parallel_map_at_most(at_a_time, &layers, |&(r, ci)| job.layer(r, ci)).into_iter().zip(&layers) {
@@ -965,6 +970,23 @@ impl Layers<'_> {
             let row = self.q.labels.get(y * w + x0..y * w + x1)?;
             mask.extend(row.iter().enumerate().map(|(k, &l)| l != TRANSPARENT && !thin.get(y * w + x0 + k).copied().unwrap_or(false) && part(l)));
         }
+        // Overlapping: a part of the mask with no pixel of this color holds only colors above it,
+        // whose own layers fill it, and is left out. The layer reaches under the colors above it
+        // only within the areas that hold its own color.
+        if overlapping {
+            let (comp, sizes) = contour::label(&mask, bw, bh);
+            let mut own = vec![false; sizes.len()];
+            for (k, &c) in comp.iter().enumerate() {
+                if self.q.labels.get((y0 + k / bw) * w + x0 + k % bw).is_some_and(|&l| l as usize == ci)
+                    && let Some(o) = own.get_mut(c as usize)
+                {
+                    *o = true;
+                }
+            }
+            for (m, &c) in mask.iter_mut().zip(&comp) {
+                *m = *m && own.get(c as usize).copied().unwrap_or(false);
+            }
+        }
         // Without Fills, areas are outlined with a 1 px stroke.
         let area_stroke = (params.strokes && !params.fills).then_some(1.0);
         let min_hole = params.noise.max(1) as i64;

crates/trace/src/tests.rs

@@ -214,6 +214,70 @@ fn overlapping_layers_stack() {
     assert_eq!(ov.paths[1].color, [0, 0, 0]);
 }
 
+/// Overlapping with Shapes: a black disc on the light half is written once. The blue layer, between
+/// light and black in the stack, does not reach under it: the disc holds no blue.
+#[test]
+fn overlapping_writes_no_copy_of_a_shape_under_a_color_it_does_not_touch() {
+    let (light, blue) = ([240, 240, 220, 255], [40, 120, 240, 255]);
+    let img = Raster::from_fn(600, 400, |x, y| {
+        let disc = |cx: f64, cy: f64| Point::new(f64::from(x) + 0.5, f64::from(y) + 0.5).distance(Point::new(cx, cy)) <= 12.0;
+        if disc(100.0, 100.0) || disc(200.0, 300.0) || (400..500).contains(&x) && (150..250).contains(&y) {
+            BLACK
+        } else if x >= 300 {
+            blue
+        } else {
+            light
+        }
+    });
+    let params = TraceParams {
+        mode: Mode::Color,
+        palette: Palette::Limited,
+        colors: 3,
+        paths: 90.0,
+        corners: 50.0,
+        noise: 2,
+        method: Method::Overlapping,
+        shapes: true,
+        ..TraceParams::default()
+    };
+    let res = trace(&img, &params);
+    let ellipses = res.paths.iter().filter(|p| matches!(p.shape, Some(TracedShape::Ellipse { .. }))).count();
+    let rectangles = res.paths.iter().filter(|p| matches!(p.shape, Some(TracedShape::Rectangle { .. }))).count();
+    // The two discs; the light ground, the blue half (the black square touches it) and the square.
+    assert_eq!((ellipses, rectangles), (2, 3), "{:?}", res.paths.iter().map(|p| (p.color, p.shape)).collect::<Vec<_>>());
+    let bounds: Vec<_> = res.paths.iter().filter(|p| p.shape.is_some()).filter_map(|p| p.path.bounds()).collect();
+    for (i, a) in bounds.iter().enumerate() {
+        for b in &bounds[i + 1..] {
+            let same = [(a.x0, b.x0), (a.y0, b.y0), (a.x1, b.x1), (a.y1, b.y1)].iter().all(|(u, v)| (u - v).abs() <= 0.5);
+            assert!(!same, "two shapes at {a:?}");
+        }
+    }
+}
+
+/// Overlapping: a black line on the light ground gets no blue path under it, though blue lies
+/// between light and black in the stack.
+#[test]
+fn overlapping_layers_leave_out_areas_without_their_color() {
+    let (light, blue) = ([240, 240, 220, 255], [40, 120, 240, 255]);
+    let img = Raster::from_fn(400, 400, |x, y| {
+        if (170..230).contains(&x) && (170..230).contains(&y) {
+            blue
+        } else if (50..350).contains(&x) && (320..330).contains(&y) {
+            BLACK
+        } else {
+            light
+        }
+    });
+    let params = TraceParams { mode: Mode::Color, palette: Palette::Limited, colors: 3, method: Method::Overlapping, ..TraceParams::default() };
+    let res = trace(&img, &params);
+    let blues: Vec<_> = res.paths.iter().filter(|p| p.color[0] < 100 && p.color[2] > 150).collect();
+    assert!(!blues.is_empty(), "{:?}", res.palette);
+    for p in blues {
+        let b = p.path.bounds().unwrap();
+        assert!(b.x1 <= 50.0 || b.x0 >= 350.0 || b.y1 <= 320.0 || b.y0 >= 330.0, "a blue path over the line: {b:?}");
+    }
+}
+
 #[test]
 fn transparent_pixels_are_not_traced() {
     let img = Raster::from_fn(40, 40, |x, _| if x < 20 { [0, 0, 0, 0] } else { BLACK });
@@ -474,9 +538,8 @@ fn parallel_layers_keep_their_order() {
     for _ in 0..4 {
         assert_eq!(trace(&img, &p), a);
     }
-    // Overlapping: each layer's shapes cover the layers above, so the pixel counts fall upwards.
-    let firsts: Vec<usize> = a.palette.iter().filter_map(|c| a.paths.iter().find(|t| t.color == *c).map(|t| t.pixels)).collect();
-    assert!(firsts.windows(2).all(|w| w[0] >= w[1]), "{firsts:?}");
+    // Overlapping: the bottom layer comes first and covers the whole image.
+    assert_eq!(a.paths.first().map(|t| t.pixels), Some(90 * 90));
 }
 
 /// A PNG whose header says `w` × `h` pixels, with no image data.
@@ -1099,11 +1162,12 @@ fn stacked_layers_go_by_lightness_and_leave_ignored_white_alone() {
     let params = TraceParams { mode: Mode::Grayscale, colors: 4, noise: 0, method: Method::Overlapping, ..TraceParams::default() };
     let tones = |res: &TraceResult| res.paths.iter().map(|p| p.color[0]).collect::<Vec<u8>>();
     // The largest colour is light, so the stack starts at the light end.
-    // (Each layer reaches under those above it: under the black stripe too, a part of its own.)
-    assert_eq!(tones(&trace(&img, &params)), [255, 200, 200, 120, 120, 0]);
+    // (Each layer reaches under those above it only where it is part of the area: the black stripe
+    // holds no gray, so no gray layer reaches under it.)
+    assert_eq!(tones(&trace(&img, &params)), [255, 200, 120, 0]);
     // Without white, no layer reaches under where it was.
     let res = trace(&img, &TraceParams { ignore_white: true, ..params });
-    assert_eq!(tones(&res), [200, 200, 120, 120, 0]);
+    assert_eq!(tones(&res), [200, 120, 0]);
     for p in &res.paths {
         let b = p.path.bounds().unwrap();
         assert!(b.x1 <= 20.5 || b.x0 >= 39.5, "a {} layer covers the white stripe: {b:?}", p.color[0]);

docs/target-app-parity.md

@@ -796,6 +796,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
 - **Image Trace small contours:** closed fits too small to form an editable subpath retain their positioned outline, including holes. Source-generated island, hole/winding and stepped-region regressions cover Corners and Noise; defaults, parity scores and estimates are unchanged.
 - **Create Strokes baseline (2026-10-17, measured):** the ignored centre-lines class covers 23 exact code-drawn scenes in aa and hard variants, including junctions, crossings, fills meeting strokes and varying widths. [Baseline and metric definitions](image-trace-quality.md#centre-lines). Topology is recovered from emitted geometry; width-profile metrics remain unavailable. No behaviour change or score increase.
 - **Image Trace, collapsed loops keep few segments:** a loop whose fit collapses to one cubic keeps the fewest segments that hold its area and course, so a higher Paths gives no fewer segments; a code-drawn Paths sweep in `crates/trace` (see [quality notes](image-trace-quality.md#collapsed-loops-keep-few-segments)).
+- **Image Trace, stacked layers only under their own areas:** with Method Overlapping, a layer reaches under the colors above it only within the connected areas that hold its own color, so a shape that sits apart from a color is no longer written again, at the same bounds, in that color, and the edge of that color's layer no longer shows through that shape's anti-aliased rim. *Measured* by `crates/trace/tests/quality` on each preset's own inputs: High Fidelity Photo 121 paths to 69.6 with ΔE 1.74 to 1.72, Low Fidelity Photo 37.1 paths to 26.1. Code-drawn disc, square and line tests in `crates/trace`.
 - **Image Trace, notches read only their own piece:** the notch rule of `subpixel::outline` reads only the shared piece, so both regions of an edge cut a corner pixel alike and leave no gap; a code-drawn noisy multi-colour coverage test in `crates/trace` (see [quality notes](image-trace-quality.md#notches-read-only-their-own-piece)).
 - **Image Trace, same shift from either side of an edge:** a crack's sub-pixel shift is averaged from its two sides, so abutting regions on one- and two-pixel slivers get the same points and leave nothing uncovered; code-drawn regressions in `crates/trace` (see [quality notes](image-trace-quality.md#same-sub-pixel-shift-from-either-side-of-an-edge)).
 - **Image Trace, cut-out edges:** in Color and Grayscale, a semi-transparent pixel (alpha 128 to 254) on the edge of a cut-out image, beside a pixel that is less than half opaque, traces as a color of the fully opaque pixels within two pixels of it when its own color is that color or a darker shade of it, and of several such colors, as the one the most of those pixels have (art across a transparent gap does not count), so a darkened edge tone no longer traces as a 1 px sliver of an unrelated palette color, while a thin semi-transparent outline of another color keeps its own; code-drawn cut-out, opaque-edge and thin-outline tests in `crates/trace`.