security(compose): bound dense mask working set

build_mask derives its working area from sparse mask geometry grown by
feather, so two distant changed pixels could span a canvas-sized
rectangle. It then allocated f32 planes (coverage, pixel-mask values,
blur scratch) over that area, gigabytes during even a one-pixel render.

Check the estimated working bytes before the dense allocation and fall
back to the constant default mask above 512 MiB, so pathological
geometry degrades feather detail instead of exhausting memory.

Finding: 8c04c5ef-a5bd-48f4-bfcf-83615bcf11c0 (CWE-770)

crates/compose/src/masks.rs

@@ -221,7 +221,14 @@ fn build_mask(layer: &Layer, canvas: Rect) -> Surface {
     }
     let area = area.intersect(&canvas);
     let mut s = Surface::with_default(FORMAT, &[p_def * v_out]);
-    if !area.is_empty() {
+    // Bound the transient dense working set: `area` is derived from sparse mask geometry grown
+    // by feather, so two distant changed pixels could span a canvas-sized rectangle. Building
+    // f32 planes over it (coverage, pixel-mask values, blur scratch) could be gigabytes during
+    // a one-pixel render. Above the budget, fall back to the constant default mask rather than
+    // allocating; feather detail is lost only for pathological geometry.
+    const MAX_MASK_WORKING_BYTES: u64 = 512 << 20;
+    let working = (area.width() as u64).saturating_mul(area.height() as u64).saturating_mul(std::mem::size_of::<f32>() as u64).saturating_mul(3);
+    if !area.is_empty() && working <= MAX_MASK_WORKING_BYTES {
         let (w, h) = (area.width() as usize, area.height() as usize);
         // A slot initializer must not enter Rayon: another tile requesting the
         // same mask could be stolen onto its stack and wait for its own build.