Cryptomatte: effect.pickCryptomatte picks the ID under a point into the Selection (#473)

Engine part of viewer picking: replace / add / remove the top-coverage ID at a
layer-space point in one undo step, at the time the Selection was read. The
aux channels come from the render's own path (Renderer::layer_aux, shared with
the effect host; footage at its source time, 3D precomps) through a renderer
set up like Session::render (expressions, render prefs, layer cache). An
optional scale snaps the point to the pixel a preview at that resolution
samples. New effects::crypto_select quotes names and falls back to <hex> for
unnamed, wildcard-looking or ambiguous names. effect.cryptomatteAt is the
read-only hover query.

crates/effects/src/channel3d.rs

@@ -326,7 +326,7 @@ fn wild(pat: &str, name: &str) -> bool {
 /// Split a selection at its commas. A name in double or single quotes stays whole, commas
 /// included (Blender allows `Green, Glossy`), and a backslash takes the next character
 /// literally (`Green\, Glossy`, `\"`). Unquoted text is trimmed; the quotes are dropped.
-fn selection_names(sel: &str) -> Vec<String> {
+pub(crate) fn selection_names(sel: &str) -> Vec<String> {
     let mut out = vec![];
     let mut cur = String::new();
     let mut quote: Option<char> = None;

crates/effects/src/crypto_select.rs

@@ -0,0 +1,122 @@
+//! Editing a Cryptomatte effect's Selection text from a pick in the viewer (#473): replace it
+//! with the picked ID, add it, or remove it, in the grammar `selection_hashes` reads.
+
+use crate::channel3d::{selection_hashes, selection_names};
+
+/// How a pick changes the selection.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum PickMode {
+    Replace,
+    Add,
+    Remove,
+}
+
+/// A name as a selection token: quoted (with `\` escapes) when it has a comma, a quote or a
+/// backslash, which the selection parser would otherwise split on or read as syntax.
+fn quote(name: &str) -> String {
+    if !name.contains([',', '"', '\'', '\\']) {
+        return name.to_string();
+    }
+    let mut out = String::from("\"");
+    for c in name.chars() {
+        if c == '"' || c == '\\' {
+            out.push('\\');
+        }
+        out.push(c);
+    }
+    out.push('"');
+    out
+}
+
+/// The selection token for the ID `hash` named `name` in `manifest`: the (quoted) name when it
+/// reads back as exactly that ID, else `<hex>`. That covers unnamed IDs, names with `*` (which
+/// would select by wildcard), names that look like `<hex>`, names with surrounding spaces (the
+/// parser trims them) and names shared by several IDs.
+pub fn token_for(name: Option<&str>, hash: u32, manifest: Option<&[(String, u32)]>) -> String {
+    match name {
+        Some(n) if !n.is_empty() && !n.contains('*') && selection_hashes(&quote(n), manifest) == [hash] => quote(n),
+        _ => format!("<{hash:08x}>"),
+    }
+}
+
+/// `sel` edited by a pick of the ID `hash` whose token is `token` (see [`token_for`]). The text
+/// stays as typed when the edit wouldn't change what it selects: Replace when it selects just
+/// that ID, Add when it already selects it, Remove when no name or `<hex>` token selects it (a
+/// wildcard that matches it stays). Other tokens (wildcards included) keep their text,
+/// re-quoted where needed, separated by `, `.
+pub fn edit_selection(sel: &str, token: &str, hash: u32, mode: PickMode, manifest: Option<&[(String, u32)]>) -> String {
+    let tokens: Vec<String> = selection_names(sel).into_iter().filter(|t| !t.is_empty()).collect();
+    let selects = |t: &String| !t.contains('*') && selection_hashes(&quote(t), manifest).contains(&hash);
+    let kept: Vec<String> = match mode {
+        PickMode::Replace if selection_hashes(sel, manifest) == [hash] => return sel.to_string(),
+        PickMode::Replace => return token.to_string(),
+        PickMode::Add if selection_hashes(sel, manifest).contains(&hash) => return sel.to_string(),
+        PickMode::Add => tokens.iter().map(|t| quote(t)).chain([token.to_string()]).collect(),
+        PickMode::Remove if !tokens.iter().any(selects) => return sel.to_string(),
+        PickMode::Remove => tokens.iter().filter(|t| !selects(t)).map(|t| quote(t)).collect(),
+    };
+    kept.join(", ")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use effectcraft_raster::channels3d::crypto_hash;
+
+    fn manifest() -> Vec<(String, u32)> {
+        ["bunny", "Green, Glossy", r#"say "hi""#, r"back\slash", "  pad ", "star*", "<0000abcd>", "twin"]
+            .iter()
+            .map(|n| (n.to_string(), crypto_hash(n)))
+            .chain([("twin".to_string(), 7)])
+            .collect()
+    }
+
+    /// Every name's token reads back as exactly its ID, through the effect's own parser.
+    #[test]
+    fn tokens_round_trip_through_selection_hashes() {
+        let m = manifest();
+        for (n, h) in &m {
+            let t = token_for(Some(n), *h, Some(&m));
+            assert_eq!(selection_hashes(&t, Some(&m)), [*h], "{n:?} → {t:?}");
+        }
+        assert_eq!(token_for(Some("bunny"), crypto_hash("bunny"), Some(&m)), "bunny");
+        assert_eq!(token_for(Some("Green, Glossy"), crypto_hash("Green, Glossy"), Some(&m)), r#""Green, Glossy""#);
+        assert_eq!(token_for(Some(r#"say "hi""#), crypto_hash(r#"say "hi""#), Some(&m)), r#""say \"hi\"""#);
+        // Wildcard-looking, hash-looking, empty, unnamed and shared names: the hash.
+        assert_eq!(token_for(Some("star*"), crypto_hash("star*"), Some(&m)), format!("<{:08x}>", crypto_hash("star*")));
+        assert_eq!(token_for(Some("<0000abcd>"), crypto_hash("<0000abcd>"), Some(&m)), format!("<{:08x}>", crypto_hash("<0000abcd>")));
+        assert_eq!(token_for(Some(""), 5, Some(&m)), "<00000005>");
+        assert_eq!(token_for(Some("  pad "), crypto_hash("  pad "), Some(&m)), format!("<{:08x}>", crypto_hash("  pad ")));
+        assert_eq!(token_for(None, 0xdead_beef, None), "<deadbeef>");
+        assert_eq!(token_for(Some("twin"), 7, Some(&m)), "<00000007>", "the name selects the other twin");
+    }
+
+    #[test]
+    fn add_and_remove_keep_other_tokens_and_skip_duplicates() {
+        let m = manifest();
+        let (b, g) = (crypto_hash("bunny"), crypto_hash("Green, Glossy"));
+        let tok = |n: &str| token_for(Some(n), crypto_hash(n), Some(&m));
+        let edit = |sel: &str, n: &str, mode| edit_selection(sel, &tok(n), crypto_hash(n), mode, Some(&m));
+        assert_eq!(edit("anything, else", "bunny", PickMode::Replace), "bunny");
+        assert_eq!(edit(" bunny ", "bunny", PickMode::Replace), " bunny ", "same meaning: unchanged");
+        assert_eq!(edit("", "bunny", PickMode::Add), "bunny");
+        assert_eq!(edit("bunny", "Green, Glossy", PickMode::Add), r#"bunny, "Green, Glossy""#);
+        // Already selected: by name, by hash, by a wildcard. The text stays as typed.
+        assert_eq!(edit("bunny,floor", "bunny", PickMode::Add), "bunny,floor");
+        assert_eq!(edit(&format!("<{b:08x}>"), "bunny", PickMode::Add), format!("<{b:08x}>"));
+        assert_eq!(edit("bun*", "bunny", PickMode::Add), "bun*");
+        // Other tokens keep their meaning: quoted where needed, wildcards stay wildcards.
+        assert_eq!(edit(r#"'a, b', fl*, "x, *""#, "bunny", PickMode::Add), r#""a, b", fl*, "x, *", bunny"#);
+        assert_eq!(edit(r#""Green, Glossy", bunny"#, "bunny", PickMode::Remove), r#""Green, Glossy""#);
+        assert_eq!(edit(&format!("bunny, <{b:08x}>, x, bunny"), "bunny", PickMode::Remove), "x");
+        assert_eq!(edit(&format!("<{g:08x}>"), "Green, Glossy", PickMode::Remove), "");
+        // Not selected, or only by a wildcard: unchanged.
+        assert_eq!(edit("a ,b", "bunny", PickMode::Remove), "a ,b");
+        assert_eq!(edit("bun*, x", "bunny", PickMode::Remove), "bun*, x");
+        for sel in [r#""a, b", fl*, bunny"#, r#"x, "Green, Glossy", "say \"hi\"", "back\\slash""#] {
+            let before = selection_hashes(sel, Some(&m));
+            let after = selection_hashes(&edit(sel, "  pad ", PickMode::Add), Some(&m));
+            assert_eq!(after, [before, vec![crypto_hash("  pad ")]].concat(), "{sel}");
+        }
+    }
+}

crates/effects/src/lib.rs

@@ -21,6 +21,7 @@ mod color2;
 mod color3;
 mod color_fx;
 pub mod controls;
+pub mod crypto_select;
 mod distort;
 mod distort2;
 mod distort3;

crates/engine/src/commands/effect.rs

@@ -186,6 +186,91 @@ fn pick_color(s: &mut Session, p: &Value) -> Result<Value> {
     Ok(json!({"prop": uid, "color": color}))
 }
 
+/// What a Cryptomatte effect has at a layer-space point: the ID with the most coverage in its
+/// Cryptomatte layer, with the aux channels (for the manifest) and the effect's Selection.
+struct CryptoHit {
+    at: (effectcraft_project::ItemId, LayerId, Tick),
+    aux: std::sync::Arc<effectcraft_raster::AuxChannels>,
+    layer: String,
+    hash: u32,
+    selection: String,
+    prop: Uid,
+    /// The Selection has an expression: setting its value wouldn't change what it reads.
+    driven: bool,
+}
+
+impl CryptoHit {
+    fn manifest(&self) -> Option<&[(String, u32)]> {
+        self.aux.manifests.iter().find(|(l, _)| *l == self.layer).map(|(_, m)| m.as_slice())
+    }
+    fn reply(&self, sel: &str) -> Value {
+        let selected = effectcraft_effects::selection_hashes(sel, self.manifest()).contains(&self.hash);
+        let name = self.aux.manifest_name(&self.layer, self.hash);
+        json!({"name": name, "hash": format!("{:08x}", self.hash), "layer": self.layer, "selected": selected})
+    }
+}
+
+/// The ID under `{x, y}` (effect space, which is layer space on footage and precomp layers) for
+/// the Cryptomatte effect `{layer?, effect?}`, from the aux channels its render reads. The point
+/// snaps to the centre of the pixel a render at `scale` (the viewer's resolution) samples.
+fn crypto_hit(s: &Session, p: &Value, cmd: &str) -> Result<Option<CryptoHit>> {
+    let (cid, lid, fx_uid) = find_fx(s, p, cmd)?;
+    let (Some(x), Some(y)) = (f_p(p, "x"), f_p(p, "y")) else { return Err(bad(cmd, "missing `x` / `y` (layer pixels)")) };
+    let scale = f_p(p, "scale").unwrap_or(1.0);
+    if !(scale.is_finite() && scale > 0.0 && scale <= 1.0) {
+        return Err(bad(cmd, "`scale` is the preview's resolution factor, above 0 and at most 1"));
+    }
+    let comp = s.project.comp(cid).ok_or(EngineError::NoComp)?;
+    let layer = comp.layer(lid).ok_or(EngineError::NoComp)?;
+    let g = layer.effects().and_then(|fx| fx.groups().find(|g| g.uid == fx_uid)).ok_or_else(|| bad(cmd, "no such effect"))?;
+    let prop = g.prop("selection").filter(|_| g.match_id == "ec.3d.cryptomatte").ok_or_else(|| bad(cmd, "not a Cryptomatte effect"))?;
+    let t = super::time_p(s, p, Some(comp));
+    let ctx = effectcraft_render::EvalCtx { project: &s.project, comp_id: cid, comp, time: t, expr: s.expr.as_deref(), footage: None };
+    let params = effectcraft_effects::flatten_params(g, &mut |pr| ctx.value(layer, pr));
+    let aux = s.renderer(effectcraft_render::RenderOpts { scale, ..Default::default() }).layer_aux(&ctx, layer);
+    let want = effectcraft_effects::CRYPTO_LAYERS.get(params.e("layer") as usize).copied().unwrap_or("CryptoAsset");
+    let Some((aux, layer_name)) = aux.and_then(|a| a.resolve_crypto_layer(want).map(|l| (a, l))) else {
+        return Err(bad(cmd, "this layer has no Cryptomatte channels (a multi-layer OpenEXR with Cryptomatte, or a 3D precomp)"));
+    };
+    let snap = |v: f64| ((v * scale).floor() + 0.5) / scale;
+    let Some(hash) = aux.index_at(snap(x), snap(y)).and_then(|i| aux.crypto_id_at(&layer_name, i)) else { return Ok(None) };
+    let (selection, driven) = (params.s("selection").to_string(), prop.expr.as_ref().is_some_and(|e| e.enabled));
+    Ok(Some(CryptoHit { at: (cid, lid, t), aux, layer: layer_name, hash, selection, prop: prop.uid, driven }))
+}
+
+/// `effect.cryptomatteAt`: the ID under a point, for the viewer's hover hint.
+fn crypto_at(s: &mut Session, p: &Value) -> Result<Value> {
+    Ok(crypto_hit(s, p, "effect.cryptomatteAt")?.map_or(Value::Null, |h| h.reply(&h.selection)))
+}
+
+/// The Cryptomatte eyedropper: replace the effect's Selection with the ID under a point, add it
+/// or remove it, in one undo step at the time it was read (none when its meaning stays).
+fn pick_crypto(s: &mut Session, p: &Value) -> Result<Value> {
+    use effectcraft_effects::crypto_select::{PickMode, edit_selection, token_for};
+    let cmd = "effect.pickCryptomatte";
+    let mode = match str_p(p, "mode").unwrap_or("replace") {
+        "replace" => PickMode::Replace,
+        "add" => PickMode::Add,
+        "remove" => PickMode::Remove,
+        m => return Err(bad(cmd, format!("unknown `mode` {m:?} (replace, add or remove)"))),
+    };
+    let h = crypto_hit(s, p, cmd)?.ok_or_else(|| bad(cmd, "no Cryptomatte ID there (empty or outside the layer)"))?;
+    if h.driven {
+        return Err(bad(cmd, "the Selection is driven by an expression; disable it to pick"));
+    }
+    let m = h.manifest();
+    let sel = edit_selection(&h.selection, &token_for(h.aux.manifest_name(&h.layer, h.hash), h.hash, m), h.hash, mode, m);
+    let changed = sel != h.selection;
+    if changed {
+        let (cid, lid, t) = h.at;
+        s.execute("prop.set", json!({"comp": cid.0, "layer": lid.0, "prop": h.prop, "value": sel, "time": t.seconds(), "timeBase": "comp"}))?;
+    }
+    let mut out = h.reply(&sel);
+    out["selection"] = json!(sel);
+    out["changed"] = json!(changed);
+    Ok(out)
+}
+
 fn remove(s: &mut Session, p: &Value) -> Result<Value> {
     let cid = super::comp_id(s, p)?;
     let fx = effects_p(s, p, "effect.remove")?;
@@ -632,6 +717,21 @@ pub fn specs() -> Vec<CommandSpec> {
             has_layers,
             pick_color
         ),
+        cmd!(
+            "effect.pickCryptomatte",
+            "Pick Cryptomatte",
+            [],
+            None,
+            "{layer?, effect?: index|uid|name, x, y (effect space), mode?: replace|add|remove (default replace), scale? (preview resolution, default 1), time?} → {name, hash, layer, selection, changed, selected} — edits the Cryptomatte effect's Selection with the ID under the point",
+            has_layers,
+            pick_crypto
+        ),
+        crate::query!(
+            "effect.cryptomatteAt",
+            "Cryptomatte At",
+            "{layer?, effect?: index|uid|name, x, y (effect space), scale?, time?} → {name, hash, layer, selected} | null — the Cryptomatte ID under the point",
+            crypto_at
+        ),
         cmd!("effect.applyLast", "Last Effect", ["Effect"], Some("Cmd+Alt+Shift+E"), "{layers?}", has_layers, last),
         cmd!("effect.removeAll", "Remove All", ["Effect"], Some("Cmd+Shift+E"), "{layers?}", has_layers, remove_all),
         cmd!("effect.remove", "Remove Effect", [], None, "{layer?, effect: index|uid|name}", has_layers, remove),

crates/engine/src/lib.rs

@@ -1036,12 +1036,18 @@ impl Session {
 
     /// Render a comp frame with the session's footage + expression hosts.
     pub fn render(&self, comp: ItemId, t: Tick, opts: RenderOpts) -> Image {
+        self.renderer(opts).comp_frame(comp, t)
+    }
+
+    /// The renderer [`Session::render`] uses: the session's footage, expressions, layer cache and
+    /// accelerator, and the render prefs (nested switches, draft shadows) over `opts`.
+    pub fn renderer(&self, opts: RenderOpts) -> Renderer<'_> {
         let opts = RenderOpts { nested_switches: self.prefs.general.switches_affect_nested_comps, draft_shadows: self.prefs.three_d.realtime_shadows, ..opts };
         let mut r = Renderer::new(&self.project, self.footage.as_ref(), opts);
         r.expr = self.expr.as_deref();
         r.cache = Some(&self.layer_cache);
         r.accel = self.accel.as_deref();
-        r.comp_frame(comp, t)
+        r
     }
 
     /// Resolve a comp reference (item id number or name; `None`/null = the active comp).
@@ -1242,6 +1248,8 @@ mod tests_codec_options;
 #[cfg(test)]
 mod tests_color_view;
 #[cfg(test)]
+mod tests_crypto_pick;
+#[cfg(test)]
 mod tests_delete_keys;
 #[cfg(test)]
 mod tests_disk_cache;

crates/engine/src/tests_crypto_pick.rs

@@ -0,0 +1,181 @@
+//! `effect.pickCryptomatte` / `effect.cryptomatteAt` (#473): the Cryptomatte eyedropper reads the
+//! aux channels the effect renders from (footage at its source time, a 3D precomp's generated
+//! pass) and edits the Selection text in one undo step.
+
+use std::sync::Arc;
+
+use effectcraft_project::{Footage, FootageKind, ItemId};
+use effectcraft_raster::channels3d::{crypto_float, crypto_hash};
+use effectcraft_raster::{AuxChannels, Image};
+use effectcraft_time::{FrameRate, Tick};
+use serde_json::{Value, json};
+
+use crate::Session;
+use crate::render::FootageSource;
+
+const COMMA: &str = "Green, \"Glossy\"";
+
+/// 8 × 4 footage with a 16 × 8 Cryptomatte (two aux pixels per layer pixel). Layer pixels in x:
+/// 0-1 `left` (0.6) over `right` (0.4); 2-3 `left` (0.3) under the comma name (0.7), ranks out of
+/// order; 4-5 `star*`; 6 an ID missing from the manifest; 7 a 0.5 / 0.5 tie of `left` and
+/// `right`. Rows 3+ are empty. From 1 s of source time `late` replaces columns 0-1.
+struct Crypto;
+
+fn aux_at(t: Tick) -> AuxChannels {
+    let (w, h) = (16usize, 8usize);
+    let mut planes = vec![vec![0.0f32; w * h]; 4];
+    for (i, x) in (0..w * h).map(|i| (i, i % w)).filter(|(i, _)| i / w < 6) {
+        let late = if t >= Tick::from_seconds_f64(1.0) { "late" } else { "left" };
+        let (a, ca, b, cb) = match x {
+            0..4 => (late, 0.6, "right", 0.4),
+            4..8 => ("left", 0.3, COMMA, 0.7),
+            8..12 => ("star*", 1.0, "", 0.0),
+            12..14 => ("ghost", 1.0, "", 0.0),
+            _ => ("left", 0.5, "right", 0.5),
+        };
+        for (k, v) in [crypto_float(crypto_hash(a)), ca, crypto_float(crypto_hash(b)), cb].into_iter().enumerate() {
+            planes[k][i] = v;
+        }
+    }
+    let mut aux = AuxChannels::new(w as u32, h as u32, 1.0);
+    for (k, c) in ["R", "G", "B", "A"].iter().enumerate() {
+        aux.channels.push((format!("CryptoObject00.{c}"), planes[k].clone()));
+    }
+    aux.manifests = vec![("CryptoObject".into(), ["left", "right", "late", COMMA, "star*"].iter().map(|n| (n.to_string(), crypto_hash(n))).collect())];
+    aux
+}
+
+impl FootageSource for Crypto {
+    fn frame(&self, _: ItemId, _: &Footage, _: Tick) -> Option<Arc<Image>> {
+        Some(Arc::new(Image::new(8, 4)))
+    }
+    fn aux(&self, _: ItemId, _: &Footage, t: Tick) -> Option<Arc<AuxChannels>> {
+        Some(Arc::new(aux_at(t)))
+    }
+}
+
+struct Probe;
+impl crate::Importer for Probe {
+    fn probe(&self, path: &str) -> Result<Footage, String> {
+        let duration = Tick::from_seconds_f64(2.0);
+        Ok(Footage {
+            path: path.into(),
+            kind: FootageKind::Video,
+            width: 8,
+            height: 4,
+            has_video: true,
+            frame_rate: FrameRate::from_f64(24.0),
+            duration,
+            ..Default::default()
+        })
+    }
+}
+
+fn session() -> Session {
+    let mut s = Session { footage: Arc::new(Crypto), importer: Some(Arc::new(Probe)), ..Default::default() };
+    let item = s.execute("file.import", json!({"paths": ["/renders/crypto.exr"]})).unwrap()["items"][0].clone();
+    s.execute("comp.new", json!({"name": "C", "width": 8, "height": 4, "frameRate": 24, "duration": 2})).unwrap();
+    s.execute("layer.addItem", json!({"item": item})).unwrap();
+    s.execute_checked("effect.apply", json!({"effect": "Cryptomatte"})).unwrap();
+    s
+}
+
+fn pick(s: &mut Session, x: f64, mode: &str) -> Value {
+    s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": x, "y": 1.5, "mode": mode})).unwrap()
+}
+
+fn selection(s: &Session) -> String {
+    let l = &s.active_comp().unwrap().layers[0];
+    let g = l.effects().unwrap().groups().next().unwrap();
+    match &g.prop("selection").unwrap().value {
+        effectcraft_keyframe::Value::Str(t) => t.clone(),
+        v => panic!("{v:?}"),
+    }
+}
+
+#[test]
+fn picks_replace_add_and_remove_in_one_undo_step_each() {
+    let mut s = session();
+    let steps = s.history.undo.len();
+    let r = pick(&mut s, 0.5, "replace");
+    assert_eq!((r["name"].as_str(), r["selection"].as_str(), r["changed"].as_bool()), (Some("left"), Some("left"), Some(true)), "{r}");
+    assert_eq!(r["hash"], format!("{:08x}", crypto_hash("left")));
+    assert_eq!(s.history.undo.len(), steps + 1);
+    // The highest coverage wins even when it is not rank 0; the name is quoted.
+    assert_eq!(pick(&mut s, 2.5, "add")["selection"], r#"left, "Green, \"Glossy\"""#);
+    // Adding it again changes nothing and leaves no undo step.
+    let again = pick(&mut s, 3.5, "add");
+    assert_eq!((again["changed"].as_bool(), s.history.undo.len()), (Some(false), steps + 2));
+    // A `*` name and an unnamed ID go in as hashes; a tie goes to rank 0.
+    pick(&mut s, 4.5, "add");
+    assert_eq!(pick(&mut s, 6.5, "add")["selection"], format!(r#"left, "Green, \"Glossy\"", <{:08x}>, <{:08x}>"#, crypto_hash("star*"), crypto_hash("ghost")));
+    assert_eq!(s.execute("effect.cryptomatteAt", json!({"effect": 1, "x": 7.5, "y": 0.5})).unwrap()["name"], "left");
+    let r = pick(&mut s, 0.0, "remove");
+    assert_eq!(r["selection"], format!(r#""Green, \"Glossy\"", <{:08x}>, <{:08x}>"#, crypto_hash("star*"), crypto_hash("ghost")));
+    assert_eq!(r["selected"], false);
+    // The text parses back to exactly the picked IDs.
+    let manifest = aux_at(Tick::ZERO).manifests[0].1.clone();
+    let got = effectcraft_effects::selection_hashes(&selection(&s), Some(&manifest));
+    assert_eq!(got, [COMMA, "star*", "ghost"].map(crypto_hash));
+    assert_eq!(s.history.undo.len(), steps + 5);
+    s.execute("edit.undo", json!({})).unwrap();
+    assert!(selection(&s).starts_with("left, "), "{}", selection(&s));
+    pick(&mut s, 2.0, "replace");
+    assert_eq!(selection(&s), r#""Green, \"Glossy\"""#);
+    // The same selection with other spacing: no change, no undo step.
+    s.execute_checked("prop.set", json!({"path": "effects/#1/selection", "value": " \"Green, \\\"Glossy\\\"\" "})).unwrap();
+    let n = s.history.undo.len();
+    assert_eq!((pick(&mut s, 2.0, "replace")["changed"].as_bool(), s.history.undo.len()), (Some(false), n));
+}
+
+#[test]
+fn bad_points_layers_and_modes_are_errors_and_change_nothing() {
+    let mut s = session();
+    let at = |s: &mut Session, x: f64, y: f64| s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": x, "y": y}));
+    for (x, y) in [(-0.1, 1.0), (8.0, 1.0), (1.0, 4.0), (1.0, 3.5), (1e300, -1e300)] {
+        assert!(at(&mut s, x, y).is_err(), "{x},{y}");
+        assert_eq!(s.execute("effect.cryptomatteAt", json!({"effect": 1, "x": x, "y": y})).unwrap(), Value::Null);
+    }
+    assert!(s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 1, "y": 1, "mode": "toggle"})).is_err());
+    assert!(s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 1})).is_err());
+    for scale in [0.0, -1.0, 2.0] {
+        assert!(s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 1, "y": 1, "scale": scale})).is_err(), "{scale}");
+    }
+    s.execute_checked("effect.apply", json!({"effect": "Gaussian Blur"})).unwrap();
+    assert!(s.execute_checked("effect.pickCryptomatte", json!({"effect": 2, "x": 1, "y": 1})).is_err(), "not a Cryptomatte effect");
+    assert_eq!(selection(&s), "");
+    // A layer without aux channels.
+    s.execute("layer.newSolid", json!({"name": "S"})).unwrap();
+    s.execute_checked("effect.apply", json!({"effect": "Cryptomatte"})).unwrap();
+    let e = s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 1, "y": 1})).unwrap_err().to_string();
+    assert!(e.contains("no Cryptomatte channels"), "{e}");
+    // An expression-driven Selection would ignore the edit.
+    let mut s = session();
+    s.execute_checked("prop.setExpression", json!({"path": "effects/#1/selection", "expression": "\"right\""})).unwrap();
+    assert!(s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 1, "y": 1})).is_err());
+    assert_eq!(s.execute("effect.cryptomatteAt", json!({"effect": 1, "x": 1, "y": 1})).unwrap()["selected"], false);
+}
+
+/// The pick reads the frame the render shows: Time Remap's source time, not the layer time.
+#[test]
+fn time_remap_picks_the_shown_source_frame() {
+    let mut s = session();
+    assert_eq!(pick(&mut s, 0.5, "replace")["name"], "left");
+    s.execute_checked("layer.enableTimeRemap", json!({})).unwrap();
+    s.execute_checked("prop.set", json!({"path": "timeRemap", "value": 1.5})).unwrap();
+    assert_eq!(pick(&mut s, 0.5, "replace")["name"], "late");
+}
+
+/// A precomp of a 3D comp: the generated CryptoObject pass names its layers.
+#[test]
+fn picks_on_a_precomp_of_a_3d_comp() {
+    let mut s = Session::default();
+    s.execute("comp.new", json!({"name": "Outer", "width": 40, "height": 20, "frameRate": 24, "duration": 1})).unwrap();
+    let l = s.execute("layer.newSolid", json!({"name": "Red, Ball", "width": 40, "height": 20})).unwrap()["layer"].clone();
+    s.execute("layer.setSwitch", json!({"layers": [l], "switch": "threeD", "value": true})).unwrap();
+    s.execute("layer.precompose", json!({"layers": [l], "name": "Inner"})).unwrap();
+    s.execute("layer.select", json!({"layers": [s.active_comp().unwrap().layers[0].id.0]})).ok();
+    s.execute_checked("effect.apply", json!({"effect": "Cryptomatte"})).unwrap();
+    let r = s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 20, "y": 10})).unwrap();
+    assert_eq!((r["name"].as_str(), r["selection"].as_str()), (Some("Red, Ball"), Some("\"Red, Ball\"")), "{r}");
+}

crates/host/tests/cryptomatte_pick.rs

@@ -0,0 +1,127 @@
+//! The Cryptomatte eyedropper on real OpenEXR footage (#473): `effect.pickCryptomatte` reads the
+//! file's Cryptomatte ranks and manifest, edits the Selection, and the rendered matte follows,
+//! including undo.
+
+use effectcraft_engine::Session;
+use effectcraft_engine::time::Tick;
+use effectcraft_raster::channels3d::{crypto_float, crypto_hash};
+use serde_json::json;
+
+const RIGHT: &str = "Right, \"Box\"";
+
+/// An 8 × 4 EXR: white RGBA, and a CryptoObject layer with `Left` in columns before `split` and
+/// the comma-and-quotes name from there, both named in its manifest.
+fn crypto_exr(tag: &str, split: usize) -> String {
+    use exr::prelude::*;
+    let (w, h) = (8usize, 4usize);
+    let ids: Vec<f32> = (0..w * h).map(|i| crypto_float(crypto_hash(if i % w < split { "Left" } else { RIGHT }))).collect();
+    let ch = |n: &str, v: Vec<f32>| AnyChannel::new(n, FlatSamples::F32(v));
+    let one = || vec![1.0f32; w * h];
+    let channels = AnyChannels::sort(
+        vec![ch("R", one()), ch("G", one()), ch("B", one()), ch("A", one()), ch("CryptoObject00.R", ids), ch("CryptoObject00.G", one())].into(),
+    );
+    let mut layer = Layer::new((w, h), LayerAttributes::default(), Encoding::FAST_LOSSLESS, channels);
+    let manifest = json!({"Left": format!("{:08x}", crypto_hash("Left")), RIGHT: format!("{:08x}", crypto_hash(RIGHT))}).to_string();
+    for (k, v) in [("name", "CryptoObject".to_string()), ("manifest", manifest)] {
+        layer.attributes.other.insert(Text::from(format!("cryptomatte/f834d0a/{k}").as_str()), AttributeValue::Text(Text::from(v.as_str())));
+    }
+    // A folder of its own: numbered siblings would import as an image sequence.
+    let dir = std::env::temp_dir().join(format!("ec-crypto-pick-{tag}-{}", std::process::id()));
+    std::fs::create_dir_all(&dir).unwrap();
+    let path = dir.join("crypto.exr");
+    Image::from_layer(layer).write().to_file(&path).unwrap();
+    path.to_string_lossy().to_string()
+}
+
+/// A session with the EXR as a layer of a 2 s comp, under a Cryptomatte effect.
+fn footage(tag: &str, split: usize) -> (Session, String) {
+    let path = crypto_exr(tag, split);
+    let mut s = effectcraft_host::session();
+    let r = s.execute("file.import", json!({"paths": [path]})).unwrap();
+    assert_eq!(r["errors"], json!([]), "{r}");
+    s.execute("comp.new", json!({"name": "Crypto", "width": 8, "height": 4, "frameRate": 24, "duration": 2})).unwrap();
+    s.execute("layer.addItem", json!({"item": r["items"][0]})).unwrap();
+    s.execute_checked("effect.apply", json!({"effect": "Cryptomatte"})).unwrap();
+    (s, path)
+}
+
+/// The matte's alpha over the left and right halves.
+fn matte(s: &Session) -> [f32; 2] {
+    let img = s.render(s.active_comp_id().unwrap(), Tick::ZERO, Default::default());
+    [img.get(1, 1)[3], img.get(6, 2)[3]]
+}
+
+#[test]
+fn picking_on_a_cryptomatte_exr_edits_the_selection_and_the_matte() {
+    let (mut s, path) = footage("edit", 4);
+    s.execute_checked("prop.set", json!({"path": "effects/#1/display", "value": 1})).unwrap();
+    assert_eq!(matte(&s), [0.0, 0.0], "nothing selected");
+    let pick = |s: &mut Session, x: f64, mode: &str| s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": x, "y": 1, "mode": mode})).unwrap();
+
+    assert_eq!(pick(&mut s, 1.0, "replace")["selection"], "Left");
+    assert_eq!(matte(&s), [1.0, 0.0]);
+    let r = pick(&mut s, 6.0, "add");
+    assert_eq!((r["name"].as_str(), r["layer"].as_str()), (Some(RIGHT), Some("CryptoObject")), "{r}");
+    assert_eq!(r["selection"], r#"Left, "Right, \"Box\"""#);
+    assert_eq!(matte(&s), [1.0, 1.0], "the quoted name selects the right half");
+    pick(&mut s, 2.0, "remove");
+    assert_eq!(matte(&s), [0.0, 1.0]);
+    s.execute("edit.undo", json!({})).unwrap();
+    assert_eq!(matte(&s), [1.0, 1.0]);
+    s.execute("edit.undo", json!({})).unwrap();
+    assert_eq!(matte(&s), [1.0, 0.0]);
+    let _ = std::fs::remove_dir_all(std::path::Path::new(&path).parent().unwrap());
+}
+
+/// At half resolution a preview pixel shows the aux texel at its centre (layer x 3 for preview
+/// pixel 1): `scale` makes the pick read that one, so what is clicked is what lights up.
+#[test]
+fn a_pick_at_preview_resolution_reads_the_pixel_the_preview_shows() {
+    let (mut s, path) = footage("half", 3);
+    s.execute_checked("prop.set", json!({"path": "effects/#1/matteOnly", "value": true})).unwrap();
+    let r = s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 2.5, "y": 1.5, "scale": 0.5})).unwrap();
+    assert_eq!(r["name"], RIGHT, "{r}");
+    let opts = effectcraft_engine::render::RenderOpts { scale: 0.5, ..Default::default() };
+    assert_eq!(s.render(s.active_comp_id().unwrap(), Tick::ZERO, opts).get(1, 0)[3], 1.0);
+    // At full resolution the same point is Left's pixel.
+    let r = s.execute_checked("effect.cryptomatteAt", json!({"effect": 1, "x": 2.5, "y": 1.5})).unwrap();
+    assert_eq!(r["name"], "Left");
+    let _ = std::fs::remove_dir_all(std::path::Path::new(&path).parent().unwrap());
+}
+
+/// A keyframed Selection picked at `time` is edited at that time, not at the current time.
+#[test]
+fn a_keyframed_selection_is_edited_where_it_was_read() {
+    let (mut s, path) = footage("key", 4);
+    for t in [0.0, 1.0] {
+        s.execute_checked("prop.addKey", json!({"path": "effects/#1/selection", "time": t, "value": format!("<{:08x}>", crypto_hash(RIGHT))})).unwrap();
+    }
+    let r = s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 0.5, "y": 1.5, "mode": "add", "time": 1.0})).unwrap();
+    assert_eq!((r["changed"].as_bool(), r["selected"].as_bool()), (Some(true), Some(true)), "{r}");
+    let sel = |s: &mut Session, t: f64| {
+        let r = s.execute("effect.cryptomatteAt", json!({"effect": 1, "x": 0.5, "y": 1.5, "time": t})).unwrap();
+        r["selected"].as_bool()
+    };
+    assert_eq!((sel(&mut s, 0.0), sel(&mut s, 1.0)), (Some(false), Some(true)), "only the key at 1 s gains Left");
+    let _ = std::fs::remove_dir_all(std::path::Path::new(&path).parent().unwrap());
+}
+
+/// A precomp whose 3D layer an expression moves: the pick finds it where the render keys it
+/// (the pick's aux pass runs the session's expressions).
+#[test]
+fn a_pick_on_a_precomp_follows_expressions() {
+    let mut s = effectcraft_host::session();
+    let inner = s.execute("comp.new", json!({"name": "Inner", "width": 40, "height": 20, "frameRate": 24, "duration": 1})).unwrap()["comp"].clone();
+    let l = s.execute("layer.newSolid", json!({"name": "Ball", "width": 6, "height": 6})).unwrap()["layer"].clone();
+    s.execute("layer.setSwitch", json!({"layers": [l], "switch": "threeD", "value": true})).unwrap();
+    s.execute_checked("prop.setExpression", json!({"layer": l, "path": "transform/position", "expression": "[33, 10, 0]"})).unwrap();
+    s.execute("comp.new", json!({"name": "Outer", "width": 40, "height": 20, "frameRate": 24, "duration": 1})).unwrap();
+    s.execute("layer.addItem", json!({"item": inner})).unwrap();
+    s.execute_checked("effect.apply", json!({"effect": "Cryptomatte"})).unwrap();
+    s.execute_checked("prop.set", json!({"path": "effects/#1/matteOnly", "value": true})).unwrap();
+    let r = s.execute_checked("effect.pickCryptomatte", json!({"effect": 1, "x": 33.5, "y": 10.5})).unwrap();
+    assert_eq!(r["selection"], "Ball", "{r}");
+    let img = s.render(s.active_comp_id().unwrap(), Tick::ZERO, Default::default());
+    assert_eq!((img.get(33, 10)[3], img.get(20, 10)[3]), (1.0, 0.0));
+    assert_eq!(s.execute("effect.cryptomatteAt", json!({"effect": 1, "x": 20.5, "y": 10.5})).unwrap(), serde_json::Value::Null);
+}

crates/raster/src/channels3d.rs

@@ -148,6 +148,31 @@ impl AuxChannels {
         self.crypto_rank_planes(layer).iter().map(|(id, c)| (id[i], c[i])).collect()
     }
 
+    /// The Cryptomatte layer an effect asking for `want` (`CryptoObject`, …) reads: an exact
+    /// match, else one ending in it (`ViewLayer.CryptoObject`), else the first layer.
+    pub fn resolve_crypto_layer(&self, want: &str) -> Option<String> {
+        let layers = self.crypto_layers();
+        layers.iter().find(|l| l.as_str() == want || l.ends_with(want)).or(layers.first()).cloned()
+    }
+
+    /// The ID (its hash, `f32::to_bits`) with the most coverage at aux pixel `i` of Cryptomatte
+    /// layer `layer`, summed over the ranks it appears in. Ties go to the lower rank; IDs with
+    /// no positive coverage don't count. `None` when nothing covers the pixel.
+    pub fn crypto_id_at(&self, layer: &str, i: usize) -> Option<u32> {
+        let mut ids: Vec<(u32, f32)> = vec![];
+        for (id, cov) in self.crypto_rank_planes(layer) {
+            let (Some(id), Some(&c)) = (id.get(i), cov.get(i)) else { continue };
+            if c > 0.0 {
+                match ids.iter_mut().find(|(h, _)| *h == id.to_bits()) {
+                    Some((_, sum)) => *sum += c,
+                    None => ids.push((id.to_bits(), c)),
+                }
+            }
+        }
+        // `min_by` keeps the first of equal elements: a tie goes to the lower rank.
+        ids.into_iter().min_by(|a, b| b.1.total_cmp(&a.1)).map(|(h, _)| h)
+    }
+
     /// Aux pixel index of a layer-space point (nearest; `None` outside).
     #[inline]
     pub fn index_at(&self, lx: f64, ly: f64) -> Option<usize> {
@@ -211,6 +236,28 @@ pub fn crypto_float(hash: u32) -> f32 {
 mod tests {
     use super::*;
 
+    /// The picked ID: most coverage (summed over ranks), ties to the lower rank, nothing when
+    /// no rank covers the pixel; layers resolve as the Cryptomatte effect does.
+    #[test]
+    fn crypto_id_at_takes_the_top_coverage_and_layers_resolve_by_suffix() {
+        let mut a = AuxChannels::new(4, 1, 1.0);
+        let f = |n: &str| crypto_float(crypto_hash(n));
+        let (x, y, z) = (f("x"), f("y"), f("z"));
+        for (c, v) in [("R", [x, x, x, x]), ("G", [0.3, 0.5, 0.0, 0.2]), ("B", [y, y, y, y]), ("A", [0.6, 0.5, 0.0, 0.3])] {
+            a.channels.push((format!("ViewLayer.CryptoObject00.{c}"), v.to_vec()));
+        }
+        for (c, v) in [("R", [z, z, z, x]), ("G", [0.1, 0.0, -1.0, 0.2])] {
+            a.channels.push((format!("ViewLayer.CryptoObject01.{c}"), v.to_vec()));
+        }
+        a.channels.push(("Other00.R".into(), vec![0.0; 4]));
+        let l = a.resolve_crypto_layer("CryptoObject").unwrap();
+        assert_eq!(l, "ViewLayer.CryptoObject");
+        assert_eq!(a.resolve_crypto_layer("CryptoMaterial").as_deref(), Some("Other"), "falls back to the first");
+        let at = |i| a.crypto_id_at(&l, i);
+        assert_eq!([at(0), at(1), at(2), at(3), at(9)], [Some(crypto_hash("y")), Some(crypto_hash("x")), None, Some(crypto_hash("x")), None]);
+        assert_eq!(AuxChannels::new(1, 1, 1.0).resolve_crypto_layer("CryptoObject"), None);
+    }
+
     #[test]
     fn murmur3_reference_values() {
         // Published MurmurHash3_x86_32 test vectors.

crates/render/src/lib.rs

@@ -314,26 +314,7 @@ impl EffectHost for FxHost<'_, '_, '_> {
     }
 
     fn aux(&self) -> Option<Arc<effectcraft_raster::AuxChannels>> {
-        match &self.layer.source {
-            LayerSource::Footage { item } => {
-                let ItemKind::Footage(f) = &self.r.project.item(*item)?.kind else { return None };
-                let a = self.r.footage.aux(*item, f, self.ctx.source_time(self.layer))?;
-                // Aux pixels per layer pixel (the layer is the footage at its nominal size).
-                if f.width > 0 && a.width > 0 {
-                    let mut a = (*a).clone();
-                    a.scale = a.width as f64 / f.width as f64;
-                    return Some(Arc::new(a));
-                }
-                Some(a)
-            }
-            LayerSource::Comp { item } => {
-                if self.r.depth > MAX_FX_DEPTH || self.r.project.comp_contains(*item, self.ctx.comp_id) {
-                    return None;
-                }
-                self.r.nested().comp_aux(*item, self.ctx.nested_time(self.layer)).map(Arc::new)
-            }
-            _ => None,
-        }
+        self.r.layer_aux(self.ctx, self.layer)
     }
 
     fn params_at(&self, layer_time: f64) -> Option<Params> {
@@ -2103,6 +2084,34 @@ impl<'a> Renderer<'a> {
         Some(three_d::compose::aux_pass(self, &ctx))
     }
 
+    /// The auxiliary channels effects on `layer` read ([`EffectHost::aux`]): its multi-layer
+    /// OpenEXR footage's at the layer's source time (Time Remap included), or a precomp's
+    /// generated 3D channels at its nested time. `None` for other layers.
+    pub fn layer_aux(&self, ctx: &EvalCtx, layer: &Layer) -> Option<Arc<effectcraft_raster::AuxChannels>> {
+        match &layer.source {
+            LayerSource::Footage { item } => {
+                let ItemKind::Footage(f) = &self.project.item(*item)?.kind else { return None };
+                let a = self.footage.aux(*item, f, ctx.source_time(layer))?;
+                // Aux pixels per layer pixel (the layer is the footage at its nominal size). The
+                // planes are only copied when that changes the scale.
+                let scale = a.width as f64 / f.width as f64;
+                if f.width > 0 && a.width > 0 && a.scale != scale {
+                    let mut a = (*a).clone();
+                    a.scale = scale;
+                    return Some(Arc::new(a));
+                }
+                Some(a)
+            }
+            LayerSource::Comp { item } => {
+                if self.depth > MAX_FX_DEPTH || self.project.comp_contains(*item, ctx.comp_id) {
+                    return None;
+                }
+                self.nested().comp_aux(*item, ctx.nested_time(layer)).map(Arc::new)
+            }
+            _ => None,
+        }
+    }
+
     /// Evaluation context of comp `comp_id` at comp time `t`.
     pub fn eval_ctx(&self, comp_id: ItemId, t: Tick) -> Option<EvalCtx<'a>> {
         Some(self.ctx(comp_id, self.project.comp(comp_id)?, t))

docs/agents.md

@@ -180,6 +180,7 @@ Short recipes; every step is one tool call.
 | Animate | `add_keyframe {"layer":2,"path":"transform/scale","keys":[{"time":0,"value":[0,0]},{"time":0.5,"value":[100,100]}],"interpolation":"easyEase"}` |
 | Effect with settings | `list_effects {"filter":"glow"}` → `add_effect {"layer":2,"effect":"Glow","values":{"threshold":40,"radius":30}}` → `set_property {"layer":2,"path":"effects/#1/intensity","value":2}` |
 | Key a green screen | `add_effect {"layer":2,"effect":"Keylight (1.2)"}` → `execute_command effect.pickColor {"layer":2,"effect":1,"param":"screenColour","x":20,"y":20,"average":true}` (a screen pixel, layer space; samples the effect's input) → `set_property {"layer":2,"path":"effects/#1/screenMatte/clipBlack","value":10}` → `render_frame {"transparent":true}` |
+| Key objects by Cryptomatte | `add_effect {"layer":2,"effect":"Cryptomatte"}` → `execute_command effect.pickCryptomatte {"layer":2,"effect":1,"x":120,"y":80}` (the ID under that layer pixel replaces the Selection; `"mode":"add"` / `"remove"` edit it; `effect.cryptomatteAt` with the same point only reports the name) → `render_frame` |
 | Drive with an expression | `set_property {"layer":2,"path":"transform/rotation","expression":"time*90"}` |
 | Inspect a layer | `get_layer {"layer":2,"flat":true}` (every property with its `path`, value, key count) |
 | Check the result | `render_frame {"time":1.5,"max_side":640}`; in bridge mode `screenshot {"panel":"Timeline"}` |
@@ -323,8 +324,8 @@ build real projects through these interfaces; they are worked examples of everyt
 ### Effect positions
 
 Effect points, Puppet pins (`puppet.addPin`, `puppet.recordPin`), Paint, Liquify and Roto Brush
-strokes and `effect.pickColor` are in **effect space**: layer pixels from the top-left of the
-layer's effect bounds. For footage, solids and precomps that is the layer's own pixel grid. Shape
+strokes, `effect.pickColor` and `effect.pickCryptomatte` are in **effect space**: layer pixels
+from the top-left of the layer's effect bounds. For footage, solids and precomps that is the layer's own pixel grid. Shape
 and text layers have comp-sized bounds centred on their content's origin, so in a `w` × `h` comp
 the point `[w / 2, h / 2]` is that origin (a new effect's default point). `prop.get` / `prop.set`
 read and write these values as they are.