From 371e67f51eadb2e70d7cdbf39c0e8ac2066b1ed3 Mon Sep 17 00:00:00 2001 From: Kristian Myram Date: Thu, 15 Oct 2026 02:25:28 +0200 Subject: [PATCH] S2.46: document Roto Brush with SAM 2 How to install and use it, how the port was verified, speed and the benchmark. Co-Authored-By: Claude Sonnet 5.5 --- docs/agents.md | 2 +- docs/roto-brush-sam2.md | 57 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 1 deletion(-) create mode 100644 docs/roto-brush-sam2.md diff --git a/docs/agents.md b/docs/agents.md index 864e7cf59..e0c0055ec 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -506,7 +506,7 @@ Trained model (Roto Brush 2.0 / 3.0): licence, size, URL, SHA-256 and whether it is installed, selected and active. - `roto.model.download {id}` fetches and verifies the official weights in the background; `roto.model.install {path, id?}` installs a file you already have. -- `roto.model.select {id: classical|mobilesam|sam2.1-hiera-tiny}` chooses which one Roto Brush uses. Loading runs +- `roto.model.select {id: classical|mobilesam|sam2.1-hiera-tiny}` chooses which one Roto Brush uses (SAM 2 also tracks the object through the clip, see [roto-brush-sam2.md](roto-brush-sam2.md); `roto.status` then reports `source` and per-frame `tracker.frames` with `confidence` and `occluded`). Loading runs in the background; `"wait": true` (also on download and install) returns once it is in use. - `roto.model.remove {id}` deletes the installed weights. - The `version` property (`effects/#1/version`: 0 = 1.0, 1 = 2.0, 2 = 3.0) picks between the diff --git a/docs/roto-brush-sam2.md b/docs/roto-brush-sam2.md new file mode 100644 index 000000000..b2c356e74 --- /dev/null +++ b/docs/roto-brush-sam2.md @@ -0,0 +1,57 @@ +# Roto Brush with SAM 2 + +Roto Brush can follow an object through a clip with Meta's **SAM 2.1 Tiny** (Hiera-Tiny, 39M parameters, +Apache-2.0). It is optional: nothing changes until you install the model and choose it. + +## Using it +1. Settings ▸ Roto Brush ▸ SAM 2.1 Tiny ▸ **Install** (downloads `model.safetensors`, 155.9 MB, from the pinned + Hugging Face commit; the SHA-256 is checked and the licence notice is kept next to the file). Or install from a + file you downloaded. Agents: `roto.model.install {id: "sam2.1-hiera-tiny"}`. +2. Choose it (`roto.model.select {id: "sam2.1-hiera-tiny"}`). +3. Paint strokes on the base frame as usual, then **Propagate**. One tracking pass follows the object through the + whole span in both directions, using SAM 2's streaming memory, so it keeps tracking through occlusion and when the + object leaves the frame and returns. Each frame's probability map is then refined by the same graph cut as every other + Roto Brush frame: your strokes stay hard constraints and edges snap to colour. +4. `roto.status {frame}` reports `source` (`tracker` or `classical`) and, for tracked frames, `tracker.frames[f]`: + `confidence` (0–1, the model's object score times its predicted mask quality) and `occluded` (the object is not + in the frame: an empty matte). + +If the model is missing, fails, or is deselected, the classic engine runs exactly as before (`roto.status` says +`classical`, and the job's `trackerNote` says why). + +## How it works +`crates/sam2` is a Rust port (on [candle](https://github.com/huggingface/candle), native only) of the Hugging Face +`transformers` SAM 2 implementation, Apache-2.0 (see its `NOTICE`). Every stage is checked against PyTorch by +`crates/sam2/tests/reference.rs` (they skip without the weights; reference tensors come from +`tools/sam2_reference.py` and `tools/sam2_video_reference.py`): + +| Stage | Max abs error vs PyTorch | +|---|---| +| Hiera backbone, 12 blocks | 2.6e-5 | +| Neck features / position encodings | 9.7e-6 / 6e-8 | +| Prompt embeddings | 1.2e-7 | +| Mask decoder (logits up to 11.6) | 1.1e-4 | +| Memory encoder | 3.5e-6 | +| Memory attention (rotary) | 2.6e-5 | +| 7-frame clip, forwards and backwards (9 frame results) | logits 0.0038, IoU 1.0000 | + +SAM 2 is prompted with at most three foreground and three background points per frame: it was trained on a handful +of clicks, and measured confidence collapses with many (0.99 with 1–7 points, 0.25 with 16, 0.00 with 32). + +## Speed (this PC: 16 threads, CPU only) +A 1024 × 1024 frame takes about **3.4 s** to encode (release build); a click on an encoded frame about 120 ms. Tracking a +clip costs one encode per frame, so a 300-frame clip takes on the order of 17 minutes on a CPU. Candle has no AMD/Vulkan +backend; GPU work is Phase 6 of the plan (the early Hiera blocks are memory-bound and have headroom on the CPU too). + +## Benchmark +`effectcraft-cli bench --roto [--frames N] [--sam2 model.safetensors]` runs synthetic clips with known mattes. 12 frames, +debug build, mean IoU (after the object returns): + +| Clip | Classical | SAM 2.1 Tiny | +|---|---|---| +| move | 0.982 | 0.993 | +| light | 0.981 | 0.992 | +| occlusion | 0.978 | 0.988 | +| reenter | 0.632 (0.000) | 0.975 (0.967) | + +MobileSAM's column appears when it is installed.