S2.46: document Roto Brush with SAM 2

How to install and use it, how the port was verified, speed and the benchmark.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs/agents.md

@@ -506,7 +506,7 @@ Trained model (Roto Brush 2.0 / 3.0):
   licence, size, URL, SHA-256 and whether it is installed, selected and active.
 - `roto.model.download {id}` fetches and verifies the official weights in the background;
   `roto.model.install {path, id?}` installs a file you already have.
-- `roto.model.select {id: classical|mobilesam|sam2.1-hiera-tiny}` chooses which one Roto Brush uses. Loading runs
+- `roto.model.select {id: classical|mobilesam|sam2.1-hiera-tiny}` chooses which one Roto Brush uses (SAM 2 also tracks the object through the clip, see [roto-brush-sam2.md](roto-brush-sam2.md); `roto.status` then reports `source` and per-frame `tracker.frames` with `confidence` and `occluded`). Loading runs
   in the background; `"wait": true` (also on download and install) returns once it is in use.
 - `roto.model.remove {id}` deletes the installed weights.
 - The `version` property (`effects/#1/version`: 0 = 1.0, 1 = 2.0, 2 = 3.0) picks between the

docs/roto-brush-sam2.md

@@ -0,0 +1,57 @@
+# Roto Brush with SAM 2
+
+Roto Brush can follow an object through a clip with Meta's **SAM 2.1 Tiny** (Hiera-Tiny, 39M parameters,
+Apache-2.0). It is optional: nothing changes until you install the model and choose it.
+
+## Using it
+1. Settings ▸ Roto Brush ▸ SAM 2.1 Tiny ▸ **Install** (downloads `model.safetensors`, 155.9 MB, from the pinned
+   Hugging Face commit; the SHA-256 is checked and the licence notice is kept next to the file). Or install from a
+   file you downloaded. Agents: `roto.model.install {id: "sam2.1-hiera-tiny"}`.
+2. Choose it (`roto.model.select {id: "sam2.1-hiera-tiny"}`).
+3. Paint strokes on the base frame as usual, then **Propagate**. One tracking pass follows the object through the
+   whole span in both directions, using SAM 2's streaming memory, so it keeps tracking through occlusion and when the
+   object leaves the frame and returns. Each frame's probability map is then refined by the same graph cut as every other
+   Roto Brush frame: your strokes stay hard constraints and edges snap to colour.
+4. `roto.status {frame}` reports `source` (`tracker` or `classical`) and, for tracked frames, `tracker.frames[f]`:
+   `confidence` (0–1, the model's object score times its predicted mask quality) and `occluded` (the object is not
+   in the frame: an empty matte).
+
+If the model is missing, fails, or is deselected, the classic engine runs exactly as before (`roto.status` says
+`classical`, and the job's `trackerNote` says why).
+
+## How it works
+`crates/sam2` is a Rust port (on [candle](https://github.com/huggingface/candle), native only) of the Hugging Face
+`transformers` SAM 2 implementation, Apache-2.0 (see its `NOTICE`). Every stage is checked against PyTorch by
+`crates/sam2/tests/reference.rs` (they skip without the weights; reference tensors come from
+`tools/sam2_reference.py` and `tools/sam2_video_reference.py`):
+
+| Stage | Max abs error vs PyTorch |
+|---|---|
+| Hiera backbone, 12 blocks | 2.6e-5 |
+| Neck features / position encodings | 9.7e-6 / 6e-8 |
+| Prompt embeddings | 1.2e-7 |
+| Mask decoder (logits up to 11.6) | 1.1e-4 |
+| Memory encoder | 3.5e-6 |
+| Memory attention (rotary) | 2.6e-5 |
+| 7-frame clip, forwards and backwards (9 frame results) | logits 0.0038, IoU 1.0000 |
+
+SAM 2 is prompted with at most three foreground and three background points per frame: it was trained on a handful
+of clicks, and measured confidence collapses with many (0.99 with 1–7 points, 0.25 with 16, 0.00 with 32).
+
+## Speed (this PC: 16 threads, CPU only)
+A 1024 × 1024 frame takes about **3.4 s** to encode (release build); a click on an encoded frame about 120 ms. Tracking a
+clip costs one encode per frame, so a 300-frame clip takes on the order of 17 minutes on a CPU. Candle has no AMD/Vulkan
+backend; GPU work is Phase 6 of the plan (the early Hiera blocks are memory-bound and have headroom on the CPU too).
+
+## Benchmark
+`effectcraft-cli bench --roto [--frames N] [--sam2 model.safetensors]` runs synthetic clips with known mattes. 12 frames,
+debug build, mean IoU (after the object returns):
+
+| Clip | Classical | SAM 2.1 Tiny |
+|---|---|---|
+| move | 0.982 | 0.993 |
+| light | 0.981 | 0.992 |
+| occlusion | 0.978 | 0.988 |
+| reenter | 0.632 (0.000) | 0.975 (0.967) |
+
+MobileSAM's column appears when it is installed.