effectcraft/docs/roto-brush-sam2.md

Roto Brush with SAM 2

Roto Brush can follow an object through a clip with Meta's SAM 2.1 Tiny (Hiera-Tiny, 39M parameters, Apache-2.0). It is optional: nothing changes until you install the model and choose it.

Using it

1. Settings ▸ Roto Brush ▸ SAM 2.1 Tiny ▸ Install (downloads model.safetensors, 155.9 MB, from the pinned Hugging Face commit; the SHA-256 is checked and the licence notice is kept next to the file). Or install from a file you downloaded. Agents: roto.model.install {id: "sam2.1-hiera-tiny"}.
2. Choose it (roto.model.select {id: "sam2.1-hiera-tiny"}).
3. Paint strokes on the base frame as usual, then Propagate. One tracking pass follows the object through the whole span in both directions, using SAM 2's streaming memory, so it keeps tracking through occlusion and when the object leaves the frame and returns. Each frame's probability map is then refined by the same graph cut as every other Roto Brush frame: your strokes stay hard constraints and edges snap to colour.
4. roto.status {frame} reports source (tracker, model for a frame the model segmented on its own, or classical) and, for tracked frames, tracker.frames[f]: confidence (0–1, the model's object score times its predicted mask quality) and occluded (the object is not in the frame: an empty matte). model says what Settings chose and whether it is installed, loaded and loading; notice says when it is not the one in use for the layer. tracker.saved names the model the saved frames came from.

Propagate never falls back to the classic engine without saying so. With SAM 2 chosen but not installed, or failing to load, it refuses with the reason. While it is still loading, wait: true (agents) loads it first; without wait it refuses, and the load goes on in the background. Deselecting SAM 2 (choosing Classic) runs Propagate exactly as before, and Roto Brush 1.0 never uses a model.

Tracked frames are saved in the project with the instance, as Freeze's are, so a saved project renders the same when it reopens, in the app or with effectcraft-cli, without the model being loaded or chosen there. They apply while the strokes and span they came from are unchanged. Changing a stroke or the span makes them stale until the next Propagate. Choosing another model, or the classic engine, leaves them in place and roto.status says so; a classic Propagate replaces them with classic mattes. Object Matte saves nothing.

Sizes

Four checkpoints are in the registry (Settings ▸ Roto Brush); the size is detected from the file, and each is pinned to a Hugging Face commit with its SHA-256. All run the same tracker; a bigger backbone costs time and memory.

Model	Download	GPU encode (RX 7900 GRE)	Agreement with PyTorch
SAM 2.1 Tiny (sam2.1-hiera-tiny)	155.9 MB	0.24 s	encoder 6.8e-6
SAM 2.1 Small (sam2.1-hiera-small)	184.3 MB	0.20 s	encoder 1.0e-5
SAM 2.1 Base+ (sam2.1-hiera-base-plus)	323.5 MB	0.37 s	encoder 2.3e-5
SAM 2.1 Large (sam2.1-hiera-large)	897.9 MB	0.72 s	encoder 5.2e-4

Settings marks the size that suits the computer: Small when a usable GPU is found (it encodes no slower than Tiny there and is more accurate), Tiny otherwise; Base+ and Large say so when they would be slow on a CPU. The list carries recommended and hint per model (roto.models).

Half-precision weights (optional)

EFFECTCRAFT_SAM2_F16=1 keeps the weight matrices on the GPU as half precision, halving their memory (Large: about 900 MB to 450 MB). It is off by default: the video logit error against PyTorch grows from about 0.003 to 0.06 (Tiny) and 0.12 (Large) with the same masks, and the speed gain is under 10% (Large encode 0.72 s to 0.66 s).

What stays on the CPU

The mask decoder takes about 110 ms of a tracked frame (attention layers 42 ms, the two upscaling convolutions 50 ms). Rewriting the upscaling convolutions as a matrix product and a pixel shuffle did not make them faster, and a GPU port would save under 0.1 s of a 0.7 s frame, so it stays on the CPU.

Every size was checked stage by stage against PyTorch (patch embedding, each Hiera block, neck, prompt encoder, decoder, memory encoder, memory attention) and on a whole clip forward and backward, on the CPU and the GPU: the video logit error is at most 0.005 with the same masks. Large carries much larger activations (peaks near 300 in the third stage), so its f32 rounding error is larger in absolute terms; its tests use a bound relative to the peak. Large needs the GPU to run per block (the encode submits once per block so intermediates are freed); a GPU error of any kind falls back to the CPU.

Corrections re-track only what they change

A correction stroke changes the keys of the frames from it on (in both directions from the base), so Propagate tracks only those again: the window starts at the nearest prompted frame at or before the first frame it needs (the corrected frame, which restarts the object with its own prompt) and runs to the last frame it needs. Frames before the correction keep their tracked results, and a Propagate with nothing changed does no tracking. The older non-conditioning memory frames before the corrected frame are not rebuilt, so frames just after it can differ slightly from a from-scratch pass. Tracked frames of a session are in memory; a project reopened in a new process re-tracks the whole span after a stroke change (its saved tracking is stale as a whole).

Memory and limits

The weights take about 156 MB. A model idle for 10 minutes is unloaded (a toast says so; the choice stays) and loads again when a Propagate or a stroke needs it. The tracker holds the memory banks of the objects it follows (about 2 MB per stored frame, pruned to the frames later ones can still look at) and refuses to grow past 2 GiB, or past a time limit when one is set, with a clear error instead of running out of memory. Two runs of the same prompts give bit-identical masks on the CPU.

How it works

crates/sam2 is a Rust port (on candle, native only) of the Hugging Face transformers SAM 2 implementation, Apache-2.0 (see its NOTICE). Every stage is checked against PyTorch by crates/sam2/tests/reference.rs (they skip without the weights; reference tensors come from tools/sam2_reference.py and tools/sam2_video_reference.py):

Stage	Max abs error vs PyTorch
Hiera backbone, 12 blocks	2.6e-5
Neck features / position encodings	9.7e-6 / 6e-8
Prompt embeddings	1.2e-7
Mask decoder (logits up to 11.6)	1.1e-4
Memory encoder	3.5e-6
Memory attention (rotary)	2.6e-5
7-frame clip, forwards and backwards (9 frame results)	logits 0.0038, IoU 1.0000

SAM 2 is prompted with at most three foreground and three background points per frame: it was trained on a handful of clicks, and measured confidence collapses with many (0.99 with 1–7 points, 0.25 with 16, 0.00 with 32).

Speed (this PC: AMD RX 7900 GRE and a 16-thread CPU, release build)

SAM 2 runs on the GPU when there is one (wgpu compute kernels, so AMD, NVIDIA and Intel alike; candle itself has no AMD backend) and on the CPU otherwise. The model loads on the GPU automatically; EFFECTCRAFT_SAM2_DEVICE=cpu (or gpu) overrides it. A GPU that fails (no adapter, out of memory, a lost device) makes the CPU take over for the rest of the session, and roto.status reports tracker.device (the GPU's name, or CPU).

Step	CPU	GPU
Encode one 1024 × 1024 frame	2.05 s	0.24 s
Memory attention (full bank of 7 memory frames)	2.3 s	0.29 s
Memory encoder	0.24 s	0.05 s
Decoder (CPU on both)	0.11 s	0.11 s
A tracked frame in all	4.3 s	0.7 s
A click on an encoded frame	0.10 s	0.10 s

A 300-frame clip takes about 21 minutes on the CPU and about 3.5 minutes on this GPU; the decoder, still on the CPU, is the next 16 % of a GPU frame. The CPU numbers came down from 3.4 s per encode at the start of the speed work (parallel GELU and bias add, cached position encodings, pooling without permutes); chunking the attention queries was tried and did not help. The GPU kernels (crates/sam2/src/gpu) are tested against CPU definitions and PyTorch: the encoder's 64 × 64 level matches PyTorch to 7e-6, the memory attention to 1.8e-5, and a clip tracked on the GPU matches Sam2VideoModel as the CPU does (worst logit error 0.0032). cargo run --release -p effectcraft-sam2 --example sam2_bench times the pieces on both devices (EFFECTCRAFT_SAM2_PROFILE=1 prints a frame's breakdown).

Benchmark

effectcraft-cli bench --roto [--frames N] [--sam2 model.safetensors (any size; default: the first installed)] [--mobilesam mobile_sam.pt] runs synthetic clips with known mattes through all three. 24 frames, debug build, SAM 2 on the GPU; mean IoU (mean IoU after the object returns), seconds per frame:

Clip	Classical	MobileSAM	SAM 2.1 Tiny
move	0.980	0.989	0.992
light	0.979	0.990	0.992
occlusion (a bar hides the disk, then clears it)	0.491 (0.000)	0.494 (0.000)	0.989 (0.988)
reenter (the disk leaves and comes back)	0.618 (0.000)	0.620 (0.000)	0.982 (0.976)
s/frame	0.01	0.5–1.3	0.8

MobileSAM prompts every frame independently from the flow-warped matte, so it follows the object while the flow does and loses it when it is hidden; SAM 2's streaming memory is what lets it re-acquire the object. The synthetic clips are easy (one textured disc), so the "move" and "light" rows mostly show all three working; the last two rows are the difference.

Corrections needed (--fixes)

bench --roto --fixes also counts the correction strokes each tracker needs before every frame reaches IoU 0.9: after each run it paints what a user would on the first frame under the target (a short foreground stroke through the true object's centre, and a background stroke on the biggest false-positive blob), up to 6. 16 frames, debug build:

Clip	Classical	MobileSAM	SAM 2.1 Tiny
move	0	0	0
light	0	0	0
occlusion	more than 6	more than 6	0
reenter	more than 6	more than 6	0

The simulated user is simple (one stroke pair per round, always on the first bad frame), so "more than 6" means this correction model never recovers the flow-based engines on those clips, not that no human could. Peak memory is not measured.