mirror of
https://github.com/storytold/effectcraft.git
synced 2026-10-29 04:26:55 -06:00
bflatasticandClaude Opus 5.5
7579de21c6
- New L1 crate effectcraft-segment: the MaskModel interface (prompts → foreground probability), a registry of open-licensed models (licence must be MIT/Apache/BSD/…; URL, size, SHA-256), a PyTorch checkpoint reader (zip + pickle subset), dependency-free SHA-256, and MobileSAM (TinyViT-5M + SAM prompt encoder / two-way transformer / mask decoder) on rayon and ndarray's safe GEMM. Loads the official mobile_sam.pt as published; checked against candle (embedding 0.3% RMS, 99.9% equal mask pixels); ~0.7 s/frame encoder at 32 threads, cached per frame, ~0.1 s decoder. - track::roto: segment_with / propagate_with prompt a model with the strokes or the flow-warped matte (box, core points, mask); its probability is a strong, unblurred prior for the same graph cut (strokes stay hard constraints); a failing model or one disagreeing with the flow (IoU < 0.5) is ignored. Prior gets a weight. - effects::roto: Version 2.0 / 3.0 use the model chosen in Settings (set_model / model_for); the model id is part of the chain seed. - engine: roto_models (roto.models, roto.model.select / install / download / remove), weights in the models folder (host::config_dir()/models, shared by the app, CLI and MCP), background loading, downloads through the system curl, verified; Settings ▸ Roto Brush page (prefs roto.model). host::config_dir moved from the desktop app. - Tests: segment (kernels vs direct sums, checkpoint reader, SHA-256 vectors, registry), track (fake model: prior shapes the cut, failing / disagreeing models ignored; real weights with EFFECTCRAFT_MOBILESAM: base IoU 0.989, propagated ≥ 0.980 vs classic 0.973), engine (registry, verification, selection; real install + load), MCP. - Docs: README, ROADMAP, parity (MSK-4 done: 90/2/0), preferences, agents, architecture, crates/segment/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
39 lines
2.4 KiB
Markdown
39 lines
2.4 KiB
Markdown
# effectcraft-segment
|
|
|
|
Trained segmentation models for Roto Brush, behind one swappable interface.
|
|
|
|
- **`MaskModel`**: what Roto Brush asks of a model: given a frame and prompts (foreground /
|
|
background points, a box, a prior mask), the foreground probability of every pixel. Roto Brush
|
|
(`effectcraft-track`) only talks to this trait, so models can be added or swapped without
|
|
touching it. The classical graph-cut segmenter stays built in as the fallback.
|
|
- **`MODELS`**: the registry. Every entry must be open source under a licence compatible with
|
|
EffectCraft's MIT OR Apache-2.0 (`ALLOWED_LICENCES`), with its homepage, licence URL, the
|
|
official weights URL, size and SHA-256. `load` refuses anything else.
|
|
- **Weights are never bundled.** They are downloaded on demand (or installed from a file) by the
|
|
engine (`roto.model.*`, Settings ▸ Roto Brush) and verified against the registry's SHA-256
|
|
(`sha256`, dependency-free).
|
|
- **`pt`**: reads PyTorch checkpoints (`torch.save` zip archives and the pickle subset state dicts
|
|
use), so official weights load exactly as published.
|
|
- **`mobilesam`**: MobileSAM in plain Rust: the TinyViT-5M image encoder and Segment Anything's
|
|
prompt encoder and two-way-transformer mask decoder, on `nn` kernels (rayon, and ndarray's safe
|
|
`general_mat_mul`, which picks AVX/FMA kernels at run time). No ML framework, no unsafe code.
|
|
|
|
## Models
|
|
|
|
| id | Model | Licence | Weights | Size |
|
|
|---|---|---|---|---|
|
|
| `mobilesam` | [MobileSAM](https://github.com/ChaoningZhang/MobileSAM) (C. Zhang et al., 2023): Segment Anything with a 5M-parameter TinyViT encoder | Apache-2.0 | `mobile_sam.pt` from the official repository | 40.7 MB |
|
|
|
|
The implementation follows the published architectures (MobileSAM, TinyViT and Segment Anything
|
|
papers and their Apache-2.0 reference code); its output was checked against an independent
|
|
implementation (Hugging Face candle's, MIT/Apache-2.0) on the same weights: embeddings within 0.3%
|
|
RMS (exact vs. tanh GELU) and 99.9% of mask pixels equal.
|
|
|
|
## Adding a model
|
|
|
|
1. Pick a model whose code **and** weights are open source under an allowed licence.
|
|
2. Add a `ModelInfo` to `MODELS` (pin the official file's SHA-256 and size).
|
|
3. Implement `MaskModel` for it and add its loader to `load`.
|
|
4. Test it against a reference implementation; Roto Brush, Settings and the `roto.model.*`
|
|
commands pick it up from the registry.
|