Describe merged the detector's per-instance masks. Those stay near 0 even well inside what they
found: a sky came out about 87 % selected, a volleyball net about 70 %, a person about 94 %.
Two instances that touch both fade out where they meet, so a line showed between people standing
side by side.
- The detector's semantic head is now ported: semantic_projection over the pixel decoder, the
transformers implementation's semantic_seg. It matches the reference to about 1e-3, like the
instance masks. tools/sam3_reference.py writes it and the reference test checks it; older
reference files still pass.
- Each instance found (score above the threshold, as before) takes the semantic logits where it
is the strongest instance and near (above -4). The semantic mask is sure inside and runs on
across the boundary between two instances. Far from every instance nothing changes, so
Describe selects nothing it didn't find before.
On 16 prompts over three photos (a mountain landscape, two gym photos), the median coverage
inside each selection rises, e.g.:
- sky 0.87 -> 1.00
- mountains 0.81 -> 0.99
- net 0.71 -> 0.99
- floor 0.92 -> 1.00
- person 0.94 -> 0.99
Each selection gains 0 to 1.9 % of the photo at its soft rim and loses none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI masks from Meta's SAM 3, run in LightCraft itself: a new
`lightcraft-segment` crate ports the Hugging Face transformers
implementation (Apache-2.0) to candle, on the GPU through Metal on macOS
(CPU elsewhere), with no C/C++ and no `unsafe`.
- Object: click to include, ⌥-click to leave a part out (SAM 3's
interactive prompt encoder + two-way mask decoder).
- Describe: a text prompt selects every instance ("sky", "the red car";
"car, road" merges several) via the CLIP text encoder and DETR detector.
- Both as new masks and as Add / Subtract / Intersect components; + / −
buttons under the selected mask combine selections.
- The segmentation is stored with the mask (288² quantized logits, ~10-30
KB), so renders, exports and the web build never need the model.
- A zoomed-in detail pass (background, ~5 s on an M4 Pro) re-segments the
area around a selection from a sharper source: 5-10x finer edges on
small objects.
- Per-selection Edge slider (hard ↔ soft); hovering a mask in the list
shows it in red (the always-on overlay is now off by default, O toggles).
- The engine feature `sam` (on in the desktop app) gates the model; the
image is encoded once per photo in the background, clicks take ~35 ms.
- Weights are not bundled (SAM License): tools/install-sam3.sh / .ps1
download and verify facebook/sam3; docs/ai-masks.md.
- Clearer errors (missing file, missing model) and a stderr logger in the
app (LIGHTCRAFT_LOG=info).
candle is pinned at 0.9.2: later releases make candle-core depend on
tokenizers with the Oniguruma C library. Every stage is checked against
the Python reference (tests/reference.rs, tools/sam3_reference.py): on
Metal the backbone matches to 7e-5 relative error, masks to ~5e-5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C8BkS7JDqa1EMcjW6spDGA
