On an Intel Mac with an AMD Radeon Pro 5300M the Metal compiler crashes
on candle's mlx_gemm kernel, candle-metal-kernels panics on the compile
error, and the worker reported "the AI model failed unexpectedly" and
loaded the model on Metal again for every next job, so AI masks never
worked there.
The worker now retries a job that fails on the GPU (a panic or a candle
error) once on the CPU, after freeing the model. When that works, the
model stays on the CPU for the session and the job's outcome carries a
notice that segment_poll shows once; when the CPU fails too, the GPU
was not the cause and the next job uses it again. Missing or damaged
model files, render and remote-worker errors are not retried. Only
builds with a GPU path (macOS) retry; LIGHTCRAFT_GPU=0 and
LIGHTCRAFT_GPU_BACKEND=off now keep SAM 3 on the CPU from the start
(lightcraft_gpu::env_off_reason), like AI Denoise and the face models.
lightcraft-segment: Sam3::load_cpu, GPU_BUILD; text_logits returns the
crate's Error so Describe failures are classified like clicks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe merged the detector's per-instance masks. Those stay near 0 even well inside what they
found: a sky came out about 87 % selected, a volleyball net about 70 %, a person about 94 %.
Two instances that touch both fade out where they meet, so a line showed between people standing
side by side.
- The detector's semantic head is now ported: semantic_projection over the pixel decoder, the
transformers implementation's semantic_seg. It matches the reference to about 1e-3, like the
instance masks. tools/sam3_reference.py writes it and the reference test checks it; older
reference files still pass.
- Each instance found (score above the threshold, as before) takes the semantic logits where it
is the strongest instance and near (above -4). The semantic mask is sure inside and runs on
across the boundary between two instances. Far from every instance nothing changes, so
Describe selects nothing it didn't find before.
On 16 prompts over three photos (a mountain landscape, two gym photos), the median coverage
inside each selection rises, e.g.:
- sky 0.87 -> 1.00
- mountains 0.81 -> 0.99
- net 0.71 -> 0.99
- floor 0.92 -> 1.00
- person 0.94 -> 0.99
Each selection gains 0 to 1.9 % of the photo at its soft rim and loses none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker binary gets the never-crash deny list and forbid(unsafe_code) like
every production crate root. docs/ai-masks.md spells out who can reach the
unauthenticated loopback listener.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With no mirror configured the dialog was titled "Download the SAM 3
Model?", only said to put "the files" in the folder (see
docs/ai-masks.md) and offered Close. It is now titled "Install the SAM 3
Model", names the three files (model.safetensors, vocab.json,
merges.txt from facebook/sam3), and has a Show in Finder / Explorer /
File Manager button that creates and shows the model folder, plus a link
to the installation guide. Object and Describe work as soon as the files
are there.
The download location itself (a hosted mirror) is still a maintainer
decision: DEFAULT_MIRRORS stays empty.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01912WaK6Rmew3wxZuPyX7E8
The downloader in lightcraft-segment (mirrors, resume, pinned SHA-256, pure-Rust
HTTPS) was already generic apart from four SAM 3 constants. Move it to
crates/fetch (L0, native only) so other model features can use it without
depending on the SAM 3 inference code. lightcraft-segment keeps SAM3_FILES,
DEFAULT_MIRRORS, MIRRORS_ENV and wraps mirrors(); no behaviour change. The
downloader's 13 tests moved with it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Engine (crates/engine/src/segment/):
- One worker thread owns the model and the encoded photo; the session
sends jobs (with the render job for the model input) and gets results
over channels, never locking what the model holds. In the app
(Segmenter::background) clicks, descriptions, prepares and detail
passes return at once and Session::segment_poll applies results (one
undo step per settled selection, pending clicks shown on the photo);
the CLI/MCP wait for them. Newer requests for the same selection
supersede queued ones.
- Every job runs under catch_unwind: a candle panic becomes an error and
the model is reloaded next time; busy counters are reset by drop
guards; a dead worker is restarted. The model unloads after 10 min idle.
- segment.model.status / .download {acknowledged: true} / .cancel: the
download runs on its own thread; status via atomics (try_lock only).
Without the model every AI request fails at once with "The SAM 3 model
is not installed…"; renders/exports never need it; `seg` params make
AI masks from stored segmentations without it.
- Hostile documents: at most 4 detail patches per component (capped on
deserialize and in the renderer), at most 64 Object clicks.
UI: Object / Describe without the model open "Download the SAM 3
Model?" (size, folder, SAM License link; Not Now / Download, progress,
cancel, Try Again with the reason, Continue once installed); progress in
the Masking panel when the dialog is closed. Reverted the PR's unrelated
default change (mask overlay off + hover-red) to main's behaviour.
tools/install-sam3.*: --fail so error pages never land in files, 416
restarts instead of looping, the weights are verified before they are
moved into place (a bad file is deleted, never resumed from).
Docs: docs/ai-masks.md (in-app download, mirrors, SAM License summary,
no weights in the repo), NOTICE, parity LR-MASK-OBJECTS, What's New,
ROADMAP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI masks from Meta's SAM 3, run in LightCraft itself: a new
`lightcraft-segment` crate ports the Hugging Face transformers
implementation (Apache-2.0) to candle, on the GPU through Metal on macOS
(CPU elsewhere), with no C/C++ and no `unsafe`.
- Object: click to include, ⌥-click to leave a part out (SAM 3's
interactive prompt encoder + two-way mask decoder).
- Describe: a text prompt selects every instance ("sky", "the red car";
"car, road" merges several) via the CLIP text encoder and DETR detector.
- Both as new masks and as Add / Subtract / Intersect components; + / −
buttons under the selected mask combine selections.
- The segmentation is stored with the mask (288² quantized logits, ~10-30
KB), so renders, exports and the web build never need the model.
- A zoomed-in detail pass (background, ~5 s on an M4 Pro) re-segments the
area around a selection from a sharper source: 5-10x finer edges on
small objects.
- Per-selection Edge slider (hard ↔ soft); hovering a mask in the list
shows it in red (the always-on overlay is now off by default, O toggles).
- The engine feature `sam` (on in the desktop app) gates the model; the
image is encoded once per photo in the background, clicks take ~35 ms.
- Weights are not bundled (SAM License): tools/install-sam3.sh / .ps1
download and verify facebook/sam3; docs/ai-masks.md.
- Clearer errors (missing file, missing model) and a stderr logger in the
app (LIGHTCRAFT_LOG=info).
candle is pinned at 0.9.2: later releases make candle-core depend on
tokenizers with the Oniguruma C library. Every stage is checked against
the Python reference (tests/reference.rs, tools/sam3_reference.py): on
Metal the backbone matches to 7e-5 relative error, masks to ~5e-5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C8BkS7JDqa1EMcjW6spDGA
aiongg