aiongg
f3bb7a9f30
* CI: dev profile keeps line tables for workspace crates and no debug info for dependencies * Control server tests: persistent_requests_and_slow_replies_outlive_the_idle_read_timeout no longer races the idle timeout Each request (and the closing half-shutdown) reaches the socket before the worker reads, so on a slow machine only the reply waits outlast the 100 ms idle timeout, as the test intends. The idle test sends its input the same way, and the no-frames test allows 5 s instead of 2 s to give up. * CI: one integration-test binary per crate Each crate's tests/*.rs become modules of tests/it/main.rs (51 test files into 13 binaries, plus 6 that stay their own). Inner #![cfg] attributes move to the mod lines. Files that change state for the whole process keep a binary of their own: codecs jxl_tail_heap and raw sr2_heap (dhat global allocator), gpu shutdown, shutdown_first and memory, ui-egui shutdown. Docs, comments and the GPU probe name the new paths and targets (--test it <module>::); the CI report names an it binary's skips after the crate. * CI: tests under nextest, doctests apart, one wasm check xtask ci runs the tests with cargo nextest run --lib --bins --tests when cargo-nextest is installed, else cargo test with the same targets and a one-line notice; doctests run after them with cargo test --workspace --doc. The heif and hardware GPU steps use the same runner. Examples stay out of the test build (clippy --all-targets still checks them). Concurrent tests under nextest follow the RUST_TEST_THREADS cap (NEXTEST_TEST_THREADS). The wasm check is one cargo check --keep-going over the same crates, and clippy runs with --keep-going. * CI: tests run under cargo test by default, nextest with LIGHTCRAFT_NEXTEST=1 (per-process scene cache)
16 KiB
AI masks (Object and Describe)
LightCraft selects objects with SAM 3 (Segment Anything with Concepts, Meta 2025), run in
pure Rust by crates/segment on candle: on the GPU
through Metal on macOS, on the CPU elsewhere (for now). If the Mac's GPU can't run it, it runs on
the CPU (see the worker below).
The model is optional. It is not part of LightCraft, and nothing requires it: without it, Object and Describe offer to download it, every other feature (and every existing mask, see below) works, and no render, export or command ever waits for it.
Using them
In the Masking panel (M):
- Object: click the thing you want; every click refines the selection. ⌥-click (Alt) a part to leave it out. Clicks show as green (include) and red (leave out) dots.
- Describe: type what to select —
sky,trees,the red car— and press Return. Every instance the model finds (score above 0.5) joins the mask; if nothing matches, nothing is created and you are told so. Around what it found, the mask follows the model's semantic mask for the phrase, which is sure inside and runs on where two instances touch (the instances' own masks alone left a sky or a net about 70–90 % selected and a thin line between two people side by side). - Several things at once:
car, road(comma-separated) selects each and merges them. - + / − under the selected mask (and the Add / Subtract / Intersect menus) combine
selections: Describe
car, click the mask, + ▸ Describe…road; − ▸ Describe…peopletakes the people out of a sky mask; − ▸ Object then clicks remove one object. - Edge (per selection, −100…100): below 0 a crisper border, above 0 a feathered one (up to 2 % of the long edge, so previews and exports match).
- Detail: about a second after the last click (or right after a description), the photo around the selection is analyzed again, zoomed in, from a sharper source (~5 s on an M4 Pro, in the background); the result is kept as a high-resolution patch (at most 4 per selection), so small objects get 5–10× finer edges than one pass over the whole photo gives.
The first Object or Describe on a photo analyzes it (the image encoder runs once per photo and look: ~3 s on an Apple M4 Pro, plus a one-time ~6 s for compiling GPU kernels and loading the model after launch). That starts in the background as soon as you pick Object; after it, each click takes ~35 ms and each description ~0.6 s. Everything the model does runs on its own worker thread: the window stays responsive, a click shows its dot at once and the selection follows when it's computed ("Selecting…" / "Analyzing the photo…" under the mask tiles). The model is unloaded after 10 minutes without use (it takes several GB of memory) and loaded again when needed.
The result is stored with the mask: the model's 288 × 288 mask logits over the uncropped photo (quantized, compressed, ~10–30 KB). Renders and exports sample them at any resolution and never need the model, so masks survive moving the library to a machine without it, and the web build renders them. Editing the photo's look later does not recompute a mask (click again to update).
Remote Metal inference
Native builds with sam can keep the editor, RAW files, library, and exports local while
a Mac runs SAM 3. Build both ends from the same revision. On the Mac, install the model
as described below, then run:
cargo build --release --locked -p lightcraft-segment --bin lightcraft-sam3-worker
target/release/lightcraft-sam3-worker "$HOME/Library/Application Support/LightCraft/models/sam3" 127.0.0.1:8793
On the editing computer, start an SSH tunnel in one terminal and LightCraft in another:
ssh -N -T -o ExitOnForwardFailure=yes -L 127.0.0.1:8793:127.0.0.1:8793 user@your-mac
LIGHTCRAFT_SAM3_REMOTE=127.0.0.1:8793 lightcraft
Object clicks, Describe prompts, and detail refinement use the remote worker. Only the rendered model input (long edge at most 1008 pixels) and prompts cross the tunnel. The worker caches the current photo; detail crops do not replace that cache. Returned masks stay in the local library, so later adjustments and exports work without the connection.
The worker requires Metal and accepts one editing connection at a time. It loads weights on demand and releases them when the client disconnects or remains idle for ten minutes. Both endpoints require loopback addresses: SSH provides authentication and encryption. The listener has no independent authentication; do not expose it through a public proxy. Any process of any user on the worker Mac can connect to it, and on the editing computer a local process listening on the port while the tunnel is down receives the rendered model input, so use this only on machines whose local users you trust.
No local weights are required in remote mode. An unavailable worker produces an explicit
error, not a silent CPU fallback. Restore the tunnel and retry. Unset
LIGHTCRAFT_SAM3_REMOTE to use the local model again. This offloads segmentation only,
not RAW development, denoising, or upscaling. The model licence below still applies.
Getting the model
In the app (recommended)
The first time you pick Object or Describe without the model, LightCraft asks:
Download the SAM 3 model? Object and Describe masks use SAM 3, Meta's segmentation model. It isn't part of LightCraft, and everything else works without it. Download it once (about 3.4 GB) · folder. Licence: SAM License (Meta) — Meta's terms, not LightCraft's. Downloading it means accepting them. [Read the SAM License] — Not Now / Download
Nothing is downloaded until you choose Download. The download runs in the background (close
the dialog and keep working; the Masking panel shows its progress), can be cancelled, resumes
where it stopped if interrupted (also after quitting), and tries the next download location when
one fails or stalls. Each file is written as <name>.part and moved into the model folder only
when complete and verified — model.safetensors must have the exact size and SHA-256 of Meta's
official checkpoint; a file that doesn't match is deleted, never used. If no location works, the
dialog says why and LightCraft carries on without the model.
Agents and scripts use the same commands (control channel, lightcraft-cli mcp --connect):
segment.model.status, segment.model.download {"acknowledged": true} (refused without the
explicit acknowledged: pass it only after the user agreed to the download and the licence),
segment.model.cancel.
Download locations
The model is fetched from an ordered list of mirrors: base URLs where <base>/model.safetensors,
<base>/vocab.json and <base>/merges.txt live. The list is, in order:
LIGHTCRAFT_SAM3_MIRRORS(environment variable; URLs separated by commas or spaces);- the file
models/sam3-mirrors.txtin LightCraft's settings folder (one URL per line,#comments) —~/Library/Application Support/LightCraft/(macOS),%APPDATA%\LightCraft\(Windows),~/.config/lightcraft/(Linux); - the built-in list,
DEFAULT_MIRRORSincrates/segment/src/fetch/mod.rs.
Maintainers: the built-in list is empty until LightCraft's own CDN locations exist (see the
TODO(maintainer)there): add them in order of preference, host the three files unchanged, and pinvocab.json/merges.txt(size + SHA-256) inSAM3_FILESat the same time. Until then the in-app download needs a user-configured mirror, and the dialog says so: it names the three files to put in the model folder by hand, with a button that shows that folder (created if needed) and a link to this guide.
Hugging Face's facebook/sam3 can't be a default: it is gated (each person must accept the
licence there, wait for approval and download with their own token). Someone with access can
use it as their mirror:
LIGHTCRAFT_SAM3_MIRRORS=https://huggingface.co/facebook/sam3/resolve/main \
LIGHTCRAFT_SAM3_TOKEN=hf_... lightcraft
(LIGHTCRAFT_SAM3_TOKEN is sent as a bearer token only over https, and only to the mirror's own
host — not to the storage host it redirects to.)
The download is plain HTTP/1.1 over TLS in pure Rust (rustls with the RustCrypto provider and
the Mozilla root certificates; no OpenSSL, ring or aws-lc, nothing compiled from C), with a
15 s connect timeout and a 30 s stall timeout; it follows redirects but doesn't use HTTP proxies.
By hand
Put model.safetensors, vocab.json and merges.txt from facebook/sam3 in the model folder:
models/sam3/ in the settings folder above, or the folder LIGHTCRAFT_SAM3_DIR names. Developers
can use the installer scripts, which download from Hugging Face with your token and verify the
weights:
HF_TOKEN=hf_... tools/install-sam3.sh # macOS, Linux
tools/install-sam3.sh --check # verify an installation
$env:HF_TOKEN = "hf_..."; .\tools\install-sam3.ps1 # Windows
Licence of the model
LightCraft's code is MIT OR Apache-2.0; crates/segment (a port of Apache-2.0 code) is
Apache-2.0. The SAM 3 weights are neither: they are Meta's, released under the SAM
License (https://github.com/facebookresearch/sam3/blob/main/LICENSE), a custom licence that
is not an OSI open-source licence. It grants a non-exclusive, worldwide, non-transferable,
royalty-free licence to use, reproduce, distribute, copy, modify and create derivative works of
the "SAM Materials", subject to its terms, including: anyone redistributing them must provide a
copy of the licence with them, and derivative works are distributed under its terms; uses
subject to ITAR, military or warfare purposes, nuclear industries, espionage, guns or illegal
weapons, and reverse engineering are prohibited, and trade controls (sanctions, export rules)
apply; the materials come "as is", without warranty; Meta may terminate the licence on a breach
(then the materials must be deleted); California law governs. Read the licence itself — this
summary is not legal advice.
Therefore:
- The weights are never part of this repository, its releases or installers, and must not be committed.
- They reach a user's computer only when that user asks for them, after the dialog names the licence (or when they install them by hand).
- Whoever hosts a mirror redistributes the SAM Materials and must follow the licence (ship it next to the files).
Commands (control channel, MCP)
| Command | Params | |
|---|---|---|
mask.add |
{kind: "object", points?: [[x,y],…], exclude?: [[x,y],…], seg?} |
an Object mask (empty until clicked) |
mask.add |
{kind: "prompt", text, seg?} |
a Describe mask |
mask.addComponent |
{op: add|subtract|intersect, kind: "object"|"prompt", …} |
the same as a component |
mask.objectPoint |
{x, y, exclude?: bool, id?} |
one click on the selected mask's Object selection (≤ 64 clicks) |
mask.refineDetail |
{id?, component?} |
the zoomed-in detail pass → {started} |
segment.prepare |
{} |
load the model and analyze the active photo → {busy} |
segment.model.status |
{} |
{available, installed, remote, dir, loaded, busy, analyzing, sizeBytes, license, licenseUrl, mirrors, download} |
segment.model.download |
{acknowledged: true} |
start the download in the background → {started} |
segment.model.cancel |
{} |
stop it (it resumes next time) → {cancelled} |
remote reports the configured endpoint, not its health. installed describes local weights only.
download is {running, done, total, file, error, cancelled, finished}: cancelled is true when the last
download was cancelled (its error says so in words; decide on the flag). An AI mask command refused because
the model is not installed (or still downloading) fails with an error that starts with "The SAM 3 model is
not installed"; inside LightCraft it is ErrorKind::ModelMissing(Sam3), which is what the app offers the
download on.
Coordinates are normalized to the uncropped, oriented photo, like every mask shape. seg passes
a stored segmentation (as mask.list / the develop settings hold it): the mask is made without
the model. Without the model, the AI kinds fail with an error starting "The SAM 3 model is not
installed" that says how to get it.
In the desktop app the commands don't wait for the model: mask.objectPoint returns
{pending: true} and the selection is applied when it's computed (one undo step per settled
selection); mask.add {kind: "prompt"} returns {pending: true} and the mask appears when
something is found (or a message says nothing was). lightcraft-cli and the headless MCP server
wait and return the result (when built with the sam feature; the default CLI has no AI masks).
Implementation notes
crates/segmentports the Hugging Facetransformersimplementation (Apache-2.0; seeNOTICE— the ported files say so and that they were modified): the 32-layer ViT backbone with 2-D RoPE and windowed attention, the feature pyramids, the SAM 2-style prompt encoder and two-way mask decoder for clicks, and the CLIP text encoder, DETR encoder/decoder (box relative-position bias, presence token), pixel decoder and semantic head for text. The CLIP tokenizer is a small BPE intokenizer.rs. The downloader (original code) is thelightcraft-fetchcrate, shared by any model that is downloaded;fetch/here says what SAM 3 needs.- candle is pinned at 0.9.2: later releases make
candle-coredepend ontokenizerswith the Oniguruma C library, and the product is pure Rust. Weights are read with positional reads (no memory map:unsafestays inlightcraft-sysmem), only the tensors a path needs, and every tensor's byte range is checked against the file's length when it is opened (a truncated or damaged file is an error, never a crash). - The engine (
crates/engine/src/segment/) owns one worker thread with the model and the last encoded photo; the session sends it jobs and receives results over channels, never holding a lock the model holds. Each job runs undercatch_unwind: a panic inside candle becomes an error and the model is reloaded on the next request; busy counters are reset by drop guards. - GPU fallback (issue #1974): a job that fails on the GPU (a panic or a candle error, e.g. an
AMD GPU in an Intel Mac whose Metal compiler crashes on a matmul kernel) frees the model and runs
once more on the CPU. If that works, the model stays on the CPU until LightCraft restarts and
the user is told once ("The GPU couldn't run the AI mask model…"); if the CPU fails too, the
error is reported and the next job tries the GPU again. Missing or damaged model files and
remote-worker errors are not retried.
LIGHTCRAFT_GPU=0orLIGHTCRAFT_GPU_BACKEND=offkeep the model on the CPU from the start. - Hostile documents: a selection keeps at most 4 detail patches (when read and when rendered), each at most 1024² logits; more than 64 clicks are never sent to the model.
- Accuracy:
crates/segment/tests/it/reference.rscompares every stage against the Python reference (tools/sam3_reference.py); on Metal the backbone matches to 7e-5 relative error and masks to ~5e-5. f16 was tried and rejected (overflow, ~7 % faster). - The engine feature
sam(on in the desktop app, off for web and CLI) gates the model; without it the commands report that the build has no AI masks.
Developers: the accuracy test against the Python reference needs torch and
transformers (tools/requirements-sam3-reference.txt):
python3 -m venv .venv-sam3 && .venv-sam3/bin/pip install -r tools/requirements-sam3-reference.txt
.venv-sam3/bin/python tools/sam3_reference.py photo.jpg ref.safetensors
LIGHTCRAFT_SAM3_DIR=<model dir> LIGHTCRAFT_SAM3_REF=ref.safetensors cargo test -p lightcraft-segment --release -- --nocapture