Conflict in crates/track/src/camtrack/bundle.rs: kept both the branch's
PtSys/preconditioner/iterative_schur block and upstream's new
add_focal_prior helper. Adapted to upstream's explicit linalg imports
(conjugate_gradient, and I3/log_rot/mt/norm in the tests) and moved the
explicit point elimination into reduce_points so adjust_impl stays under the
cognitive-complexity lint.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflict in crates/track/src/camtrack/bundle.rs (imports): upstream replaced
the linalg glob import with an explicit list; kept that list and added the
branch's SolveError import (use super::{Distortion, SolveError}). Input
validation, size budget and Result-returning adjust/cost are unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflict in crates/track/src/face.rs: upstream moved the feature-blob search
into FaceTracker::feature_blobs. Re-applied the branch there: the per-pixel
point-in-polygon loop is replaced by non_skin_inside (one crossing pass per
window row) and the roll rotation is hoisted with face_mapper.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dropping the points a later stroke of the other kind painted over could leave
a stroke with none, even though most of it was still painted: a small Alt tap
in the middle of a large foreground dot removed its only point, and the
selected model was skipped for the classical cut. Points are now kept by the
resolved ownership under them, and a stroke that loses every sampled point
but still owns pixels is prompted from the one deepest inside them. A stroke
painted over completely still prompts nothing.
A trained model is no longer prompted with both labels at one spot: points
along a stroke that a later stroke of the other kind painted over are
dropped, and the propagated matte's points, box and mask take this frame's
strokes over it.
A Refine Edge stroke over an earlier Alt (erase) stroke now restores the band
there; the band carried from the previous frame stays under this frame's
strokes.
A green stroke over an earlier Alt (red) stroke now restores the area: where
strokes overlap, the one painted last sets the hard constraint, for the cut,
the colour samples and the cleanup seeds alike.
Upstream split lib.rs files into modules and moved engine commands/ to cmd/; this branch's change merges without textual conflicts.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The fine stage searched a 5x5 neighbourhood per pyramid level and seed.
When the best coarse match is reliable (NCC score >= 0.8) and lies within
one top-level pixel of the motion prediction, a 3x3 neighbourhood is
enough; otherwise the 5x5 search stays.
Fixes#2349
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Reuse consecutive caller-owned pyramids for regular Warp motion and trajectory tracking. Preserve the original three-level motion search by setting aside the trajectory-only extra level, without cloning planes. Keep detailed analysis at its existing distinct resolutions and test bitwise motion and complete JSON equality against separate pyramids.
Fixes#2322
Solve expensive camera systems with matrix-free Schur products and a block-preconditioned iterative solver. Check the true linear residual and fall back to the direct solve when convergence fails. Keep short systems on the existing path and compare reprojections, camera parameters and outlier decisions against direct solves.
Fixes#2328
Share the unchanged Jacobi factors between full SVD and null-vector callers. Select the same right singular vector directly instead of normalizing U and sorting copies that the caller discards.
Fixes#2329
Detect narrow lower-triangle bandwidth and skip structural zeros while preserving dot-product lanes and signed-zero arithmetic. Keep a dense fallback for wide systems and compare factors and solutions against the original algorithm.
Fixes#2325
Return errors for invalid bundle observations and oversized workspaces, reject non-finite projections and mismatched correspondence slices, and prevent track frame overflow. Cover the reported failures with synthetic regression tests.
Fixes#2330
Use a bounded inline array for the current camera parameters and preserve the existing accumulation order. Extra parameters spill safely if the parameter set grows.
Fixes#2327
The normalized cross-correlation only had a fast path for an exactly
identity shape, which tracked footage never has after the first frame, so
every candidate went through Plane::sample. All candidates of a search are
the reference centre plus integer offsets, so each template pixel has the
same fractional offsets for all of them. Build one table of taps per
(level, seed, shape) and score each candidate with integer-indexed lerps,
with one bounding-box check up front (outside the plane the old path is
used).
lk_refine only reads the level-0 gradients around the match, so compute
them for that window (with an interior fast path in Plane::gradients_in)
and fall back to the full gradients if the iteration leaves it.
The results are unchanged: same lerp form and accumulation order.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
step() cloned the whole RGBA frame to remember it (33 MB at 1080p,
133 MB at 4K) and then cropped that clone, although only the region
around the outline is ever read. The tracker now stores the previous
frame already cropped to the region the next step will use, computed
from the updated outline with the same margin and rounding as before,
so the crops, and therefore the tracking result, are unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Store the two four-element DLT rows in arrays while preserving arithmetic and accumulation order. This removes two heap allocations for every triangulated view.
Fixes#2326
The feature mask tested every non-skin window pixel against the 48-edge
face outline (about 25 M edge tests per frame). A scanline hits the same
edges for every pixel of a row, so the crossings are now computed once
per row with the same expression and comparisons as inside_polygon, and
a pixel's parity follows from how many lie to its right. The face-frame
rotation is also worked out once per frame instead of once per blob
pixel.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Every pixel of a (2r+1)^2 window shares one fractional offset per level
and iteration, but lk_point called Plane::sample (with a libm floor, a
bounds test and the clamp path) up to three times per pixel. The offset
and the clamped column/row index tables are now computed once per
window and the pixels run the same lerps on integer indices.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The nightly FreeBSD run measured 119 ms/frame against a 100 ms limit: the mean included
Tracker::new and any scheduling stalls on the two-core VM. The best single step with a
250 ms bound still catches an order-of-magnitude regression.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: with a trained model installed, Object Matte's optional box reached
MobileSAM only as background points along the box's edges (an approximation),
although the model's prompt encoder takes a box prompt directly.
Fix: a new Roto Brush stroke kind, Box (two opposite corners), that
stroke_prompt turns into the model prompt's bbox (scaled to the matte grid,
corners in any order, degenerate or non-finite boxes ignored). The graph cut
never rasterizes it. Object Matte emits it instead of the edge strokes when a
model segments; the classical fallback keeps the edge strokes. A prior matte's
box no longer replaces a box the user drew on that frame.
Tests: effects object_matte::the_box_reaches_the_model_as_a_box_prompt (a
stand-in model sees the box, at full and half resolution, and no edge points);
track roto::tests::box_stroke_is_the_model_box_prompt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- effectcraft-segment: a FaceModel interface (find a face in a region, follow it; a Topology
maps the model's points to the tracker's landmarks and outline), a TensorFlow Lite reader and
interpreter (float ops, float16 weights folded; matches ai-edge-litert to ~1e-6), and
MediaPipe Face Landmarker (Google, Apache-2.0: BlazeFace + Face Mesh V2, 3.8 MB, pinned
SHA-256). The registry gains a task and authors per model; installed weights get a NOTICE.
- effectcraft-track: FaceTracker::new_with uses the model, falls back to the classical fitter
when it finds no face; chin and jaw come from the outline either way so measurements agree.
- Engine: roto_models becomes models (both tasks); face.models / face.model.* commands,
Settings ▸ Face Tracking, `wait` on select/install/download for scripts; track.mask reports
faceModel.
- Credits: ATTRIBUTION.md lists the optional models and the Apache-2.0 specs the code follows.
- Docs: README, ROADMAP (G7), gaps, parity (TRK-3 score unchanged pending G1), agents,
preferences, architecture, segment README.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New L1 crate effectcraft-segment: the MaskModel interface (prompts → foreground probability),
a registry of open-licensed models (licence must be MIT/Apache/BSD/…; URL, size, SHA-256),
a PyTorch checkpoint reader (zip + pickle subset), dependency-free SHA-256, and MobileSAM
(TinyViT-5M + SAM prompt encoder / two-way transformer / mask decoder) on rayon and ndarray's
safe GEMM. Loads the official mobile_sam.pt as published; checked against candle
(embedding 0.3% RMS, 99.9% equal mask pixels); ~0.7 s/frame encoder at 32 threads, cached
per frame, ~0.1 s decoder.
- track::roto: segment_with / propagate_with prompt a model with the strokes or the flow-warped
matte (box, core points, mask); its probability is a strong, unblurred prior for the same
graph cut (strokes stay hard constraints); a failing model or one disagreeing with the flow
(IoU < 0.5) is ignored. Prior gets a weight.
- effects::roto: Version 2.0 / 3.0 use the model chosen in Settings (set_model / model_for);
the model id is part of the chain seed.
- engine: roto_models (roto.models, roto.model.select / install / download / remove), weights in
the models folder (host::config_dir()/models, shared by the app, CLI and MCP), background
loading, downloads through the system curl, verified; Settings ▸ Roto Brush page (prefs
roto.model). host::config_dir moved from the desktop app.
- Tests: segment (kernels vs direct sums, checkpoint reader, SHA-256 vectors, registry), track
(fake model: prior shapes the cut, failing / disagreeing models ignored; real weights with
EFFECTCRAFT_MOBILESAM: base IoU 0.989, propagated ≥ 0.980 vs classic 0.973), engine (registry,
verification, selection; real install + load), MCP.
- Docs: README, ROADMAP, parity (MSK-4 done: 90/2/0), preferences, agents, architecture,
crates/segment/README.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The camera solver, KLT pyramid and subspace (warp stabilizer) tracker expect()ed data-structure
invariants: a track point seen in a frame, a solved camera, a pyramid level. A missing value now
skips that point/track (or, for finish(), returns a SolveError: "no frame was solved") instead of
panicking. Denies the panic lints in the crate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Viewer frames render in long-lived Web Workers holding a project replica
(engine::remote: structural JSON diff/patch, Mirror, FrameServer); the
frame cache hands jobs to a RemoteFrames renderer (UI-thread fallback).
Roto Brush propagation runs in a job worker and streams segmentations
back (FrameSeg binary form), relayed to frame workers. Control-channel
wait:true commands whose job is offloaded reply when it ends. The Media
Browser browses a pluggable Browser: the file system on the desktop;
browser storage and File System Access folders on the web.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Structure from motion for the 3D Camera Tracker in effectcraft-track
(camtrack): long KLT feature tracks with re-detection, parallax keyframes,
two-view initialisation (normalised 8-point essential matrix in RANSAC or
planar homography decomposition), incremental resection + triangulation,
sparse Levenberg-Marquardt bundle adjustment with the Schur complement and
Huber loss, focal length search (Fixed / Variable Zoom / Specify Angle of
View), tripod pan solve, Auto Detect, ground plane / target / world mapping.
Synthetic-scene tests (path, focal, error, tripod, flat, rendered frames).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Library groundwork for mask tracking (MSK-3) and the Warp Stabilizer (TRK-2),
clean-room from published methods:
- track::klt: Shi-Tomasi corners, pyramidal Lucas-Kanade with forward-backward check
- track::fit: RANSAC + least-squares translation/scale/similarity/affine/homography
- track:😷 MaskTracker following the pixels inside a mask outline per frame
- track::stabilize: incremental clip analysis (serde WarpAnalysis), Gaussian path
smoothing (Matsushita 2006), No Motion, preserve scale, crop / auto-scale with
max-scale relaxation
- path::interp: Mask Interpolation correspondence (arc-length vertex insertion,
shape-context matching) and rigid/linear in-betweens
Engine commands, UI and the effect itself are not wired yet (stubs unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tracker panel (Window > Tracker): Track Camera / Warp Stabilizer (disabled),
Track Motion / Stabilize Motion, Motion Source, Current Track, Track Type,
Position/Rotation/Scale, Motion Target with Edit Target..., Options...
(Motion Tracker Options dialog), Analyze (1 frame back, back, forward,
1 frame forward; stop while running) with progress, Reset and Apply (Apply
Dimensions dialog). Track point widgets in the viewer: feature and search
regions with corner handles, attach point crosshair and track path, all
draggable through track.setPoint (one undo step per drag). Automation ids for
every widget; headless UI test.
Tracker: featureless regions no longer drift (ties keep the prediction, only
confident matches predict motion). docs/agents.md: tracking example.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
