aleleba/effectcraft
1
0
Fork 0
mirror of https://github.com/storytold/effectcraft.git synced 2026-10-24 18:18:17 -06:00
Code Issues Packages Projects Releases Wiki Activity

Merge main into M13.5 long-tail polish

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Thomas
2026-10-03 00:25:09 -04:00
co-authored by Claude Opus 5.5
parent 08daf72e3f b6e1f3ad0c
commit a4f32f8df2
177 changed files with 25734 additions and 510 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/agents.md
+28
View File
@@ -147,6 +147,34 @@ effectcraft-cli exec warp.analyze '{"layer":"#1","wait":true}' shaky.ecproj --sa
effectcraft-cli exec warp.status '{"layer":"#1"}' shaky.ecproj --json
```
### 3D Camera Tracker
Animation ▸ Track Camera and the Tracker panel's button (`track.camera {layer?, shotType?:
fixed|variable|specify, aov?, solveMethod?: auto|typical|flat|tripod, detailed?, wait?}`) apply
Effect ▸ Perspective ▸ 3D Camera Tracker (or reuse the layer's) and analyse it in the background:
"Analyzing in background (step 1 of 2)" tracks features, "Solving camera" solves.
`camera.analyze {layer?, effect?, wait?}` re-runs it, `camera.cancel` stops it without writing
anything. `camera.solveStatus` reports `progress`/`banner`, `analyzed`, `solved`, `methodUsed`,
`averageError` (pixels), `focalLength`, `horizontalAngleOfView`, the current frame's `camera`
(position, orientation, zoom) and `groundPlane`. `camera.points {time?}` lists the solved points
visible now with their `id`, `comp` position, `depth`, `world` position and `error`.
Select points with `camera.selectPoints {points, add?, toggle?}` (the viewer's click / Shift-click /
marquee), then the right-click menu's commands: `camera.setGroundPlane {points?}` (Set Ground Plane
and Origin), `camera.createFromSolve {kind: text|solid|null|shadowCatcher, points? | target:
{center, normal, size?}, multiple?}` (Create Text / Solid / Null / Shadow Catcher, Camera and Light,
Create Multiple …), `camera.deletePoints {points?, wait?}` (re-solves; with Auto-delete Points
Across Time the same feature's other tracks go too) and `camera.create` (the Create Camera button).
Each is one undo step; the first create adds the one-node "3D Tracker Camera" keyed on every frame.
Changing the layer's frames clears the analysis; changing `effects/#1/shotType`,
`effects/#1/horizontalAngleOfView`, `effects/#1/advanced/solveMethod` re-solves the stored tracks.
```sh
effectcraft-cli run shot.ecproj track.camera '{"layer":"#1","wait":true}' \
camera.points '{}' --json
effectcraft-cli run shot.ecproj camera.createFromSolve '{"kind":"solid","points":[12,40,77]}' --save
```
### Roto Brush & Refine Edge
The Roto Brush tool (Alt+W cycles Roto Brush / Refine Edge) paints in the Layer panel; agents use
docs/architecture.md
+64 -5
View File
@@ -16,18 +16,22 @@ cpal or muda. Everything in L0 to L4, the egui UI and the web app also build for
| L0 | `time` | `Tick` (254 016 000 000 per second), rational frame rates incl. NTSC, SMPTE and drop-frame timecode |
| L0 | `geom` | Vectors, matrices, quaternions, the layer transform (anchor, position, scale, orientation, rotation) |
| L0 | `color` | sRGB and linear, HSL/HSV, luminance, the 38 blend modes, label colors |
| L0 | `vp9enc` | VP9 intra-frame encoder (profile 0, 8-bit 4:2:0; lossless at quality 100) for WebM export, from the VP9 bitstream specification |
| L0 | `opusenc` | Opus (CELT) encoder: RFC 6716 CELT-only fullband 48 kHz 20 ms packets, mono/stereo, and the RFC 7845 `OpusHead`, for WebM export audio |
| L1 | `raster` | Premultiplied float images, sampling, affine and projective warps, blurs, compositing (parallel with rayon) |
| L1 | `keyframe` | Animated values, keyframes with temporal ease and spatial Bezier, roving, hold, velocity |
| L1 | `psd` | Photoshop PSD/PSB reader (layers, groups, masks, blend modes, text, layer effects, adjustment layers; 8/16/32-bit; RGB/CMYK/Gray/Lab) and a minimal writer, from Adobe's published format specification |
| L1 | `path` | Bezier paths, path operators (trim, offset, round corners, zig zag, twist, merge…), stroking, coverage masks |
| L2 | `project` | The document: items, compositions, layers, the property tree, render queue model, `.ecproj` serde |
| L2 | `text` | Fonts, shaping, layout, per-glyph geometry, text animators and selectors |
| L2 | `effects` | The effect registry (241 effects) and their CPU implementations |
| L2 | `svg` | SVG import (W3C SVG 1.1/2 static subset: shapes, paths, transforms, `use`, CSS, gradients) and rasterisation at any scale |
| L2 | `model` | 3D models for Advanced 3D: glTF 2.0 (`.gltf`/`.glb`) and OBJ/MTL import (meshes, PBR metallic-roughness materials and textures, node hierarchy, skins, animations), parametric primitives, extruded/bevelled outline meshes and polygon triangulation |
| L2 | `track` | Motion tracking: feature/search region point tracking (pyramid normalized cross-correlation, Lucas–Kanade sub-pixel refinement), confidence, homography/affine/similarity solves |
| L3 | `render` | Evaluation and compositing: sources, masks, effects, transforms, 3D, motion blur, mattes, blending, layer cache, audio mixdown |
| L3 | `media` | Footage decoding (FilmCraft's pure-Rust codecs), image sequences, frame cache |
| L3 | `media` | Footage decoding (FilmCraft's pure-Rust codecs), image sequences, Photoshop and SVG stills, frame cache |
| L3 | `expr` | The expression engine (JavaScript via boa) with the After Effects object model |
| L3 | `export` | Render queue encoding: H.264, ProRes, PNG/JPEG/TIFF/EXR sequences, GIF, audio |
| L3 | `export` | Render queue encoding: H.264, ProRes, WebM (VP9 + alpha, Opus), PNG/JPEG/TIFF/EXR sequences, GIF, WAV/AIFF |
| L3 | `gpu` | The GPU compositor and GPU effects on wgpu compute shaders (Metal, Vulkan, Direct3D 12, WebGPU), checked against the CPU renderer |
| L3 | `lottie` | Lottie JSON / dotLottie import and export (layers, precomps, eased and spatial keyframes, shapes, masks, mattes) with a warnings list for what Lottie cannot express |
| L4 | `engine` | `Session`: project, undo history, editor state, the command registry and menus |
@@ -106,6 +110,14 @@ its evaluated inputs, excluding the transform. Static and transform-only layers
editing one layer re-renders only that layer. Effects that read the clock directly are declared in
`effects::TIME_DEPENDENT`, and a test checks every registered effect against that list.
A persistent **disk cache** (`render::disk_cache`, Settings ▸ Media & Disk Cache) backs both
the layer cache and the viewer's RAM preview: layer buffers that were slow to render and every
preview frame are written (LZ4, atomic rename, checksummed) under 128-bit content keys — the
comp's content hash (the comp, what it uses, footage file size and time) plus frame, scale and
view for frames; the layer key salted with the footage fingerprint for layers — so entries
survive restarts and never go stale. LRU eviction keeps it under the size limit; Edit ▸ Purge
empties it, `cache.diskStats` reports it, and the timeline draws disk-only frames in blue.
**Time effects** (Echo, Posterize Time, Timewarp…) read the layer at other times through
`EffectHost::self_at`: the renderer renders the layer's source and masks (plus, optionally, the
effects before it) at that layer time via `Renderer::layer_input`, cached under separate *input*
@@ -154,7 +166,8 @@ chain with one upload and one readback. Tests render scenes on both paths and co
(≤ 1/255 at 8 bpc, ≤ 1e-3 at 32 bpc); they skip without an adapter. The desktop viewer builds the
`Gpu` on egui-wgpu's device and shows frames from GPU textures without reading them back
(`ui-egui::frames`); headless renders, the CLI (unless `--gpu`) and CI use the CPU. On the web
(WebGPU) the GPU composites viewer frames; steps that need a readback fall back to the CPU.
(WebGPU) the GPU composites viewer frames; steps that need a readback fall back to the CPU (the
Info panel's pixel readout reads GPU frames back asynchronously).
**Colour and bit depth** (`crates/render/src/color.rs`, `crates/color/src/space.rs`). Pixels are
`f32`, but 8 and 16 bpc projects clamp and quantise each layer after its source and masks and
@@ -207,6 +220,40 @@ auto-scale; cached per analysis and settings) and warps each frame; Synthesize E
borders from neighbouring frames read with `EffectHost::self_at`. The effect lives in the
effects crate, hence the `effects → track` edge.
**Essential Graphics** (`project::essential`, `engine::commands::essential`): a comp's
`essential` field lists its exposed controls (properties, Media Replacement, groups, comments).
Every precomp layer of such a comp gets an Essential Properties group (`GroupKind::Essential`,
children `eg<control id>`), kept in step by `Session::edit`; editing a child overrides it for
that instance. The renderer draws an instance with overrides from a copy of the project with the
overridden source properties set (`essential::with_overrides`). Templates are `.ectemplate`
ZIPs: `manifest.json` (controls), `project.ecproj` (the comp and its dependencies), `media/…` and
`poster.png`; importing offsets every id past the project's (`essential::offset_ids`).
**Responsive Design — Time**: a time-stretched precomp maps time piecewise so its protected
marker regions play at 100 % (`eval::responsive_source_time`).
**Proxies and interpretation**: an item's `proxy` (footage or comp) is decoded in its place at
the source's nominal size when `RenderOpts::proxy` (Render Settings ▸ Proxy Use) allows it.
Interpret Footage's Separate Fields turns each field into a frame at twice the rate (the other
lines interpolated); pixel aspect stretches footage, solids and precomps in the comp
(`EvalCtx::par_ratio`, not inherited by children); Invert Alpha and Interpret As Linear Light
apply after decoding.
**3D Camera Tracker** (`effects::camera_tracker`, `engine::camera_track`, `effectcraft-track`'s
`camtrack`): `camera.analyze` / `track.camera` render the layer's input to the effect on a
background thread and run structure from motion in two steps. Step 1 follows Shi–Tomasi features
through the clip with pyramidal Lucas–Kanade (forward–backward checked, re-detected where the frame
has none) into long 2D tracks. Step 2 solves the camera: parallax keyframes; a two-view start from
the essential matrix (normalised 8-point in RANSAC) or a planar homography decomposition; incremental
resection and triangulation; a sparse Levenberg–Marquardt bundle adjustment (Schur complement over
the points, Huber loss) with the focal length fixed, shared, or per frame; a log-spaced and
golden-section focal search for Fixed Angle of View / Variable Zoom; a rotation-only model for tripod
pans, chosen by Auto Detect when it explains the tracks as well. Tracks and solve are stored as JSON
in hidden effect parameters with keys: changing the layer's frames clears both, changing Shot Type,
Angle of View, Solve Method or deleting points only re-solves. The solve's canonical frame maps to
comp space so the first frame's camera is the default comp camera (or so a chosen ground plane is
the X-Z plane at the origin); `camera.createFromSolve` keys a one-node "3D Tracker Camera" on every
frame and places text, solids, nulls or a shadow catcher and light on the target plane.
**Roto Brush & Refine Edge** (`effectcraft_track::roto`, `effects::roto`, `engine::roto`):
strokes (foreground, background, Refine Edge) are stored as JSON in the effect's hidden Strokes
parameter with the base frame and segmentation span. Each frame is segmented by graph cut
@@ -224,7 +271,11 @@ in the effect.
Expressions are JavaScript, run by boa, with After Effects' object model (`thisComp`, `thisLayer`,
`time`, `value`, `wiggle`, `loopOut`, vector maths on arrays, and so on). A syntax error keeps the
text, disables the expression and shows a warning, like After Effects.
text, disables the expression and shows a warning, like After Effects. `sampleImage` renders the
sampled layer through the evaluating renderer's footage source (`EvalCtx::footage`), cached per
thread and frame; `footage(name)` reads data footage (JSON, CSV, TSV imported with File ▸ Import,
text kept in the project) through `sourceData`, `sourceText` and `dataValue`. `expr.errors`
lists failing expressions for the viewer's error bar.
## 5a. Scripting
@@ -259,6 +310,14 @@ inspect and click the interface. See [agents.md](agents.md) and
Settings (`engine::prefs`), keyboard shortcut presets (`engine::shortcuts`) and auto-save /
crash recovery (`engine::autosave`) live in the engine too; they persist through a
`ConfigStore` the frontend provides (a directory on the desktop, `localStorage` on the web).
`ConfigStore` the frontend provides (a directory on the desktop; the Origin Private File System,
or IndexedDB, on the web). Auto-saves go through the store's `FileOps` (the file system by
default, the browser storage on the web).
Renders and analyses run on background threads on the desktop. Where there are no threads (the
browser), `engine::offload` sends them to a second engine instance instead: the session's
`Offload` ships the serialized project, the footage it reads and the job (`WorkerRequest`) to a
Web Worker, and the worker's replies (progress, item status, the analysed property group) are
applied on the UI thread.
The shortcut dispatcher and the menus read the active preset. See
[preferences.md](preferences.md).
docs/parity.md
+48 -22
View File
@@ -1,9 +1,35 @@
# Parity with After Effects
How close EffectCraft is to After Effects 2026, feature by feature, and how much work is left.
Audited against the code at commit `5273f1b` (2 October 2026). Every feature in our catalog was
graded from the code itself: a disabled menu entry, or a setting that is stored but never
rendered, does not count as done.
## Current status (audit at commit `fa26ad9`, 2 October 2026, late)
| Measure | Value |
|---|---|
| **Feature parity, weighted by tier** (P0 ×3, P1 ×2, P2 ×1) | **≈ 93%** |
| Unweighted | ≈ 91% |
| P0 / P1 / P2 | ≈ 96% / 90% / 65% |
| Features done / partial / missing | 41 / 50 / 1 of 92 (missing: a plug-in API) |
| **Effects** | **298 of 298** After Effects 2026 effects exist (some still simplified) |
| Disabled menu entries left | 26 commands (31 menu items) |
| Remaining work | ≈ 50–55 agent-hours at the pace measured so far (≈ 150 by the original conservative audit scale) |
| **Wall-clock estimate** | **≈ 11–14 hours** with five agents in parallel; ≈ 6–8 hours for 100% of P0 + P1 |
By area: Layers 98%, Output 95%, Compositions 95%, Automation 96%, Paint 95%, Text 95%, Import 96%,
Animation 94%, Masks 94%, Preview 94%, Interface 93%, Shapes 93%, 3D 91%, Audio 90%, Project 90%,
Tracking 88%, Effects 85%, Web 65%.
What is left, in priority order: the web app's depth (threads, storage, audio, non-blocking
renders); stroke taper/wave and multi-segment dashes; variable mask feather points; camera iris/bokeh and focus-link commands; GPU
rendering of 3D runs and adjustment layers; Render Queue field render, crop/resize and templates;
approximated effects (Subspace Warp, Key Cleaner) and Liquify's viewer brush; Advanced 3D motion blur
and blend modes; a few viewer and colour-management menu items; OpenType features and variable
font axes; AI/EPS/PDF import; more codecs; 59 preferences not yet wired; ScriptUI;
Lumetri Scopes, Footage, Media Browser and Metadata panels; Content-Aware Fill; face tracking; and
the "better than After Effects" items (a plug-in API, branching history, GPU particles).
The sections below are the original audit (morning of 2 October, ≈ 64%) and its updates, kept for
history.
## Summary
@@ -15,7 +41,7 @@ rendered, does not count as done.
| P1 (professional daily use) | ≈ 38% |
| P2 (long tail) | ≈ 17% |
| Features done / partial / missing | 18 / 60 / 14 of 92 |
| **Effects** | **297 of 298** After Effects 2026 effects by name (99.7%, after M9.11 and Warp Stabilizer; was 257); only the 3D Camera Tracker is left |
| **Effects** | **298 of 298** After Effects 2026 effects by name (100%, after M9.11, Warp Stabilizer and the 3D Camera Tracker; was 257) |
| Remaining work | ≈ **167 agent-hours** (139 h of features + ≈ 20% for 1:1 polish against After Effects) |
| Wall-clock estimate | ≈ **42–50 hours** with five agents working in parallel and one integrating; ≈ 31–35 h for P0 + P1 only |
@@ -73,39 +99,39 @@ library groundwork for mask tracking and Warp Stabilizer.
| Remaining (audit-scale agent-hours) | ≈ 167 | ≈ 100 |
| Wall clock, five parallel agents | 42–50 h | **≈ 8–10 h at today's measured pace; ≈ 25 h by the audit's conservative figures** |
What is left is concentrated in large systems: the 3D camera tracker,
the Warp Stabilizer and mask-tracking UI on top of
the new tracking library, SVG/PSD import, WebM, a disk cache, and wasm threads. (The missing
effect categories, 3D Channel, Immersive Video and OCIO, landed in M9.11; the scripting object
model in M14.4; Roto Brush and Refine Edge in M6.7; native macOS menus, the Composition Flowchart,
Timeline column/search depth, the Home screen and every After Effects workspace in the UI-polish wave.)
What is left is concentrated in large systems: the Warp Stabilizer and mask-tracking UI on top of
the new tracking library, and wasm threads. (The missing effect categories, 3D Channel, Immersive
Video and OCIO, landed in M9.11; the scripting object model in M14.4; Roto Brush and Refine Edge in
M6.7; native macOS menus, the Composition Flowchart, Timeline column/search depth, the Home screen
and every After Effects workspace in the UI-polish wave; PSD/SVG import, WebM/WAV/AIFF output and
the disk cache in the formats wave; the 3D Camera Tracker in M12.6.)
## By area
| Area | Weighted parity | Remaining (agent-hours) | Biggest gaps |
|---|---|---|---|
| Layers | 88% | 4.8 | frame blending, collapse transformations, slip edit |
| Output | 85% | 4.2 | multiple output modules, pre-render, WebM, audio-only |
| Output | 90% | 3.0 | multiple output modules, pre-render (WebM with VP9 alpha + Opus and WAV/AIFF audio-only landed: VP9 is intra-only, Opus CELT-only) |
| Audio | 85% | 0.5 | audio to keyframes |
| Import | 78% | 4.5 | PSD, SVG as shapes, Lottie |
| Import | 88% | 2.0 | AI/EPS/PDF vector footage, PSD smart objects and 3D layers (PSD as footage/composition/retain layer sizes, SVG footage and Create Shapes from Vector Layer landed) |
| Automation | ≈ 90% | 0.8 | scripting covers the documented core object model (AUT-2, M14.4: `app`, project items, comps, layers, properties and keyframes, text documents, markers, render queue, Script Console, `effectcraft-cli script`, MCP `run_script`); still missing: ScriptUI panels/dialogs, `.jsxbin`, sockets |
| Shapes | ≈ 80% | 1.5 | Lottie can't carry stroke taper/wave (stroke Taper and Wave, Dash 2/Gap 2/Dash 3/Gap 3 and radial-gradient Highlight Length/Angle landed in M13.5; pen tool for shape paths and vertex editing in M6.5) |
| Compositions | 72% | 6.0 | Essential Graphics panel (the marker dialog and Composition Flowchart landed) |
| Compositions | ≈ 80% | 3.5 | Mocha-style planar tracks for templates, Essential Graphics' rare controls (font menus, mirrored properties) (the marker dialog, Composition Flowchart, Essential Graphics with master properties, `.ectemplate` templates and Responsive Design — Time landed: CMP-6, CMP-7) |
| Animation | 67% | 10.0 | puppet, Wiggler/Smoother/Motion Sketch (motion-path handles and the graph editor transform box landed in M5.8; keyframe colour labels and Select Keyframe Label Group, Graph Editor snapping to markers / layer ends in M13.5) |
| Text | ≈ 91% | 0.8 | no OpenType feature panel, no extruded strokes, variable-axis animation changes outlines but not advances (vertical Roman / Tate-Chu-Yoko, forced LTR paragraphs, caret on animated and path text, Variable Font Axes and Lottie style runs landed in M13.5; extruded, bevelled text in M7.6; per-character styles, paragraph settings, on-canvas editing and the `sourceText` style API in M9.9–M9.10) |
| Web | 60% | 2.5 | threads, browser storage, audio |
| 3D | 74% | 11.3 | 3D camera tracker, multi-view layouts, Advanced 3D motion blur and blend modes (and iris bokeh in Advanced 3D's depth of field), cameras/lights from models; Classic 3D iris-shaped bokeh with highlights, progressive DOF on tilted layers and the focus-link commands landed in M13.5; Advanced 3D (glTF/OBJ models, primitives, extruded text and shapes, PBR, image-based light, shadow maps, GPU rasteriser) in M7.4–M7.6 |
| Effects | ≈ 75% | 5.0 | GPU versions of more effects and the 3D Camera Tracker (every other After Effects effect exists since M9.11 and M12.5) |
| Web | 85% | 1.0 | viewer frames and Roto Brush propagation still on the page's thread; GPU effects in the browser (browser storage, Web Audio, Web Worker renders/analyses, WebGPU viewer and offline install landed in M15.2) |
| 3D | 78% | 9.3 | multi-view layouts, Advanced 3D motion blur and blend modes (and iris shapes in Advanced 3D's depth of field), cameras/lights from models; Classic 3D iris-shaped bokeh with highlights, progressive depth of field on tilted layers and the focus-link commands landed in M13.5; Advanced 3D (glTF/OBJ models, primitives, extruded text and shapes, PBR, image-based light, shadow maps, GPU rasteriser) in M7.4–M7.6 |
| Effects | ≈ 75% | 5.0 | GPU versions of more effects (every After Effects effect exists since M9.11, M12.5 and M12.6) |
| Interface | 74% | 5.0 | a richer Learn area (Timeline outline and Project panel columns scroll horizontally, the Layer Style dialog, ROI resize handles, Pan Behind snapping and 3D Reference Axes landed in M13.5; native macOS menu bar, Timeline columns/search/reveal-add, Home screen with recent projects and all AE workspaces landed; viewer rulers/snapping/channels/snapshots landed in M0.13) |
| Project | 53% | 8.1 | 8/16/32-bit pipeline and colour management, auto-save, proxies, folder moves |
| Project | ≈ 60% | 6.5 | 8/16/32-bit pipeline and colour management, auto-save, folder moves (proxies and Interpret Footage fields / pixel aspect / alpha guess landed: PRJ-8, PRJ-3) |
| Masks & roto | 74% | 5.0 | Roto Brush's learned (3.0) segmentation model (variable-width mask feather points with the Mask Feather tool landed in M13.5; mask tracking and Mask Interpolation landed in M6.6; Roto Brush & Refine Edge with graph-cut segmentation, flow propagation, edge matting, decontamination and Freeze in M6.7) |
| Preview | 55% | 6.1 | GPU 3D and adjustment layers, disk cache (region of interest, snapshots, exposure and Fast Previews landed in M0.13) |
| Tracking | ≈ 70% | 4.0 | 3D camera tracker, face tracking and Subspace Warp's mesh warp (Rolling Shutter Repair landed in M9.11; point tracker, mask tracking and Warp Stabilizer landed in M6.x / M12.5) |
| Preview | 62% | 5.0 | GPU 3D and adjustment layers (persistent disk cache with the blue cache bar landed; region of interest, snapshots, exposure and Fast Previews landed in M0.13) |
| Tracking | ≈ 85% | 2.5 | face tracking, Subspace Warp's mesh warp, lens distortion in the camera solve (Rolling Shutter Repair landed in M9.11; point tracker, mask tracking, Warp Stabilizer and the 3D Camera Tracker landed in M6.x / M12.5 / M12.6) |
| Paint | 0% | 4.0 | Brush, Clone Stamp, Eraser |
## Effects still missing
The 3D Camera Tracker (built on the tracking library by other work). Boris FX
None: the 3D Camera Tracker landed in M12.6. Boris FX
Mocha and Cineware are third-party and not counted.
### Added in M9.11 (39 effects)
@@ -152,8 +178,8 @@ taper/wave, Advanced 3D's DOF has no iris shapes, and variable axes don't change
9. A real 8/16/32-bit pipeline with linear blending and colour management.
10. ~~Drag-to-dock and floating panels, saved workspaces, native macOS menus.~~
11. Auto-save, crash recovery, recent projects (in progress).
12. Expression gaps: `sampleImage`, `footage()` (the `sourceText` style API landed in M9.9).
13. Lottie (in progress), WebM, SVG and PSD import.
12. ~~Expression gaps: `sampleImage`, `footage()`~~ (landed with data footage, the error bar and the Expression Language menu; the `sourceText` style API landed in M9.9).
13. ~~Lottie, WebM, SVG and PSD import~~ (landed; a disk cache too).
14. Puppet and paint tools (in progress).
15. Preferences and a shortcut editor that can rebind (in progress); real Wiggler, Smoother and Motion Sketch; the marker dialog.
docs/web.md
+138 -30
View File
@@ -2,9 +2,10 @@
`apps/effectcraft-web` runs the same engine and egui UI as the desktop app in the browser: the
workspace compiled to `wasm32-unknown-unknown`, started by eframe's web runner on WebGPU (WebGL2
fallback). It is a static site: one `.wasm`, its `wasm-bindgen` JavaScript glue, `index.html` and an
icon. Nothing is uploaded anywhere; projects, media and renders stay in the page and on the user's
machine.
fallback). It is a static site: one `.wasm`, its `wasm-bindgen` JavaScript glue, `index.html`, a
few small scripts (worker, audio worklet, service worker), a web manifest and icons. Nothing is
uploaded anywhere: projects, media, settings and renders stay in the browser (its private storage)
and on the user's machine.
## Build and run
@@ -15,48 +16,136 @@ machine.
| wasm-opt (optional) | any recent binaryen | `brew install binaryen`; used by `cargo xtask web` when found |
```sh
cargo xtask web # release build → <target>/web/dist (index.html, effectcraft_web.js, effectcraft_web_bg.wasm, favicon.svg)
cargo xtask web # release build → <target>/web/dist
cargo xtask web --dev # unoptimised build (faster to compile, slow to run)
cargo xtask web --serve 8765 # build, then serve dist on http://127.0.0.1:8765/
```
`<target>` is `$CARGO_TARGET_DIR` or `target`. `--serve` is a tiny localhost-only static server that
sends `Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy`, so the page is cross-origin
`<target>` is `$CARGO_TARGET_DIR` or `target`. `dist` holds `index.html`, `effectcraft_web.js` +
`effectcraft_web_bg.wasm` (+ `snippets/`, the browser glue `js/host.js`), `worker.js`,
`audio-worklet.js`, `sw.js` (its precache list and version filled in by the build),
`manifest.webmanifest`, `favicon.svg` and `icon-256.png` / `icon-512.png` (copied from
`assets/app-icon`). `--serve` is a tiny localhost-only static server that sends
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy`, so the page is cross-origin
isolated like a production deployment should be. Any static server works
(`python3 -m http.server -d target/web/dist 8765`); it must serve `.wasm` as `application/wasm`.
Service workers need a secure context: `https://`, or `http://localhost` / `127.0.0.1`.
The release `.wasm` is about 31 MB before `wasm-opt` (20 MB code: the effects, codecs, the
expression engine and the UI; 8.5 MB data, mostly the bundled fonts; 2 MB function names for
readable panics). Served compressed it is a fraction of that.
The release `.wasm` is about 38 MB before `wasm-opt` (code: the effects, codecs, the expression
engine and the UI; data, mostly the bundled fonts; function names for readable panics). Served
compressed it is a fraction of that, and the service worker caches it after the first visit.
The web crate is empty on non-wasm targets, so `cargo test --workspace` and clippy are unaffected;
`cargo xtask wasm` (part of `cargo xtask ci`) checks it, with every L0–L4 crate and
`effectcraft-ui-egui`, for `wasm32-unknown-unknown`.
On other targets the web crate holds only its portable storage model (`store.rs`, unit-tested by
`cargo test --workspace`); `cargo xtask wasm` (part of `cargo xtask ci`) checks the whole crate,
with every L0–L4 crate and `effectcraft-ui-egui`, for `wasm32-unknown-unknown`.
URL flags: `?empty` (start without the demo project), `?home` (show the start screen).
URL flags: `?empty` (a blank project; the session is not restored or recorded), `?demo` (the demo
project instead of the last session), `?home` (show the start screen), `?storage=indexeddb` /
`?storage=memory` (force a storage backend), `?noworkers` (render and analyse on the page's
thread), `?nosw` (don't register the service worker).
## How the desktop pieces map to the browser
| Desktop | Web |
|---|---|
| frame render thread pool (`ui-egui/src/frames.rs`) | `Frames::pump`: queued frames render on the UI thread after each egui frame, most important first (the viewer's frame, then prefetch), within 40 ms (24 ms while playing) |
| Render Queue on a background thread | `Session::start_render` always runs inline on wasm32: the page waits until the render is done |
| Render Queue on a background thread | a Web Worker running a second engine instance (below); progress streams back, the page never waits |
| tracker / mask tracker / Warp Stabilizer / 3D Camera Tracker / Roto Brush Freeze threads | the same workers; the analysed property group comes back as one undo step |
| `rayon` parallel loops (raster, effects, export batches) | rayon's global pool falls back to the calling thread when threads are unavailable: same code, serial |
| `std::time::Instant` / `SystemTime` (panic on wasm32) | `web-time` (re-exports `std::time` on native) |
| `std::fs` project reads/writes (`FsServices`) | `files::WebServices`: an in-memory file table; saving downloads the `.ecproj` |
| `std::fs` project reads/writes (`FsServices`) | `files::WebServices`: the virtual file table, persisted (below); saving also downloads the `.ecproj` |
| config directory: settings, shortcut presets, recent projects, the recovery sentinel | `store::WebConfig` in browser storage |
| auto-save folder | `/EffectCraft Auto-Save/` in browser storage (`ConfigStore::files`) |
| media reads (`MediaPool`, `probe`) | `files::WebImporter`: bytes from the file table, handed to `MediaPool::add_bytes` / `probe_bytes` |
| export writes (`effectcraft-export`) | the job's `sink` (`effectcraft_host::FileExporter { sink }`): files land in the table and download after the render; several files (an image sequence) download as one stored `.zip` |
| rfd file dialogs | `<input type=file>` for File ▸ Open / Import; drop files anywhere on the page (`.ecproj` opens, everything else imports); "Save As" / "Output To" pick a download name |
| system fonts | not scanned; the bundled fonts (Inter, Noto Serif, JetBrains Mono) are always there |
| TCP control channel / MCP | `window.effectcraft` (below) |
| audio output | none yet (if `cpal` lands for desktop playback it must stay behind a cfg/feature off wasm32; a WebAudio backend would replace it) |
| cpal audio output | Web Audio (below) |
| GPU compositor on the desktop's wgpu device | the same compositor on eframe's WebGPU device (below) |
### Threads
### Persistence
wasm threads need a cross-origin isolated page **and** a wasm build with atomics (a nightly
`build-std` toolchain), so this build is single-threaded: frames, effects and exports run on the UI
thread. A long Render Queue job freezes the page until it finishes; keep web renders short or
small (resolution, time span).
Everything that lives in files on the desktop lives in the browser's **Origin Private File
System** (OPFS); where OPFS can't be written from the page (older Safari), **IndexedDB**; with
neither (some private windows), memory only. `effectcraft.info().storage.backend` says which.
The page asks for persistent storage (`navigator.storage.persist()`), which browsers grant to
installed apps, so the store isn't evicted under storage pressure.
The engine's storage traits are synchronous and the browser's are not, so the app works on an
in-memory mirror (`store.rs`): every stored entry is read before the app starts, reads and writes
hit the mirror, and changes flush in the background (one write per changed key, in order; OPFS
writes are atomic: a swap file replaced on close). Keys:
| Key | What |
|---|---|
| `config/prefs.json`, `config/shortcuts.json` | Settings (with File ▸ Open Recent's list) and keyboard shortcut presets |
| `config/session.lock` | the crash-recovery sentinel |
| `config/session.ecproj`, `config/session.json` | the session snapshot: the open project and editor state (active comp, time, selection), written at most once a second while they change |
| `files/<path>` | the file table: imported media, projects saved with File ▸ Save / Save As or `saveToBrowser`, auto-saves |
On the next visit the app reopens the snapshot (unsaved changes stay unsaved: the project is
dirty), registers every stored media file with the media pool so its footage decodes, and File ▸
Open Recent reopens saved projects from storage. Without a snapshot (first visit) it opens the
demo project. Render outputs are not stored: they download.
### Background jobs: Web Workers
wasm threads need shared memory, which needs a nightly `build-std` toolchain, so this build is
single-threaded per instance. Long jobs run in **Web Workers** instead, each with its own instance
of the same module (the page compiles it once and posts the `WebAssembly.Module`, so nothing is
compiled twice) and its own memory (`crates/engine/src/offload.rs`, `src/worker.rs`):
1. The session's `Offload` serializes a `WorkerRequest`: the project JSON, the footage paths it
reads, and the job (`render` with the resolved queue items and output paths; `warp`, `camera`,
`track`, `maskTrack`, `rotoFreeze` with the target and analysis parameters).
2. `js/host.js` takes an idle worker from its pool (or starts one), sends it the footage bytes it
doesn't have yet, then the request.
3. The worker runs the job on a plain engine session, blocking, and posts `WorkerReply`s: render
progress (at most every 100 ms), item started/finished, rendered files (transferred), analysis
progress, the analysed property group, `failed`, `done`.
4. The page applies them every frame: `Session::poll_render` updates the queue items and the
progress bar, `Session::poll_offload` writes an analysis result as one undo step (the effect /
tracker / mask group is replaced; other edits made meanwhile are kept) and the downloads start.
Stopping a job terminates its worker (a worker can't be interrupted mid-frame): the item being
rendered becomes "User Stopped", an analysis writes nothing. `renderQueue.render {wait: true}` (the
default for agents) and analysis commands with `wait: true` still run on the page's thread and
return when done. Roto Brush propagation fills the session's segmentation cache rather than the
project, so it runs on the page's thread too, as do viewer frames (`Frames::pump`).
### Audio
Audio preview goes through **Web Audio**: an AudioWorklet (`audio-worklet.js`; a
ScriptProcessorNode where worklets are unavailable) plays interleaved stereo blocks. The mixdown
is the desktop's (`AudioPlayback`, `mix_comp`): without a feeder thread, `AudioPlayback::pump`
mixes ahead on every UI frame (≈ 350 ms queued) and the output device takes what keeps it
≈ 150 ms ahead of the playhead. A/V sync follows `AudioContext.currentTime`: the playback clock
is the frames played by the context clock minus its output latency, and the viewer shows the
frame under the audible sample, as on the desktop. The Audio panel's meters read the same feed.
Browsers start an AudioContext suspended until the user interacts with the page; the app resumes
it on the first pointer or key event, so audio plays from the first preview after any click or
key press.
### GPU
eframe starts on WebGPU where the browser has it (WebGL2 otherwise), and the GPU compositor
(`effectcraft-gpu`) runs on the same device: viewer frames are composited by compute shaders and
drawn straight from their texture, with no readback. Steps that need the CPU mid-render (3D runs,
adjustment layers, CPU-only effects) fall back to the CPU renderer for that frame, because the
page's thread can't wait for a GPU readback. The Info panel's pixel readout and the eyedroppers
read GPU frames back asynchronously (`Gpu::read_display_async`: the pixels arrive a frame
later). On WebGL2, or without a usable adapter, everything renders on the CPU.
`effectcraft.info().gpu` reports `{compositor, viewerOnGpu}`.
### Offline and install (PWA)
`manifest.webmanifest` (name, icons, standalone display) makes the app installable; `sw.js`
precaches every file of the build into a cache named after the build's hash, serves them
cache-first (the cached responses keep the server's COOP/COEP headers, so the page stays
cross-origin isolated offline), and drops older caches when a new build activates. After the
first visit the app loads without a network.
## `window.effectcraft`: the agent / test API
@@ -70,34 +159,53 @@ with the method's `result`, errors reject with the message.
| `effectcraft.commands()` / `effectcraft.inspect()` | `engine.commands` / `ui.inspect` |
| `effectcraft.renderFrame(params)` | `render.frame` with `base64: true`: `{comp, time, width, height, png}` |
| `effectcraft.screenshot(params)` | `ui.screenshot`; without `path` the PNG comes back inline as `png` (base64) |
| `effectcraft.addFile(fileOrUrl, name?)` | put a `File`/`Blob` (or fetched URL) into the file table and open (`.ecproj`) or import it; resolves with its path |
| `effectcraft.addFile(fileOrUrl, name?)` | put a `File`/`Blob` (or fetched URL) into the file table (stored) and open (`.ecproj`) or import it; resolves with its path |
| `effectcraft.files()` / `effectcraft.readFile(path)` | the file table `[{path, size}]` / a file's bytes (`Uint8Array`), e.g. a render |
| `effectcraft.info()` | graphics backend, version, `crossOriginIsolated`, load timings |
| `effectcraft.saveToBrowser(path?)` | save the project to browser storage without downloading it (default: its path, or `/<name>.ecproj`); resolves with `{path, bytes}` |
| `effectcraft.listStored()` | `{backend, usage, quota, persisted, pending, files: [{path, size, modified}], config: [name]}` |
| `effectcraft.removeStored(path)` | delete a stored file |
| `effectcraft.flush()` | resolves once every change is written to browser storage |
| `effectcraft.info()` | graphics backend, `gpu`, `storage`, `audio` (`{state, sampleRate, backend, posted, played, underruns}`), `workers`, `restored`, `webgpu`, `serviceWorker`, version, `crossOriginIsolated`, load timings |
`window.effectcraftLoad` holds `{wasmMs, readyMs}` (module fetch + compile, and until the app runs).
```js
await effectcraft.execute("layer.newSolid", {color: "#ff8800"});
await effectcraft.request("ui.menu.invoke", {id: "renderQueue.add", params: {format: "gif", resolution: 0.25}});
await effectcraft.execute("renderQueue.render", {}); // the GIF downloads
await effectcraft.execute("renderQueue.render", {wait: false}); // renders in a worker; the GIF downloads
await effectcraft.saveToBrowser("/my-project.ecproj"); // File ▸ Open Recent has it after a reload
```
## Browser test
`apps/effectcraft-web/tests/smoke.mjs` drives headless Chrome over the DevTools protocol (Node ≥ 22,
no npm packages): load, the demo comp in the viewer, `render.frame`, Render Queue GIF and PNG
sequence (`.zip`) downloads, Save As and re-opening the saved project. It writes screenshots and
`report.json`:
no npm packages): load, the GPU path (WebGPU → GPU compositor, viewer on the GPU), the demo comp
in the viewer, `render.frame`, Render Queue GIF and PNG sequence (`.zip`) downloads, a background
render in a worker while measuring the page's event-loop gaps (must stay under 400 ms) and its
progress, the AudioContext starting on a (simulated) key press and a tone playing through it
with the meters moving, Save As and re-opening the saved project, persistence across a reload
(session snapshot, a stored project, imported media, a setting, Open Recent) and an offline
reload through the service worker. It writes screenshots and `report.json`:
```sh
cargo xtask web --serve 8765 &
node apps/effectcraft-web/tests/smoke.mjs --url http://127.0.0.1:8765/ --out target/web/smoke
```
Native unit tests cover the storage model (`apps/effectcraft-web/src/store.rs`: write
coalescing, file table, `ConfigStore` / `FileOps`, auto-save and crash recovery through the
browser store) and the worker protocol (`crates/engine/src/offload.rs`: serde round trips of
every request and reply, a render through an in-process offload).
## Gaps
- Single-threaded (above); Render Queue renders block the page.
- No audio playback.
- Imported media live in memory for the session only (nothing is persisted; reload = fresh start).
- Viewer frames, RAM preview and Roto Brush propagation still render on the page's thread (one
frame at a time between UI frames).
- GPU effects run only in the desktop GPU path: in the browser, effects run on the CPU (inside
workers for renders) because the page's thread can't wait for mid-render readbacks.
- No disk cache (Settings ▸ Media & Disk Cache) in the browser: the layer cache stays in memory.
- Cancelling an analysis drops its partial result (the worker is terminated).
- Stored media count against the origin's storage quota; `effectcraft.listStored()` shows usage,
`removeStored` frees it (there is no storage manager in the UI yet).
- `engine.execute` rejects the spread Render Settings / Output Module keys of `renderQueue.add`
(`resolution`, `timeSpan`…) through its strict parameter check; use `ui.menu.invoke` as above.