diff --git a/ATTRIBUTION.md b/ATTRIBUTION.md index abe06308d..38a3d624e 100644 --- a/ATTRIBUTION.md +++ b/ATTRIBUTION.md @@ -12,6 +12,9 @@ iconography, images or other Adobe assets; open licences only; every asset attri | `assets/fonts/Inter-Regular.ttf` | The Inter Project Authors | https://github.com/rsms/inter | OFL-1.1 (`assets/fonts/OFL-Inter.txt`) | | `assets/fonts/Inter-Medium.ttf` | The Inter Project Authors | https://github.com/rsms/inter | OFL-1.1 (`assets/fonts/OFL-Inter.txt`) | | `assets/fonts/Inter-SemiBold.ttf` | The Inter Project Authors | https://github.com/rsms/inter | OFL-1.1 (`assets/fonts/OFL-Inter.txt`) | +| `assets/fonts/Inter-Bold.ttf` | The Inter Project Authors | https://github.com/rsms/inter | OFL-1.1 (`assets/fonts/OFL-Inter.txt`) | +| `assets/fonts/Inter-Italic.ttf` | The Inter Project Authors | https://github.com/rsms/inter | OFL-1.1 (`assets/fonts/OFL-Inter.txt`) | +| `assets/fonts/NotoSerif-Regular.ttf` | The Noto Project Authors | https://github.com/notofonts/latin-greek-cyrillic | OFL-1.1 (`assets/fonts/OFL-NotoSerif.txt`) | | `assets/fonts/JetBrainsMono-Regular.ttf` | The JetBrains Mono Project Authors | https://github.com/JetBrains/JetBrainsMono | OFL-1.1 (`assets/fonts/OFL-JetBrainsMono.txt`) | | `docs/images/filmcraft-hero.png` | FilmCraft contributors | Original work: FilmCraft screenshot; frames from public-domain films (Night of the Living Dead 1968, Carnival of Souls 1962), CC0 Chopin (Musopen) | MIT OR Apache-2.0; film frames public domain | | `docs/images/filmcraft-color.png` | FilmCraft contributors | Original work: FilmCraft screenshot; frames from Charade (1963, public domain) | MIT OR Apache-2.0; film frames public domain | @@ -23,6 +26,26 @@ iconography, images or other Adobe assets; open licences only; every asset attri | `docs/images/filmcraft-effects.png` | FilmCraft contributors | Original work: FilmCraft screenshot crop; NASA Earth Views (US Government work, public domain) | MIT OR Apache-2.0; footage public domain | | `docs/images/filmcraft-timeline.png` | FilmCraft contributors | Original work: FilmCraft UI crop; thumbnails from Night of the Living Dead (1968) and Carnival of Souls (1962), CC0 Chopin waveform | MIT OR Apache-2.0; film frames public domain | +## Golden test images + +Reference frames for the renderer golden tests (`crates/golden`), rendered by FilmCraft itself from +procedural demo footage (`crates/media/src/generators.rs`); no external media. Regenerate with +`FILMCRAFT_BLESS=1 cargo test -p filmcraft-golden`. + +| Asset | Author | Source | Licence | +|---|---|---|---| +| `crates/golden/goldens/blend_modes.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `blend_modes`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/crop.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `crop`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/gaussian_blur.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `gaussian_blur`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/graphic_shapes.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `graphic_shapes`) from procedurally generated media and bundled OFL fonts | MIT OR Apache-2.0 | +| `crates/golden/goldens/graphic_title.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `graphic_title`) from procedurally generated media and bundled OFL fonts | MIT OR Apache-2.0 | +| `crates/golden/goldens/lumetri_basic.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `lumetri_basic`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/text_burnin.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `text_burnin`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/transform_opacity.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `transform_opacity`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/transition_cross_dissolve.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `transition_cross_dissolve`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/transition_dip_to_black.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `transition_dip_to_black`) from procedurally generated media | MIT OR Apache-2.0 | +| `crates/golden/goldens/transition_wipe.png` | FilmCraft contributors | Original work: golden reference rendered by FilmCraft (crates/golden/tests/golden.rs, scene `transition_wipe`) from procedurally generated media | MIT OR Apache-2.0 | + ## Third-party media shown in screenshots The README screenshots show frames from these works. The media files themselves are not in the @@ -44,6 +67,7 @@ is traced or derived from Adobe artwork. | Asset | Where | |---|---| | UI icons (tools, transport, panels, header) | `crates/ui-egui/src/icons.rs`: vector paths on a 16×16 grid | +| Align / distribute / paragraph-alignment glyphs in the graphics panel | `crates/ui-egui/src/panels/graphics.rs` (`align_glyph`): bars and lines drawn with the egui painter | | Demo footage (ocean sunset, aurora, city night, dunes, forest, plasma), bars and tone, counting leader, colour matte | `crates/media/src/generators.rs` | | Lumetri "Look" presets (Teal & Orange, Warm Film, …) | `crates/render/src/effects.rs` (`apply_look`): procedural colour transforms, not LUT files | | Colour palette and layout metrics | `crates/ui-egui/src/theme.rs`: colour values and sizes only; no artwork | diff --git a/Cargo.lock b/Cargo.lock index aca993685..4097c7b84 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1103,6 +1103,24 @@ dependencies = [ "winit", ] +[[package]] +name = "egui_kittest" +version = "0.36.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb3f3effb2726fee23449293e67acbf6cc6ce35525b1722bd5743bb348114d5" +dependencies = [ + "eframe", + "egui", + "egui-wgpu", + "image", + "kittest", + "log", + "pollster 1.0.1", + "serde", + "toml 1.1.6+spec-1.1.0", + "wgpu", +] + [[package]] name = "either" version = "1.18.0" @@ -1306,6 +1324,7 @@ name = "filmcraft-aac" version = "0.1.0" dependencies = [ "filmcraft-bitstream", + "filmcraft-testkit", "rayon", "thiserror 2.0.21", ] @@ -1313,6 +1332,9 @@ dependencies = [ [[package]] name = "filmcraft-audio-dsp" version = "0.1.0" +dependencies = [ + "filmcraft-testkit", +] [[package]] name = "filmcraft-automation" @@ -1343,9 +1365,9 @@ version = "0.1.0" dependencies = [ "filmcraft-color", "filmcraft-project", + "filmcraft-text", "filmcraft-time", "proptest", - "skrifa", "thiserror 2.0.21", ] @@ -1378,6 +1400,7 @@ dependencies = [ "filmcraft-opus", "filmcraft-prores", "filmcraft-time", + "filmcraft-vp9", "image", "log", "symphonia", @@ -1419,6 +1442,7 @@ dependencies = [ "filmcraft-media", "filmcraft-project", "filmcraft-render", + "filmcraft-text", "filmcraft-time", "log", "serde", @@ -1439,6 +1463,7 @@ dependencies = [ "filmcraft-project", "filmcraft-prores", "filmcraft-render", + "filmcraft-testkit", "filmcraft-time", "image", "rayon", @@ -1477,6 +1502,21 @@ dependencies = [ "serde", ] +[[package]] +name = "filmcraft-golden" +version = "0.1.0" +dependencies = [ + "filmcraft-geom", + "filmcraft-gpu", + "filmcraft-media", + "filmcraft-project", + "filmcraft-render", + "filmcraft-testkit", + "filmcraft-time", + "pollster 0.4.0", + "wgpu", +] + [[package]] name = "filmcraft-gpu" version = "0.1.0" @@ -1498,6 +1538,7 @@ name = "filmcraft-h264" version = "0.1.0" dependencies = [ "filmcraft-bitstream", + "filmcraft-testkit", "rayon", "thiserror 2.0.21", ] @@ -1507,6 +1548,7 @@ name = "filmcraft-h264enc" version = "0.1.0" dependencies = [ "filmcraft-bitstream", + "filmcraft-testkit", "rayon", "thiserror 2.0.21", ] @@ -1516,6 +1558,7 @@ name = "filmcraft-hevc" version = "0.1.0" dependencies = [ "filmcraft-bitstream", + "filmcraft-testkit", "rayon", "thiserror 2.0.21", ] @@ -1541,6 +1584,7 @@ name = "filmcraft-isobmff" version = "0.1.0" dependencies = [ "filmcraft-bitstream", + "filmcraft-testkit", "proptest", "serde_json", "thiserror 2.0.21", @@ -1550,6 +1594,7 @@ dependencies = [ name = "filmcraft-matroska" version = "0.1.0" dependencies = [ + "filmcraft-testkit", "serde_json", ] @@ -1569,6 +1614,9 @@ dependencies = [ [[package]] name = "filmcraft-opus" version = "0.1.0" +dependencies = [ + "filmcraft-testkit", +] [[package]] name = "filmcraft-project" @@ -1589,6 +1637,7 @@ name = "filmcraft-prores" version = "0.1.0" dependencies = [ "filmcraft-bitstream", + "filmcraft-testkit", "rayon", "thiserror 2.0.21", ] @@ -1604,6 +1653,7 @@ dependencies = [ "filmcraft-geom", "filmcraft-media", "filmcraft-project", + "filmcraft-text", "filmcraft-time", "image", "rayon", @@ -1611,6 +1661,24 @@ dependencies = [ "serde_json", ] +[[package]] +name = "filmcraft-testkit" +version = "0.1.0" +dependencies = [ + "png", +] + +[[package]] +name = "filmcraft-text" +version = "0.1.0" +dependencies = [ + "filmcraft-color", + "harfrust", + "skrifa", + "unicode-bidi", + "unicode-linebreak", +] + [[package]] name = "filmcraft-time" version = "0.1.0" @@ -1625,6 +1693,7 @@ version = "0.1.0" dependencies = [ "eframe", "egui", + "egui_kittest", "filmcraft-audio-dsp", "filmcraft-codecs", "filmcraft-color", @@ -1636,7 +1705,9 @@ dependencies = [ "filmcraft-media", "filmcraft-project", "filmcraft-render", + "filmcraft-text", "filmcraft-time", + "image", "log", "pollster 0.4.0", "rayon", @@ -1645,6 +1716,15 @@ dependencies = [ "serde_json", ] +[[package]] +name = "filmcraft-vp9" +version = "0.1.0" +dependencies = [ + "filmcraft-bitstream", + "rayon", + "thiserror 2.0.21", +] + [[package]] name = "find-msvc-tools" version = "0.1.14" @@ -2458,6 +2538,16 @@ version = "3.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e2db585e1d738fc771bf08a151420d3ed193d9d895a36df7f6f8a9456b911ddc" +[[package]] +name = "kittest" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90ceaa75eb0036a32b6b9833962eb18137449e9817e2e586006471925b727fd5" +dependencies = [ + "accesskit", + "accesskit_consumer 0.35.0", +] + [[package]] name = "kurbo" version = "0.13.1" @@ -3955,6 +4045,15 @@ dependencies = [ "serde", ] +[[package]] +name = "serde_spanned" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6662b5879511e06e8999a8a235d848113e942c9124f211511b16466ee2995f26" +dependencies = [ + "serde_core", +] + [[package]] name = "shlex" version = "2.0.1" @@ -4332,7 +4431,7 @@ dependencies = [ "cfg-expr", "heck 0.5.0", "pkg-config", - "toml", + "toml 0.8.2", "version-compare", ] @@ -4465,11 +4564,24 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "185d8ab0dfbb35cf1399a6344d8484209c088f75f8f68230da55d48d95d43e3d" dependencies = [ "serde", - "serde_spanned", + "serde_spanned 0.6.9", "toml_datetime 0.6.3", "toml_edit 0.20.2", ] +[[package]] +name = "toml" +version = "1.1.6+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "920602543f0911ab71da12c50d59701da54c196d1a2bf5cb4b75667f137a406a" +dependencies = [ + "serde_core", + "serde_spanned 1.1.1", + "toml_datetime 1.1.1+spec-1.1.0", + "toml_parser", + "winnow 1.0.4", +] + [[package]] name = "toml_datetime" version = "0.6.3" @@ -4507,7 +4619,7 @@ checksum = "396e4d48bbb2b7554c944bde63101b5ae446cff6ec4a24227428f15eb72ef338" dependencies = [ "indexmap", "serde", - "serde_spanned", + "serde_spanned 0.6.9", "toml_datetime 0.6.3", "winnow 0.5.40", ] @@ -4591,6 +4703,12 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94" +[[package]] +name = "unicode-bidi" +version = "0.3.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" + [[package]] name = "unicode-general-category" version = "1.1.0" @@ -4603,6 +4721,12 @@ version = "1.0.26" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954" +[[package]] +name = "unicode-linebreak" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b09c83c3c29d37506a3e260c08c03743a6bb66a9cd432c6934ab501a190571f" + [[package]] name = "unicode-segmentation" version = "1.13.3" @@ -5643,6 +5767,7 @@ checksum = "e450f9b2ed1dff33c94c12589a87338689467b9c4f5d8a5710bd09a847d2c8a7" name = "xtask" version = "0.1.0" dependencies = [ + "filmcraft-testkit", "serde_json", ] diff --git a/Cargo.toml b/Cargo.toml index 71bb13049..00ddb8419 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -7,7 +7,7 @@ default-members = ["crates/*", "apps/*", "xtask"] version = "0.1.0" edition = "2024" license = "MIT OR Apache-2.0" -rust-version = "1.90" +rust-version = "1.95" repository = "https://github.com/storytold/filmcraft" [workspace.dependencies] @@ -20,6 +20,7 @@ filmcraft-isobmff = { path = "crates/isobmff" } filmcraft-matroska = { path = "crates/matroska" } filmcraft-h264 = { path = "crates/h264" } filmcraft-hevc = { path = "crates/hevc" } +filmcraft-vp9 = { path = "crates/vp9" } filmcraft-prores = { path = "crates/prores" } filmcraft-aac = { path = "crates/aac" } filmcraft-opus = { path = "crates/opus" } @@ -31,6 +32,7 @@ filmcraft-codecs = { path = "crates/codecs" } filmcraft-project = { path = "crates/project" } filmcraft-edit = { path = "crates/edit" } filmcraft-interchange = { path = "crates/interchange" } +filmcraft-text = { path = "crates/text" } filmcraft-captions = { path = "crates/captions" } filmcraft-render = { path = "crates/render" } filmcraft-gpu = { path = "crates/gpu" } @@ -39,6 +41,7 @@ filmcraft-format = { path = "crates/format" } filmcraft-engine = { path = "crates/engine" } filmcraft-ui-egui = { path = "crates/ui-egui" } filmcraft-automation = { path = "crates/automation" } +filmcraft-testkit = { path = "crates/testkit" } # foundation thiserror = "2" @@ -90,6 +93,9 @@ opt-level = 3 [profile.dev.package.filmcraft-opus] opt-level = 3 +[profile.dev.package.filmcraft-vp9] +opt-level = 3 + [profile.release] lto = "thin" codegen-units = 1 diff --git a/ROADMAP.md b/ROADMAP.md index f86088ad3..07f802140 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,7 +2,7 @@ Progress toward feature parity with Adobe Premiere Pro, with estimates. Updated as milestones land. -**Last updated:** 2026-09-30 (midday) · **Overall parity:** ~32% · **Code:** ~71k lines of Rust, 400+ tests +**Last updated:** 2026-10-01 · **Overall parity:** ~45% · **Code:** ~125k lines of Rust in 31 crates, 800+ tests ## Estimate to parity @@ -31,24 +31,28 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining | M0 | Skeleton + visual shell | ✅ | Workspace, 20 crates, dock/workspaces, Premiere 26 look, native menus, control channel, MCP, xtask gates (layers, wasm) | — | — | | M1 | Media I/O | ✅ | MP4/MOV demux+mux, WAV, stills, MJPEG, symphonia audio (MP3/FLAC/ALAC/Vorbis), GOP seek + frame cache, import | Media Browser polish | 2 | | M2 | H.264 decoder | ✅ | Own decoder, bit-exact on 37+ streams, 500–600 fps 1080p | — | — | -| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers | Trim mode + dynamic trimming, multicam, subclips, keyboard shortcut editor | 10–15 | +| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets | Multicam, subclip polish, ~220 Premiere default shortcuts whose commands don't exist yet | 6–10 | | M4 | Playback | 🟡 | Audio-clock master, prefetch, J/K/L, dropped-frame stats, playback resolution | Render previews, render bar, 4K/8K tuning | 8–12 | | M5 | Effects, keyframes, GPU | 🟡 | ~60 CPU effects, 30 transitions, keyframes + value/velocity graphs, wgpu compositor | Full ~150-effect catalogue, WGSL parity for all, masks + tracking, adjustment layers, presets, Warp Stabilizer, Morph Cut | 30–45 | | M6 | Export | ✅ | Own H.264 encoder (High/Main/Baseline, B-frames, VBR/CBR/2-pass) → MP4 + own AAC; ProRes, MJPEG, PNG, GIF, WAV; background jobs | Preset library, queue UI, smart render | 6–8 | -| M7 | Audio | 🟡 | Mixer basics, peak meters, BS.1770 loudness meters (M/S/I LUFS, true peak; matches ffmpeg), DSP crate, 12 clip audio effects running on it (EQ, filters, dynamics, limiter, delay, reverb, DeNoise, DeHum, pitch) with keyframes | Effects UI polish (EQ curve), remaining effects (multiband, convolution reverb), track effects/mixers with automation, Essential Sound, audio transitions, 5.1 | 6–10 | +| M7 | Audio | 🟡 | Mixer graph (tracks → submixes → Mix, pre/post-fader inserts and sends, latency-compensated, sample-accurate, ~6× realtime for 24 tracks × 3 effects on one core), Audio Track Mixer + Audio Clip Mixer panels, track automation (Off/Read/Latch/Touch/Write, recorded live while playing, thinned to keyframes, timeline lanes with pen editing), solo/solo-safe, channel mapping basics, peak + BS.1770 loudness meters (match ffmpeg), DSP crate with 12 clip/track effects, Audio Gain (set/adjust/normalize), Constant Power / Constant Gain / Exponential Fade | Essential Sound, 5.1 panner and multichannel buses, voice-over record, effect editor windows (EQ curve), remaining effects (multiband, convolution reverb), clip-mixer automation recording | 6–9 | | M8 | Colour | 🟡 | Lumetri: basic, creative + looks, RGB & hue curves, wheels, HSL secondary, vignette; basic scopes | LUT import UI, colour match, colour management + HDR (PQ/HLG, log, tone mapping) | 8–12 | -| M9 | More codecs | 🟡 | ProRes decode+encode, AAC decode+encode, HEVC Main/Main 10 decoder (bit-exact on 41 fixtures, ~225 fps 1080p), Matroska/WebM import (H.264/HEVC/ProRes/MJPEG + AAC/Opus/FLAC/MP3/Vorbis/PCM), Opus decoder (SILK/CELT/hybrid, 5.1/7.1 multistream; all RFC 8251 vectors range-exact; WebM/MKV/MP4) | VP9 (in progress), Ogg Opus files, AV1 (rav1d), DNxHR, MXF, hardware decode | 12–20 | -| M10 | Graphics & captions | 🟡 | Caption tracks (Subtitle/CEA-608/708/Teletext formats, track style), SRT/WebVTT/SCC import+export (frame-exact, property-tested), caption editing (add/split/merge/trim/move, sync-locked insert/extract), Text panel Captions tab, burn-in in Program monitor and export | Text engine with shaping, Type tool, Essential Graphics, rolls/crawls, MCC/STL/TTML, 608/708 embedding, speech-to-text | 18–27 | +| M9 | More codecs | 🟡 | ProRes decode+encode, AAC decode+encode, HEVC Main/Main 10 decoder (bit-exact on 41 fixtures, ~225 fps 1080p), VP9 decoder (profiles 0–3, 8/10/12-bit, bit-exact on 50+ fixtures; WebM/MKV `V_VP9` and MP4 `vp09` import with key-frame-checked seeking), Matroska/WebM import (H.264/HEVC/VP9/ProRes/MJPEG + AAC/Opus/FLAC/MP3/Vorbis/PCM), Opus decoder (SILK/CELT/hybrid, 5.1/7.1 multistream; all RFC 8251 vectors range-exact; WebM/MKV/MP4) | VP9 frame threading, Ogg Opus files, AV1 (rav1d), DNxHR, MXF, hardware decode | 12–20 | +| M10 | Graphics & captions | 🟡 | Caption tracks (Subtitle/CEA-608/708/Teletext formats, track style), SRT/WebVTT/SCC import+export (frame-exact, property-tested), caption editing (add/split/merge/trim/move, sync-locked insert/extract), Text panel Captions tab, burn-in in Program monitor and export; text engine (`crates/text`: bundled + system fonts, harfrust shaping, bidi, line breaking, paragraph layout, glyph cache; 3-line 1080p title ≈ 0.15 ms warm); graphic clips with text + shape layers (fill, 2 strokes, background, shadow, keyframable transform), Type tool with on-monitor editing, shape/pen tools, Properties/Essential Graphics editor, align/distribute | Responsive design pins, rolls/crawls, per-character styles, motion graphics templates, MCC/STL/TTML, 608/708 embedding, speech-to-text | 12–18 | | M11 | Interchange & project management | 🟡 | `.fcproj` schema versions + migrations, atomic saves, Save a Copy/Revert, auto-save ring + crash-recovery journal (Preferences ▸ Auto Save, recovery prompt), FCP7 XML, FCPXML, EDL, OTIO | Relink/offline, project manager, proxies | 8 | | M12–M16 | Web (WASM), platform, long tail | 🟡 | L0–L4 crates compile to wasm32 | Web app shell (file access, WebCodecs, audio), ~850 remaining commands and dialogs, performance hardening | 40–65 | ## Running now -- VP9 decoder (`crates/vp9`), from the public VP9 bitstream spec -- Interchange (`crates/interchange`): CMX 3600 EDL, FCP7 XML, FCPXML, OTIO import and export +- Playback: fix a UI hang after preview renders, then a load-robust playback benchmark and profiling (target: 3 × 1080p streams with 0 dropped frames) ## Log +- **2026-10-01 (later):** text engine (shaping, bidi, line breaking) + Type/Shape/Pen tools + graphic clips + graphics panel; Trim Monitor + dynamic trimming; Keyboard Shortcuts editor; audio mixer with automation. Fixed: MP4 muxer wrote unreadable all-empty sample tables; system font scan race. + +- **2026-10-01:** Audio mixing (M7.2/M7.5/M7.6 basics): mixer graph with submixes, sends, inserts and latency compensation; Audio Track / Clip Mixer panels; Latch/Touch/Write automation recorded live; timeline track keyframes; Audio Gain dialog. +- **2026-10-01:** recovered from a machine crash with no lost work. Merged: VP9 decoder (profiles 0–3, bit-exact, WebM/MKV/MP4), Opus decoder (RFC 8251 range-exact; WebM/MKV/MP4), captions (SRT/VTT/SCC, burn-in), render bar + render previews, project schema versioning + atomic saves + auto-save + crash-recovery journal, test infrastructure (golden images, loudness oracle vs ffmpeg, headless scripted UI tests; fixed a GPU stale-texture bug), public contributor docs and licence files. + - **2026-09-30 (evening):** Opus decoder (RFC 6716/8251, range-exact on every conformance vector, ~80–110× realtime 48 kHz stereo) wired into WebM/MKV and MP4 import. - **2026-09-30 (afternoon):** Matroska/WebM import; LUFS meters; clip audio effects on the DSP crate. - **2026-09-30 (midday):** HEVC decoder, Matroska/WebM demuxer and audio DSP merged; HEVC import wired; asset rules (AGENTS.md, ATTRIBUTION.md, `cargo xtask assets`); README with hero screenshot and the Craft family. diff --git a/apps/filmcraft/src/main.rs b/apps/filmcraft/src/main.rs index c5f691d66..f39a8a423 100644 --- a/apps/filmcraft/src/main.rs +++ b/apps/filmcraft/src/main.rs @@ -111,12 +111,16 @@ fn main() -> eframe::Result { app.hooks.pick_save_as = Some(Box::new(|filter: &str, exts: &[&str], name: &str| { rfd::FileDialog::new().add_filter(filter, exts).set_file_name(name).save_file().map(|p| p.to_string_lossy().to_string()) })); + app.hooks.pick_open_file = Some(Box::new(|filter: &str, exts: &[&str]| { + rfd::FileDialog::new().add_filter(filter, exts).pick_file().map(|p| p.to_string_lossy().to_string()) + })); app.hooks.pick_open_project = Some(Box::new(|| rfd::FileDialog::new().add_filter("FilmCraft Project", &["fcproj"]).pick_file().map(|p| p.to_string_lossy().to_string()))); #[cfg(target_os = "macos")] { - let rx = native_menu::install(&app, cc.egui_ctx.clone()); + let (rx, update) = native_menu::install(&app, cc.egui_ctx.clone()); app.command_inbox = Some(rx); + app.hooks.shortcuts_changed = Some(update); app.ui.show_menu_bar = false; } if let Some(port) = control_port { diff --git a/apps/filmcraft/src/native_menu.rs b/apps/filmcraft/src/native_menu.rs index 3f9a6543e..fd1422948 100644 --- a/apps/filmcraft/src/native_menu.rs +++ b/apps/filmcraft/src/native_menu.rs @@ -19,8 +19,12 @@ fn accel(s: &str) -> Option { mapped.parse().ok() } -pub fn install(app: &FilmcraftApp, ctx: egui::Context) -> Receiver { +/// Updates native key equivalents when the active keyboard shortcuts change. +pub type ShortcutUpdater = Box; + +pub fn install(app: &FilmcraftApp, ctx: egui::Context) -> (Receiver, ShortcutUpdater) { let items = menu_items(app); + let mut native: Vec<(String, MenuItem)> = Vec::new(); let bar = Menu::new(); let app_menu = Submenu::new("FilmCraft", true); let _ = app_menu.append_items(&[ @@ -40,6 +44,7 @@ pub fn install(app: &FilmcraftApp, ctx: egui::Context) -> Receiver { let mut subs: Vec<(String, Submenu)> = Vec::new(); for it in mine { let mi = MenuItem::with_id(it.id.clone(), &it.label, true, it.shortcut.as_deref().and_then(accel)); + native.push((it.id.clone(), mi.clone())); if let Some(name) = it.path.get(1) { if let Some((_, s)) = subs.iter().find(|(n, _)| n == name) { let _ = s.append(&mi); @@ -69,5 +74,12 @@ pub fn install(app: &FilmcraftApp, ctx: egui::Context) -> Receiver { ctx.request_repaint(); } }); - rx + let update: ShortcutUpdater = Box::new(move |items: &[Item]| { + for (id, mi) in &native { + if let Some(it) = items.iter().find(|i| &i.id == id) { + let _ = mi.set_accelerator(it.shortcut.as_deref().and_then(accel)); + } + } + }); + (rx, update) } diff --git a/assets/fonts/Inter-Bold.ttf b/assets/fonts/Inter-Bold.ttf new file mode 100644 index 000000000..9fb9b751e Binary files /dev/null and b/assets/fonts/Inter-Bold.ttf differ diff --git a/assets/fonts/Inter-Bold.ttf.attribution b/assets/fonts/Inter-Bold.ttf.attribution new file mode 100644 index 000000000..6f2ca0b1d --- /dev/null +++ b/assets/fonts/Inter-Bold.ttf.attribution @@ -0,0 +1,8 @@ +asset: Inter-Bold.ttf +title: Inter Bold (text engine / graphics) +author: The Inter Project Authors (Rasmus Andersson et al.) +source: https://github.com/rsms/inter (release v4.1, extras/ttf) +license: OFL-1.1 +license-file: assets/fonts/OFL-Inter.txt +added: 2026-10-01 by Claude (agent) for Brandon Thomas +notes: Unmodified font file, bundled via include_bytes! in the text engine (crates/text/src/fonts.rs) for graphics text, titles and burn-ins. diff --git a/assets/fonts/Inter-Italic.ttf b/assets/fonts/Inter-Italic.ttf new file mode 100644 index 000000000..e7ed211d2 Binary files /dev/null and b/assets/fonts/Inter-Italic.ttf differ diff --git a/assets/fonts/Inter-Italic.ttf.attribution b/assets/fonts/Inter-Italic.ttf.attribution new file mode 100644 index 000000000..e346bc98d --- /dev/null +++ b/assets/fonts/Inter-Italic.ttf.attribution @@ -0,0 +1,8 @@ +asset: Inter-Italic.ttf +title: Inter Italic (text engine / graphics) +author: The Inter Project Authors (Rasmus Andersson et al.) +source: https://github.com/rsms/inter (release v4.1, extras/ttf) +license: OFL-1.1 +license-file: assets/fonts/OFL-Inter.txt +added: 2026-10-01 by Claude (agent) for Brandon Thomas +notes: Unmodified font file, bundled via include_bytes! in the text engine (crates/text/src/fonts.rs) for graphics text, titles and burn-ins. diff --git a/assets/fonts/Inter-Medium.ttf.attribution b/assets/fonts/Inter-Medium.ttf.attribution index 20dc0cf41..707fc56c8 100644 --- a/assets/fonts/Inter-Medium.ttf.attribution +++ b/assets/fonts/Inter-Medium.ttf.attribution @@ -5,4 +5,4 @@ source: https://github.com/rsms/inter license: OFL-1.1 license-file: assets/fonts/OFL-Inter.txt added: 2026-09-30 by Claude (agent) for Brandon Thomas -notes: Unmodified font file, embedded in the UI via include_bytes! (crates/ui-egui/src/theme.rs). +notes: Unmodified font file, embedded via include_bytes! in the text engine (crates/text/src/fonts.rs), which also supplies it to the UI (crates/ui-egui/src/theme.rs). diff --git a/assets/fonts/Inter-Regular.ttf.attribution b/assets/fonts/Inter-Regular.ttf.attribution index c4f496fbc..052b01130 100644 --- a/assets/fonts/Inter-Regular.ttf.attribution +++ b/assets/fonts/Inter-Regular.ttf.attribution @@ -5,4 +5,4 @@ source: https://github.com/rsms/inter license: OFL-1.1 license-file: assets/fonts/OFL-Inter.txt added: 2026-09-30 by Claude (agent) for Brandon Thomas -notes: Unmodified font file, embedded in the UI via include_bytes! (crates/ui-egui/src/theme.rs). +notes: Unmodified font file, embedded via include_bytes! in the text engine (crates/text/src/fonts.rs), which also supplies it to the UI (crates/ui-egui/src/theme.rs). diff --git a/assets/fonts/Inter-SemiBold.ttf.attribution b/assets/fonts/Inter-SemiBold.ttf.attribution index 771822f07..e010cad7e 100644 --- a/assets/fonts/Inter-SemiBold.ttf.attribution +++ b/assets/fonts/Inter-SemiBold.ttf.attribution @@ -5,4 +5,4 @@ source: https://github.com/rsms/inter license: OFL-1.1 license-file: assets/fonts/OFL-Inter.txt added: 2026-09-30 by Claude (agent) for Brandon Thomas -notes: Unmodified font file, embedded via include_bytes! in the UI (crates/ui-egui/src/theme.rs) and as the caption burn-in font (crates/captions/src/font.rs). +notes: Unmodified font file, embedded via include_bytes! in the text engine (crates/text/src/fonts.rs; caption burn-in, graphics), which also supplies it to the UI (crates/ui-egui/src/theme.rs). diff --git a/assets/fonts/JetBrainsMono-Regular.ttf.attribution b/assets/fonts/JetBrainsMono-Regular.ttf.attribution index ab720f6cf..f185af97b 100644 --- a/assets/fonts/JetBrainsMono-Regular.ttf.attribution +++ b/assets/fonts/JetBrainsMono-Regular.ttf.attribution @@ -5,4 +5,4 @@ source: https://github.com/JetBrains/JetBrainsMono license: OFL-1.1 license-file: assets/fonts/OFL-JetBrainsMono.txt added: 2026-09-30 by Claude (agent) for Brandon Thomas -notes: Unmodified font file, embedded in the UI via include_bytes! (crates/ui-egui/src/theme.rs). +notes: Unmodified font file, embedded via include_bytes! in the text engine (crates/text/src/fonts.rs), which also supplies it to the UI (crates/ui-egui/src/theme.rs). diff --git a/assets/fonts/NotoSerif-Regular.ttf b/assets/fonts/NotoSerif-Regular.ttf new file mode 100644 index 000000000..9fe4b2986 Binary files /dev/null and b/assets/fonts/NotoSerif-Regular.ttf differ diff --git a/assets/fonts/NotoSerif-Regular.ttf.attribution b/assets/fonts/NotoSerif-Regular.ttf.attribution new file mode 100644 index 000000000..8d09443bb --- /dev/null +++ b/assets/fonts/NotoSerif-Regular.ttf.attribution @@ -0,0 +1,8 @@ +asset: NotoSerif-Regular.ttf +title: Noto Serif Regular (serif face of the text engine / graphics) +author: The Noto Project Authors +source: https://github.com/notofonts/notofonts.github.io (fonts/NotoSerif/unhinted/ttf) +license: OFL-1.1 +license-file: assets/fonts/OFL-NotoSerif.txt +added: 2026-10-01 by Claude (agent) for Brandon Thomas +notes: Unmodified font file, bundled via include_bytes! in the text engine (crates/text/src/fonts.rs) for graphics text, titles and burn-ins. diff --git a/assets/fonts/OFL-NotoSerif.txt b/assets/fonts/OFL-NotoSerif.txt new file mode 100644 index 000000000..6843f3187 --- /dev/null +++ b/assets/fonts/OFL-NotoSerif.txt @@ -0,0 +1,93 @@ +Copyright 2022 The Noto Project Authors (https://github.com/notofonts/latin-greek-cyrillic) + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +https://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/crates/aac/Cargo.toml b/crates/aac/Cargo.toml index 1f1203da8..1bc264263 100644 --- a/crates/aac/Cargo.toml +++ b/crates/aac/Cargo.toml @@ -17,3 +17,6 @@ rayon = ["dep:rayon"] filmcraft-bitstream = { workspace = true } thiserror = { workspace = true } rayon = { workspace = true, optional = true } + +[dev-dependencies] +filmcraft-testkit = { workspace = true } diff --git a/crates/aac/tests/common/mod.rs b/crates/aac/tests/common/mod.rs index 7ffc04c20..eb0551728 100644 --- a/crates/aac/tests/common/mod.rs +++ b/crates/aac/tests/common/mod.rs @@ -5,19 +5,13 @@ use std::process::Command; use filmcraft_aac::{Decoder, Encoder, EncoderConfig, EncoderStats, split_adts}; +/// ffmpeg (see `filmcraft_testkit::oracle`); `None` after printing `SKIPPED`. pub fn ffmpeg() -> Option { - for p in ["/opt/homebrew/bin/ffmpeg", "/usr/local/bin/ffmpeg", "/usr/bin/ffmpeg"] { - if Path::new(p).exists() { - return Some(PathBuf::from(p)); - } - } - None + filmcraft_testkit::ffmpeg_or_skip("aac oracle") } pub fn fixtures() -> PathBuf { - let d = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../target/fixtures/aac"); - std::fs::create_dir_all(&d).unwrap(); - d + filmcraft_testkit::fixtures_dir("aac") } pub struct Rng(pub u64); diff --git a/crates/audio-dsp/Cargo.toml b/crates/audio-dsp/Cargo.toml index bc1a0483b..aef31dba1 100644 --- a/crates/audio-dsp/Cargo.toml +++ b/crates/audio-dsp/Cargo.toml @@ -11,3 +11,6 @@ workspace = true # Intentionally dependency-free (layer L1): planar `&mut [f32]` slices in, planar slices out. [dependencies] + +[dev-dependencies] +filmcraft-testkit = { workspace = true } diff --git a/crates/audio-dsp/README.md b/crates/audio-dsp/README.md index 99c85a0fd..e1ae13a6c 100644 --- a/crates/audio-dsp/README.md +++ b/crates/audio-dsp/README.md @@ -24,6 +24,9 @@ Householder FDN, phase vocoder). - Gated integrated loudness (−70 LUFS / −10 LU) and LRA (−70 / −20 LU, 10th–95th percentile) use a 0.01 LU histogram with exact energy sums: fixed memory for programmes of any length. - True peak: 4× (2× at 88.2/96 kHz) polyphase Kaiser-sinc interpolation. +- Oracle-tested against ffmpeg's `ebur128` filter (`tests/loudness_oracle.rs`, skipped without ffmpeg): + M/S within 0.0005 LU, I within 0.007 LU, LRA within 0.04 LU and true peak within 0.045 dB (0.053 dB of the analytic value on an fs/4 tone) on + sine, pink-noise, speech-like and gated stereo signals at 44.1/48/96 kHz. - `normalize_gain_db(measured, target)` and `normalize_gain_db_peak_limited(...)` for Normalize / Essential Sound loudness auto-match. diff --git a/crates/audio-dsp/tests/loudness_oracle.rs b/crates/audio-dsp/tests/loudness_oracle.rs new file mode 100644 index 000000000..1a3b82765 --- /dev/null +++ b/crates/audio-dsp/tests/loudness_oracle.rs @@ -0,0 +1,285 @@ +//! Oracle test: BS.1770-4 / EBU R128 loudness against ffmpeg's `ebur128` filter. +//! +//! Each signal is generated here, written as a 32-bit float WAV and measured twice: by +//! `LoudnessMeter` and by `ffmpeg -af ebur128=peak=true:metadata=1,ametadata=print` (three decimals +//! per 100 ms frame). ffmpeg is an external oracle only (never linked). +//! +//! Tolerances (EBU Tech 3341 allows ±0.1 LU for M/S/I and +0.2/−0.4 dB for true peak; Tech 3342 +//! allows ±1 LU for LRA): +//! - momentary (every 100 ms once 400 ms have been seen) and short-term (once 3 s have been seen): +//! ±0.1 LU wherever ffmpeg reads above −70 LUFS; +//! - integrated: ±0.1 LU; +//! - loudness range: ±0.5 LU; +//! - true peak: ±0.2 dB (both are 4× oversampling estimates with different interpolators). + +use filmcraft_audio_dsp::loudness::LoudnessMeter; +use std::f64::consts::PI; +use std::path::Path; +use std::process::Command; + +const TOL_MS: f64 = 0.1; +const TOL_I: f64 = 0.1; +const TOL_LRA: f64 = 0.5; +const TOL_TP_DB: f64 = 0.2; + +/// ffmpeg's per-100 ms frame values. +#[derive(Clone, Copy, Debug, Default)] +struct Frame { + m: f64, + s: f64, + i: f64, + lra: f64, + true_peak: f64, +} + +fn ffmpeg_ebur128(ff: &Path, wav: &Path) -> Vec { + let out = Command::new(ff) + .args(["-hide_banner", "-nostats", "-v", "error", "-i"]) + .arg(wav) + .args(["-af", "ebur128=peak=true:metadata=1,ametadata=mode=print:file=-", "-f", "null", "-"]) + .output() + .expect("run ffmpeg"); + assert!(out.status.success(), "ffmpeg ebur128 failed: {}", String::from_utf8_lossy(&out.stderr)); + let mut frames = Vec::new(); + let mut cur: Option = None; + for line in String::from_utf8_lossy(&out.stdout).lines() { + if line.starts_with("frame:") { + frames.extend(cur.take()); + cur = Some(Frame::default()); + continue; + } + let (Some(f), Some((k, v))) = (cur.as_mut(), line.strip_prefix("lavfi.r128.").and_then(|l| l.split_once('='))) else { continue }; + let v: f64 = v.trim().parse().unwrap_or(f64::NAN); + match k { + "M" => f.m = v, + "S" => f.s = v, + "I" => f.i = v, + "LRA" => f.lra = v, + "true_peak" => f.true_peak = v, + _ => {} + } + } + frames.extend(cur); + frames +} + +/// Our measurements after each 100 ms block: (M, S) per block, plus the final meter. +fn ours(chans: &[Vec], sr: u32) -> (Vec<(f64, f64)>, LoudnessMeter) { + let mut m = LoudnessMeter::new(sr as f64, chans.len()); + let block = sr as usize / 10; + let n = chans[0].len(); + let mut per = Vec::new(); + let mut pos = 0; + while pos < n { + let end = (pos + block).min(n); + let slices: Vec<&[f32]> = chans.iter().map(|c| &c[pos..end]).collect(); + m.process(&slices); + per.push((m.momentary(), m.short_term())); + pos = end; + } + (per, m) +} + +struct Report { + max_dm: f64, + max_ds: f64, + i: (f64, f64), + lra: (f64, f64), + tp: (f64, f64), + /// True-peak reference used (analytic value when known, else ffmpeg). + tp_ref: f64, +} + +/// Measure `chans` with both meters and check the tolerances. `tp_truth`: the analytic true peak +/// (dBTP) when known; it then replaces ffmpeg as the true-peak reference. +fn compare(ff: &Path, name: &'static str, chans: &[Vec], sr: u32, tp_truth: Option) -> Report { + let dir = filmcraft_testkit::fixtures_dir("audio-dsp/loudness"); + let wav = dir.join(format!("{name}_{sr}.wav")); + let tmp = filmcraft_testkit::temp_path(&wav); + let refs: Vec<&[f32]> = chans.iter().map(|c| c.as_slice()).collect(); + filmcraft_testkit::wav::write_f32(&tmp, &refs, sr).unwrap(); + let frames = ffmpeg_ebur128(ff, &tmp); + let _ = std::fs::remove_file(&tmp); + let (per, meter) = ours(chans, sr); + assert!(frames.len() + 1 >= per.len() && !frames.is_empty(), "{name}: ffmpeg gave {} frames, we have {} blocks", frames.len(), per.len()); + let (mut max_dm, mut max_ds, mut n_m, mut n_s) = (0f64, 0f64, 0, 0); + for (k, (f, (m, s))) in frames.iter().zip(&per).enumerate() { + let full_blocks = k + 1; + if full_blocks >= 4 && f.m > -70.0 { + let d = (f.m - m).abs(); + assert!(d <= TOL_MS, "{name}: momentary at {:.1} s: ffmpeg {:.3}, ours {m:.3}", full_blocks as f64 / 10.0, f.m); + max_dm = max_dm.max(d); + n_m += 1; + } + if full_blocks >= 30 && f.s > -70.0 { + let d = (f.s - s).abs(); + assert!(d <= TOL_MS, "{name}: short-term at {:.1} s: ffmpeg {:.3}, ours {s:.3}", full_blocks as f64 / 10.0, f.s); + max_ds = max_ds.max(d); + n_s += 1; + } + } + let last = frames.last().unwrap(); + let ff_tp_db = 20.0 * last.true_peak.log10(); + let r = Report { + tp_ref: tp_truth.unwrap_or(ff_tp_db), + max_dm, + max_ds, + i: (meter.integrated(), last.i), + lra: (meter.loudness_range(), last.lra), + tp: (meter.true_peak_dbtp(), ff_tp_db), + }; + eprintln!( + "{name} @ {sr} Hz: I ours {:.3} / ffmpeg {:.3} LUFS; LRA {:.2} / {:.2} LU; TP {:.3} / {:.3} dBTP; max |ΔM| {:.4} over {n_m}, max |ΔS| {:.4} over {n_s}", + r.i.0, r.i.1, r.lra.0, r.lra.1, r.tp.0, r.tp.1, r.max_dm, r.max_ds + ); + assert!(n_m > 0, "{name}: no momentary values compared"); + assert!((r.i.0 - r.i.1).abs() <= TOL_I, "{name}: integrated ours {:.3} vs ffmpeg {:.3}", r.i.0, r.i.1); + assert!((r.lra.0 - r.lra.1).abs() <= TOL_LRA, "{name}: LRA ours {:.3} vs ffmpeg {:.3}", r.lra.0, r.lra.1); + match tp_truth { + Some(t) => { + eprintln!("{name}: analytic true peak {t:.3} dBTP: ours off by {:+.3} dB, ffmpeg off by {:+.3} dB", r.tp.0 - t, r.tp.1 - t); + assert!((r.tp.0 - t).abs() <= TOL_TP_DB, "{name}: true peak ours {:.3} vs analytic {t:.3} dBTP", r.tp.0); + } + None => assert!((r.tp.0 - r.tp.1).abs() <= TOL_TP_DB, "{name}: true peak ours {:.3} vs ffmpeg {:.3} dBTP", r.tp.0, r.tp.1), + } + r +} + +// ---- signals ---- + +struct Rng(u64); +impl Rng { + fn next(&mut self) -> f64 { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 >> 11) as f64 / (1u64 << 53) as f64 * 2.0 - 1.0 + } +} + +fn db(x: f64) -> f64 { + 10f64.powf(x / 20.0) +} + +fn sine(freq: f64, amp: f64, sr: u32, secs: f64, phase: f64) -> Vec { + (0..(secs * sr as f64) as usize).map(|i| (amp * (2.0 * PI * freq * i as f64 / sr as f64 + phase).sin()) as f32).collect() +} + +/// Pink noise (Paul Kellet's refined filter on white noise), scaled to `amp` RMS-ish. +fn pink(sr: u32, secs: f64, amp: f64, seed: u64) -> Vec { + let mut r = Rng(seed); + let (mut b0, mut b1, mut b2, mut b3, mut b4, mut b5, mut b6) = (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0); + (0..(secs * sr as f64) as usize) + .map(|_| { + let w = r.next(); + b0 = 0.99886 * b0 + w * 0.0555179; + b1 = 0.99332 * b1 + w * 0.0750759; + b2 = 0.96900 * b2 + w * 0.1538520; + b3 = 0.86650 * b3 + w * 0.3104856; + b4 = 0.55000 * b4 + w * 0.5329522; + b5 = -0.7616 * b5 - w * 0.0168980; + let y = b0 + b1 + b2 + b3 + b4 + b5 + b6 + w * 0.5362; + b6 = w * 0.115926; + (y * 0.11 * amp) as f32 + }) + .collect() +} + +/// Speech-like: syllable bursts (80–300 ms) of a pitched, formant-ish signal with pauses +/// (50–600 ms) and phrase-level level changes, so gating and the M/S windows are exercised. +fn speech(sr: u32, secs: f64, seed: u64) -> Vec { + let mut r = Rng(seed); + let n = (secs * sr as f64) as usize; + let mut out = vec![0f32; n]; + let mut pos = 0usize; + let mut phrase_gain = 1.0; + while pos < n { + if r.next() > 0.7 { + phrase_gain = db(-12.0 * (r.next() + 1.0) / 2.0); + } + let syl = ((0.08 + 0.22 * (r.next() + 1.0) / 2.0) * sr as f64) as usize; + let f0 = 110.0 + 60.0 * (r.next() + 1.0) / 2.0; + let (f1, f2) = (500.0 + 400.0 * r.next(), 1500.0 + 700.0 * r.next()); + for i in 0..syl.min(n - pos) { + let t = i as f64 / sr as f64; + let env = (PI * i as f64 / syl as f64).sin().powi(2); + let mut v = 0.0; + for h in 1..=20 { + let f = f0 * h as f64; + if f > sr as f64 / 2.2 { + break; + } + let formant = (-((f - f1) / 200.0).powi(2)).exp() + 0.5 * (-((f - f2) / 300.0).powi(2)).exp() + 0.05; + v += formant * (2.0 * PI * f * t).sin() / h as f64; + } + out[pos + i] = (0.3 * phrase_gain * env * v + 0.003 * r.next()) as f32; + } + pos += syl; + let gap = ((0.05 + 0.55 * (r.next() + 1.0) / 2.0) * sr as f64) as usize; + pos += gap; + } + out +} + +fn concat(parts: &[Vec]) -> Vec { + parts.concat() +} + +#[test] +fn loudness_matches_ffmpeg_ebur128() { + let ff = filmcraft_testkit::require_ffmpeg!(); + let sr = 48_000; + let mut reports = Vec::new(); + + // 1. Steady 997 Hz sine, −20 dBFS, both channels (dual mono adds +3 dB). + let s = sine(997.0, db(-20.0), sr, 10.0, 0.0); + reports.push(compare(&ff, "sine_997", &[s.clone(), s], sr, Some(-20.0))); + + // 2. Independent pink noise per channel. + reports.push(compare(&ff, "pink_noise", &[pink(sr, 15.0, 0.5, 1), pink(sr, 15.0, 0.5, 2)], sr, None)); + + // 3. Speech-like bursts and pauses (exercises the relative gate and LRA). + let sp = speech(sr, 25.0, 7); + let sp2: Vec = sp.iter().map(|v| v * 0.7).collect(); + reports.push(compare(&ff, "speech_like", &[sp, sp2], sr, None)); + + // 4. Stereo with silence and quiet passages: loud tone, digital silence, a −30 dBFS tone on L + // only, a −50 dBFS passage (below the relative gate) and a −80 dBFS passage (below the + // absolute gate). + let l = concat(&[ + sine(440.0, db(-18.0), sr, 5.0, 0.0), + vec![0.0; 3 * sr as usize], + sine(1000.0, db(-30.0), sr, 5.0, 0.0), + pink(sr, 4.0, db(-50.0), 3), + pink(sr, 4.0, db(-80.0), 4), + ]); + let r = concat(&[ + sine(440.0, db(-18.0), sr, 5.0, 0.3), + vec![0.0; 3 * sr as usize], + vec![0.0; 5 * sr as usize], + pink(sr, 4.0, db(-50.0), 5), + pink(sr, 4.0, db(-80.0), 6), + ]); + reports.push(compare(&ff, "gated_stereo", &[l, r], sr, None)); + + // 5. Inter-sample peaks: fs/4 at 45° phase, 0 dBFS true peak but −3 dBFS sample peak. + let x = sine(sr as f64 / 4.0, 0.9, sr, 5.0, PI / 4.0); + reports.push(compare(&ff, "intersample_peak", &[x.clone(), x], sr, Some(20.0 * 0.9f64.log10()))); + + // 6. Mono speech-like at 44.1 kHz (K-weighting derived for another rate; true peak at 4×). + let sr2 = 44_100; + reports.push(compare(&ff, "speech_mono_44k", &[speech(sr2, 20.0, 11)], sr2, None)); + + // 7. Pink noise at 96 kHz (2× true-peak oversampling). + reports.push(compare(&ff, "pink_noise_96k", &[pink(96_000, 8.0, 0.4, 21), pink(96_000, 8.0, 0.4, 22)], 96_000, None)); + + let worst = |f: fn(&Report) -> f64| reports.iter().map(f).fold(0f64, f64::max); + eprintln!( + "worst: |ΔM| {:.3}, |ΔS| {:.3}, |ΔI| {:.3}, |ΔLRA| {:.3}, |ΔTP| {:.3} dB", + worst(|r| r.max_dm), + worst(|r| r.max_ds), + worst(|r| (r.i.0 - r.i.1).abs()), + worst(|r| (r.lra.0 - r.lra.1).abs()), + worst(|r| (r.tp.0 - r.tp_ref).abs()), + ); +} diff --git a/crates/captions/Cargo.toml b/crates/captions/Cargo.toml index 55ce943c3..dad44780c 100644 --- a/crates/captions/Cargo.toml +++ b/crates/captions/Cargo.toml @@ -13,9 +13,8 @@ workspace = true filmcraft-time = { workspace = true } filmcraft-color = { workspace = true } filmcraft-project = { workspace = true } +filmcraft-text = { workspace = true } thiserror = { workspace = true } -# Font tables, glyph outlines and metrics (MIT OR Apache-2.0). Rasterisation is our own. -skrifa = { version = "0.44", default-features = false, features = ["std"] } [dev-dependencies] proptest = { workspace = true } diff --git a/crates/captions/README.md b/crates/captions/README.md index 919e057c9..b988d65fb 100644 --- a/crates/captions/README.md +++ b/crates/captions/README.md @@ -1,7 +1,7 @@ # filmcraft-captions Caption files and caption burn-in for FilmCraft (layer L2: depends on `filmcraft-time`, -`filmcraft-color`, `filmcraft-project` and `skrifa`). +`filmcraft-color`, `filmcraft-project` and `filmcraft-text`). The caption *model* (caption tracks in a sequence, captions, track style) lives in `filmcraft-project` (`caption.rs`) because it is part of the `.fcproj` document. This crate reads @@ -40,16 +40,13 @@ rasterises it to a small premultiplied linear-light RGBA overlay. `filmcraft-ren overlays of visible caption tracks over the finished frame (Program monitor and export burn-in). Text is set in **Inter SemiBold** (`assets/fonts/Inter-SemiBold.ttf`, SIL OFL 1.1, attributed in -`ATTRIBUTION.md`). `skrifa` (MIT OR Apache-2.0) provides glyph outlines and metrics; the -rasteriser (`raster.rs`, non-zero winding, 5 sub-scanlines with exact horizontal coverage) is our -own. Glyph masks are cached per character, size and quarter-pixel offset. There is no shaping or -kerning yet; that arrives with the M10.1 text engine. +`ATTRIBUTION.md`) by the `filmcraft-text` engine: shaped with kerning and ligatures, bidi-ordered, +and drawn from its sub-pixel positioned glyph cache. ## Tests - Unit tests per module (timestamps, decoding, sloppy SRT, WebVTT blocks and voices, CEA-608 - tables, PAC round trips, known SCC pop-on and roll-up streams, rasteriser coverage, glyphs, - layout). + tables, PAC round trips, known SCC pop-on and roll-up streams, layout). - `tests/roundtrip.rs`: property tests — SRT (also with BOM + CRLF), WebVTT (ids, speakers, settings, STYLE blocks) and SCC (drop-frame and non-drop, gaps and back-to-back captions) round-trip exactly; readers never panic on random bytes; SRT → VTT → SCC keeps text and frames. diff --git a/crates/captions/src/burn.rs b/crates/captions/src/burn.rs index 8fb7d75ca..5c43e5251 100644 --- a/crates/captions/src/burn.rs +++ b/crates/captions/src/burn.rs @@ -10,7 +10,7 @@ use filmcraft_color::srgb_to_linear; use filmcraft_project::{Caption, CaptionAlign, CaptionAnchor, CaptionStyle, CaptionTrack, Sequence}; use filmcraft_time::Tick; -use crate::font; +use filmcraft_text::{ParagraphStyle, TextStyle, layout, render}; /// A rendered caption: premultiplied linear-light RGBA f32 pixels at `(x, y)` in the frame. #[derive(Clone, Debug, PartialEq)] @@ -49,6 +49,15 @@ impl Overlay { } } +/// The caption face: Inter SemiBold at `px`. +fn caption_style(px: f32) -> TextStyle { + TextStyle { style: "SemiBold".into(), size: px, ..Default::default() } +} + +fn measure(text: &str, px: f32) -> f32 { + filmcraft_text::measure(text, &caption_style(px)) +} + fn lin(c: [u8; 4]) -> [f32; 4] { let a = c[3] as f32 / 255.0; [srgb_to_linear(c[0] as f32 / 255.0) * a, srgb_to_linear(c[1] as f32 / 255.0) * a, srgb_to_linear(c[2] as f32 / 255.0) * a, a] @@ -60,7 +69,7 @@ fn wrap(line: &str, px: f32, max_w: f32) -> Vec { let mut cur = String::new(); for word in line.split(' ') { let cand = if cur.is_empty() { word.to_string() } else { format!("{cur} {word}") }; - if !cur.is_empty() && font::measure(&cand, px) > max_w { + if !cur.is_empty() && measure(&cand, px) > max_w { out.push(std::mem::take(&mut cur)); cur = word.to_string(); } else { @@ -115,7 +124,8 @@ pub fn render_caption(c: &Caption, style: &CaptionStyle, w: usize, h: usize) -> if lines.is_empty() { return None; } - let (asc, desc) = font::metrics(px); + let vm = filmcraft_text::fonts::face(filmcraft_text::resolve("Inter", "SemiBold").face).metrics(px); + let (asc, desc) = (vm.ascent, vm.descent); let lh = px * style.line_spacing.max(0.8); let pad_x = (px * 0.3).round(); let block_h = lh * lines.len() as f32; @@ -131,7 +141,7 @@ pub fn render_caption(c: &Caption, style: &CaptionStyle, w: usize, h: usize) -> .max(0.0) .round(); let side = w as f32 * 0.05; - let widths: Vec = lines.iter().map(|l| font::measure(l, px)).collect(); + let widths: Vec = lines.iter().map(|l| measure(l, px)).collect(); let xs: Vec = widths .iter() .map(|&lw| match align { @@ -166,28 +176,10 @@ pub fn render_caption(c: &Caption, style: &CaptionStyle, w: usize, h: usize) -> } } let baseline = lt + (lh - (asc + desc)) / 2.0 + asc; - let mut pen = xs[i]; - for ch in line.chars() { - if let Some(g) = font::glyph(ch, px, pen.fract()) { - let gx = pen.floor() as i32 + g.left - ox; - let gy = baseline.round() as i32 + g.top - oy; - for y in 0..g.mask.h { - let ty = gy + y as i32; - if ty < 0 || ty >= oh as i32 { - continue; - } - for x in 0..g.mask.w { - let tx = gx + x as i32; - if tx < 0 || tx >= ow as i32 { - continue; - } - let j = ty as usize * ow + tx as usize; - cover[j] = (cover[j] + g.mask.get(x, y)).min(1.0); - } - } - } - pen += font::advance(ch, px); - } + let l = layout(line, &caption_style(px), &ParagraphStyle::default()); + let mut m = filmcraft_text::Mask { w: ow, h: oh, a: std::mem::take(&mut cover) }; + render::draw(&l, &render::at(xs[i], baseline.round()), &mut m, (ox, oy)); + cover = m.a; } let stroke = if outline > 0.0 { dilate(&cover, ow, oh, outline) } else { Vec::new() }; let (tc, bc, oc) = (lin(style.color), lin(style.background_color), lin(style.outline_color)); diff --git a/crates/captions/src/font.rs b/crates/captions/src/font.rs deleted file mode 100644 index 2aa1cf843..000000000 --- a/crates/captions/src/font.rs +++ /dev/null @@ -1,137 +0,0 @@ -//! The caption font: Inter SemiBold (SIL OFL 1.1, `assets/fonts/Inter-SemiBold.ttf`, attributed in -//! ATTRIBUTION.md), read with `skrifa` and rasterised by [`crate::raster`]. Glyph masks are -//! cached by (char, size, sub-pixel offset). - -use std::collections::HashMap; -use std::sync::{Arc, Mutex, OnceLock}; - -use skrifa::instance::{LocationRef, Size}; -use skrifa::outline::{DrawSettings, OutlinePen}; -use skrifa::{FontRef, MetadataProvider}; - -use crate::raster::{Mask, Path, fill}; - -static INTER_SEMIBOLD: &[u8] = include_bytes!("../../../assets/fonts/Inter-SemiBold.ttf"); - -fn font() -> Option> { - FontRef::new(INTER_SEMIBOLD).ok() -} - -/// Vertical metrics at a pixel size: (ascent, descent), both positive. -pub fn metrics(px: f32) -> (f32, f32) { - match font() { - Some(f) => { - let m = f.metrics(Size::new(px), LocationRef::default()); - (m.ascent, -m.descent) - } - None => (px * 0.8, px * 0.2), - } -} - -/// Horizontal advance of a character at a pixel size. -pub fn advance(c: char, px: f32) -> f32 { - let Some(f) = font() else { return px * 0.5 }; - let gid = f.charmap().map(c).or_else(|| f.charmap().map('?')); - gid.and_then(|g| f.glyph_metrics(Size::new(px), LocationRef::default()).advance_width(g)).unwrap_or(px * 0.5) -} - -/// Width of a single line of text. -pub fn measure(text: &str, px: f32) -> f32 { - text.chars().map(|c| advance(c, px)).sum() -} - -struct Pen { - p: Path, -} - -impl OutlinePen for Pen { - fn move_to(&mut self, x: f32, y: f32) { - self.p.move_to(x, -y); - } - fn line_to(&mut self, x: f32, y: f32) { - self.p.line_to(x, -y); - } - fn quad_to(&mut self, cx0: f32, cy0: f32, x: f32, y: f32) { - self.p.quad_to(cx0, -cy0, x, -y); - } - fn curve_to(&mut self, cx0: f32, cy0: f32, cx1: f32, cy1: f32, x: f32, y: f32) { - self.p.cubic_to(cx0, -cy0, cx1, -cy1, x, -y); - } - fn close(&mut self) { - self.p.close(); - } -} - -/// A rasterised glyph: coverage mask placed with its top-left at `(left, top)` relative to the -/// pen position on the baseline. -#[derive(Debug)] -pub struct Glyph { - pub mask: Mask, - pub left: i32, - pub top: i32, -} - -type Key = (char, u32, u8); - -fn cache() -> &'static Mutex>>> { - static C: OnceLock>>>> = OnceLock::new(); - C.get_or_init(Default::default) -} - -/// Rasterise `c` at `px` with the pen at fractional x offset `frac` (0..1, quantised to ¼ px). -pub fn glyph(c: char, px: f32, frac: f32) -> Option> { - let size_q = (px * 4.0).round().max(1.0) as u32; - let frac_q = ((frac.rem_euclid(1.0) * 4.0).round() as u8) % 4; - let key = (c, size_q, frac_q); - if let Some(g) = cache().lock().unwrap_or_else(|e| e.into_inner()).get(&key) { - return g.clone(); - } - let g = render_glyph(c, size_q as f32 / 4.0, frac_q as f32 / 4.0).map(Arc::new); - let mut m = cache().lock().unwrap_or_else(|e| e.into_inner()); - if m.len() > 8192 { - m.clear(); - } - m.insert(key, g.clone()); - g -} - -fn render_glyph(c: char, px: f32, frac: f32) -> Option { - if c.is_whitespace() { - return None; - } - let f = font()?; - let gid = f.charmap().map(c).or_else(|| f.charmap().map('?'))?; - let outline = f.outline_glyphs().get(gid)?; - let mut pen = Pen { p: Path::default() }; - outline.draw(DrawSettings::unhinted(Size::new(px), LocationRef::default()), &mut pen).ok()?; - pen.p.close(); - let (x0, y0, x1, y1) = pen.p.bounds()?; - let left = (x0 + frac).floor() as i32; - let top = y0.floor() as i32; - let w = ((x1 + frac).ceil() as i32 - left).max(1) as usize; - let h = (y1.ceil() as i32 - top).max(1) as usize; - let mask = fill(&pen.p, w, h, frac - left as f32, -top as f32); - Some(Glyph { mask, left, top }) -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn inter_loads_and_draws() { - let (a, d) = metrics(100.0); - assert!(a > 80.0 && a < 110.0, "ascent {a}"); - assert!(d > 10.0 && d < 40.0, "descent {d}"); - assert!(advance('W', 100.0) > advance('i', 100.0)); - let g = glyph('H', 40.0, 0.0).unwrap(); - // the H has two stems: the middle row has ink on both sides and a gap between them - let y = g.mask.h / 4; - let row: Vec = (0..g.mask.w).map(|x| g.mask.get(x, y)).collect(); - assert!(row[1] > 0.9 && row[g.mask.w - 2] > 0.9, "{row:?}"); - assert!(row[g.mask.w / 2] < 0.1, "{row:?}"); - assert!(g.top < 0, "glyph rises above the baseline"); - assert!(glyph(' ', 40.0, 0.0).is_none()); - assert!(measure("Hello", 30.0) > 50.0); - } -} diff --git a/crates/captions/src/lib.rs b/crates/captions/src/lib.rs index 4c8b24b74..d10c98d12 100644 --- a/crates/captions/src/lib.rs +++ b/crates/captions/src/lib.rs @@ -10,13 +10,11 @@ //! (drop-frame or not) convert exactly; [`Document::snap_to_frames`] snaps to a sequence's frames. //! - **Model:** [`track_from_document`] / [`document_from_track`] convert to and from the project's //! [`CaptionTrack`]. -//! - **Burn-in:** [`burn`] lays a caption out with the track style and rasterises it with the -//! bundled Inter font (outlines via `skrifa`, coverage rasteriser in [`raster`]). +//! - **Burn-in:** [`burn`] lays a caption out with the track style and draws it with the +//! `filmcraft-text` engine (Inter SemiBold, shaped and kerned). pub mod burn; pub mod cea608; -pub mod font; -pub mod raster; pub mod scc; pub mod srt; pub mod vtt; diff --git a/crates/captions/src/raster.rs b/crates/captions/src/raster.rs deleted file mode 100644 index ff8dcc383..000000000 --- a/crates/captions/src/raster.rs +++ /dev/null @@ -1,228 +0,0 @@ -//! A small anti-aliased path rasteriser for glyph outlines. -//! -//! Curves are flattened to line segments; each pixel row is sampled at [`SUB`] sub-scanlines, and -//! on each sub-scanline the non-zero winding spans between edge crossings are accumulated with -//! exact horizontal coverage at the span ends. Original implementation for FilmCraft. - -/// Vertical sub-samples per pixel row. -pub const SUB: usize = 5; - -/// A flattened path: line segments `(x0, y0, x1, y1)` in pixel space, y down. -#[derive(Clone, Debug, Default)] -pub struct Path { - pub segs: Vec<[f32; 4]>, - start: (f32, f32), - cur: (f32, f32), - open: bool, -} - -impl Path { - pub fn move_to(&mut self, x: f32, y: f32) { - self.close(); - self.start = (x, y); - self.cur = (x, y); - self.open = true; - } - pub fn line_to(&mut self, x: f32, y: f32) { - if (x, y) != self.cur { - self.segs.push([self.cur.0, self.cur.1, x, y]); - } - self.cur = (x, y); - } - pub fn quad_to(&mut self, cx: f32, cy: f32, x: f32, y: f32) { - let (x0, y0) = self.cur; - let dd = ((x0 - 2.0 * cx + x).abs() + (y0 - 2.0 * cy + y).abs()).max(0.01); - let n = ((dd * 2.0).sqrt().ceil() as usize).clamp(1, 32); - for i in 1..=n { - let t = i as f32 / n as f32; - let u = 1.0 - t; - self.line_to(u * u * x0 + 2.0 * u * t * cx + t * t * x, u * u * y0 + 2.0 * u * t * cy + t * t * y); - } - } - pub fn cubic_to(&mut self, c1x: f32, c1y: f32, c2x: f32, c2y: f32, x: f32, y: f32) { - let (x0, y0) = self.cur; - let dd = (x0 - 2.0 * c1x + c2x).abs() + (y0 - 2.0 * c1y + c2y).abs() + (c1x - 2.0 * c2x + x).abs() + (c1y - 2.0 * c2y + y).abs(); - let n = (((dd.max(0.01)) * 3.0).sqrt().ceil() as usize).clamp(1, 48); - for i in 1..=n { - let t = i as f32 / n as f32; - let u = 1.0 - t; - let (a, b, c, d) = (u * u * u, 3.0 * u * u * t, 3.0 * u * t * t, t * t * t); - self.line_to(a * x0 + b * c1x + c * c2x + d * x, a * y0 + b * c1y + c * c2y + d * y); - } - } - pub fn close(&mut self) { - if self.open && self.cur != self.start { - self.segs.push([self.cur.0, self.cur.1, self.start.0, self.start.1]); - } - self.cur = self.start; - self.open = false; - } - /// Bounding box `(x0, y0, x1, y1)` of all segments. - pub fn bounds(&self) -> Option<(f32, f32, f32, f32)> { - let mut it = self.segs.iter(); - let f = it.next()?; - let mut b = (f[0].min(f[2]), f[1].min(f[3]), f[0].max(f[2]), f[1].max(f[3])); - for s in it { - b = (b.0.min(s[0]).min(s[2]), b.1.min(s[1]).min(s[3]), b.2.max(s[0]).max(s[2]), b.3.max(s[1]).max(s[3])); - } - Some(b) - } -} - -/// Coverage mask (0..1), row-major. -#[derive(Clone, Debug, PartialEq)] -pub struct Mask { - pub w: usize, - pub h: usize, - pub a: Vec, -} - -impl Mask { - pub fn new(w: usize, h: usize) -> Self { - Self { w, h, a: vec![0.0; w * h] } - } - pub fn get(&self, x: usize, y: usize) -> f32 { - self.a[y * self.w + x] - } -} - -/// Fill `path` (non-zero winding) into a `w`×`h` mask, offsetting the path by `(dx, dy)`. -pub fn fill(path: &Path, w: usize, h: usize, dx: f32, dy: f32) -> Mask { - let mut m = Mask::new(w, h); - if w == 0 || h == 0 { - return m; - } - // edges: (ytop, ybot, x at ytop, dx/dy, winding) - let mut edges: Vec<(f32, f32, f32, f32, i32)> = path - .segs - .iter() - .filter(|s| s[1] != s[3]) - .map(|s| { - let (x0, y0, x1, y1) = (s[0] + dx, s[1] + dy, s[2] + dx, s[3] + dy); - let (dir, xa, ya, xb, yb) = if y0 < y1 { (1, x0, y0, x1, y1) } else { (-1, x1, y1, x0, y0) }; - (ya, yb, xa, (xb - xa) / (yb - ya), dir) - }) - .collect(); - edges.sort_by(|a, b| a.0.total_cmp(&b.0)); - let mut xs: Vec<(f32, i32)> = Vec::new(); - let mut row = vec![0.0f32; w + 1]; - let weight = 1.0 / SUB as f32; - for y in 0..h { - row.iter_mut().for_each(|v| *v = 0.0); - let mut any = false; - for k in 0..SUB { - let sy = y as f32 + (k as f32 + 0.5) / SUB as f32; - xs.clear(); - for e in &edges { - if e.0 > sy { - break; // sorted by top - } - if sy < e.1 { - xs.push((e.2 + (sy - e.0) * e.3, e.4)); - } - } - if xs.len() < 2 { - continue; - } - xs.sort_by(|a, b| a.0.total_cmp(&b.0)); - let mut wind = 0; - let mut span_start = 0.0f32; - for &(x, d) in &xs { - let before = wind; - wind += d; - if before == 0 && wind != 0 { - span_start = x; - } else if before != 0 && wind == 0 { - add_span(&mut row, span_start, x, weight, w); - any = true; - } - } - } - if any { - let out = &mut m.a[y * w..(y + 1) * w]; - for (o, v) in out.iter_mut().zip(&row) { - *o = v.min(1.0); - } - } - } - m -} - -/// Add coverage `weight` over `[x0, x1)` with fractional ends. -fn add_span(row: &mut [f32], x0: f32, x1: f32, weight: f32, w: usize) { - let x0 = x0.clamp(0.0, w as f32); - let x1 = x1.clamp(0.0, w as f32); - if x1 <= x0 { - return; - } - let i0 = x0.floor() as usize; - let i1 = x1.floor() as usize; - if i0 == i1 { - row[i0] += (x1 - x0) * weight; - return; - } - row[i0] += (i0 as f32 + 1.0 - x0) * weight; - for v in &mut row[i0 + 1..i1] { - *v += weight; - } - if i1 < w { - row[i1] += (x1 - i1 as f32) * weight; - } -} - -#[cfg(test)] -mod tests { - use super::*; - - fn rect(x0: f32, y0: f32, x1: f32, y1: f32) -> Path { - let mut p = Path::default(); - p.move_to(x0, y0); - p.line_to(x1, y0); - p.line_to(x1, y1); - p.line_to(x0, y1); - p.close(); - p - } - - #[test] - fn rectangle_coverage() { - let m = fill(&rect(1.5, 2.0, 4.0, 5.0), 6, 6, 0.0, 0.0); - assert!((m.get(1, 3) - 0.5).abs() < 1e-5); - assert!((m.get(2, 3) - 1.0).abs() < 1e-5); - assert_eq!(m.get(4, 3), 0.0); - assert_eq!(m.get(2, 1), 0.0); - assert_eq!(m.get(2, 5), 0.0); - let total: f32 = m.a.iter().sum(); - assert!((total - 2.5 * 3.0).abs() < 1e-3, "{total}"); - } - - #[test] - fn hole_with_opposite_winding() { - let mut p = rect(0.0, 0.0, 10.0, 10.0); - // inner square wound the other way - p.move_to(3.0, 3.0); - p.line_to(3.0, 7.0); - p.line_to(7.0, 7.0); - p.line_to(7.0, 3.0); - p.close(); - let m = fill(&p, 10, 10, 0.0, 0.0); - assert_eq!(m.get(5, 5), 0.0); - assert!((m.get(1, 5) - 1.0).abs() < 1e-5); - } - - #[test] - fn circle_area() { - let mut p = Path::default(); - let (cx, cy, r) = (10.0f32, 10.0f32, 6.0f32); - p.move_to(cx + r, cy); - for i in 1..=64 { - let a = i as f32 / 64.0 * std::f32::consts::TAU; - p.line_to(cx + r * a.cos(), cy + r * a.sin()); - } - p.close(); - let m = fill(&p, 20, 20, 0.0, 0.0); - let total: f32 = m.a.iter().sum(); - let want = std::f32::consts::PI * r * r; - assert!((total - want).abs() / want < 0.02, "{total} vs {want}"); - } -} diff --git a/crates/codecs/Cargo.toml b/crates/codecs/Cargo.toml index ee250a8c5..843bd4f1a 100644 --- a/crates/codecs/Cargo.toml +++ b/crates/codecs/Cargo.toml @@ -19,6 +19,7 @@ filmcraft-matroska = { workspace = true } filmcraft-bitstream = { workspace = true } filmcraft-h264 = { workspace = true } filmcraft-hevc = { workspace = true } +filmcraft-vp9 = { workspace = true } filmcraft-prores = { workspace = true } filmcraft-aac = { workspace = true } filmcraft-opus = { workspace = true } diff --git a/crates/codecs/src/gop.rs b/crates/codecs/src/gop.rs index 16c6ef59f..34a603882 100644 --- a/crates/codecs/src/gop.rs +++ b/crates/codecs/src/gop.rs @@ -180,7 +180,7 @@ impl GopCache { if st.intra { return self.intra_frame(st, s, i, want_pts); } - let key = s.sync_before(i); + let mut key = s.sync_before(i); // Continue the running decoder when it has passed the wanted sample's sync sample and // either has not reached the sample yet or has been fed it without outputting it yet // (a frame-threaded decoder holds many pictures in flight). Otherwise the frame was @@ -192,6 +192,15 @@ impl GopCache { let continuing = running && (st.next > key || near) && (i >= st.next || (st.start <= i && want_pts > st.out_max)); if !continuing { SEEKS.fetch_add(1, Ordering::Relaxed); + // The container's sync flags may be wrong for the codec (an MP4 without `stss` marks + // every sample): step back to a sample the decoder can start from. + while key > 0 { + let data = s.read(key)?; + if st.decoder.as_ref().expect("decoder").is_random_access(&data) != Some(false) { + break; + } + key = s.sync_before(key - 1); + } if let Some(d) = st.decoder.as_mut() { d.reset(); } diff --git a/crates/codecs/src/lib.rs b/crates/codecs/src/lib.rs index 9dd95d64c..7a1baad0d 100644 --- a/crates/codecs/src/lib.rs +++ b/crates/codecs/src/lib.rs @@ -1,6 +1,6 @@ //! The container + codec hub. //! -//! - [`VideoDecoder`]: the trait every video codec implements (our own H.264/ProRes/…, MJPEG, and +//! - [`VideoDecoder`]: the trait every video codec implements (our own H.264/HEVC/VP9/ProRes/…, MJPEG, and //! OS hardware decoders registered by the platform layer). Factories are tried in registration //! order, so a hardware decoder can take precedence over the pure-Rust one. //! - [`Mp4Source`]: a [`MediaSource`](filmcraft_media::MediaSource) over MP4/MOV using @@ -51,7 +51,7 @@ impl From for filmcraft_media::MediaError { fn factories() -> &'static RwLock> { static F: std::sync::OnceLock>> = std::sync::OnceLock::new(); - F.get_or_init(|| RwLock::new(vec![video::h264_factory, video::hevc_factory, video::prores_factory, video::mjpeg_factory])) + F.get_or_init(|| RwLock::new(vec![video::h264_factory, video::hevc_factory, video::vp9_factory, video::prores_factory, video::mjpeg_factory])) } /// Register a video decoder factory (tried before previously registered ones). diff --git a/crates/codecs/src/mkv.rs b/crates/codecs/src/mkv.rs index e5134fd95..b6c56a395 100644 --- a/crates/codecs/src/mkv.rs +++ b/crates/codecs/src/mkv.rs @@ -1,7 +1,7 @@ //! Matroska/WebM media source (`filmcraft-matroska` demux, GOP-aware video via [`crate::gop`]). //! //! Video codecs are mapped onto ISO-BMFF sample entries so the same decoder factories serve both -//! containers (H.264, HEVC, ProRes, MJPEG; VP9/AV1 once their decoders land). Audio: AAC and Opus +//! containers (H.264, HEVC, VP9, ProRes, MJPEG; AV1 once its decoder lands). Audio: AAC and Opus //! via our decoders, PCM directly, MP3/FLAC/Vorbis via the bootstrap decoders. //! //! Opus (`A_OPUS`): `CodecPrivate` is the `OpusHead`; output is always 48 kHz. The demuxer @@ -14,7 +14,7 @@ use std::sync::{Arc, Mutex}; use filmcraft_color::{ColorInfo, Matrix, Primaries, Range, Transfer}; use filmcraft_frame::{AudioBuffer, VideoFrame}; -use filmcraft_isobmff::{AvcConfig, FourCc, HevcConfig, PcmConfig, SampleEntry}; +use filmcraft_isobmff::{AvcConfig, CodecConfig, FourCc, HevcConfig, PcmConfig, SampleEntry, VpcConfig}; use filmcraft_matroska::{Codec, MkvFile, TrackKind}; use filmcraft_media::{AudioStreamInfo, FrameRequest, MediaError, MediaInfo, MediaKind, MediaSource, SharedSource, VideoStreamInfo}; use filmcraft_time::{FrameRate, Tick}; @@ -105,8 +105,42 @@ fn from_tick(t: &filmcraft_matroska::Track, time: Tick) -> i64 { time.to_rational_floor(n, d) } -fn sample_entry(c: &Codec, w: u16, h: u16) -> Option { +/// VP9 configuration from the Matroska `CodecPrivate` feature list (ID / length / value triples: +/// 1 profile, 2 level, 3 bit depth, 4 chroma subsampling) and the track's `Colour`. +fn vp9_config(private: &[u8], v: Option<&filmcraft_matroska::VideoInfo>) -> VpcConfig { + let mut c = VpcConfig { bit_depth: 8, colour_primaries: 2, transfer_characteristics: 2, matrix_coefficients: 2, ..Default::default() }; + let mut p = 0; + while p + 2 <= private.len() { + let (id, len) = (private[p], private[p + 1] as usize); + let Some(val) = private.get(p + 2..p + 2 + len) else { break }; + let x = val.first().copied().unwrap_or(0); + match id { + 1 => c.profile = x, + 2 => c.level = x, + 3 => c.bit_depth = x, + 4 => c.chroma_subsampling = x, + _ => {} + } + p += 2 + len; + } + if let Some(col) = v.and_then(|v| v.colour.as_ref()) { + if let Some(t) = col.transfer_characteristics { + c.transfer_characteristics = t as u8; + } + if let Some(pr) = col.primaries { + c.colour_primaries = pr as u8; + } + if let Some(m) = col.matrix_coefficients { + c.matrix_coefficients = m as u8; + } + c.full_range = col.full_range(); + } + c +} + +fn sample_entry(c: &Codec, v: Option<&filmcraft_matroska::VideoInfo>, w: u16, h: u16) -> Option { Some(match c { + Codec::Vp9 { private } => SampleEntry::video(FourCc(*b"vp09"), CodecConfig::Vp9(vp9_config(private, v)), w, h), Codec::Avc { avcc } => SampleEntry::avc(AvcConfig::parse(avcc).ok()?, w, h), Codec::Hevc { hvcc } => SampleEntry::hevc(HevcConfig::parse(hvcc).ok()?, w, h), Codec::ProRes { fourcc } => SampleEntry::prores(FourCc(fourcc.unwrap_or(*b"apcn")), w, h), @@ -190,7 +224,7 @@ impl MkvSource { if explicit { explicit_color = Some(color); } - ventry = sample_entry(&t.codec, w as u16, h as u16); + ventry = sample_entry(&t.codec, t.video.as_ref(), w as u16, h as u16); let secs = file.duration_ns().unwrap_or(0) as f64 / 1e9; let bitrate = (secs > 0.0).then(|| (t.samples.iter().map(|s| s.size as u64).sum::() as f64 * 8.0 / secs) as u64); VideoStreamInfo { diff --git a/crates/codecs/src/mp4.rs b/crates/codecs/src/mp4.rs index 3cc08b874..8d8f46097 100644 --- a/crates/codecs/src/mp4.rs +++ b/crates/codecs/src/mp4.rs @@ -48,9 +48,17 @@ pub fn sniff(b: &[u8]) -> bool { fn color_from(entry: &filmcraft_isobmff::SampleEntry, w: u32, h: u32) -> ColorInfo { let mut c = ColorInfo { matrix: filmcraft_frame::default_matrix(w, h), transfer: Transfer::Bt709, primaries: Primaries::Bt709, range: Range::Limited }; - if let Some(v) = &entry.video - && let Some(col) = &v.color - { + // VP9 carries its colour description in vpcC (used when there is no colr box). + let vpc = match &entry.codec { + CodecConfig::Vp9(v) => Some(filmcraft_isobmff::ColorInfo::Nclx { + primaries: v.colour_primaries as u16, + transfer: v.transfer_characteristics as u16, + matrix: v.matrix_coefficients as u16, + full_range: v.full_range, + }), + _ => None, + }; + if let Some(col) = entry.video.as_ref().and_then(|v| v.color.as_ref()).or(vpc.as_ref()) { let (p, t, m, full) = match col { filmcraft_isobmff::ColorInfo::Nclx { primaries, transfer, matrix, full_range } => (*primaries, *transfer, *matrix, *full_range), filmcraft_isobmff::ColorInfo::Nclc { primaries, transfer, matrix } => (*primaries, *transfer, *matrix, false), @@ -369,6 +377,7 @@ fn codec_label(c: &CodecConfig) -> String { } ), CodecConfig::Hevc(_) => "HEVC".into(), + CodecConfig::Vp9(c) => format!("VP9 (Profile {})", c.profile), CodecConfig::ProRes { fourcc } => match &fourcc.0 { b"apco" => "Apple ProRes 422 Proxy".into(), b"apcs" => "Apple ProRes 422 LT".into(), @@ -399,6 +408,14 @@ fn pixfmt_label(c: &CodecConfig) -> String { 244 => "YUV 4:4:4".into(), _ => "YUV 4:2:0 8-bit".into(), }, + CodecConfig::Vp9(c) => { + let sub = match c.chroma_subsampling { + 2 => "4:2:2", + 3 => "4:4:4", + _ => "4:2:0", + }; + format!("YUV {sub} {}-bit", c.bit_depth.max(8)) + } CodecConfig::ProRes { fourcc } if fourcc.0[2] == b'4' => "YUVA 4:4:4 12-bit".into(), CodecConfig::ProRes { .. } => "YUV 4:2:2 10-bit".into(), CodecConfig::Jpeg { .. } => "YUV 4:2:x 8-bit".into(), diff --git a/crates/codecs/src/tests.rs b/crates/codecs/src/tests.rs index 601f2048b..fc35862aa 100644 --- a/crates/codecs/src/tests.rs +++ b/crates/codecs/src/tests.rs @@ -179,6 +179,169 @@ fn matroska_prores_pcm() { check_mkv("blue_prores_pcm.mkv", &["-c:v", "prores_ks", "-profile:v", "2"], &["-c:a", "pcm_s16le"]); } +/// Path of a fixture under `target/fixtures/codecs/`, generated with ffmpeg when missing +/// (`None` when ffmpeg or the encoder is unavailable). +fn fixture_path(name: &str, args: &[&str]) -> Option { + fixture(name, args)?; + Some(PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target/fixtures/codecs").join(name)) +} + +/// ffmpeg's decode of `src` as raw planar video (the oracle). +fn reference_yuv(src: &std::path::Path, pix_fmt: &str) -> Option> { + let name = format!("{}.{pix_fmt}.yuv", src.file_name()?.to_str()?); + let path = src.to_str()?; + let b = fixture(&name, &["-i", path, "-fps_mode", "passthrough", "-f", "rawvideo", "-pix_fmt", pix_fmt])?; + Some(b.to_vec()) +} + +/// Planes of a decoded YUV frame as little-endian bytes (ffmpeg rawvideo layout). +fn yuv_bytes(f: &filmcraft_frame::VideoFrame) -> Vec { + match &f.data { + filmcraft_frame::PixelData::Yuv8 { planes, .. } => planes.iter().flat_map(|p| p.iter().copied()).collect(), + filmcraft_frame::PixelData::Yuv16 { planes, .. } => planes.iter().flat_map(|p| p.iter().flat_map(|s| s.to_le_bytes())).collect(), + _ => panic!("not a YUV frame: {}", f.format_label()), + } +} + +/// Random access into a VP9 file: frames requested out of order (seeks back and forth across key +/// frames, then sequential playback) are sample-exact against ffmpeg's decode. +fn check_vp9_seeks(name: &str, container: &str, pix_fmt: &str, extra: &[&str], frames: usize) { + let dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target/fixtures/codecs"); + let log = dir.join(format!("{name}.passlog")).to_string_lossy().into_owned(); + let mut args = vec!["-f", "lavfi", "-i", "testsrc2=s=352x288:r=25,noise=alls=8:allf=t", "-frames:v"]; + let n = frames.to_string(); + args.push(&n); + args.extend_from_slice(&["-c:v", "libvpx-vp9", "-pix_fmt", pix_fmt, "-g", "25", "-b:v", "600k"]); + args.extend_from_slice(extra); + if extra.contains(&"-auto-alt-ref") { + // libvpx only uses alternate reference frames in two-pass mode: run the first pass. + let Some(ff) = ffmpeg() else { return }; + if !dir.join(name).exists() { + let _ = std::fs::create_dir_all(&dir); + let ok = Command::new(ff).args(["-y", "-v", "error"]).args(&args).args(["-pass", "1", "-passlogfile"]).arg(&log).args(["-f", "null", "-"]).status(); + if !ok.is_ok_and(|s| s.success()) { + eprintln!("first pass for {name} failed; skipping"); + return; + } + } + args.extend_from_slice(&["-pass", "2", "-passlogfile"]); + args.push(&log); + } + let Some(path) = fixture_path(name, &args) else { return }; + let Some(reference) = reference_yuv(&path, pix_fmt) else { return }; + let bytes: Arc<[u8]> = std::fs::read(&path).unwrap().into(); + let src = crate::open_bytes(name, bytes).unwrap(); + let info = src.info().clone(); + assert_eq!(info.container, container); + let v = info.video.as_ref().unwrap(); + assert!(v.codec.contains("VP9"), "{}", v.codec); + assert_eq!((v.width, v.height), (352, 288)); + let frame_len = reference.len() / frames; + let rate = info.frame_rate(); + let order: Vec = [frames - 3, 3, 40, 26, 24, frames - 1, 0, 12].into_iter().chain(30..45).filter(|&k| k < frames).collect(); + for k in order { + let f = src.video_frame(FrameRequest::full(rate.tick_of(k as i64))).unwrap(); + assert_eq!((f.width, f.height), (352, 288)); + let got = yuv_bytes(&f); + assert!(got == reference[k * frame_len..(k + 1) * frame_len], "{name}: frame {k} differs from ffmpeg"); + } +} + +#[test] +fn webm_vp9_seeks_bit_exact() { + check_vp9_seeks("noise_vp9.webm", "WebM", "yuv420p", &["-deadline", "realtime", "-speed", "8"], 60); +} + +#[test] +fn webm_vp9_altref_superframes_seek_bit_exact() { + // Good-quality encode with alternate reference frames: hidden frames packed into superframes. + check_vp9_seeks("noise_vp9_altref.webm", "WebM", "yuv420p", &["-speed", "4", "-auto-alt-ref", "1", "-lag-in-frames", "16"], 60); +} + +#[test] +fn mp4_vp9_10bit_seeks_bit_exact() { + check_vp9_seeks("noise_vp9_10bit.mp4", "MPEG-4", "yuv420p10le", &["-profile:v", "2", "-deadline", "realtime", "-speed", "8"], 50); +} + +#[test] +fn mkv_vp9_444_12bit_seeks_bit_exact() { + check_vp9_seeks("noise_vp9_444_12bit.mkv", "Matroska", "yuv444p12le", &["-profile:v", "3", "-deadline", "realtime", "-speed", "8"], 30); +} + +#[test] +fn mkv_vp9_422_seeks_bit_exact() { + check_vp9_seeks("noise_vp9_422.mkv", "Matroska", "yuv422p", &["-profile:v", "1", "-deadline", "realtime", "-speed", "8"], 30); +} + +/// Every sample flagged as sync (as in an MP4 without `stss`): the GOP cache must still start +/// decoding at a real VP9 key frame. +#[test] +fn vp9_random_access_ignores_bogus_sync_flags() { + use crate::gop::{GopCache, VideoSamples}; + let Some(path) = fixture_path( + "noise_vp9_g20.ivf", + &["-f", "lavfi", "-i", "testsrc2=s=176x144:r=25", "-frames:v", "50", "-c:v", "libvpx-vp9", "-g", "20", "-deadline", "realtime", "-speed", "8"], + ) else { + return; + }; + let Some(reference) = reference_yuv(&path, "yuv420p") else { return }; + let data = std::fs::read(&path).unwrap(); + let mut chunks = Vec::new(); + let mut p = 32; + while p + 12 <= data.len() { + let sz = u32::from_le_bytes(data[p..p + 4].try_into().unwrap()) as usize; + chunks.push(data[p + 12..p + 12 + sz].to_vec()); + p += 12 + sz; + } + struct AllSync(Vec>); + impl VideoSamples for AllSync { + fn count(&self) -> usize { + self.0.len() + } + fn pts(&self, i: usize) -> i64 { + i as i64 + } + fn sync_before(&self, i: usize) -> usize { + i + } + fn sample_at(&self, t: i64) -> Option { + (t >= 0 && (t as usize) < self.0.len()).then_some(t as usize) + } + fn read(&self, i: usize) -> crate::Result> { + Ok(self.0[i].clone()) + } + fn make_decoder(&self) -> crate::Result> { + Ok(Box::new(crate::video::Vp9Decoder::new(None))) + } + } + let s = AllSync(chunks); + let cache = GopCache::new(None); + let frame_len = 176 * 144 * 3 / 2; + for k in [37usize, 5, 49, 21, 20, 19] { + let f = cache.frame(&s, k as i64).unwrap(); + assert!(yuv_bytes(&f) == reference[k * frame_len..(k + 1) * frame_len], "frame {k} differs from ffmpeg"); + } +} + +/// VP9 RGB (profile 1, colour space sRGB; planes G, B, R) decodes to RGBA in the right order. +#[test] +fn mkv_vp9_rgb() { + let Some(b) = fixture( + "orange_vp9_gbrp.mkv", + &["-f", "lavfi", "-i", "color=c=0xe08020:s=128x96:r=25:d=0.4", "-c:v", "libvpx-vp9", "-pix_fmt", "gbrp", "-deadline", "realtime", "-lossless", "1"], + ) else { + return; + }; + let src = crate::open_bytes("orange_vp9_gbrp.mkv", b).unwrap(); + let f = src.video_frame(FrameRequest::full(Tick(TICKS_PER_SECOND / 5))).unwrap(); + let px = f.to_rgba8(); + let c = &px[(48 * 128 + 64) * 4..][..3]; + // (the lavfi colour source is converted to RGB by ffmpeg, which rounds by a level) + for (got, want) in c.iter().zip([0xe0u8, 0x80, 0x20]) { + assert!((*got as i32 - want as i32).abs() <= 2, "{c:?}"); + } +} + /// ffmpeg + libopus decode of a container fixture (pre-skip / codec delay / edit list applied) as /// interleaved f32 at 48 kHz: the reference for our Opus path. fn opus_reference(name: &str) -> Option> { diff --git a/crates/codecs/src/video.rs b/crates/codecs/src/video.rs index 16d1ad38f..9542951d1 100644 --- a/crates/codecs/src/video.rs +++ b/crates/codecs/src/video.rs @@ -25,6 +25,12 @@ pub trait VideoDecoder: Send { fn intra_only(&self) -> bool { false } + /// Whether decoding can start at `sample` (`None`: unknown, trust the container's sync flags). + /// Containers may flag samples as sync that the codec cannot start from (an MP4 without + /// `stss` marks every sample), so codecs that can tell say so. + fn is_random_access(&self, _sample: &[u8]) -> Option { + None + } } /// A factory returns `None` when it does not handle the entry. @@ -220,6 +226,141 @@ pub fn hevc_factory(e: &SampleEntry) -> Option>> { } } +/// Our pure-Rust VP9 decoder (profiles 0-3, 8/10/12-bit; tile columns and loop filter decode in +/// parallel). +pub struct Vp9Decoder { + dec: filmcraft_vp9::Decoder, + /// Container colour (vpcC / Matroska `Colour`): transfer and primaries are not in the VP9 + /// bitstream. + transfer: Option, + primaries: Option, +} + +/// Colour primaries from an ISO/IEC 23091-2 code. +pub(crate) fn primaries_from_code(c: u8) -> Option { + use filmcraft_color::Primaries; + match c { + 1 => Some(Primaries::Bt709), + 5 => Some(Primaries::Bt601_625), + 6 => Some(Primaries::Bt601_525), + 9 => Some(Primaries::Bt2020), + 12 => Some(Primaries::P3D65), + _ => None, + } +} + +impl Vp9Decoder { + pub fn new(cfg: Option<&filmcraft_isobmff::VpcConfig>) -> Self { + Self { + dec: filmcraft_vp9::Decoder::new(), + transfer: cfg.and_then(|c| filmcraft_color::Transfer::from_code(c.transfer_characteristics)), + primaries: cfg.and_then(|c| primaries_from_code(c.colour_primaries)), + } + } + + fn convert(&self, p: filmcraft_vp9::Picture) -> DecodedFrame { + use filmcraft_color::{Matrix, Primaries, Range}; + use filmcraft_frame::{Chroma, PixelData}; + use filmcraft_vp9::Plane; + use std::sync::Arc; + let (w, h) = (p.width as usize, p.height as usize); + let (cw, ch) = (p.chroma_width as usize, p.chroma_height as usize); + let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(p.width, p.height), ..filmcraft_color::ColorInfo::REC709 }; + // color_space (7.2.2): 1 BT.601, 2 BT.709, 3 SMPTE-170, 4 SMPTE-240, 5 BT.2020, 7 sRGB. + match p.color.color_space { + 1 | 3 => color.matrix = Matrix::Bt601, + 2 | 4 => color.matrix = Matrix::Bt709, + 5 => { + color.matrix = Matrix::Bt2020Ncl; + color.primaries = Primaries::Bt2020; + } + _ => {} + } + if let Some(t) = self.transfer { + color.transfer = t; + } + if let Some(pr) = self.primaries { + color.primaries = pr; + } + if p.color.full_range { + color.range = Range::Full; + } + let pts = p.pts; + if p.color.color_space == 7 { + // RGB (profiles 1 / 3, 4:4:4): the planes carry G, B, R. + let shift = p.bit_depth - 8; + let mut rgba = Vec::with_capacity(w * h * 4); + for i in 0..w * h { + rgba.extend_from_slice(&[(p.v.get(i) >> shift) as u8, (p.y.get(i) >> shift) as u8, (p.u.get(i) >> shift) as u8, 255]); + } + let mut frame = VideoFrame::rgba8(p.width, p.height, rgba); + frame.color = filmcraft_color::ColorInfo { range: Range::Full, transfer: filmcraft_color::Transfer::Srgb, ..color }; + return DecodedFrame { pts, frame }; + } + // 4:4:0 has no frame format of its own: chroma rows are repeated to 4:4:4. + let (chroma, rows_440) = match (p.subsampling_x, p.subsampling_y) { + (true, true) => (Chroma::C420, false), + (true, false) => (Chroma::C422, false), + (false, false) => (Chroma::C444, false), + (false, true) => (Chroma::C444, true), + }; + fn expand(v: &[T], cw: usize, ch: usize, h: usize, rows_440: bool) -> Vec { + if !rows_440 { + return v[..cw * ch].to_vec(); + } + let mut out = Vec::with_capacity(cw * h); + for y in 0..h { + out.extend_from_slice(&v[(y >> 1) * cw..(y >> 1) * cw + cw]); + } + out + } + let data = match (&p.y, &p.u, &p.v) { + (Plane::U8(y), Plane::U8(u), Plane::U8(v)) => PixelData::Yuv8 { + planes: [Arc::new(y[..w * h].to_vec()), Arc::new(expand(u, cw, ch, h, rows_440)), Arc::new(expand(v, cw, ch, h, rows_440))], + chroma, + alpha: None, + }, + (Plane::U16(y), Plane::U16(u), Plane::U16(v)) => PixelData::Yuv16 { + planes: [Arc::new(y[..w * h].to_vec()), Arc::new(expand(u, cw, ch, h, rows_440)), Arc::new(expand(v, cw, ch, h, rows_440))], + chroma, + bits: p.bit_depth, + alpha: None, + }, + _ => unreachable!("VP9 planes share one sample type"), + }; + // render_size (the intended display size) is not applied: the container's display + // dimensions / pixel aspect describe the presentation. + let par = (1, 1); + DecodedFrame { pts, frame: VideoFrame { width: p.width, height: p.height, data, color, par, pts: filmcraft_time::Tick::ZERO } } + } +} + +impl VideoDecoder for Vp9Decoder { + fn decode(&mut self, sample: &[u8], pts: i64) -> Result> { + let pics = self.dec.decode(sample, pts).map_err(|e| CodecError::Decode(e.to_string()))?; + Ok(pics.into_iter().map(|p| self.convert(p)).collect()) + } + fn flush(&mut self) -> Vec { + self.dec.flush().into_iter().map(|p| self.convert(p)).collect() + } + fn reset(&mut self) { + self.dec.reset(); + } + fn name(&self) -> &str { + "FilmCraft VP9" + } + fn is_random_access(&self, sample: &[u8]) -> Option { + Some(filmcraft_vp9::is_keyframe(sample)) + } +} + +pub fn vp9_factory(e: &SampleEntry) -> Option>> { + match &e.codec { + CodecConfig::Vp9(c) => Some(Ok(Box::new(Vp9Decoder::new(Some(c))) as Box)), + _ => None, + } +} + /// Our ProRes decoder (every frame is intra; slices decode in parallel). pub struct ProResDecoder; diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml index b8fb90c7d..287c1a325 100644 --- a/crates/engine/Cargo.toml +++ b/crates/engine/Cargo.toml @@ -19,6 +19,7 @@ filmcraft-codecs = { workspace = true } filmcraft-project = { workspace = true } filmcraft-edit = { workspace = true } filmcraft-render = { workspace = true } +filmcraft-text = { workspace = true } filmcraft-export = { workspace = true } filmcraft-format = { workspace = true } filmcraft-interchange = { workspace = true } diff --git a/crates/engine/src/autosave.rs b/crates/engine/src/autosave.rs index 436841ab7..e66b7c2d6 100644 --- a/crates/engine/src/autosave.rs +++ b/crates/engine/src/autosave.rs @@ -63,15 +63,98 @@ impl AutoSavePrefs { } } +/// Preferences ▸ Playback (the parts trim-mode loop playback uses). +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", default)] +pub struct PlaybackPrefs { + /// "Preroll: N seconds" (Play Around, trim-mode loop playback). + pub preroll_seconds: f64, + /// "Postroll: N seconds". + pub postroll_seconds: f64, +} + +impl Default for PlaybackPrefs { + fn default() -> Self { + Self { preroll_seconds: 3.0, postroll_seconds: 2.0 } + } +} + +/// Preferences ▸ Trim. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", default)] +pub struct TrimPrefs { + /// "Large Trim Offset: N frames" (Trim Forward/Backward Many, the Trim Monitor's ±N buttons). + pub large_trim_offset: u32, + /// "Playhead position determines trim monitor loop playback" (else the edit point does). + pub playhead_determines_loop: bool, +} + +impl Default for TrimPrefs { + fn default() -> Self { + Self { large_trim_offset: 5, playhead_determines_loop: false } + } +} + /// User preferences (persisted as JSON in the per-user data directory). Keys are dotted camelCase -/// paths: `autoSave.enabled`, `autoSave.intervalMinutes`, … +/// paths: `autoSave.enabled`, `autoSave.intervalMinutes`, `playback.prerollSeconds`, … #[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] #[serde(rename_all = "camelCase", default)] pub struct Preferences { pub auto_save: AutoSavePrefs, + pub audio: AudioPrefs, + pub playback: PlaybackPrefs, + pub trim: TrimPrefs, +} + +/// Preferences ▸ Audio (the mixer-automation part) and the Track Mixer panel-menu toggle. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", default)] +pub struct AudioPrefs { + /// "Automatch Time" (s): how long Touch takes to return to the existing automation. + pub automatch_time: f64, + /// "Large Volume Adjustment" (dB) for the Increase/Decrease Clip Volume Many commands. + pub large_volume_adjustment: f64, + /// Automation Keyframe Optimization: "Linear keyframe thinning". + pub linear_keyframe_thinning: bool, + /// "Minimum time interval thinning". + pub minimum_time_interval_thinning: bool, + /// "Minimum time" (ms). + pub minimum_time_ms: u32, + /// Track Mixer ▸ "Switch to Touch after Write". + pub switch_to_touch_after_write: bool, +} + +impl Default for AudioPrefs { + fn default() -> Self { + Self { + automatch_time: 1.0, + large_volume_adjustment: 6.0, + linear_keyframe_thinning: true, + minimum_time_interval_thinning: false, + minimum_time_ms: 20, + switch_to_touch_after_write: true, + } + } +} + +impl AudioPrefs { + fn clamp(&mut self) { + self.automatch_time = if self.automatch_time.is_finite() { self.automatch_time.clamp(0.0, 30.0) } else { 1.0 }; + self.large_volume_adjustment = if self.large_volume_adjustment.is_finite() { self.large_volume_adjustment.clamp(0.0, 96.0) } else { 6.0 }; + self.minimum_time_ms = self.minimum_time_ms.clamp(1, 10_000); + } } impl Preferences { + fn clamp(&mut self) { + self.auto_save.clamp(); + self.audio.clamp(); + let secs = |v: f64| if v.is_finite() { v.clamp(0.0, 60.0) } else { 0.0 }; + self.playback.preroll_seconds = secs(self.playback.preroll_seconds); + self.playback.postroll_seconds = secs(self.playback.postroll_seconds); + self.trim.large_trim_offset = self.trim.large_trim_offset.clamp(1, 1000); + } + pub fn to_value(&self) -> Value { serde_json::to_value(self).unwrap_or_default() } @@ -117,14 +200,14 @@ impl Preferences { }; *slot = value; let mut p: Preferences = serde_json::from_value(v).map_err(|e| format!("`{key}`: {e}"))?; - p.auto_save.clamp(); + p.clamp(); *self = p; Ok(()) } pub fn load(path: &Path) -> Self { let mut p: Preferences = fs::read(path).ok().and_then(|b| serde_json::from_slice(&b).ok()).unwrap_or_default(); - p.auto_save.clamp(); + p.clamp(); p } diff --git a/crates/engine/src/commands.rs b/crates/engine/src/commands.rs index 687721f3b..7f3ae7e09 100644 --- a/crates/engine/src/commands.rs +++ b/crates/engine/src/commands.rs @@ -876,20 +876,16 @@ fn build() -> Vec { })?; Ok(Value::Null) }), - cmd!("clip.audioGain", "Audio Gain…", ["Clip"], Some("G"), r#"{"db":f64,"relative":bool}"#, has_selection, |s, p| { - let db = f64_p(p, "db").unwrap_or(0.0); - let rel = bool_p(p, "relative").unwrap_or(true); - let sel = with_links(s, &s.state.selection.clone()); - s.edit_sequence("Audio Gain", |q, _, _| { - for t in q.audio_tracks.iter_mut() { - for i in t.items.iter_mut().filter(|i| sel.contains(&i.id)) { - i.gain_db = if rel { i.gain_db + db } else { db }; - } - } - Ok(()) - })?; - Ok(Value::Null) - }), + cmd!( + "clip.audioGain", + "Audio Gain…", + ["Clip"], + Some("G"), + r#"{"mode":"set|adjust|normalizeMax|normalizeAll"?,"db":f64,"relative":bool?}"#, + has_selection, + |s, p| crate::mixer::audio_gain(s, p) + ), + query!("clip.audioPeak", "Audio Clip Peak Amplitude", r#"{"clips":[id]?}"#, |s, p| crate::mixer::audio_peak(s, p)), cmd!("clip.frameHold", "Add Frame Hold", ["Clip", "Video Options"], None, "{}", has_selection, |s, _| { let t = s.playhead(); let sel = s.state.selection.clone(); @@ -916,7 +912,7 @@ fn build() -> Vec { s.events.push(crate::Event::OpenSequence(id)); Ok(Value::Null) }), - cmd!("sequence.close", "Close Sequence", [], None, r#"{"item":id}"#, has_seq, |s, p| { + cmd!("sequence.close", "Close Sequence", [], None, r#"{"item":id?}"#, has_seq, |s, p| { let id = item_p(p, "item").or(s.state.active_sequence).ok_or(EngineError::NoSequence)?; s.state.open_sequences.retain(|x| *x != id); if s.state.active_sequence == Some(id) { @@ -1252,8 +1248,10 @@ fn build() -> Vec { Ok(Value::Null) }), // ================= Source monitor ================= - cmd!("source.open", "Open in Source Monitor", [], None, r#"{"item":id}"#, always, |s, p| { - let id = item_p(p, "item").ok_or_else(|| bad("source.open", "need `item`"))?; + cmd!("source.open", "Open in Source Monitor", [], None, r#"{"item":id?}"#, always, |s, p| { + let id = item_p(p, "item") + .or(s.state.project_selection.first().copied()) + .ok_or_else(|| bad("source.open", "need `item` (or a Project panel selection)"))?; s.project.item(id).ok_or_else(|| bad("source.open", "no such item"))?; s.state.source_item = Some(id); s.state.source_playhead = Tick::ZERO; @@ -1313,6 +1311,9 @@ fn build() -> Vec { let clips: Vec = p.get("clips").and_then(Value::as_array).map(|a| a.iter().filter_map(|v| v.as_u64().map(ClipId)).collect()).unwrap_or_default(); let clips = with_links(s, &clips); + // selecting clips leaves trim mode (Premiere: clip and edit point selections are exclusive) + s.state.edit_points.clear(); + s.state.trim_shift = Default::default(); if bool_p(p, "toggle").unwrap_or(false) { for c in clips { if let Some(i) = s.state.selection.iter().position(|x| *x == c) { @@ -1404,16 +1405,43 @@ fn build() -> Vec { }), cmd!("trim.clear", "Clear Edit Point Selection", [], None, "{}", has_edit_points, |s, _| { s.state.edit_points.clear(); + s.state.trim_shift = Default::default(); + s.trim_play.around = None; Ok(Value::Null) }), cmd!("trim.toggleType", "Toggle Trim Type", [], Some("Ctrl+T"), "{}", has_edit_points, |s, _| crate::trim::toggle_type(s)), cmd!("trim.backward", "Trim Backward", [], Some("Alt+Left"), "{}", has_edit_points, |s, _| crate::trim::nudge(s, &json!({"frames": -1}))), cmd!("trim.forward", "Trim Forward", [], Some("Alt+Right"), "{}", has_edit_points, |s, _| crate::trim::nudge(s, &json!({"frames": 1}))), - cmd!("trim.backwardMany", "Trim Backward Many", [], Some("Alt+Shift+Left"), "{}", has_edit_points, |s, _| crate::trim::nudge( - s, - &json!({"frames": -5}) - )), - cmd!("trim.forwardMany", "Trim Forward Many", [], Some("Alt+Shift+Right"), "{}", has_edit_points, |s, _| crate::trim::nudge(s, &json!({"frames": 5}))), + cmd!("trim.backwardMany", "Trim Backward Many", [], Some("Alt+Shift+Left"), "{}", has_edit_points, |s, _| { + let n = -(s.prefs.trim.large_trim_offset as i64); + crate::trim::nudge(s, &json!({"frames": n})) + }), + cmd!("trim.forwardMany", "Trim Forward Many", [], Some("Alt+Shift+Right"), "{}", has_edit_points, |s, _| { + let n = s.prefs.trim.large_trim_offset as i64; + crate::trim::nudge(s, &json!({"frames": n})) + }), + cmd!("trim.applyDefaultTransition", "Apply Default Transitions to Selection", ["Sequence"], None, "{}", has_edit_points, |s, _| { + crate::trim::apply_default_transitions(s) + }), + cmd!("trim.shuttle", "Dynamic Trim (Shuttle)", [], None, r#"{"direction":"forward|reverse","slow":bool?,"clock":seconds}"#, has_edit_points, |s, p| { + crate::trim::shuttle(s, p) + }), + cmd!("trim.shuttleStop", "Dynamic Trim Stop", [], None, r#"{"clock":seconds?}"#, has_seq, |s, p| crate::trim::stop(s, p)), + cmd!("trim.cancelDynamic", "Cancel Dynamic Trim", [], None, "{}", has_seq, |s, _| crate::trim::cancel(s)), + cmd!("trim.playAround", "Play Around Edit", [], None, r#"{"clock":seconds,"loop":bool=true,"toggle":bool?}"#, has_seq, |s, p| { + crate::trim::play_around(s, p) + }), + CommandSpec { + id: "trim.tick", + label: "Advance Trim Playback", + menu: &[], + shortcut: None, + params: r#"{"clock":seconds}"#, + enabled: has_seq, + run: |s, p| crate::trim::tick(s, p.get("clock").and_then(Value::as_f64).unwrap_or(0.0)), + journal: false, + }, + query!("trim.monitor", "Trim Monitor State", "{}", |s, _| Ok(crate::trim::monitor_info(s))), cmd!("trim.nudge", "Trim by Frames", [], None, r#"{"frames":i64}"#, has_edit_points, |s, p| crate::trim::nudge(s, p)), cmd!("trim.extendToPlayhead", "Extend Selected Edit to Playhead", [], Some("E"), "{}", has_edit_points, |s, _| { crate::trim::extend_to_playhead(s) }), cmd!("trim.ripplePrevious", "Ripple Trim Previous Edit to Playhead", [], Some("Q"), "{}", has_seq, |s, _| crate::trim::to_playhead( @@ -1575,6 +1603,9 @@ fn build() -> Vec { cmd!("effects.apply", "Apply Effect", [], None, r#"{"clips":[id]?,"effect":"gaussian_blur|Gaussian Blur|…"}"#, has_seq, |s, p| { let name = str_p(p, "effect").ok_or_else(|| bad("effects.apply", "need `effect`"))?; let def = filmcraft_project::effect::find_effect_by_name(name).ok_or_else(|| bad("effects.apply", format!("unknown effect `{name}`")))?; + if filmcraft_project::graphic::is_layer_id(def.id) { + return Err(bad("effects.apply", "graphic layers are added with graphics.newText / graphics.newShape")); + } if matches!(def.kind, filmcraft_project::EffectKind::VideoTransition | filmcraft_project::EffectKind::AudioTransition) { let mut q = p.clone(); q["effect"] = json!(def.id); @@ -1795,7 +1826,10 @@ fn build() -> Vec { Ok(Value::Array( command_specs() .iter() - .map(|c| json!({"id": c.id, "label": c.label, "menu": c.menu, "shortcut": c.shortcut, "params": c.params, "enabled": (c.enabled)(s).is_ok()})) + .map(|c| { + let all: Vec<&str> = s.shortcuts.for_command(c.id).iter().map(|b| b.keys.as_str()).collect(); + json!({"id": c.id, "label": c.label, "menu": c.menu, "shortcut": s.shortcuts.primary(c.id), "shortcuts": all, "defaultShortcut": c.shortcut, "params": c.params, "enabled": (c.enabled)(s).is_ok()}) + }) .collect(), )) }), @@ -1814,6 +1848,9 @@ fn build() -> Vec { )), ]; v.extend(crate::captions::commands()); + v.extend(crate::mixer::commands()); + v.extend(crate::graphics::commands()); + v.extend(crate::shortcuts::commands()); // Labels as individual commands (Edit ▸ Label ▸ ) for l in Label::ALL { let _ = l; @@ -1982,7 +2019,7 @@ fn mark(s: &mut Session, p: &Value, is_in: bool) -> Result { Ok(Value::Null) } -fn json_to_param(template: &ParamValue, v: &Value) -> Option { +pub(crate) fn json_to_param(template: &ParamValue, v: &Value) -> Option { Some(match template { ParamValue::Float(_) => ParamValue::Float(v.as_f64()?), ParamValue::Vec2(_) => { diff --git a/crates/engine/src/graphics.rs b/crates/engine/src/graphics.rs new file mode 100644 index 000000000..f71fb8da1 --- /dev/null +++ b/crates/engine/src/graphics.rs @@ -0,0 +1,551 @@ +//! Graphics commands (`graphics.*`, `fonts.list`): graphic clips with text and shape layers. +//! +//! A graphic clip's layers are the `graphic_text` / `graphic_shape` effect instances on the clip +//! (see `filmcraft_project::graphic`). Commands address a clip by `clip` (default: the first +//! selected graphic clip, else the topmost graphic clip under the playhead) and a layer by +//! `layer` = index among the clip's graphic layers, 0 = back (default: the selected layer, else +//! the frontmost). Property values are keyframe-aware: on an animated property they set the value +//! at the playhead. + +use filmcraft_geom::Vec2; +use filmcraft_project::graphic::{self, LayerContent, SHAPE_OPTS, eval_layer, layer_display_name, layer_indices, new_shape_layer, new_text_layer}; +use filmcraft_project::{ClipId, ItemId, ItemKind, Label, ParamValue, Sequence, TrackKind}; +use filmcraft_render::graphic_clip::{layer_local_bounds, layer_quad}; +use filmcraft_time::{Tick, TimeRange}; +use serde_json::{Value, json}; + +use crate::commands::{CommandSpec, always, bad, f64_p, has_seq, str_p, time_p, u64_p}; +use crate::{EngineError, Result, Session}; + +type Run = fn(&mut Session, &Value) -> Result; +type Enabled = fn(&Session) -> std::result::Result<(), String>; + +fn spec( + id: &'static str, + label: &'static str, + menu: &'static [&'static str], + shortcut: Option<&'static str>, + params: &'static str, + enabled: Enabled, + run: Run, +) -> CommandSpec { + CommandSpec { id, label, menu, shortcut, params, enabled, run, journal: true } +} +fn query(id: &'static str, label: &'static str, params: &'static str, run: Run) -> CommandSpec { + CommandSpec { id, label, menu: &[], shortcut: None, params, enabled: always, run, journal: false } +} + +/// Whether a track item is a graphic clip. +pub fn is_graphic(s: &Session, seq: &Sequence, c: ClipId) -> bool { + seq.find_item(c).and_then(|(_, it)| s.project.item(it.item)).is_some_and(|p| matches!(p.kind, ItemKind::Graphic { .. })) +} + +/// The graphic clip a command acts on: `clip`, else the first selected graphic clip, else the +/// topmost graphic clip under the playhead. +pub fn target_clip(s: &Session, p: &Value) -> Option { + let seq = s.active_sequence()?; + if let Some(c) = u64_p(p, "clip").map(ClipId) { + return is_graphic(s, seq, c).then_some(c); + } + if let Some(c) = s.state.selection.iter().copied().find(|c| is_graphic(s, seq, *c)) { + return Some(c); + } + let t = s.playhead(); + seq.video_tracks.iter().rev().filter_map(|tr| tr.item_at(t)).map(|it| it.id).find(|c| is_graphic(s, seq, *c)) +} + +fn has_graphic(s: &Session) -> std::result::Result<(), String> { + has_seq(s)?; + target_clip(s, &Value::Null).map(|_| ()).ok_or_else(|| "select a graphic clip".into()) +} + +/// Effect index of graphic layer `layer` (or the selected / frontmost layer) of `clip`. +fn layer_effect_index(s: &Session, clip: ClipId, p: &Value) -> Result<(usize, usize)> { + let seq = s.active_sequence().ok_or(EngineError::NoSequence)?; + let (_, it) = seq.find_item(clip).ok_or_else(|| bad("graphics", "no such clip"))?; + let idx = layer_indices(&it.effects); + if idx.is_empty() { + return Err(bad("graphics", "the graphic has no layers")); + } + let l = match p.get("layer") { + Some(Value::Number(n)) => n.as_u64().unwrap_or(0) as usize, + Some(Value::String(name)) => (0..idx.len()) + .find(|&i| layer_display_name(&it.effects[idx[i]], i).eq_ignore_ascii_case(name)) + .ok_or_else(|| bad("graphics", format!("no layer named `{name}`")))?, + _ => s.state.graphic_layers.first().copied().filter(|l| *l < idx.len()).unwrap_or(idx.len() - 1), + }; + let e = *idx.get(l).ok_or_else(|| bad("graphics", format!("no layer {l}")))?; + Ok((l, e)) +} + +fn vec2_p(p: &Value, k: &str) -> Option { + let a = p.get(k)?.as_array()?; + Some(Vec2::new(a.first()?.as_f64()?, a.get(1)?.as_f64()?)) +} + +/// Find or create the graphic source item for the active sequence's frame size. +fn graphic_source(p: &mut filmcraft_project::Project, w: u32, h: u32, rate: filmcraft_time::FrameRate) -> ItemId { + if let Some((id, _)) = + p.items.iter().find(|(_, i)| matches!(i.kind, ItemKind::Graphic { width, height, rate: r } if width == w && height == h && r == rate)) + { + return *id; + } + p.add_item("Graphic", Label::Rose, ItemKind::Graphic { width: w, height: h, rate }, None) +} + +/// Place a new graphic clip holding `layer` at the playhead: on the first video track above the +/// topmost clip at the playhead that is free for the duration (a track is added if needed). +fn new_graphic_clip(s: &mut Session, layer: filmcraft_project::EffectInstance, name: &str, p: &Value) -> Result { + let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?; + let t = time_p(s, p, "").unwrap_or_else(|| s.playhead()); + let seconds = f64_p(p, "seconds").unwrap_or(5.0).max(0.01); + let want_track = u64_p(p, "track").map(|v| v as usize); + let name = name.to_string(); + s.edit("New Graphic", |pr, st| { + let (w, h, rate) = { + let q = pr.sequence(seq_id).ok_or(EngineError::NoSequence)?; + (q.settings.width, q.settings.height, q.settings.frame_rate) + }; + let src = graphic_source(pr, w, h, rate); + let t = rate.snap(t); + let dur = rate.snap_nearest(Tick::from_seconds_f64(seconds)).max(rate.frame_duration()); + let mut ti = pr.make_track_item(src, TrackKind::Video, t, TimeRange::new(Tick::ZERO, dur), rate).ok_or_else(|| bad("graphics.newText", "bad item"))?; + ti.name = name; + ti.effects.push(layer); + for e in &mut ti.effects { + filmcraft_project::resolve_auto_points(e, (w, h), (w, h)); + } + let id = ti.id; + let track_id = filmcraft_project::TrackId(pr.alloc_id()); + let q = pr.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?; + let range = TimeRange::new(t, dur); + let free = |tr: &filmcraft_project::Track| !tr.items.iter().any(|i| i.range().overlaps(&range)); + let idx = match want_track { + Some(i) if i < q.video_tracks.len() && free(&q.video_tracks[i]) => i, + Some(i) if i < q.video_tracks.len() => return Err(bad("graphics.newText", format!("V{} is not free here", i + 1))), + _ => { + let top = q.video_tracks.iter().rposition(|tr| tr.item_at(t).is_some()).map_or(0, |k| k + 1); + match (top..q.video_tracks.len()).find(|&i| free(&q.video_tracks[i]) && !q.video_tracks[i].locked) { + Some(i) => i, + None => { + let n = q.video_tracks.len() + 1; + q.video_tracks.push(filmcraft_project::Track::new(track_id, TrackKind::Video, format!("Video {n}"))); + q.video_tracks.len() - 1 + } + } + } + }; + q.video_tracks[idx].items.push(ti); + q.video_tracks[idx].sort(); + q.check().map_err(EngineError::Other)?; + st.selection = vec![id]; + st.graphic_layers = vec![0]; + Ok(id) + }) +} + +/// Add a layer to an existing graphic clip; returns the new layer index. +fn add_layer(s: &mut Session, clip: ClipId, mut layer: filmcraft_project::EffectInstance) -> Result { + let frame = s.active_sequence().map(|q| (q.settings.width, q.settings.height)).unwrap_or((1920, 1080)); + filmcraft_project::resolve_auto_points(&mut layer, frame, frame); + s.edit_sequence("Add Graphic Layer", |q, _, st| { + let (_, it) = q.find_item_mut(clip).ok_or(filmcraft_edit::EditError::NoItem(clip))?; + // new layers go in front: after the last graphic layer + let pos = layer_indices(&it.effects).last().map_or(it.effects.len(), |l| l + 1); + it.effects.insert(pos, layer); + let n = layer_indices(&it.effects).len() - 1; + st.selection = vec![clip]; + st.graphic_layers = vec![n]; + Ok(n) + }) +} + +/// Map friendly property names to parameter ids. +fn param_id(k: &str) -> &str { + match k { + "fontSize" => "size", + "fontStyle" | "style" => "font_style", + "fillColor" | "color" => "fill_color", + "strokeColor" => "stroke_color", + "strokeWidth" => "stroke_width", + "backgroundColor" => "background_color", + "shadowColor" => "shadow_color", + "baselineShift" => "baseline_shift", + "fauxBold" => "faux_bold", + "fauxItalic" => "faux_italic", + "boxWidth" => "box_width", + "anchorPoint" => "anchor", + "cornerRadius" => "corner_radius", + other => other, + } +} + +/// JSON value for a property, accepting names for choices ("center", "small caps", "ellipse"…). +fn to_param(template: &ParamValue, id: &str, v: &Value) -> Option { + if let (ParamValue::Choice(_), Value::String(name)) = (template, v) { + let opts: &[&str] = match id { + "align" => graphic::ALIGN_OPTS, + "caps" => graphic::CAPS_OPTS, + "stroke_type" | "stroke2_type" => graphic::STROKE_OPTS, + "shape" => SHAPE_OPTS, + _ => &[], + }; + let n = name.to_ascii_lowercase().replace('_', " "); + let n = if n == "centre" { "center".to_string() } else { n }; + return opts.iter().position(|o| o.to_ascii_lowercase() == n).map(|i| ParamValue::Choice(i as u32)); + } + crate::commands::json_to_param(template, v) +} + +/// Set properties `props` on a layer (keyframe-aware at time `tl`). +fn set_props(s: &mut Session, clip: ClipId, eidx: usize, props: &serde_json::Map, tl: Tick, label: &str) -> Result<()> { + let props = props.clone(); + s.edit_sequence(label, |q, _, _| { + let (_, it) = q.find_item_mut(clip).ok_or(filmcraft_edit::EditError::NoItem(clip))?; + let mt = it.source_time_at(tl.clamp(it.start, it.end() - Tick(1))); + let e = it.effects.get_mut(eidx).ok_or_else(|| bad("graphics.set", "no such layer"))?; + for (k, v) in &props { + if k == "enabled" { + e.enabled = v.as_bool().unwrap_or(true); + continue; + } + let id = param_id(k); + let prm = e.params.get_mut(id).ok_or_else(|| bad("graphics.set", format!("no property `{k}`")))?; + let pv = to_param(&prm.value, id, v).ok_or_else(|| bad("graphics.set", format!("`{k}`: value has the wrong type")))?; + prm.set_at(mt, pv); + } + Ok(()) + }) +} + +/// Layers of a graphic clip with their evaluated bounds (sequence/canvas pixels). +fn list_layers(s: &Session, clip: ClipId) -> Result { + let seq = s.active_sequence().ok_or(EngineError::NoSequence)?; + let (_, it) = seq.find_item(clip).ok_or_else(|| bad("graphics.list", "no such clip"))?; + let size = match s.project.item(it.item).map(|p| &p.kind) { + Some(ItemKind::Graphic { width, height, .. }) => (*width, *height), + _ => (seq.settings.width, seq.settings.height), + }; + let ph = s.playhead(); + let mt = it.source_time_at(ph.clamp(it.start, it.end() - Tick(1))); + let layers: Vec = layer_indices(&it.effects) + .iter() + .enumerate() + .filter_map(|(i, &ei)| { + let e = &it.effects[ei]; + let sp = eval_layer(e, mt, size)?; + let q = layer_quad(&sp); + let (kind, text) = match &sp.content { + LayerContent::Text(t) => ("text", Some(t.text.clone())), + LayerContent::Shape(sh) => (SHAPE_OPTS.get(sh.shape as usize).copied().unwrap_or("Shape"), None), + }; + Some(json!({ + "layer": i, + "effectIndex": ei, + "name": layer_display_name(e, i), + "kind": kind, + "text": text, + "enabled": e.enabled, + "position": [sp.transform.position.x, sp.transform.position.y], + "localBounds": layer_local_bounds(&sp), + "quad": q.iter().map(|p| [p.x, p.y]).collect::>(), + })) + }) + .collect(); + Ok(json!({"clip": clip.0, "name": it.name, "start": it.start.0, "duration": it.duration.0, "canvas": [size.0, size.1], "layers": layers})) +} + +fn quad_bounds(q: &[Vec2; 4]) -> [f64; 4] { + let mut b = [f64::MAX, f64::MAX, f64::MIN, f64::MIN]; + for p in q { + b = [b[0].min(p.x), b[1].min(p.y), b[2].max(p.x), b[3].max(p.y)]; + } + b +} + +/// Layer bounds of `layers` (layer indices) of `clip` at the playhead: (effect index, position, bounds). +fn layer_boxes(s: &Session, clip: ClipId, layers: &[usize]) -> Result<(Vec<(usize, Vec2, [f64; 4])>, (u32, u32))> { + let seq = s.active_sequence().ok_or(EngineError::NoSequence)?; + let (_, it) = seq.find_item(clip).ok_or_else(|| bad("graphics.align", "no such clip"))?; + let size = match s.project.item(it.item).map(|p| &p.kind) { + Some(ItemKind::Graphic { width, height, .. }) => (*width, *height), + _ => (seq.settings.width, seq.settings.height), + }; + let mt = it.source_time_at(s.playhead().clamp(it.start, it.end() - Tick(1))); + let idx = layer_indices(&it.effects); + let mut out = Vec::new(); + for &l in layers { + let ei = *idx.get(l).ok_or_else(|| bad("graphics.align", format!("no layer {l}")))?; + let sp = eval_layer(&it.effects[ei], mt, size).ok_or_else(|| bad("graphics.align", "bad layer"))?; + out.push((ei, sp.transform.position, quad_bounds(&layer_quad(&sp)))); + } + Ok((out, size)) +} + +fn layers_p(s: &Session, clip: ClipId, p: &Value) -> Result> { + if let Some(a) = p.get("layers").and_then(Value::as_array) { + return Ok(a.iter().filter_map(Value::as_u64).map(|v| v as usize).collect()); + } + if !s.state.graphic_layers.is_empty() { + return Ok(s.state.graphic_layers.clone()); + } + let (l, _) = layer_effect_index(s, clip, p)?; + Ok(vec![l]) +} + +fn move_layers(s: &mut Session, clip: ClipId, moves: Vec<(usize, Vec2)>, label: &str) -> Result<()> { + let tl = s.playhead(); + s.edit_sequence(label, |q, _, _| { + let (_, it) = q.find_item_mut(clip).ok_or(filmcraft_edit::EditError::NoItem(clip))?; + let mt = it.source_time_at(tl.clamp(it.start, it.end() - Tick(1))); + for (ei, pos) in &moves { + if let Some(prm) = it.effects.get_mut(*ei).and_then(|e| e.params.get_mut("position")) { + prm.set_at(mt, ParamValue::Vec2(*pos)); + } + } + Ok(()) + }) +} + +pub fn commands() -> Vec { + vec![ + spec( + "graphics.newText", + "Text", + &["Graphics and Titles", "New Layer"], + Some("Cmd+T"), + r#"{"text":str="New Text","position":[x,y]?,"clip":id?,"newClip":bool?,"size":px=100,"font":str?,"fontStyle":str?,"seconds":f64=5,"track":index?,"time":ticks?}"#, + has_seq, + |s, p| { + let text = str_p(p, "text").unwrap_or("New Text").to_string(); + let (w, h) = s.active_sequence().map(|q| (q.settings.width, q.settings.height)).unwrap_or((1920, 1080)); + let pos = vec2_p(p, "position").unwrap_or(Vec2::new(w as f64 / 2.0, h as f64 / 2.0)); + let mut layer = new_text_layer(&text, pos, f64_p(p, "size").unwrap_or(100.0)); + for (k, id) in [("font", "font"), ("fontStyle", "font_style")] { + if let Some(v) = str_p(p, k) { + layer.params.insert(id.into(), filmcraft_project::Param::new(ParamValue::Text(v.into()))); + } + } + let into = if p.get("newClip").and_then(Value::as_bool) == Some(true) { None } else { u64_p(p, "clip").map(ClipId) }; + let into = into.filter(|c| s.active_sequence().is_some_and(|q| is_graphic(s, q, *c))); + let (clip, layer_i) = match into { + Some(c) => (c, add_layer(s, c, layer)?), + None => { + let name = text.lines().next().filter(|l| !l.trim().is_empty()).unwrap_or("Graphic").to_string(); + (new_graphic_clip(s, layer, &name, p)?, 0) + } + }; + Ok(json!({"clip": clip.0, "layer": layer_i})) + }, + ), + spec( + "graphics.newShape", + "Shape", + &["Graphics and Titles", "New Layer"], + None, + r#"{"shape":"rectangle|ellipse|polygon|path","position":[x,y]?,"size":[w,h]=[400,200],"points":[[x,y],…]?,"clip":id?,"seconds":f64=5}"#, + has_seq, + |s, p| { + let shape = str_p(p, "shape").unwrap_or("rectangle").to_ascii_lowercase(); + let k = SHAPE_OPTS + .iter() + .position(|o| o.to_ascii_lowercase() == shape) + .ok_or_else(|| bad("graphics.newShape", format!("unknown shape `{shape}`")))? as u32; + let (w, h) = s.active_sequence().map(|q| (q.settings.width, q.settings.height)).unwrap_or((1920, 1080)); + let pos = vec2_p(p, "position").unwrap_or(Vec2::new(w as f64 / 2.0, h as f64 / 2.0)); + let size = vec2_p(p, "size").unwrap_or(Vec2::new(400.0, 200.0)); + let points: Vec<[f32; 2]> = p + .get("points") + .and_then(Value::as_array) + .map(|a| a.iter().filter_map(|q| Some([q.get(0)?.as_f64()? as f32, q.get(1)?.as_f64()? as f32])).collect()) + .unwrap_or_default(); + if k == 3 && points.len() < 3 { + return Err(bad("graphics.newShape", "a path needs at least 3 points")); + } + let layer = new_shape_layer(k, pos, size, points); + let into = u64_p(p, "clip").map(ClipId).filter(|c| s.active_sequence().is_some_and(|q| is_graphic(s, q, *c))); + let (clip, layer_i) = match into { + Some(c) => (c, add_layer(s, c, layer)?), + None => (new_graphic_clip(s, layer, "Shape", p)?, 0), + }; + Ok(json!({"clip": clip.0, "layer": layer_i})) + }, + ), + spec( + "graphics.setText", + "Edit Text", + &[], + None, + r#"{"clip":id?,"layer":n|name?,"text":str,"merge":bool? (coalesce with the previous Edit Text undo step)}"#, + has_graphic, + |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.setText", "no graphic clip"))?; + let (_, ei) = layer_effect_index(s, clip, p)?; + let text = str_p(p, "text").ok_or_else(|| bad("graphics.setText", "need `text`"))?.to_string(); + let merge = p.get("merge").and_then(Value::as_bool).unwrap_or(false) && s.history.undo.last().is_some_and(|h| h.0 == "Edit Text"); + let mut props = serde_json::Map::new(); + props.insert("text".into(), Value::String(text)); + let ph = s.playhead(); + set_props(s, clip, ei, &props, ph, "Edit Text")?; + if merge && s.history.undo.len() >= 2 { + // keep the snapshot from before the typing session + s.history.undo.pop(); + } + Ok(Value::Null) + }, + ), + spec( + "graphics.set", + "Set Graphic Properties", + &[], + None, + r##"{"clip":id?,"layer":n|name?,"props":{"font":"Inter","font_style":"Bold","size":120,"align":"center","tracking":50,"leading":0,"fill_color":"#ffcc00","stroke":true,"stroke_width":6,"background":true,"shadow":true,"position":[x,y],"scale":100,"rotation":0,"opacity":100,…},"time":ticks?}"##, + has_graphic, + |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.set", "no graphic clip"))?; + let (_, ei) = layer_effect_index(s, clip, p)?; + let props = p.get("props").and_then(Value::as_object).ok_or_else(|| bad("graphics.set", "need `props`"))?.clone(); + let tl = time_p(s, p, "").unwrap_or_else(|| s.playhead()); + set_props(s, clip, ei, &props, tl, "Change Graphic Property")?; + Ok(Value::Null) + }, + ), + spec("graphics.selectLayer", "Select Graphic Layer", &[], None, r#"{"clip":id?,"layers":[n]}"#, has_graphic, |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.selectLayer", "no graphic clip"))?; + let layers: Vec = + p.get("layers").and_then(Value::as_array).map(|a| a.iter().filter_map(Value::as_u64).map(|v| v as usize).collect()).unwrap_or_default(); + s.state.selection = vec![clip]; + s.state.graphic_layers = layers; + Ok(Value::Null) + }), + spec("graphics.deleteLayer", "Delete Graphic Layer", &[], None, r#"{"clip":id?,"layer":n?}"#, has_graphic, |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.deleteLayer", "no graphic clip"))?; + let (_, ei) = layer_effect_index(s, clip, p)?; + s.edit_sequence("Delete Graphic Layer", |q, _, st| { + let (_, it) = q.find_item_mut(clip).ok_or(filmcraft_edit::EditError::NoItem(clip))?; + it.effects.remove(ei); + st.graphic_layers.clear(); + Ok(()) + })?; + Ok(Value::Null) + }), + spec( + "graphics.arrangeLayer", + "Arrange Graphic Layer", + &[], + None, + r#"{"clip":id?,"layer":n?,"to":"front|back|forward|backward"|index}"#, + has_graphic, + |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.arrangeLayer", "no graphic clip"))?; + let (l, _) = layer_effect_index(s, clip, p)?; + let to = p.get("to").cloned().unwrap_or(json!("front")); + let n = s.state.graphic_layers.len(); + let _ = n; + s.edit_sequence("Arrange Graphic Layer", |q, _, st| { + let (_, it) = q.find_item_mut(clip).ok_or(filmcraft_edit::EditError::NoItem(clip))?; + let idx = layer_indices(&it.effects); + let last = idx.len() - 1; + let dest = match &to { + Value::Number(v) => (v.as_u64().unwrap_or(0) as usize).min(last), + Value::String(w) => match w.as_str() { + "front" => last, + "back" => 0, + "forward" => (l + 1).min(last), + "backward" => l.saturating_sub(1), + o => return Err(bad("graphics.arrangeLayer", format!("unknown `to` `{o}`"))), + }, + _ => last, + }; + let e = it.effects.remove(idx[l]); + let mut idx2 = layer_indices(&it.effects); + let insert_at = if dest >= idx2.len() { idx2.pop().map_or(idx[0], |x| x + 1) } else { idx2[dest] }; + it.effects.insert(insert_at, e); + st.graphic_layers = vec![dest]; + Ok(()) + })?; + Ok(Value::Null) + }, + ), + spec( + "graphics.align", + "Align Layers", + &[], + None, + r#"{"clip":id?,"layers":[n]?,"align":"left|hcenter|right|top|vcenter|bottom","to":"frame|selection"="frame" (one layer always aligns to the frame)}"#, + has_graphic, + |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.align", "no graphic clip"))?; + let layers = layers_p(s, clip, p)?; + let (boxes, size) = layer_boxes(s, clip, &layers)?; + let how = str_p(p, "align").ok_or_else(|| bad("graphics.align", "need `align`"))?; + let to_sel = str_p(p, "to") == Some("selection") && boxes.len() > 1; + let r = if to_sel { + boxes.iter().fold([f64::MAX, f64::MAX, f64::MIN, f64::MIN], |a, b| [a[0].min(b.2[0]), a[1].min(b.2[1]), a[2].max(b.2[2]), a[3].max(b.2[3])]) + } else { + [0.0, 0.0, size.0 as f64, size.1 as f64] + }; + let mut moves = Vec::new(); + for (ei, pos, b) in boxes { + let (dx, dy) = match how { + "left" => (r[0] - b[0], 0.0), + "right" => (r[2] - b[2], 0.0), + "hcenter" | "center" => ((r[0] + r[2]) / 2.0 - (b[0] + b[2]) / 2.0, 0.0), + "top" => (0.0, r[1] - b[1]), + "bottom" => (0.0, r[3] - b[3]), + "vcenter" | "middle" => (0.0, (r[1] + r[3]) / 2.0 - (b[1] + b[3]) / 2.0), + o => return Err(bad("graphics.align", format!("unknown alignment `{o}`"))), + }; + moves.push((ei, Vec2::new(pos.x + dx, pos.y + dy))); + } + move_layers(s, clip, moves, "Align Layers")?; + Ok(Value::Null) + }, + ), + spec( + "graphics.distribute", + "Distribute Layers", + &[], + None, + r#"{"clip":id?,"layers":[n] (3 or more)?,"axis":"horizontal|vertical"}"#, + has_graphic, + |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.distribute", "no graphic clip"))?; + let layers = layers_p(s, clip, p)?; + let (mut boxes, _) = layer_boxes(s, clip, &layers)?; + if boxes.len() < 3 { + return Err(bad("graphics.distribute", "select three or more layers")); + } + let vertical = str_p(p, "axis") == Some("vertical"); + let c = |b: &[f64; 4]| if vertical { (b[1] + b[3]) / 2.0 } else { (b[0] + b[2]) / 2.0 }; + boxes.sort_by(|a, b| c(&a.2).total_cmp(&c(&b.2))); + let (first, last) = (c(&boxes[0].2), c(&boxes[boxes.len() - 1].2)); + let n = boxes.len() - 1; + let moves = boxes + .iter() + .enumerate() + .map(|(i, (ei, pos, b))| { + let d = first + (last - first) * i as f64 / n as f64 - c(b); + (*ei, if vertical { Vec2::new(pos.x, pos.y + d) } else { Vec2::new(pos.x + d, pos.y) }) + }) + .collect(); + move_layers(s, clip, moves, "Distribute Layers")?; + Ok(Value::Null) + }, + ), + query("graphics.list", "List Graphic Layers", r#"{"clip":id?}"#, |s, p| { + let clip = target_clip(s, p).ok_or_else(|| bad("graphics.list", "no graphic clip"))?; + list_layers(s, clip) + }), + query("fonts.list", "List Fonts", r#"{"system":bool=true (scan the system font folders)}"#, |_, p| { + if p.get("system").and_then(Value::as_bool).unwrap_or(true) { + filmcraft_text::fonts::scan_system(); + } + Ok(json!(filmcraft_text::families().into_iter().map(|(f, st)| json!({"family": f, "styles": st})).collect::>())) + }), + ] +} + +#[cfg(test)] +#[path = "graphics_tests.rs"] +mod tests; diff --git a/crates/engine/src/graphics_tests.rs b/crates/engine/src/graphics_tests.rs new file mode 100644 index 000000000..64f21bab8 --- /dev/null +++ b/crates/engine/src/graphics_tests.rs @@ -0,0 +1,130 @@ +use serde_json::json; + +use crate::Session; +use filmcraft_project::graphic::{LayerContent, eval_layer, layer_indices}; +use filmcraft_project::{ClipId, ItemKind}; +use filmcraft_time::Tick; + +fn demo() -> Session { + let mut s = Session::default(); + s.execute("file.openDemoProject", json!({})).unwrap(); + s +} + +fn layers(s: &Session, clip: ClipId) -> Vec { + let q = s.active_sequence().unwrap(); + let (_, it) = q.find_item(clip).unwrap(); + layer_indices(&it.effects).into_iter().filter_map(|i| eval_layer(&it.effects[i], Tick::ZERO, (q.settings.width, q.settings.height))).collect() +} + +fn text_of(l: &filmcraft_project::graphic::LayerSpec) -> String { + match &l.content { + LayerContent::Text(t) => t.text.clone(), + _ => panic!("not text"), + } +} + +#[test] +fn new_text_makes_a_graphic_clip_above_the_footage() { + let mut s = demo(); + let t = s.playhead(); + let r = s.execute("graphics.newText", json!({"text": "Hello", "position": [200, 300]})).unwrap(); + let clip = ClipId(r["clip"].as_u64().unwrap()); + let q = s.active_sequence().unwrap(); + let (tid, it) = q.find_item(clip).unwrap(); + let track_index = q.video_tracks.iter().position(|tr| tr.id == tid).unwrap(); + let below_busy = q.video_tracks[..track_index].iter().any(|tr| tr.item_at(t).is_some()); + assert!(below_busy || track_index == 0, "placed above the clip under the playhead"); + assert_eq!(it.name, "Hello"); + assert!(matches!(s.project.item(it.item).unwrap().kind, ItemKind::Graphic { .. })); + assert_eq!(it.duration, q.settings.frame_rate.snap_nearest(Tick::from_seconds_f64(5.0))); + assert!(!it.has_standard_effects(), "layers are not fx"); + assert_eq!(s.state.selection, vec![clip]); + let ls = layers(&s, clip); + assert_eq!(ls.len(), 1); + assert_eq!(text_of(&ls[0]), "Hello"); + assert_eq!((ls[0].transform.position.x, ls[0].transform.position.y), (200.0, 300.0)); + + // a second text layer in the same clip, then a shape + let r2 = s.execute("graphics.newText", json!({"text": "World", "clip": clip.0})).unwrap(); + assert_eq!(r2["layer"], 1); + s.execute("graphics.newShape", json!({"shape": "ellipse", "clip": clip.0, "size": [100, 50]})).unwrap(); + assert_eq!(layers(&s, clip).len(), 3); + let list = s.execute("graphics.list", json!({"clip": clip.0})).unwrap(); + assert_eq!(list["layers"][2]["kind"], "Ellipse"); + assert_eq!(list["layers"][1]["name"], "World"); + + // undo removes the shape layer again + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(layers(&s, clip).len(), 2); +} + +#[test] +fn typing_coalesces_into_one_undo_step_and_props_set() { + let mut s = demo(); + let r = s.execute("graphics.newText", json!({"text": ""})).unwrap(); + let clip = ClipId(r["clip"].as_u64().unwrap()); + let before = s.history.undo.len(); + for (i, t) in ["T", "Ti", "Tit", "Titl", "Title"].iter().enumerate() { + s.execute("graphics.setText", json!({"clip": clip.0, "layer": 0, "text": t, "merge": i > 0})).unwrap(); + } + assert_eq!(s.history.undo.len(), before + 1, "one undo step for the typing session"); + assert_eq!(text_of(&layers(&s, clip)[0]), "Title"); + s.execute("graphics.set", json!({"clip": clip.0, "props": {"font_style": "Bold", "fontSize": 140, "align": "center", "fill_color": "#ffcc00", "stroke": true, "caps": "small caps"}})).unwrap(); + let l = &layers(&s, clip)[0]; + let LayerContent::Text(t) = &l.content else { panic!() }; + assert_eq!((t.style.as_str(), t.size, t.align, t.caps), ("Bold", 140.0, 1, 2)); + assert_eq!(l.appearance.strokes.len(), 1); + assert!((l.appearance.fill.unwrap()[1] - 0.8).abs() < 0.01); + assert!(s.execute("graphics.set", json!({"clip": clip.0, "props": {"nope": 1}})).is_err()); + s.execute("edit.undo", json!({})).unwrap(); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(text_of(&layers(&s, clip)[0]), ""); +} + +#[test] +fn align_and_distribute_layers() { + let mut s = demo(); + let (w, h) = { + let q = s.active_sequence().unwrap(); + (q.settings.width as f64, q.settings.height as f64) + }; + let r = s.execute("graphics.newShape", json!({"shape": "rectangle", "position": [300, 200], "size": [100, 60]})).unwrap(); + let clip = r["clip"].as_u64().unwrap(); + s.execute("graphics.newShape", json!({"shape": "rectangle", "clip": clip, "position": [500, 400], "size": [40, 40]})).unwrap(); + s.execute("graphics.newShape", json!({"shape": "rectangle", "clip": clip, "position": [1500, 900], "size": [40, 40]})).unwrap(); + s.execute("graphics.align", json!({"clip": clip, "layers": [0], "align": "left"})).unwrap(); + s.execute("graphics.align", json!({"clip": clip, "layers": [0], "align": "bottom"})).unwrap(); + let l = layers(&s, ClipId(clip)); + assert!((l[0].transform.position.x - 50.0).abs() < 1e-6, "left edge on the frame edge"); + assert!((l[0].transform.position.y - (h - 30.0)).abs() < 1e-6); + s.execute("graphics.align", json!({"clip": clip, "layers": [1, 2], "align": "vcenter", "to": "selection"})).unwrap(); + let l = layers(&s, ClipId(clip)); + assert!((l[1].transform.position.y - l[2].transform.position.y).abs() < 1e-6); + s.execute("graphics.distribute", json!({"clip": clip, "layers": [0, 1, 2], "axis": "horizontal"})).unwrap(); + let l = layers(&s, ClipId(clip)); + let xs: Vec = l.iter().map(|x| x.transform.position.x).collect(); + assert!(((xs[1] - xs[0]) - (xs[2] - xs[1])).abs() < 1e-6, "{xs:?}"); + let _ = w; +} + +#[test] +fn graphic_renders_in_the_program_and_survives_save() { + let mut s = demo(); + let r = s.execute("graphics.newText", json!({"text": "BIG", "size": 300, "position": [100, 500]})).unwrap(); + let clip = r["clip"].as_u64().unwrap(); + s.execute("graphics.set", json!({"clip": clip, "props": {"fill_color": "#ff0000"}})).unwrap(); + let img = s.render_program(0.25).unwrap(); + let red = img.px.chunks(4).filter(|p| p[0] > 0.9 && p[1] < 0.05 && p[2] < 0.05).count(); + assert!(red > 200, "red text in the program: {red}"); + let dir = std::env::temp_dir().join(format!("fc-gfx-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("g.fcproj"); + s.execute("file.saveAs", json!({"path": path.to_string_lossy()})).unwrap(); + let mut s2 = Session::default(); + s2.execute("file.open", json!({"path": path.to_string_lossy()})).unwrap(); + assert_eq!(s2.project.items, s.project.items, "graphic clips round-trip"); + let _ = std::fs::remove_dir_all(&dir); + let fonts = s.execute("fonts.list", json!({"system": false})).unwrap(); + assert!(fonts.as_array().unwrap().iter().any(|f| f["family"] == "Inter")); +} diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index dfaba163b..e9a9f2788 100644 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -12,9 +12,13 @@ pub mod autosave; pub mod captions; pub mod commands; pub mod demo; +pub mod graphics; pub mod interchange; pub mod media_pool; +pub mod mixer; pub mod previews; +pub mod shortcut_presets; +pub mod shortcuts; pub mod trim; use std::sync::Arc; @@ -77,6 +81,9 @@ pub struct History { pub redo: Vec<(String, Arc)>, /// Labels of all applied states, oldest first (History panel). pub limit: usize, + /// Key of the last [`Session::edit_merged`] step: a continuous gesture (a fader drag) with the + /// same key folds into that one undo step. + pub merge_key: Option, } impl History { @@ -124,9 +131,15 @@ pub struct EditorState { /// Selected edit points (trim mode). #[serde(default)] pub edit_points: Vec, + /// Trim Monitor Out/In shift counters for the selected edit point. + #[serde(default)] + pub trim_shift: trim::TrimShift, /// Selected captions (caption tracks / Captions panel). #[serde(default)] pub caption_selection: Vec, + /// Selected layers (indices among the graphic layers, 0 = back) of the selected graphic clip. + #[serde(default)] + pub graphic_layers: Vec, } /// Events for frontends (drained each frame). @@ -162,6 +175,14 @@ pub struct Session { pub loaded_schema: u32, /// Render preview files + render-bar segments (shared with the frontend's frame workers). pub previews: Arc, + /// Audio Track Mixer automation pass in progress. + pub mixrec: mixer::Recorder, + /// Dynamic (J/K/L) trimming and trim-mode loop playback in progress. + pub trim_play: trim::TrimPlayback, + /// Keyboard shortcuts (active bindings, presets; `shortcuts.*` commands). + pub shortcuts: shortcuts::Shortcuts, + /// Nesting depth of [`Session::execute`] (commands that run other commands). + exec_depth: u32, } /// A background job with shared progress. @@ -221,6 +242,10 @@ impl Session { persistence: None, loaded_schema: filmcraft_format::SCHEMA_VERSION, previews: Arc::new(previews::PreviewStore::temp()), + mixrec: Default::default(), + trim_play: Default::default(), + shortcuts: shortcuts::Shortcuts::new(), + exec_depth: 0, } } @@ -230,6 +255,7 @@ impl Session { pub fn start_autosave(&mut self, cfg: autosave::AutosaveConfig) -> std::io::Result<()> { let prefs_path = cfg.data_dir.join("preferences.json"); self.prefs = autosave::Preferences::load(&prefs_path); + self.shortcuts.set_dir(&cfg.data_dir); self.prefs_path = Some(prefs_path); self.persistence = Some(autosave::Persistence::start(&cfg, self.prefs.auto_save.clone())?); self.sync_persistence(); @@ -311,9 +337,16 @@ impl Session { /// Run a command by id. pub fn execute(&mut self, id: &str, params: Value) -> Result { let spec = commands::find(id).ok_or_else(|| EngineError::UnknownCommand(id.to_string()))?; + if self.exec_depth == 0 && spec.journal && self.trim_play.active() { + self.settle_trim_playback(id); + } (spec.enabled)(self).map_err(|why| EngineError::Disabled(id.to_string(), why))?; + self.exec_depth += 1; let r = (spec.run)(self, ¶ms); + self.exec_depth -= 1; self.sync_persistence(); + // playback reads the newest snapshot (mixer moves, mutes… are heard while playing) + self.previews.live.publish_project(self.project.clone()); if r.is_ok() && spec.journal { self.journal.push((id.to_string(), params)); if self.journal.len() > 10_000 { @@ -323,6 +356,21 @@ impl Session { r } + /// Another command arrives while trim-mode playback runs: a dynamic trim is committed first + /// (so e.g. Undo undoes it as one step); loop playback stops unless it is a trim command. + fn settle_trim_playback(&mut self, id: &str) { + const LIVE: [&str; 5] = ["trim.shuttle", "trim.tick", "trim.shuttleStop", "trim.cancelDynamic", "trim.playAround"]; + if LIVE.contains(&id) { + return; + } + if self.trim_play.dynamic.is_some() { + trim::commit(self); + } + if !id.starts_with("trim.") { + self.trim_play.around = None; + } + } + pub fn is_enabled(&self, id: &str) -> bool { commands::find(id).is_some_and(|c| (c.enabled)(self).is_ok()) } @@ -338,6 +386,25 @@ impl Session { self.history.undo.remove(0); } self.history.redo.clear(); + self.history.merge_key = None; + self.state = st; + self.bump(); + Ok(r) + } + + /// Like [`Session::edit`], but consecutive calls with the same `key` (and no other edit in + /// between) share one undo step: dragging a control is one undoable change. + pub fn edit_merged(&mut self, label: &str, key: &str, f: impl FnOnce(&mut Project, &mut EditorState) -> Result) -> Result { + let merge = self.history.merge_key.as_deref() == Some(key) && self.history.undo.last().is_some_and(|u| u.0 == label); + if !merge { + let r = self.edit(label, f)?; + self.history.merge_key = Some(key.to_string()); + return Ok(r); + } + let mut p = (*self.project).clone(); + let mut st = self.state.clone(); + let r = f(&mut p, &mut st)?; + self.project = Arc::new(p); self.state = st; self.bump(); Ok(r) @@ -367,6 +434,7 @@ impl Session { pub fn undo(&mut self) -> Option { let (label, prev) = self.history.undo.pop()?; + self.history.merge_key = None; let cur = std::mem::replace(&mut self.project, prev); self.history.redo.push((label.clone(), cur)); self.fix_state(); @@ -376,6 +444,7 @@ impl Session { pub fn redo(&mut self) -> Option { let (label, next) = self.history.redo.pop()?; + self.history.merge_key = None; let cur = std::mem::replace(&mut self.project, next); self.history.undo.push((label.clone(), cur)); self.fix_state(); @@ -479,7 +548,7 @@ pub fn media_duration(p: &Project, _pool: &MediaPool, id: ItemId) -> Option Some(s.duration()), filmcraft_project::ItemKind::Subclip { range, .. } => Some(range.end()), - filmcraft_project::ItemKind::AdjustmentLayer { .. } => None, + filmcraft_project::ItemKind::AdjustmentLayer { .. } | filmcraft_project::ItemKind::Graphic { .. } => None, } } @@ -488,6 +557,12 @@ mod autosave_tests; #[cfg(test)] mod file_tests; #[cfg(test)] +mod mixer_tests; +#[cfg(test)] mod previews_tests; #[cfg(test)] +mod shortcuts_tests; +#[cfg(test)] mod tests; +#[cfg(test)] +mod trim_tests; diff --git a/crates/engine/src/mixer.rs b/crates/engine/src/mixer.rs new file mode 100644 index 000000000..5f3781efb --- /dev/null +++ b/crates/engine/src/mixer.rs @@ -0,0 +1,1191 @@ +//! Audio Track Mixer commands and the automation recorder. +//! +//! Strips are addressed by `strip` (or `track`): `"A1"` … (audio tracks), `"S1"` … (submixes), +//! `"Mix"` (the master), a track id, or a strip name. Lanes are `volume`, `pan`, `mute`, +//! `send..level` and `fx..` (see [`filmcraft_project::mixer`]). +//! +//! **Recording.** `mixer.recordStart` begins an automation pass at the playhead (the UI calls it +//! when playback starts). While it runs, `mixer.touch` / `mixer.release` are the fader gestures: +//! the audio follows the control immediately (through [`filmcraft_render::mixer::LiveMix`]) and the +//! gesture is recorded as a stream of (time, value) points. `mixer.recordStop` (playback stops) +//! thins every stream into keyframes and writes them as one undoable step, following the strip's +//! automation mode: +//! +//! | mode | records | after release | +//! |---|---|---| +//! | Off / Read | nothing (Read plays the automation) | the control returns to the automation | +//! | Latch | from the first touch | keeps writing the last value until playback stops | +//! | Touch | while touched | ramps back to the existing automation over the automatch time | +//! | Write | every control from the start of playback | keeps writing the last value until playback stops | +//! +//! With "Switch to Touch after Write" (on by default) a Write strip changes to Touch after the +//! pass. Without a recording pass, `mixer.touch` holds the control live and `mixer.release` commits +//! it once (a static value, or a keyframe at the playhead when the lane is automated). + +use serde_json::{Value, json}; + +use filmcraft_project::mixer::{ + self as pm, FADER_MAX_DB, FADER_MIN_DB, LANE_MUTE, LANE_PAN, LANE_VOLUME, MASTER_STRIP, lane_info, parse_fx_lane, parse_send_lane, thin_points, write_lane, +}; +use filmcraft_project::{AudioChannels, AutomationMode, InputMap, ItemId, Param, ParamValue, Sequence, Track, TrackId, TrackKind, TrackSend, find_effect}; +use filmcraft_time::{TICKS_PER_SECOND, Tick}; + +use crate::commands::{CommandSpec, always, bad, bool_p, f64_p, has_seq, str_p, time_p, u64_p}; +use crate::{EngineError, Result, Session}; + +type Run = fn(&mut Session, &Value) -> Result; +type Enabled = fn(&Session) -> std::result::Result<(), String>; + +// ------------------------------------------------------------------------------------- recorder + +/// A held value ends this long before the next move (one sample at 48 kHz). +const HOLD_EPS: Tick = Tick(TICKS_PER_SECOND / 48_000); + +/// One recorded control gesture. +#[derive(Clone, Debug)] +pub struct Gesture { + pub strip: TrackId, + pub lane: String, + pub mode: AutomationMode, + /// Recorded (time, value) points. + pub points: Vec<(Tick, f64)>, + pub touching: bool, + /// Touch: released at. + pub released: Option, + last: (Tick, f64), +} + +impl Gesture { + fn new(strip: TrackId, lane: &str, mode: AutomationMode, t: Tick, v: f64) -> Self { + Gesture { strip, lane: lane.to_string(), mode, points: vec![(t, v)], touching: true, released: None, last: (t, v) } + } + /// A new value at `t`. Repeats of the same value extend the hold instead of adding points, so a + /// fader held still records a flat line, not a ramp to the next move. + fn push(&mut self, t: Tick, v: f64) { + if t < self.last.0 { + return; + } + if v == self.last.1 { + self.last.0 = t; + return; + } + if self.points.last().is_some_and(|p| self.last.0 > p.0) { + self.points.push(self.last); + } + self.points.push((t, v)); + self.last = (t, v); + } + fn flush(&mut self) { + if self.points.last().is_some_and(|p| self.last.0 > p.0) { + self.points.push(self.last); + } + } + pub fn value(&self) -> f64 { + self.last.1 + } +} + +/// State of the current automation pass. +#[derive(Clone, Debug, Default)] +pub struct Recorder { + /// Sequence being recorded (None = no pass running). + pub seq: Option, + pub start: Tick, + pub gestures: Vec, +} + +impl Recorder { + pub fn active(&self) -> bool { + self.seq.is_some() + } + fn find(&mut self, strip: TrackId, lane: &str) -> Option<&mut Gesture> { + self.gestures.iter_mut().find(|g| g.strip == strip && g.lane == lane) + } +} + +/// Lanes a Write pass records on every strip. +fn write_lanes(strip: TrackId) -> &'static [&'static str] { + if strip == MASTER_STRIP { &[LANE_VOLUME] } else { &[LANE_VOLUME, LANE_PAN, LANE_MUTE] } +} + +/// The lane value as automation plays it (no live overrides). +fn underlying(tr: &Track, lane: &str, t: Tick) -> f64 { + let v = tr.lane_value(lane, t); + if lane_info(lane).is_some_and(|i| i.hold) { + // hold lanes: last keyframe at or before t + if let Some(p) = tr.lane(lane).filter(|p| p.is_animated() && tr.mixer.mode.reads()) { + let i = p.keyframes.partition_point(|k| k.time <= t); + return p.keyframes[i.saturating_sub(1)].value.as_f64().unwrap_or(v); + } + } + v +} + +fn clamp_lane(lane: &str, v: f64) -> f64 { + match lane_info(lane) { + Some(i) if i.hold => { + if v >= 0.5 { + 1.0 + } else { + 0.0 + } + } + Some(i) => v.clamp(i.min, i.max), + None => v, + } +} + +fn seq_of(s: &Session, id: ItemId) -> Result<&Sequence> { + s.project.sequence(id).ok_or(EngineError::NoSequence) +} + +/// Start an automation pass at `t` (playback start). +pub fn record_start(s: &mut Session, t: Tick) -> Result { + let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?; + let seq = seq_of(s, seq_id)?; + let mut gestures = Vec::new(); + for id in seq.strip_ids() { + let Some(tr) = seq.strip(id) else { continue }; + if tr.mixer.mode != AutomationMode::Write { + continue; + } + for lane in write_lanes(id) { + let v = underlying(&tr, lane, t); + let mut g = Gesture::new(id, lane, AutomationMode::Write, t, v); + g.touching = false; + gestures.push(g); + } + } + s.previews.live.clear_all(); + for g in &gestures { + s.previews.live.hold(g.strip, &g.lane, g.value()); + } + let n = gestures.len(); + s.mixrec = Recorder { seq: Some(seq_id), start: t, gestures }; + Ok(json!({"recording": true, "writing": n})) +} + +/// A control moved to `v` at `t`. +pub fn touch(s: &mut Session, strip: TrackId, lane: &str, v: f64, t: Tick) -> Result { + let v = clamp_lane(lane, v); + let seq_id = s.mixrec.seq.or(s.state.active_sequence).ok_or(EngineError::NoSequence)?; + let mode = seq_of(s, seq_id)?.strip(strip).ok_or_else(|| bad("mixer.touch", "no such strip"))?.mixer.mode; + s.previews.live.hold(strip, lane, v); + if !s.mixrec.active() || !mode.writes() { + return Ok(json!({"recording": false, "value": v})); + } + match s.mixrec.find(strip, lane) { + Some(g) => { + if !g.touching { + if g.mode == AutomationMode::Touch { + // a new touch after a release: the old ramp is overwritten from here + g.released = None; + } else if t > g.last.0 { + // Write / Latch held the last value until this touch + g.last.0 = (t - HOLD_EPS).max(g.last.0); + } + } + g.touching = true; + g.push(t, v); + } + None => s.mixrec.gestures.push(Gesture::new(strip, lane, mode, t, v)), + } + Ok(json!({"recording": true, "value": v})) +} + +/// A control let go at `t`. +pub fn release(s: &mut Session, strip: TrackId, lane: &str, t: Tick) -> Result { + let automatch = Tick::from_seconds_f64(s.prefs.audio.automatch_time); + if s.mixrec.active() { + if let Some(g) = s.mixrec.find(strip, lane) { + g.touching = false; + g.push(t, g.value()); + if g.mode == AutomationMode::Touch { + g.released = Some(t); + s.previews.live.release(strip, lane, t, automatch); + } + // Latch / Write keep holding the last value until playback stops. + return Ok(json!({"recording": true})); + } + s.previews.live.clear(strip, lane); + return Ok(json!({"recording": false})); + } + // no pass: commit the held value once + let held = s.previews.live.get(strip, lane); + s.previews.live.clear(strip, lane); + match held { + Some(o) => { + set_value(s, strip, lane, o.value, t)?; + Ok(json!({"recording": false, "value": o.value})) + } + None => Ok(json!({"recording": false})), + } +} + +/// End the pass at `t`: write every gesture as keyframes (one undo step). +pub fn record_stop(s: &mut Session, t_stop: Tick) -> Result { + let rec = std::mem::take(&mut s.mixrec); + let Some(seq_id) = rec.seq.filter(|_| !rec.gestures.is_empty()) else { + s.previews.live.clear_all(); + return Ok(json!({"recording": false, "lanes": 0})); + }; + let automatch = Tick::from_seconds_f64(s.prefs.audio.automatch_time); + let prefs = s.prefs.audio.clone(); + let rate = seq_of(s, seq_id)?.settings.sample_rate.max(1) as i64; + let eps = Tick::from_units(1, rate).max(Tick(1)); + let min_interval = if prefs.minimum_time_interval_thinning { Tick(prefs.minimum_time_ms as i64 * TICKS_PER_SECOND / 1000) } else { Tick::ZERO }; + let mut written = Vec::new(); + let mut keyframes = 0usize; + let result = s.edit("Write Automation", |p, _| { + let seq = p.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?; + for mut g in rec.gestures { + if g.points.is_empty() { + continue; + } + let Some(tr) = seq.strip(g.strip).map(|c| c.into_owned()) else { continue }; + let hold = lane_info(&g.lane).is_some_and(|i| i.hold); + let held = g.value(); + let end = match g.mode { + AutomationMode::Touch => g.released.unwrap_or(t_stop).min(t_stop.max(g.points[0].0)), + _ => t_stop, + }; + g.push(end.max(g.last.0), held); + g.flush(); + let mut pts = g.points.clone(); + if g.mode == AutomationMode::Touch && automatch.0 > 0 && !hold { + let back = end + automatch; + pts.push((back, underlying(&tr, &g.lane, back))); + } + let t0 = pts[0].0; + let t1 = pts.last().map(|p| p.0).unwrap_or(t0); + let tol = if hold { + 0.0 + } else if prefs.linear_keyframe_thinning { + if g.lane == LANE_PAN { 0.25 } else { 0.05 } + } else { + 0.0 + }; + let thinned = thin_points(&pts, tol, min_interval, hold); + let before = (t0.0 > 0).then(|| underlying(&tr, &g.lane, t0 - eps)); + let after = tr.lane(&g.lane).filter(|p| p.keyframes.iter().any(|k| k.time > t1)).map(|_| underlying(&tr, &g.lane, t1 + eps)); + keyframes += thinned.len(); + let lane = g.lane.clone(); + seq.with_strip_mut(g.strip, |tr| { + if let Some(param) = tr.lane_mut(&lane) { + write_lane(param, t0, t1, &thinned, before, after, eps, hold); + } + }); + written.push(json!({"strip": g.strip.0, "lane": lane, "from": t0.0, "to": t1.0, "keyframes": thinned.len(), "points": pts.len()})); + } + if prefs.switch_to_touch_after_write { + for id in seq.strip_ids() { + seq.with_strip_mut(id, |tr| { + if tr.mixer.mode == AutomationMode::Write { + tr.mixer.mode = AutomationMode::Touch; + } + }); + } + } + Ok(()) + }); + s.previews.live.clear_all(); + s.previews.live.publish_project(s.project.clone()); + result?; + Ok(json!({"recording": false, "lanes": written.len(), "keyframes": keyframes, "written": written})) +} + +// ------------------------------------------------------------------------------------- helpers + +fn active_seq(s: &Session) -> Result<&Sequence> { + s.active_sequence().ok_or(EngineError::NoSequence) +} + +/// Parse a strip reference: id, "A1", "S1", "Mix", or a strip name. +pub fn strip_ref(seq: &Sequence, v: &Value) -> Option { + if let Some(id) = v.as_u64() { + let id = TrackId(id); + return seq.strip(id).map(|_| id); + } + let name = v.as_str()?.trim(); + if name.eq_ignore_ascii_case("mix") || name.eq_ignore_ascii_case("master") { + return Some(MASTER_STRIP); + } + let idx = |pre: &str| { + name.strip_prefix(pre).or_else(|| name.strip_prefix(&pre.to_lowercase())).and_then(|n| n.parse::().ok()).and_then(|n| n.checked_sub(1)) + }; + if let Some(i) = idx("A") { + return seq.audio_tracks.get(i).map(|t| t.id); + } + if let Some(i) = idx("S") { + return seq.submix_tracks.get(i).map(|t| t.id); + } + seq.audio_tracks.iter().chain(&seq.submix_tracks).find(|t| t.name.eq_ignore_ascii_case(name)).map(|t| t.id) +} + +/// Display reference of a strip ("A1", "S2", "Mix"). +pub fn strip_label(seq: &Sequence, id: TrackId) -> String { + if id == MASTER_STRIP { + return "Mix".into(); + } + if let Some(i) = seq.audio_tracks.iter().position(|t| t.id == id) { + return format!("A{}", i + 1); + } + if let Some(i) = seq.submix_tracks.iter().position(|t| t.id == id) { + return format!("S{}", i + 1); + } + id.0.to_string() +} + +fn strip_p(s: &Session, p: &Value, cmd: &str) -> Result { + let seq = active_seq(s)?; + let v = p.get("strip").or_else(|| p.get("track")).ok_or_else(|| bad(cmd, "need `strip` (\"A1\", \"S1\", \"Mix\" or an id)"))?; + strip_ref(seq, v).ok_or_else(|| bad(cmd, format!("no strip {v}"))) +} + +fn lane_p(s: &Session, p: &Value, strip: TrackId, cmd: &str) -> Result { + let lane = str_p(p, "lane").unwrap_or(LANE_VOLUME).to_string(); + let seq = active_seq(s)?; + let tr = seq.strip(strip).ok_or_else(|| bad(cmd, "no such strip"))?; + let ok = match parse_fx_lane(&lane) { + Some((slot, param)) => tr.effects.get(slot).and_then(|e| e.param(param)).is_some_and(|p| p.value.as_f64().is_some()), + None => lane_info(&lane).is_some() && (parse_send_lane(&lane).is_none_or(|i| i < tr.mixer.sends.len())), + }; + if !ok { + return Err(bad(cmd, format!("no lane `{lane}` on this strip (volume, pan, mute, send..level, fx..)"))); + } + if strip == MASTER_STRIP && (lane == LANE_PAN || lane == LANE_MUTE || parse_send_lane(&lane).is_some()) { + return Err(bad(cmd, "the Mix track has volume and effect lanes only")); + } + Ok(lane) +} + +fn value_p(p: &Value, lane: &str, cmd: &str) -> Result { + let v = p.get("value").ok_or_else(|| bad(cmd, "need `value`"))?; + let x = v.as_f64().or_else(|| v.as_bool().map(|b| if b { 1.0 } else { 0.0 })).ok_or_else(|| bad(cmd, "`value` must be a number"))?; + Ok(clamp_lane(lane, x)) +} + +/// Set a control outside a recording pass: the static value, or a keyframe at `t` when the lane +/// is automated and the strip reads automation. +pub fn set_value(s: &mut Session, strip: TrackId, lane: &str, v: f64, t: Tick) -> Result<()> { + let lane = lane.to_string(); + let hold = lane_info(&lane).is_some_and(|i| i.hold); + s.edit_sequence("Track Mixer Adjust", move |q, _, _| { + q.with_strip_mut(strip, |tr| { + let animated = tr.lane(&lane).is_some_and(Param::is_animated) && tr.mixer.mode.reads(); + if animated { + if let Some(p) = tr.lane_mut(&lane) { + set_keyframe(p, t, v, hold); + } + } else { + tr.set_lane_static(&lane, v); + } + }) + .ok_or_else(|| EngineError::Other("no such strip".into())) + }) +} + +fn set_keyframe(p: &mut Param, t: Tick, v: f64, hold: bool) { + p.put_keyframe(t, ParamValue::Float(v)); + if hold { + for k in p.keyframes.iter_mut() { + k.interp = filmcraft_project::Interpolation::Hold; + } + } +} + +fn mode_p(p: &Value) -> Option { + str_p(p, "mode").and_then(AutomationMode::from_name) +} + +fn channels_name(c: AudioChannels) -> &'static str { + match c { + AudioChannels::Mono => "Mono", + AudioChannels::Stereo => "Stereo", + AudioChannels::Surround51 => "5.1", + AudioChannels::Adaptive => "Adaptive", + } +} + +fn channels_from(s: &str) -> Option { + match s.to_ascii_lowercase().as_str() { + "mono" => Some(AudioChannels::Mono), + "stereo" => Some(AudioChannels::Stereo), + "5.1" | "surround51" => Some(AudioChannels::Surround51), + "adaptive" => Some(AudioChannels::Adaptive), + _ => None, + } +} + +/// JSON view of one strip at time `t`. +pub fn strip_json(s: &Session, seq: &Sequence, id: TrackId, t: Tick) -> Value { + let Some(tr) = seq.strip(id) else { return Value::Null }; + let live = &s.previews.live; + let shown = |lane: &str| live.get(id, lane).map(|o| o.value).unwrap_or_else(|| underlying(&tr, lane, t)); + let lanes: serde_json::Map = tr + .automated_lanes() + .into_iter() + .map(|k| { + let kf: Vec = tr.lane_keyframes(&k).iter().map(|kf| json!([kf.time.0, kf.value.as_f64()])).collect(); + (k, Value::Array(kf)) + }) + .collect(); + let kind = if id == MASTER_STRIP { + "mix" + } else if seq.submix_tracks.iter().any(|t| t.id == id) { + "submix" + } else { + "audio" + }; + json!({ + "id": id.0, + "ref": strip_label(seq, id), + "name": tr.name, + "kind": kind, + "channels": channels_name(tr.channels), + "mode": tr.mixer.mode.label(), + "volumeDb": tr.volume_db, + "pan": tr.pan, + "muted": tr.muted, + "solo": tr.solo, + "recordArm": tr.mixer.record_arm, + "soloSafe": tr.mixer.solo_safe, + "inputMap": tr.mixer.input_map.label(), + "output": tr.mixer.output.map(|o| strip_label(seq, o)).unwrap_or_else(|| if id == MASTER_STRIP { String::new() } else { "Mix".into() }), + "inserts": tr.effects.iter().enumerate().map(|(i, e)| json!({"slot": i, "effect": e.effect, "name": e.def().map(|d| d.name).unwrap_or(""), "enabled": e.enabled, "postFader": e.post_fader})).collect::>(), + "sends": tr.mixer.sends.iter().enumerate().map(|(i, sd)| json!({"send": i, "target": strip_label(seq, sd.target), "levelDb": sd.level_db, "pan": sd.pan, "preFader": sd.pre_fader, "muted": sd.muted})).collect::>(), + "at": {"time": t.0, "volumeDb": shown(LANE_VOLUME), "pan": shown(LANE_PAN), "mute": shown(LANE_MUTE) >= 0.5}, + "lanes": lanes, + }) +} + +fn inspect(s: &mut Session, p: &Value) -> Result { + let t = time_p(s, p, "").unwrap_or(s.playhead()); + let seq = active_seq(s)?; + let strips: Vec = seq.strip_ids().into_iter().map(|id| strip_json(s, seq, id, t)).collect(); + Ok(json!({ + "strips": strips, + "recording": s.mixrec.active(), + "gestures": s.mixrec.gestures.iter().map(|g| json!({"strip": strip_label(seq, g.strip), "lane": g.lane, "mode": g.mode.label(), "touching": g.touching, "points": g.points.len(), "value": g.value()})).collect::>(), + "latencySamples": filmcraft_render::mixer::graph_latency(seq), + "automatchTime": s.prefs.audio.automatch_time, + })) +} + +fn set_strip(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.setStrip")?; + let seq = active_seq(s)?; + let output = match p.get("output") { + None => None, + Some(Value::Null) => Some(None), + Some(v) => { + let o = strip_ref(seq, v).ok_or_else(|| bad("mixer.setStrip", format!("no output {v}")))?; + if o == MASTER_STRIP { + Some(None) + } else if o == id || !seq.submix_tracks.iter().any(|t| t.id == o) { + return Err(bad("mixer.setStrip", "output must be the Mix or a submix")); + } else if seq + .submix_tracks + .iter() + .position(|t| t.id == id) + .is_some_and(|me| seq.submix_tracks.iter().position(|t| t.id == o).is_some_and(|it| it <= me)) + { + return Err(bad("mixer.setStrip", "a submix can only feed a submix after it (no feedback)")); + } else { + Some(Some(o)) + } + } + }; + let p = p.clone(); + s.edit_sequence("Track Mixer Settings", move |q, _, _| { + q.with_strip_mut(id, |t| { + if let Some(v) = str_p(&p, "name") { + t.name = v.to_string(); + } + if let Some(v) = f64_p(&p, "volumeDb") { + t.volume_db = v.clamp(FADER_MIN_DB, FADER_MAX_DB); + } + if let Some(v) = f64_p(&p, "pan") { + t.pan = v.clamp(-100.0, 100.0); + } + if let Some(v) = bool_p(&p, "muted") { + t.muted = v; + } + if let Some(v) = bool_p(&p, "solo") { + t.solo = v; + } + if let Some(v) = bool_p(&p, "recordArm") { + t.mixer.record_arm = v; + } + if let Some(v) = bool_p(&p, "soloSafe") { + t.mixer.solo_safe = v; + } + if let Some(m) = mode_p(&p) { + t.mixer.mode = m; + } + if let Some(m) = str_p(&p, "inputMap").and_then(InputMap::from_name) { + t.mixer.input_map = m; + } + if let Some(c) = str_p(&p, "channels").and_then(channels_from) { + t.channels = c; + } + if let Some(o) = output { + t.mixer.output = o; + } + }) + .ok_or_else(|| EngineError::Other("no such strip".into())) + })?; + let seq = active_seq(s)?; + Ok(strip_json(s, seq, id, s.playhead())) +} + +fn add_submix(s: &mut Session, p: &Value) -> Result { + let name = str_p(p, "name").map(str::to_string); + let channels = str_p(p, "channels").and_then(channels_from).unwrap_or(AudioChannels::Stereo); + let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?; + let id = s.edit("Add Submix Track", |pr, _| { + let id = TrackId(pr.alloc_id()); + let seq = pr.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?; + let n = seq.submix_tracks.len() + 1; + let mut t = Track::new(id, TrackKind::Audio, name.unwrap_or_else(|| format!("Submix {n}"))); + t.channels = channels; + seq.submix_tracks.push(t); + Ok(id) + })?; + let seq = active_seq(s)?; + Ok(json!({"id": id.0, "ref": strip_label(seq, id)})) +} + +fn delete_submix(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.deleteSubmix")?; + if !active_seq(s)?.submix_tracks.iter().any(|t| t.id == id) { + return Err(bad("mixer.deleteSubmix", "not a submix")); + } + s.edit_sequence("Delete Submix Track", move |q, _, _| { + q.submix_tracks.retain(|t| t.id != id); + // re-route everything that fed it to the Mix and drop sends to it + for sid in q.strip_ids() { + q.with_strip_mut(sid, |t| { + if t.mixer.output == Some(id) { + t.mixer.output = None; + } + while let Some(i) = t.mixer.sends.iter().position(|sd| sd.target == id) { + t.remove_send(i); + } + }); + } + Ok(()) + })?; + Ok(Value::Null) +} + +fn slot_p(p: &Value, cmd: &str) -> Result { + u64_p(p, "slot").map(|v| v as usize).ok_or_else(|| bad(cmd, "need `slot` (0-based)")) +} + +fn add_insert(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.addInsert")?; + let eid = str_p(p, "effect").ok_or_else(|| bad("mixer.addInsert", "need `effect` (an audio effect id)"))?; + let def = find_effect(eid) + .filter(|d| d.kind == filmcraft_project::EffectKind::Audio && !d.intrinsic) + .ok_or_else(|| bad("mixer.addInsert", format!("`{eid}` is not an audio effect")))?; + let mut inst = def.instance(); + inst.post_fader = bool_p(p, "postFader").unwrap_or(false); + let slot = u64_p(p, "slot").map(|v| v as usize); + let n = active_seq(s)?.strip(id).map(|t| t.effects.len()).unwrap_or(0); + if n >= pm::MAX_INSERTS { + return Err(bad("mixer.addInsert", format!("all {} effect slots are in use", pm::MAX_INSERTS))); + } + let at = slot.unwrap_or(n).min(n); + s.edit_sequence(&format!("Add Track Effect {}", def.name), move |q, _, _| { + q.with_strip_mut(id, |t| t.effects.insert(at, inst)).ok_or_else(|| EngineError::Other("no such strip".into())) + })?; + Ok(json!({"slot": at})) +} + +fn remove_insert(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.removeInsert")?; + let slot = slot_p(p, "mixer.removeInsert")?; + let ok = s.edit_sequence("Remove Track Effect", move |q, _, _| Ok(q.with_strip_mut(id, |t| t.remove_insert(slot)).unwrap_or(false)))?; + if !ok { + return Err(bad("mixer.removeInsert", "no effect in that slot")); + } + Ok(Value::Null) +} + +fn set_insert(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.setInsert")?; + let slot = slot_p(p, "mixer.setInsert")?; + let p = p.clone(); + s.edit_sequence("Track Effect Settings", move |q, _, _| { + q.with_strip_mut(id, |t| -> Result<()> { + let e = t.effects.get_mut(slot).ok_or_else(|| bad("mixer.setInsert", "no effect in that slot"))?; + if let Some(v) = bool_p(&p, "enabled") { + e.enabled = v; + } + if let Some(v) = bool_p(&p, "postFader") { + e.post_fader = v; + } + if let Some(obj) = p.get("params").and_then(Value::as_object) { + for (k, v) in obj { + let par = e.param_mut(k).ok_or_else(|| bad("mixer.setInsert", format!("no parameter `{k}`")))?; + par.value = match (&par.value, v) { + (ParamValue::Bool(_), Value::Bool(b)) => ParamValue::Bool(*b), + (ParamValue::Choice(_), v) => ParamValue::Choice(v.as_u64().unwrap_or(0) as u32), + (_, v) => ParamValue::Float(v.as_f64().ok_or_else(|| bad("mixer.setInsert", format!("`{k}` needs a number")))?), + }; + } + } + Ok(()) + }) + .ok_or_else(|| EngineError::Other("no such strip".into()))? + })?; + Ok(Value::Null) +} + +fn send_target(s: &Session, p: &Value, id: TrackId, cmd: &str) -> Result { + let seq = active_seq(s)?; + let v = p.get("target").ok_or_else(|| bad(cmd, "need `target` (a submix)"))?; + let t = strip_ref(seq, v).filter(|t| seq.submix_tracks.iter().any(|x| x.id == *t)).ok_or_else(|| bad(cmd, "`target` must be a submix"))?; + let pos = |x: TrackId| seq.submix_tracks.iter().position(|y| y.id == x); + if t == id || pos(id).is_some_and(|me| pos(t).is_some_and(|it| it <= me)) { + return Err(bad(cmd, "a submix can only send to a submix after it (no feedback)")); + } + Ok(t) +} + +fn add_send(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.addSend")?; + if id == MASTER_STRIP { + return Err(bad("mixer.addSend", "the Mix track has no sends")); + } + let target = send_target(s, p, id, "mixer.addSend")?; + let mut snd = TrackSend::new(target); + snd.level_db = f64_p(p, "levelDb").unwrap_or(0.0).clamp(FADER_MIN_DB, FADER_MAX_DB); + snd.pre_fader = bool_p(p, "preFader").unwrap_or(false); + snd.pan = f64_p(p, "pan").unwrap_or(0.0).clamp(-100.0, 100.0); + if active_seq(s)?.strip(id).is_some_and(|t| t.mixer.sends.len() >= pm::MAX_SENDS) { + return Err(bad("mixer.addSend", format!("all {} send slots are in use", pm::MAX_SENDS))); + } + let i = s.edit_sequence("Add Send", move |q, _, _| { + q.with_strip_mut(id, |t| { + t.mixer.sends.push(snd); + t.mixer.sends.len() - 1 + }) + .ok_or_else(|| EngineError::Other("no such strip".into())) + })?; + Ok(json!({"send": i})) +} + +fn set_send(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.setSend")?; + let i = u64_p(p, "send").ok_or_else(|| bad("mixer.setSend", "need `send` (0-based)"))? as usize; + let target = if p.get("target").is_some() { Some(send_target(s, p, id, "mixer.setSend")?) } else { None }; + let p = p.clone(); + s.edit_sequence("Send Settings", move |q, _, _| { + q.with_strip_mut(id, |t| -> Result<()> { + let sd = t.mixer.sends.get_mut(i).ok_or_else(|| bad("mixer.setSend", "no such send"))?; + if let Some(v) = f64_p(&p, "levelDb") { + sd.level_db = v.clamp(FADER_MIN_DB, FADER_MAX_DB); + } + if let Some(v) = f64_p(&p, "pan") { + sd.pan = v.clamp(-100.0, 100.0); + } + if let Some(v) = bool_p(&p, "preFader") { + sd.pre_fader = v; + } + if let Some(v) = bool_p(&p, "muted") { + sd.muted = v; + } + if let Some(t) = target { + sd.target = t; + } + Ok(()) + }) + .ok_or_else(|| EngineError::Other("no such strip".into()))? + })?; + Ok(Value::Null) +} + +fn remove_send(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.removeSend")?; + let i = u64_p(p, "send").ok_or_else(|| bad("mixer.removeSend", "need `send` (0-based)"))? as usize; + let ok = s.edit_sequence("Remove Send", move |q, _, _| Ok(q.with_strip_mut(id, |t| t.remove_send(i)).unwrap_or(false)))?; + if !ok { + return Err(bad("mixer.removeSend", "no such send")); + } + Ok(Value::Null) +} + +fn set_kf(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.setKeyframe")?; + let lane = lane_p(s, p, id, "mixer.setKeyframe")?; + let t = time_p(s, p, "").unwrap_or(s.playhead()); + let v = value_p(p, &lane, "mixer.setKeyframe")?; + let hold = lane_info(&lane).is_some_and(|i| i.hold); + s.edit_sequence("Add Track Keyframe", move |q, _, _| { + q.with_strip_mut(id, |tr| { + if let Some(par) = tr.lane_mut(&lane) { + if par.keyframes.is_empty() { + // the first keyframe keeps the current value everywhere else + let cur = par.value.as_f64().unwrap_or(v); + par.value = ParamValue::Float(cur); + } + set_keyframe(par, t, v, hold); + } + }) + .ok_or_else(|| EngineError::Other("no such strip".into())) + })?; + Ok(json!({"time": t.0, "value": v})) +} + +fn delete_kf(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.deleteKeyframe")?; + let lane = lane_p(s, p, id, "mixer.deleteKeyframe")?; + let t = time_p(s, p, "").ok_or_else(|| bad("mixer.deleteKeyframe", "need `time`"))?; + let ok = s.edit_sequence("Delete Track Keyframe", move |q, _, _| { + Ok(q.with_strip_mut(id, |tr| tr.lane_mut(&lane).is_some_and(|par| par.remove_keyframe_at(t))).unwrap_or(false)) + })?; + if !ok { + return Err(bad("mixer.deleteKeyframe", "no keyframe at that time")); + } + Ok(Value::Null) +} + +fn move_kf(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.moveKeyframe")?; + let lane = lane_p(s, p, id, "mixer.moveKeyframe")?; + let t = time_p(s, p, "").ok_or_else(|| bad("mixer.moveKeyframe", "need `time` (the keyframe)"))?; + let nt = time_p(s, p, "new"); + let nv = p.get("value").map(|_| value_p(p, &lane, "mixer.moveKeyframe")).transpose()?; + let ok = s.edit_sequence("Move Track Keyframe", move |q, _, _| { + Ok(q.with_strip_mut(id, |tr| { + let Some(par) = tr.lane_mut(&lane) else { return false }; + let Ok(i) = par.keyframes.binary_search_by_key(&t, |k| k.time) else { return false }; + let mut k = par.keyframes.remove(i); + if let Some(nt) = nt { + // keep keyframe order: clamp between the neighbours + let lo = i.checked_sub(1).map(|j| par.keyframes[j].time + Tick(1)).unwrap_or(Tick::ZERO); + let hi = par.keyframes.get(i).map(|n| n.time - Tick(1)).unwrap_or(Tick(i64::MAX)); + k.time = nt.clamp(lo, hi); + } + if let Some(v) = nv { + k.value = ParamValue::Float(v); + } + let j = par.keyframes.partition_point(|x| x.time < k.time); + par.keyframes.insert(j, k); + true + }) + .unwrap_or(false)) + })?; + if !ok { + return Err(bad("mixer.moveKeyframe", "no keyframe at that time")); + } + Ok(Value::Null) +} + +fn clear_lane(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.clearLane")?; + let lane = lane_p(s, p, id, "mixer.clearLane")?; + let t = s.playhead(); + s.edit_sequence("Clear Track Keyframes", move |q, _, _| { + q.with_strip_mut(id, |tr| { + let v = tr.lane_value(&lane, t); + if let Some(par) = tr.lane_mut(&lane) { + par.keyframes.clear(); + } + tr.set_lane_static(&lane, v); + if parse_fx_lane(&lane).is_none() { + tr.mixer.lanes.remove(&lane); + } + }) + .ok_or_else(|| EngineError::Other("no such strip".into())) + })?; + Ok(Value::Null) +} + +fn write_automation(s: &mut Session, p: &Value) -> Result { + let id = strip_p(s, p, "mixer.writeAutomation")?; + let lane = lane_p(s, p, id, "mixer.writeAutomation")?; + let pts: Vec<(Tick, f64)> = p + .get("points") + .and_then(Value::as_array) + .ok_or_else(|| bad("mixer.writeAutomation", "need `points`: [[ticks, value], …]"))? + .iter() + .filter_map(|v| Some((Tick(v.get(0)?.as_i64()?), clamp_lane(&lane, v.get(1)?.as_f64()?)))) + .collect(); + if pts.is_empty() { + return Err(bad("mixer.writeAutomation", "no points")); + } + let hold = lane_info(&lane).is_some_and(|i| i.hold); + let tol = f64_p(p, "tolerance").unwrap_or(if hold { 0.0 } else { 0.05 }); + let thinned = thin_points(&pts, tol, Tick::ZERO, hold); + let (t0, t1) = (thinned[0].0, thinned[thinned.len() - 1].0); + let rate = active_seq(s)?.settings.sample_rate.max(1) as i64; + let eps = Tick::from_units(1, rate).max(Tick(1)); + let n = thinned.len(); + s.edit_sequence("Write Automation", move |q, _, _| { + q.with_strip_mut(id, |tr| { + let before = (t0.0 > 0).then(|| underlying(tr, &lane, t0 - eps)); + let after = tr.lane(&lane).filter(|p| p.keyframes.iter().any(|k| k.time > t1)).map(|_| underlying(tr, &lane, t1 + eps)); + if let Some(par) = tr.lane_mut(&lane) { + write_lane(par, t0, t1, &thinned, before, after, eps, hold); + } + }) + .ok_or_else(|| EngineError::Other("no such strip".into())) + })?; + Ok(json!({"keyframes": n, "points": pts.len()})) +} + +/// Audio Clip Mixer: set the clip's Volume level / Panner balance (or any scalar clip effect +/// parameter) at the playhead. With `keyframe` (or when already animated) it writes a keyframe at the +/// playhead; a drag (same clip and parameter, no other edit between) is one undo step. +fn clip_set(s: &mut Session, p: &Value) -> Result { + let clip = filmcraft_project::ClipId(u64_p(p, "clip").ok_or_else(|| bad("clipMixer.set", "need `clip`"))?); + let effect = str_p(p, "effect").unwrap_or("volume").to_string(); + let param = str_p(p, "param").unwrap_or("level").to_string(); + let v = f64_p(p, "value").ok_or_else(|| bad("clipMixer.set", "need `value`"))?; + let kf = bool_p(p, "keyframe").unwrap_or(false); + let t = time_p(s, p, "").unwrap_or(s.playhead()); + let seq_id = s.state.active_sequence.ok_or(EngineError::NoSequence)?; + let key = format!("clipMixer:{}:{effect}:{param}", clip.0); + if bool_p(p, "begin").unwrap_or(false) { + s.history.merge_key = None; + } + s.edit_merged("Audio Clip Mixer", &key, |pr, _| { + let seq = pr.sequence_mut(seq_id).ok_or(EngineError::NoSequence)?; + let (_, it) = seq.find_item_mut(clip).ok_or_else(|| bad("clipMixer.set", "no such clip"))?; + let mt = it.source_time_at(t.clamp(it.start, it.end())); + let e = it.effect_mut(&effect).ok_or_else(|| bad("clipMixer.set", format!("the clip has no `{effect}` effect")))?; + let par = e.param_mut(¶m).ok_or_else(|| bad("clipMixer.set", format!("no parameter `{param}`")))?; + let v = if param == "level" { v.clamp(FADER_MIN_DB, FADER_MAX_DB) } else { v.clamp(-100.0, 100.0) }; + if kf || par.is_animated() { + par.put_keyframe(mt, ParamValue::Float(v)); + } else { + par.value = ParamValue::Float(v); + } + Ok(()) + })?; + Ok(Value::Null) +} + +// ------------------------------------------------------------------------------------- audio gain + +/// Peak (dBFS, before clip gain) of each selected audio clip's source range. +fn clip_peaks(s: &Session, clips: &[filmcraft_project::ClipId]) -> Vec<(filmcraft_project::ClipId, f64)> { + let Some(seq) = s.active_sequence() else { return Vec::new() }; + let sr = seq.settings.sample_rate.max(1); + let provider = s.media.provider(s.project.clone(), s.services.clone()); + let mut out = Vec::new(); + for t in &seq.audio_tracks { + for it in t.items.iter().filter(|i| clips.contains(&i.id)) { + let Some(src) = filmcraft_render::SourceProvider::source(&provider, it.item) else { continue }; + if !src.info().has_audio() { + continue; + } + let s0 = it.source_in.to_units_floor(sr as i64); + let len = ((it.duration.to_units_floor(sr as i64) as f64) * it.speed.abs().max(1e-6)).ceil() as i64; + let mut peak = 0f32; + let mut pos = s0; + while pos < s0 + len { + let n = (s0 + len - pos).min(sr as i64) as usize; + if let Ok(b) = src.audio(pos, n, sr) { + peak = b.peaks().into_iter().fold(peak, f32::max); + } + pos += n as i64; + } + out.push((it.id, 20.0 * (peak.max(1e-9) as f64).log10())); + } + } + out +} + +fn gain_targets(s: &Session, p: &Value) -> Vec { + let sel: Vec = match p.get("clips").and_then(Value::as_array) { + Some(a) => a.iter().filter_map(|v| v.as_u64().map(filmcraft_project::ClipId)).collect(), + None => s.state.selection.clone(), + }; + crate::commands::with_links(s, &sel) +} + +/// `clip.audioPeak`: the peak amplitude of the selection (with its current clip gain), as the Audio +/// Gain dialog shows it. +pub fn audio_peak(s: &mut Session, p: &Value) -> Result { + let clips = gain_targets(s, p); + let seq = active_seq(s)?; + let gain = |c: filmcraft_project::ClipId| seq.find_item(c).map(|(_, i)| i.gain_db).unwrap_or(0.0); + let peaks = clip_peaks(s, &clips); + let max = peaks.iter().map(|(c, pk)| pk + gain(*c)).fold(f64::NEG_INFINITY, f64::max); + Ok(json!({ + "peakDb": if max.is_finite() { json!(max) } else { Value::Null }, + "clips": peaks.iter().map(|(c, pk)| json!({"clip": c.0, "sourcePeakDb": pk, "gainDb": gain(*c)})).collect::>(), + })) +} + +/// `clip.audioGain`: Set Gain to / Adjust Gain by / Normalize Max Peak to / Normalize All Peaks to. +pub fn audio_gain(s: &mut Session, p: &Value) -> Result { + let db = f64_p(p, "db").unwrap_or(0.0); + let mode = match str_p(p, "mode") { + Some(m) => m.to_string(), + None => if bool_p(p, "relative").unwrap_or(true) { "adjust" } else { "set" }.to_string(), + }; + let clips = gain_targets(s, p); + let new_gain: Vec<(filmcraft_project::ClipId, f64)> = match mode.as_str() { + "set" => clips.iter().map(|c| (*c, db)).collect(), + "adjust" => { + let seq = active_seq(s)?; + clips.iter().filter_map(|c| seq.find_item(*c).map(|(_, i)| (*c, i.gain_db + db))).collect() + } + "normalizeMax" | "normalizeAll" => { + let peaks = clip_peaks(s, &clips); + if peaks.is_empty() { + return Err(bad("clip.audioGain", "no audio clips with audio in the selection")); + } + let seq = active_seq(s)?; + let gain = |c: filmcraft_project::ClipId| seq.find_item(c).map(|(_, i)| i.gain_db).unwrap_or(0.0); + if mode == "normalizeAll" { + peaks.iter().map(|(c, pk)| (*c, db - pk)).collect() + } else { + let loudest = peaks.iter().map(|(c, pk)| pk + gain(*c)).fold(f64::NEG_INFINITY, f64::max); + let delta = db - loudest; + peaks.iter().map(|(c, _)| (*c, gain(*c) + delta)).collect() + } + } + m => return Err(bad("clip.audioGain", format!("unknown mode `{m}` (set, adjust, normalizeMax, normalizeAll)"))), + }; + let n = new_gain.len(); + let applied = new_gain.clone(); + s.edit_sequence("Audio Gain", move |q, _, _| { + for t in q.audio_tracks.iter_mut() { + for i in t.items.iter_mut() { + if let Some((_, g)) = new_gain.iter().find(|(c, _)| *c == i.id) { + i.gain_db = g.clamp(-96.0, 96.0); + } + } + } + Ok(()) + })?; + Ok(json!({"mode": mode, "clips": n, "gainDb": applied.iter().map(|(c, g)| json!({"clip": c.0, "gainDb": g})).collect::>()})) +} + +/// Effects panel ▸ Set Selected as Default Transition (video or audio, from the effect's kind). +fn set_default_transition(s: &mut Session, p: &Value) -> Result { + let id = str_p(p, "effect").ok_or_else(|| bad("effects.setDefaultTransition", "need `effect`"))?; + let def = find_effect(id) + .or_else(|| filmcraft_project::effect_defs().iter().find(|d| d.name.eq_ignore_ascii_case(id))) + .ok_or_else(|| bad("effects.setDefaultTransition", format!("no effect `{id}`")))?; + match def.kind { + filmcraft_project::EffectKind::AudioTransition => s.state.default_audio_transition = def.id.to_string(), + filmcraft_project::EffectKind::VideoTransition => s.state.default_video_transition = def.id.to_string(), + _ => return Err(bad("effects.setDefaultTransition", format!("`{}` is not a transition", def.name))), + } + Ok(json!({"video": s.state.default_video_transition, "audio": s.state.default_audio_transition})) +} + +fn not_recording(s: &Session) -> std::result::Result<(), String> { + has_seq(s)?; + if s.mixrec.active() { Err("an automation pass is already recording".into()) } else { Ok(()) } +} + +fn spec( + id: &'static str, + label: &'static str, + menu: &'static [&'static str], + shortcut: Option<&'static str>, + params: &'static str, + enabled: Enabled, + run: Run, + journal: bool, +) -> CommandSpec { + CommandSpec { id, label, menu, shortcut, params, enabled, run, journal } +} + +pub fn commands() -> Vec { + vec![ + spec("mixer.inspect", "Inspect Audio Track Mixer", &[], None, r#"{"time":ticks?}"#, always, inspect, false), + spec( + "mixer.setStrip", + "Track Mixer Settings", + &[], + None, + r#"{"strip":"A1"|"S1"|"Mix"|id,"name":str?,"volumeDb":f64?,"pan":f64?,"muted":bool?,"solo":bool?,"recordArm":bool?,"soloSafe":bool?,"mode":"Off|Read|Latch|Touch|Write"?,"output":"Mix"|"S1"?,"inputMap":"Stereo|Left|Right|Swap|Mono"?,"channels":"Mono|Stereo|5.1"?}"#, + has_seq, + set_strip, + true, + ), + spec( + "mixer.setValue", + "Set Mixer Control", + &[], + None, + r#"{"strip":"A1"|"S1"|"Mix"|id,"lane":"volume|pan|mute|send..level|fx..","value":f64,"time":ticks?}"#, + has_seq, + |s, p| { + let id = strip_p(s, p, "mixer.setValue")?; + let lane = lane_p(s, p, id, "mixer.setValue")?; + let v = value_p(p, &lane, "mixer.setValue")?; + let t = time_p(s, p, "").unwrap_or(s.playhead()); + if s.mixrec.active() { + touch(s, id, &lane, v, t)?; + return release(s, id, &lane, t); + } + set_value(s, id, &lane, v, t)?; + Ok(json!({"value": v})) + }, + true, + ), + spec( + "mixer.touch", + "Touch Mixer Control", + &[], + None, + r#"{"strip":"A1"|"S1"|"Mix"|id,"lane":"volume|pan|mute|send..level|fx..","value":f64,"time":ticks?}"#, + has_seq, + |s, p| { + let id = strip_p(s, p, "mixer.touch")?; + let lane = lane_p(s, p, id, "mixer.touch")?; + let v = value_p(p, &lane, "mixer.touch")?; + let t = time_p(s, p, "").unwrap_or(s.playhead()); + touch(s, id, &lane, v, t) + }, + false, + ), + spec( + "mixer.release", + "Release Mixer Control", + &[], + None, + r#"{"strip":"A1"|"S1"|"Mix"|id,"lane":str?,"time":ticks?}"#, + has_seq, + |s, p| { + let id = strip_p(s, p, "mixer.release")?; + let lane = lane_p(s, p, id, "mixer.release")?; + let t = time_p(s, p, "").unwrap_or(s.playhead()); + release(s, id, &lane, t) + }, + true, + ), + spec( + "mixer.recordStart", + "Start Automation Pass", + &[], + None, + r#"{"time":ticks?}"#, + not_recording, + |s, p| { + let t = time_p(s, p, "").unwrap_or(s.playhead()); + record_start(s, t) + }, + false, + ), + spec( + "mixer.recordStop", + "Write Automation", + &[], + None, + r#"{"time":ticks?}"#, + has_seq, + |s, p| { + let t = time_p(s, p, "").unwrap_or(s.playhead()); + record_stop(s, t) + }, + true, + ), + spec( + "clipMixer.set", + "Audio Clip Mixer Adjust", + &[], + None, + r#"{"clip":id,"effect":"volume"|"panner","param":"level"|"balance","value":f64,"keyframe":bool?,"time":ticks?,"begin":bool?}"#, + has_seq, + clip_set, + true, + ), + spec( + "effects.setDefaultTransition", + "Set Selected as Default Transition", + &[], + None, + r#"{"effect":"constant_power|constant_gain|exponential_fade|