aleleba/filmcraft
1
0
Fork 0
mirror of https://github.com/storytold/filmcraft.git synced 2026-10-24 10:10:05 -06:00
Code Issues Packages Projects Releases Wiki Activity

Merge origin/main into #278; schema v14, neural-voices off by default, review fixes

Browse Source 
- Narrations take schema v14 (main's v13 is audio streams): v13_to_v14, docs updated.
- neural-voices is off by default in the desktop app, like whisper; the model download stays
  an explicit, confirmed job.
- Removed the debug probe2 test module from tts-text.
- project_subset prunes narrations of items it drops.
- unique_wav_path is bounded (99 999 names) and returns an error instead of looping.
- narration.rs uses web_time::Instant (wasm clock test); tts.* commands get the
  "Text to Speech" shortcut category, with es/ja rows for it and Edit Narration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Thomas
2026-10-10 06:55:59 -04:00
co-authored by Claude Opus 5.5
parent abd62eefa9 814287858c
commit 664c13efeb
352 changed files with 38493 additions and 5448 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+19 -9
View File
@@ -74,7 +74,7 @@ and `filmcraft-cli`.
| `engine` | L4 | `Session`, command registry, undo history, media pool, jobs, interchange glue |
| `ui-egui` | L5 | the egui frontend: docking, panels, timeline, monitors, playback, control-channel handlers |
| `automation` | L5 | MCP server (`rmcp`, stdio), headless or bridged to the running app |
| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; Media Foundation / Direct3D 11 H.264 / HEVC on Windows; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders, and hardware H.264 encoding (VideoToolbox, opt-in) and H.265 encoding (VideoToolbox, the only H.265 encoder; the format exists only where a hardware encoder does) and NVIDIA NVENC H.264 encoding (Windows, opt-in) behind `export::VideoEncoder`; H.264 declines to the built-in encoder for what the hardware does not take. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) |
| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; Media Foundation / Direct3D 11 H.264 / HEVC / VP9 / AV1 on Windows; VA-API H.264 / HEVC on Linux, with our own parsers and DPBs driving the stateless hardware; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders, and hardware H.264 encoding (VideoToolbox, opt-in) and H.265 encoding (VideoToolbox, the only H.265 encoder; the format exists only where a hardware encoder does) and NVIDIA NVENC H.264 (Windows and 64-bit Linux, opt-in) and H.265 (Windows, FilmCraft's only H.265 encoder there) encoding behind `export::VideoEncoder`; H.264 declines to the built-in encoder for what the hardware does not take. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) |
| `filmcraft` | L6 | desktop binary: eframe/wgpu window, cpal audio output, file dialogs, native macOS menu, TCP control server |
| `filmcraft-cli` | L6 | headless CLI: `exec`, `run`, `inspect`, `describe`, `commands`, `import`, `export`, `render`, `probe`, `mcp`; `--bridge` targets the running app |
| `filmcraft-web` | L6 | the browser app (wasm32): eframe web runner on WebGPU/WebGL2, Blob-backed services, OPFS recovery, WebAudio, WebCodecs, `window.filmcraft` API ([web.md](web.md)) |
@@ -265,17 +265,25 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
Decoders run slices on rayon, so an export worker waiting inside a decode can pick up another
frame of the same source. A request that finds the shared decoder busy decodes with a private
decoder.
- **Hardware encoding.** On Windows, `platform::nvenc::export::factory` is registered with
- **Hardware encoding.** On Windows and 64-bit Linux, `platform::nvenc::export::factory` is registered with
`filmcraft_export::register_encoder` (`register()` does this once), in front of the software H.264
encoder. It takes an export only when Export ▸ Hardware encoding (`ExportSettings.hardwareEncoding`,
`HardwareEncoding::Auto`) is Auto, which is off by default: hardware streams differ from ours, and
exports are otherwise byte-identical from run to run. NVENC (`platform::nvenc`, the driver's
`nvEncodeAPI64.dll` loaded at run time) then takes RGBA frames, converted to 4:2:0 by the software
`nvEncodeAPI64.dll` / `libnvidia-encode.so.1` loaded at run time) then takes RGBA frames, converted to 4:2:0 by the software
encoder's own conversion, and produces the H.264 samples and `avcC` of an MP4 / MOV. It declines
(the software encoder runs, counted in `export.hardware.declined`) two-pass VBR, HDR, MXF,
interlaced output, sizes outside NVENC's limits, and systems without an NVIDIA GPU or driver.
A failure during an export ends it with an error: the software encoder cannot take over a hardware
stream.
stream. The same NVENC session also encodes H.265 (HEVC Main 8-bit 4:2:0 SDR, or Main 10 HDR: PQ / HLG,
BT.2020, limited range, from the float R'G'B' pictures through `rgbf_to_yuv420_10` into P010 input
buffers, with the HDR10 static metadata as SEI on every IDR for PQ; the `hvcC` is built from the SPS
the encoder wrote). An HDR sequence exports as HDR H.265 only where a registered probe says so
(`filmcraft_export::register_hdr_probe`, `hdr_available`; NVENC with 10-bit support), otherwise it is
tone-mapped to SDR as on macOS: `register()` hands `platform::nvenc::hevc_available` to
`filmcraft_export::register_format_probe`, so the H.265 format exists where an HEVC encoder does,
and choosing it is the opt-in (the Hardware encoding toggle governs H.264 only). With no software
H.265 encoder, what NVENC declines is counted and ends the export with an error that says why.
- **Compositor.** `render` is the reference for monitors, thumbnails and export. `render::plan`
turns a frame into GPU layers, each with its opacity and blend mode. All 27 blend modes run on the
GPU (`filmcraft-gpu`): Normal and Dissolve with fixed-function "over" blending, the others by
@@ -308,9 +316,11 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
not zero (`VideoFrame::alpha_region`, `to_linear_f32_region`, `blend::composite_at`), the rest
being transparent anyway; and working images are recycled through `filmcraft_frame::pool`
(`take_f32_overwritten` / `recycle_f32`: the buffer comes back with its old contents, so there is
no zero-fill) by the export pipeline once a frame is converted to 8 bits. A clip with effects,
opacity masks, frame blending, colour management, or a picture that is not placed one to one on
the output takes the general path.
no zero-fill) by the export pipeline once a frame is converted to 8 bits; a standalone export's
pipeline frees the idle float images (`pool::trim_f32`) when it goes away, so a finished export
leaves none behind (parts of a batch, such as render-preview segments, keep them for the next part). A clip
with effects, opacity masks, frame blending, colour management, or a picture that is not placed
one to one on the output takes the general path.
- **Frame scheduling.** `crates/ui-egui/src/frames.rs` runs a small pool of worker threads with
prioritised jobs: the frame on screen first, then playback prefetch, then thumbnails. The UI never
decodes. It shows the exact frame when it is ready and holds the nearest cached frame meanwhile.
@@ -439,7 +449,7 @@ stereo / mono track ──5.1 panner───┼─► 5.1 submix / 5.1 Mix ─
### 5.1.2a Text to Speech (narrations)
**Narrations** (`engine::narration`, `tts.*`; voices in `filmcraft-tts`). The Text to Speech panel (Window ▸ Text to Speech) writes a script with pause markers (`[pause 1s]`), a voice, a vocal pitch and a pace; `tts.create` synthesizes it, writes `Narration <n>.wav` (mono 32-bit float, 24 kHz, where voice-over recordings go), imports it and places it at the playhead on the first free targeted audio track, else the first free audio track, else a new one, as one undo step. The settings are kept per generated item in `Project::narrations` (schema v13). Selecting a narration clip loads it into the panel; `tts.edit` writes a new file (beside the previous one) and swaps the clip to it, keeping the clip's start, duration and speed: longer speech is kept whole in the file but plays only to the clip's end; shorter speech is padded with silence to the clip's length. The old item and narration stay, so undo is exact. `tts.preview` synthesizes without changing the project and the UI plays `Session::tts_preview` on the audio output. Built-in voices are an original formant synthesizer (robotic, no download). The natural voices (feature `neural-voices`, on in the desktop app) are Kokoro-82M on candle (`filmcraft-tts`) fed by `filmcraft-tts-text` (normalizer, CMUdict + letter-to-sound); their 336 MB package is downloaded once after the user confirms (`tts.downloadVoices`, a job). The UI synthesizes with `tts.render` (a job filling `Session::tts_cache`) and then runs create / edit / preview, which use the cache.
**Narrations** (`engine::narration`, `tts.*`; voices in `filmcraft-tts`). The Text to Speech panel (Window ▸ Text to Speech) writes a script with pause markers (`[pause 1s]`), a voice, a vocal pitch and a pace; `tts.create` synthesizes it, writes `Narration <n>.wav` (mono 32-bit float, 24 kHz, where voice-over recordings go), imports it and places it at the playhead on the first free targeted audio track, else the first free audio track, else a new one, as one undo step. The settings are kept per generated item in `Project::narrations` (schema v14). Selecting a narration clip loads it into the panel; `tts.edit` writes a new file (beside the previous one) and swaps the clip to it, keeping the clip's start, duration and speed: longer speech is kept whole in the file but plays only to the clip's end; shorter speech is padded with silence to the clip's length. The old item and narration stay, so undo is exact. `tts.preview` synthesizes without changing the project and the UI plays `Session::tts_preview` on the audio output. Built-in voices are an original formant synthesizer (robotic, no download). The natural voices (feature `neural-voices` of the desktop app, off by default like `whisper`) are Kokoro-82M on candle (`filmcraft-tts`) fed by `filmcraft-tts-text` (normalizer, CMUdict + letter-to-sound); their 336 MB package is downloaded once after the user confirms (`tts.downloadVoices`, a job). The UI synthesizes with `tts.render` (a job filling `Session::tts_cache`) and then runs create / edit / preview, which use the cache.
### 5.1.3 Remix
@@ -456,7 +466,7 @@ Clip ▸ Remix ▸ Enable Remix / Remix Properties… / Revert Remix (`clip.remi
```text
clip.essential (type + settings) ──apply()──► clip effects marked `essential`, clip gain, Volume, Panner
essentialSound.autoMatch BS.1770 integrated loudness of render::audio::clip_signal → match gain (clip gain)
essentialSound.autoMatch BS.1770 integrated loudness of render::audio::clip_signal (mono clips as one channel) → match gain (clip gain)
essentialSound.generateDucking trigger clips' summed level (10 ms hops) → activity regions → Volume keyframes
```