Pin GOP-cache eviction on the requested pts so a burst past the playhead
does not drop the target and walk the GOP again. Preroll only schedules
the frames that gate clock start. bench_playback no longer meters preroll
jobs as on-clock latency.
This is the seek stall from #848, not the ~3× per-core gap vs ffmpeg.
Problem: the menu parity figure in docs/target-app-parity.md came from an
ad hoc label-matching script over string literals; nothing repeatable
measured the real menu bar (docs/gaps.md G7).
Fix: `cargo xtask parity` dumps the shipped menu bar (new
`menus::default_menu_items`, deterministic: English, default prefs,
built-in workspaces; printed by `examples/menu_tree.rs`) and compares it
with a reference list of menu paths, by default the local, never-committed
`plan/premiere/menus.json` (any JSON of paths via --reference; parsing is
depth- and size-capped). Presence and placement are reported separately
(same path, elsewhere, missing); app/Help/recent/account/macOS items are
out of scope. Writes a markdown checklist (target/parity/ or --out) and
--min fails below a presence threshold. Without a reference it prints our
inventory only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- GpuRendering defaults to Off (opt-in per export) until measured on Windows and macOS;
docs and ROADMAP say so.
- The shared export device installs an uncaptured-error handler and a device-lost callback
that log once and retire GPU export for the session (frames go to the CPU) instead of
wgpu's default panic.
- RendererPool drops a renderer that panicked and retires the pool, so the rest of the
export renders on the CPU (regression test in gpu_hook_scenarios); the duplicate
hostile-size test that raced on the global registry is folded into it.
- Rebased onto main's render_plan/transitions: render_export_prepared reads the linear
accumulator after render_plan; transition (Composite) plans fall back to the CPU.
- es/ja rows for the new Export strings; web_time::Instant in the export pipeline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GPU export rendering was ~20 % slower in wall-clock time than CPU
rendering: one renderer behind a mutex made the export workers queue.
- Export workers share a pool of up to four renderers on one wgpu device
(one per worker was no faster: the read-back then contends for memory
bandwidth).
- render_export_prepared composites into the linear accumulator and
copies it to a reused staging buffer in one submission (no sRGB resolve
pass); read_accumulator shares the read-back, which now returns None
instead of stale pooled values when a row is missing.
- f16_to_f32_fast: same result as f16_to_f32 for all 65536 inputs (test).
- Timing counters (export::export_lock_wait, gpu::export_renderer::timing)
printed by the export bench.
RX 7900, same build, Off / Auto / Off: H.264 + AAC 4.67 / 4.71 / 4.76 s,
ProRes 422 4.20 / 4.21 / 4.21 s; CPU per frame 287-292 -> 277 (H.264),
316-317 -> 300 (ProRes). Was 4.86 -> 5.78 s. Parity, not faster: on this
single-clip scene decoding and encoding dominate (rendering is ~15 of
~290 CPU ms per frame), so GPU rendering stays Off by default.
Gates: fmt, clippy (gpu, export, ui-egui), layers, assets; gpu
export_pipeline, golden and export tests pass. Pre-existing on the
previous commit: fx_tests::gpu_effects_match_cpu_exactly fails on this
AMD GPU, and the gpu unit tests segfault in the driver when run in
parallel (pass serially apart from that test).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add off-screen GPU export renderer (filmcraft-gpu::ExportRenderer) reading back
the linear float accumulator as premultiplied RGBA f32, plug it into the
export pipeline via FrameRenderer hook and GpuRendering setting, register
in desktop app and CLI, and record export.gpu_frames statistics.
🤖 Completed by agy (Antigravity Agent, Gemini 3.8 Flash) based on draft by Merlin
Right after importing many 4K clips the Project panel and the timeline ask
every clip for a thumbnail. Up to six frame workers decoded them at once,
each with a frame-threaded software H.264 decoder (16 threads and 18
pictures in flight here, about 0.5 GB at 4K), and the decoders stayed: 42
4K clips with hardware decoding Off took the process past 6 GB during import.
Thumbnail jobs (item frames at the Media Browser / thumbnail priorities) now
run under filmcraft_media::cancel::with_background: H.264 decoders made for
them use 2 threads (thread_limited), parallelism comes from the workers
decoding different files, and the decoders of caches only thumbnails used
are released after each thumbnail (the clip served last keeps its own). A
monitor or playback request on such a cache replaces the small decoder with
a full one. Sequence frames are never background (the scopes ask for the
program frame at 40 while playing).
42 4K clips (Windows), peak private bytes: hardware Off 6.1-6.8 GB (killed)
-> 3.1-3.2 GB; hardware Auto 2.7-2.8 GB (#381) -> 2.3-2.4 GB. Thumbnails of
12 / 42 clips ready in 0.6 / 2.2 s instead of 2.2 s / not finishing.
Example crates/ui-egui/examples/thumb_burst.rs reproduces the burst.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/main into the NVENC branch. #183 already provides the
HardwareEncoding enum, ExportSettings.hardware_encoding, the
hardwareEncoding export param and the Export > Video > Hardware Encoding
checkbox, so the duplicates from this branch are dropped and NVENC uses
them. filmcraft-export is now an all-platform dependency of
filmcraft-platform (VideoToolbox and NVENC both register encoders).
Never-crash fixes in the NVENC backend:
- submit() rejects a driver input pitch narrower than a row (and unlocks
the buffer); fill_nv12 uses get_mut instead of slicing rows.
- composition offsets use saturating math and i32::try_from; frame-rate,
size and timestamp casts are checked; sizes above u16::MAX decline.
- Annex B scanning and NAL type checks use get()/first().
- A failed nvEncOpenEncodeSessionEx that returns a handle destroys it.
ffi.rs carries NVIDIA's MIT notice for nvEncodeAPI.h, listed in
ATTRIBUTION.md under a new "Third-party source code" section.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NVENC through nvEncodeAPI64.dll (API 12.1, FFI layout checked against the C header by generated
tests), registered as an Export encoder in front of the software one; Export > Hardware encoding
(off by default). Declines two-pass VBR, HDR, MXF, unsupported sizes; B-frames, IDR/GOP, VUI, avcC.
Tests: decode with our decoder (PSNR), pipeline export vs software, decline paths.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
Extends the Windows backend (#141) to VP9 (profiles 0 and 2) and AV1 (main, 8/10-bit), 4:2:0.
- codecs::hw: FrameStreamInfo (vpcC / av1C): profile, bit depth, chroma, picture size and colour as
the software decoders report them (vp9_color / av1_color are now shared with them), random-access
samples, in-band format changes; StreamInfo lets HybridDecoder handle NAL and frame-coded streams.
- platform: media_foundation::stream (what each codec needs: DXVA profile, MFT request, input
format), MfDecoder generalised; VP9 frames / AV1 temporal units go in as they are; the MFT's
declared picture size (display aperture) is checked against the bitstream's.
- Declined up front: VP9 profiles 1/3, AV1 profiles 1/2, 4:2:2/4:4:4, mono, 12-bit, no codec
extension, no DXVA profile. A key frame of another size or a different sequence header falls back
to the software decoder.
- Tests: bit-exact parity (360p / 1080p / 2160p, seeks, resets, flush, hidden frames), forced
failure, damaged samples and av1C, format change fallback, WebM / Matroska through the media stack.
- Bench: AV1 fixtures with libaom when ffmpeg lacks SVT-AV1.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
New backend in crates/platform (media_foundation/): a Direct3D-aware decoder MFT is driven at the
level of single access units (Annex B), given the process's D3D11 video device through an
IMFDXGIDeviceManager so it decodes with DXVA, and its NV12 / P010 textures are read back through a
staging texture into planar Yuv8 / Yuv16. H.264 Baseline/Main/High and HEVC Main / Main 10, 4:2:0.
- Hardware is verified: non-D3D-aware or async MFTs, streams the GPU's DXVA decoder does not list,
and system-memory output (Microsoft's MFT software fallback) are declined or fail the stream so
HybridDecoder continues with our decoder.
- mfplat.dll is loaded at run time so Windows N editions still start.
- register() / registered() / hardware_decoder_for() cover Windows; perf.stats reports the backend.
- Tests (Windows): bit-exact parity with the software decoders at 640x360, 1080p and 2160p
(resets, seeks, flush), forced mid-stream failure, damaged input, concurrency, repeated init.
- Bench: runs on Windows (ffmpeg on PATH, process/thread CPU clocks), HEVC Main 10 2160p row,
hardware counters per row.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
Problem: both branches change start_audio, so the nightly updater stops on the conflict.
Solution: keep play-ahead's structure; the rate converter lives in playback_mix (now FnMut).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: playback decoded and mixed every ~10 ms device buffer inside the
cpal callback (AAC decode, effects, opening a clip's file at a cut, rayon,
locks). On a loaded Mac any stall missed the device deadline: static and
dropouts, different on every play.
Solution: play_ahead mixes on its own thread into a lock-free ring kept
200 ms ahead; the callback only copies. An underrun plays silence, is
counted, and the mixer skips the same span so sound stays locked to the
picture. Any device width plays the channels it shares with the mix.
perf.stats reports playback.audio (callbacks, underruns, missingFrames,
minLeadMs). The program mix moved to playback_mix so bench_audio can play a
real project through a simulated 48 kHz device and count dropouts.
Context: on a real 22-sequence project, load average 38-58, 15 s per cut:
mixing in the callback overran 8-170 times, mixing ahead 0 (ring never
below 149 ms). The web build keeps mixing in its callback (no threads).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A media clip whose enabled standard effects all have a GPU implementation
plans as a GPU layer carrying them (render::plan::LayerFx): parameters are
evaluated on the CPU at the frame's time (render::gpufx::FxOp::eval, shared
with the CPU path so both clamp hostile values identically), the source is
drawn into an Rgba32Float working image at the size the CPU decodes it, the
effects run as WGSL compute passes (gpu::fx, fx.wgsl) with the CPU
reference's math, and the result is placed with Motion, opacity and blend
mode as before. Layers without effects keep the single draw.
GPU effects: Brightness & Contrast, ProcAmp, Tint, Black & White, Color
Balance, Leave Color, Change to Color, Color Pass, Color Replace, Channel
Mix, ASC CDL, Gamma Correction, Levels, Extract, Invert, Posterize, Alpha
Adjust, Gaussian / Camera / Directional Blur (+ legacy aliases), Sharpen,
Unsharp Mask, Crop, Edge Feather, Transform, H/V Flip, Mirror, Offset.
Masked effects, opacity masks, non-finite parameters and minifying
Transform/Crop zooms stay on the CPU. Devices without compute shaders run
the effects on the CPU inside the compositor.
Also fixes a debug-build overflow in Image::transformed for non-finite maps.
Parity tests (exact f32 checks and composited plans), a golden test that
plans effect chains as GPU layers, and the stack3-fx playback benchmark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New crates/platform (filmcraft-platform, L5): OS media FFI only. The one
crate allowed `unsafe` (own [lints] with unsafe_code = "deny",
#[allow(unsafe_code)] only on the videotoolbox module, SAFETY comment on
every block, safe Result-returning public API). AGENTS.md §0.3 names the
exception; docs/adr/0001-platform-ffi.md records why and the rules.
- VideoToolbox decoder for avcC / hvcC, 8/10-bit 4:2:0 and 4:2:2: session from
the sample entry's parameter sets with hardware required, asynchronous
decode (2 in flight), biplanar CVPixelBuffer copied to planar Yuv8/Yuv16,
reorder buffer of the stream's own depth, RASL skipped after a starting CRA.
Output callback under catch_unwind.
- HybridDecoder: transparent mid-stream fallback to the software decoder
(replays samples since the last restart point, drops already-returned
pictures). New codecs::software_video_decoder (built-in factories only) and
codecs::hw (switch, counters, NalStreamInfo shared with software decoders;
vui_color / sar_par helpers now used by both).
- Settings ▸ Playback ▸ Hardware decoding: Auto / Off (engine applies it to
the codecs switch); desktop app, CLI/MCP and bench call register().
- perf.stats decode.framesDecoded and decode.hardware counters.
- Tests: bit-exact parity with the software decoders (H.264 High, HEVC Main
open GOP, Main 10; resets, reseeks, flushes), forced-failure fallback,
mutation fuzz of samples and parameter sets, cross-platform fallback with a
stand-in decoder, setting Off, HEVC 4:2:2 vs ffmpeg.
- Bench --hw auto|off; results in docs/performance.md (4K CPU/frame
~86-119 ms -> ~4 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All 27 blend modes now composite on the GPU instead of sending the
whole frame to the CPU.
- render::plan: PlanLayer carries its Blend; plan_frame no longer falls
back to a CPU frame for non-Normal modes. Media, graphic clips,
multicam angles and CPU-rendered layer images (standard effects) keep
their clip's mode; inside a transition the mode is Normal, as in the
CPU reference. execute_cpu composites with each layer's mode.
- gpu: Normal keeps the single fixed-function pass. Dissolve also uses
fixed-function "over": the CPU's 64-bit pixel hash is reproduced in
WGSL on u32 pairs (pattern bit-exact). The other 25 modes copy the
accumulator region under the layer's quad into a backdrop texture and
composite in fs_blend with the CPU math (sRGB-encoded straight colour,
W3C formulas, back to linear, same premultiplied mix and clamping).
- Tests: per-mode GPU-vs-CPU parity (gpu crate: overlapping transformed
layers, partial opacity/alpha, empty/partial backdrop; golden: every
mode through plan_frame incl. an effect layer, blended clips in a
cross dissolve), exact-value edge cases (black/white, Divide by 0,
Dodge/Burn at 0 and 1, zero alpha) within 4.9e-4, Dissolve pattern
exact, off-output layers, quad bounds.
- Bench: new stack3-blend playback scenario (V2 Screen, V3 Overlay);
CPU ms/frame 151-214 -> 86-89, Full 19/173 and 138/54 -> 191/1, 192/0.
- Docs: architecture §5, gpu crate docs, testing, performance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frames are published one superblock row at a time once loop filtered; a post stage per frame
(own thread) assembles bands from the tile strips, filters them in raster order and builds the
output picture, while the next frame's tile columns decode and wait per band for the reference
rows they read. Bit-exact with 1, 3 and all threads. Draft mode skips the loop filter of frames
that refresh no reference slot. av1dec / hevcdec bench knobs and VP9 / AV1 4K playback scenarios
in the bench.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Settings > Playback > Draft decoding (off by default): the Program monitor
asks for draft frames only while playing at 1/2 resolution or lower
(FrameKey::draft, filmcraft_media::cancel::with_draft). The GOP cache runs
the decoder in draft mode for those requests and serves draft frames to draft
requests only, so paused frames, renders and exports always get exact
pictures (codecs test: exports after draft playback match ffmpeg). Draft
plans hand the GPU box-decimated YUV planes at the drawn size. perf.stats
reports draft frames, H.264 threads and the preference; the bench gains
2160p quarter and draft playback cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- HEVC inverse transform: sum basis rows of non-zero coefficients with a
const-generic size (bit-exact, tested against the direct product);
SAO edge offset gets a check-free interior loop. HEVC CPU/frame -60%.
- H.264/HEVC/VP9 planes are moved into VideoFrames, not copied.
- Catch-up decoding: frame jobs tell sources which frames are late
(filmcraft_media::cancel::with_catch_up); the GOP cache skips
non-reference samples of late frames (H.264 nal_ref_idc 0, HEVC
sub-layer non-reference pictures). Scrubs keep the last 2 s decoded.
- perf.stats: engine query + UI/control method (decode counters,
playback shown/dropped, frame-worker decode/render ms, cache use).
- Timeline waveform source peak cached; bench seek section and --only.
- docs/performance.md with before/after results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decode fps per codec (H.264, HEVC, VP9, AV1, ProRes at 1080p/2160p),
Program-monitor playback and scrubbing through the real frame scheduler,
timeline UI frame time with a 1000-clip / 20-track sequence under
egui_kittest (+ wgpu render), export throughput (H.264 + AAC, ProRes),
project save/open of 5000 clips, and peak RSS per section (each section
runs in its own process under /usr/bin/time). Writes JSON and Markdown
to target/bench. bench_playback now shares its code (examples/bench/).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Play waits for the first PREROLL_FRAMES (6) frames, at most 0.5 s, before starting the clock
and audio: the first frames after Play were due before any decoder could produce them.
- A new on-screen request for a view drops the queued and cancels the running request it
replaces (scrubbing used to decode every visited position oldest-first).
- playback_plan: when per-frame CPU work (effects; decode excluded, it is sequential anyway)
exceeds what the workers can do in real time, start only frames that can still be on time and
space them evenly, instead of every frame arriving late. Cost estimate updated from finished
and (as a lower bound) cancelled prefetch jobs. No change while the workers keep up.
- PlaybackMeter::resync: frames while the window is hidden (occluded/minimized) are not drops.
- Scopes render after the Program prefetch while playing (prio 40, was 2).
- ui.inspect playback.preroll; docs/architecture.md frame scheduling.
bench, same-session A/B, median of 3 (GPU path, Full, 8 s, load avg 22–64):
h264-1080 190/2 → 192/0 shown/dropped (3/3 runs 0 dropped)
stack3 189/3 → 192/0; h264-2160 173/19 → 192/0; demo 186/6 → 189/3
seek-storm: 10 → 15 of 40 seeks shown within 150 ms, 1581 → 1300 ms CPU per seek
after-preview live effects play (median of 2): 118/74 → 134/58, cancelled jobs 74 → 48;
plays from previews 192/0 both
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GpuCompositor::upload converted f32 RGBA layers (effects rendered on the CPU, captions,
whole-frame fallbacks) and 16-bit YUV frames (ProRes render previews) to half floats on the UI
thread every time a frame was shown: 8.3 M values per 1080p RGBA layer, 4 M per ProRes frame.
filmcraft_gpu::prepare(plan) now does those conversions on the frame worker that built the plan
(parallel chunks; 16-bit codes through a cached per-bit-depth table) and the plan cache stores
them with the plan (frames::GpuPlan), bounded by count and bytes (1 GiB). composite_prepared()
only copies bytes into textures. The texels are bit-identical (new test compares the read-back
output of both paths for f32, 10-bit 4:2:2 and fallback-image plans).
bench, same-session A/B (GPU path, after-preview Full, 2 rounds, load avg 50–95):
UI-thread present time p50/p95: plays from previews 11–31 / 34–77 ms → 3.6–5.8 / 4–14 ms;
live effects play 17–28 / 31–56 ms → 4.1 / 12 ms (first run)
plays from previews: 184/8, 183/9, 189/3, 169/23 → 192/0, 192/0, 191/1, 192/0 shown/dropped
live effects play (CPU-bound, ~6 cores at this load): too noisy to compare (0–117 shown either way)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- filmcraft_media::cancel: a per-job cancellation flag passed through a thread-local, so the
MediaSource API is unchanged. Frame workers run each job inside it.
- FrameServer::retain_queue (called by schedule_playback every refresh) now also cancels running
jobs for frames the playhead has passed. Before, a worker blocked on a source's decoder lock
went on to decode its frame after playback had moved 50–100 frames on, found it evicted and
restarted the decoder at the keyframe (decoder trace: "seek i=32 key=0 next=142").
- GopCache gives up (CodecError/MediaError::Cancelled) before seeking or decoding for a cancelled
job and between samples; decoder state stays consistent, so the next request carries on.
- The CPU compositor checks the flag between tracks and effects; cancelled results are never
cached.
- Stop drops queued prefetch jobs and cancels running ones (FrameServer::stop_prefetch). They
used to run to completion after Stop, e.g. 14 frames of live effects rendering competing with
a Render In to Out started right after.
bench, same-session A/B (GPU path, 8 s, load avg 35–41):
after-preview Full live play (1080p + Lumetri/Sharpen/Levels/Tint, CPU-bound):
38/154 → 98/94 shown/dropped, 297 → 267 ms CPU/frame, 84 jobs cancelled
after-preview plays from previews, h264-1080, stack3, h264-2160: unchanged within noise
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- crates/ui-egui/examples/bench_playback.rs drives the Program monitor's own FrameServer,
prefetch (now FrameServer::schedule_playback, shared with the monitor) and PlaybackMeter at a
simulated 60 Hz display, optionally compositing on a headless wgpu device. Scenarios:
h264-1080, stack3 (3 × 1080p, scaled/rotated/opacity), h264-2160, demo, after-preview
(live play → Render Effects In to Out → 2 plays), seek-storm. Reports latency percentiles,
on-time ratio, worker/decode wall and thread-CPU time, UI-thread present time, process CPU per
frame, decoder seeks/samples (codecs::gop_stats) and wasted jobs; --json for machine output.
Fixtures are generated with ffmpeg into target/fixtures/playback.
- FrameServer records per-job timings (thread CPU via rustix clocks) while profiling is on.
- Dropped/shown were counted per UI refresh: "shown" counted every refresh that had the frame
(322 "shown" in an 8 s play of 192 frames) and "dropped" counted a frame that arrived a few ms
after its first refresh. PlaybackMeter counts timeline frames: shown when the exact picture was
on screen while due, dropped otherwise, including frames passed over without a refresh.
Baseline (GPU path, 8 s, load avg 104–168 on 14 cores; CPU columns are load-robust):
h264-1080 Full 191/1 shown/dropped, 46 ms CPU/frame; Half 188/4
stack3 Full 189/3, 102 ms CPU/frame; Half 157/35 (3 extra decoder seeks)
h264-2160 Full 8/184, 313 ms CPU/frame, 10 seeks; demo Full 114/78, Half 191/1
after-preview: live 0/192; 1st play after render 124/67, 2nd 119/73 (UI-thread present
p50 21 ms / p95 45 ms); seek-storm: 0 of 40 seeks shown within 150 ms
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
