Extends the Windows backend (#141) to VP9 (profiles 0 and 2) and AV1 (main, 8/10-bit), 4:2:0.
- codecs::hw: FrameStreamInfo (vpcC / av1C): profile, bit depth, chroma, picture size and colour as
the software decoders report them (vp9_color / av1_color are now shared with them), random-access
samples, in-band format changes; StreamInfo lets HybridDecoder handle NAL and frame-coded streams.
- platform: media_foundation::stream (what each codec needs: DXVA profile, MFT request, input
format), MfDecoder generalised; VP9 frames / AV1 temporal units go in as they are; the MFT's
declared picture size (display aperture) is checked against the bitstream's.
- Declined up front: VP9 profiles 1/3, AV1 profiles 1/2, 4:2:2/4:4:4, mono, 12-bit, no codec
extension, no DXVA profile. A key frame of another size or a different sequence header falls back
to the software decoder.
- Tests: bit-exact parity (360p / 1080p / 2160p, seeks, resets, flush, hidden frames), forced
failure, damaged samples and av1C, format change fallback, WebM / Matroska through the media stack.
- Bench: AV1 fixtures with libaom when ffmpeg lacks SVT-AV1.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
make_sequence had ffmpeg write numbered stills straight into the shared seq_<ext> directory and
treated it as ready once frames 1 and 12 existed. Two tests generating it at once (png_sequence_exact
and missing_frames_hold_the_previous_frame) read each other's half-written PNGs; one that fails to
decode is held as missing, so frame 6 equalled frame 3. Six parallel processes on a fresh fixture
dir failed 18 of 120 runs before, 0 of 120 after.
ffmpeg now writes into a testkit temp_path directory that is renamed into place (with a .complete
marker) only when all twelve frames are there; the gap copy is private to its test.
Fixes#125
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- codecs: a resampled MP4 audio read scales the edit-list offset to the source rate in i128 and
saturates the window arithmetic; a hostile media_time overflowed i64.
- engine: media_duration follows a subclip's parent chain in a loop bounded at 16 hops instead of
recursing; a cyclic or very deep chain in a damaged project overflowed the stack.
Each has a regression test that failed before (overflow panic / stack overflow).
Fixes#66
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New backend in crates/platform (media_foundation/): a Direct3D-aware decoder MFT is driven at the
level of single access units (Annex B), given the process's D3D11 video device through an
IMFDXGIDeviceManager so it decodes with DXVA, and its NV12 / P010 textures are read back through a
staging texture into planar Yuv8 / Yuv16. H.264 Baseline/Main/High and HEVC Main / Main 10, 4:2:0.
- Hardware is verified: non-D3D-aware or async MFTs, streams the GPU's DXVA decoder does not list,
and system-memory output (Microsoft's MFT software fallback) are declined or fail the stream so
HybridDecoder continues with our decoder.
- mfplat.dll is loaded at run time so Windows N editions still start.
- register() / registered() / hardware_decoder_for() cover Windows; perf.stats reports the backend.
- Tests (Windows): bit-exact parity with the software decoders at 640x360, 1080p and 2160p
(resets, seeks, flush), forced mid-stream failure, damaged input, concurrency, repeated init.
- Bench: runs on Windows (ffmpeg on PATH, process/thread CPU clocks), HEVC Main 10 2160p row,
hardware counters per row.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
- aac: the decoder notes SBR data in fill elements (EXT_SBR_DATA /
EXT_SBR_DATA_CRC), so implicitly signalled HE-AAC is recognised;
Decoder::sbr() covers explicit signalling too.
- codecs: aac_output_rate() gives an AAC stream's output rate (the
extension rate, or twice a <= 24 kHz core when SBR data is present); the
MP4 reader reports it instead of the core rate, so sequences made from an
HE-AAC clip get 44.1 kHz instead of 22.05 kHz. Our decoder now also takes
explicitly signalled HE-AAC (it decodes the AAC-LC core) and, when the
stream rate is twice the core rate, interpolates the core to it with a
streaming half-band upsampler. The SBR high band is not reconstructed yet.
- export: the AAC bitrate is capped at 6 bits per sample and channel
(ISO/IEC 14496-3 frame limit) instead of failing with "bitrate out of
range" (320 kbps stereo at 22.05 kHz).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The probe guessed the pixel format from the codec profile alone: HEVC
reported nothing, H.264 High 4:2:2 had no bit depth, and ProRes 4444
claimed alpha because of its fourcc.
- HEVC: chroma format and bit depth from hvcC.
- H.264: from the avcC High-profile extension (ISO/IEC 14496-15 §5.3.3),
else from the first SPS; unknown (empty) if neither can be read.
- ProRes: the first frame header gives the chroma format and alpha
(alpha_channel_type), as the first DNx frame header already does.
Checked against ffprobe on ffmpeg-made HEVC 4:2:0/4:2:2 8/10-bit,
H.264 4:2:0/4:2:2 8/10-bit and ProRes 422 HQ / 4444 with and without
alpha. Unit tests cover hvcC, avcC (extension and SPS fallback) and
ProRes 4444 with and without alpha.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The clip duration came from the media duration (`mdhd`), which runs on
the decode timeline. ffmpeg / x264 MP4s with B-frames skip the reorder
delay with an edit, so every one imported a frame too long (a 120-frame
file became 121 frames, the last a held repeat). A video track delayed
by a leading empty edit was a frame short, and its last frame could not
be reached.
Use the sum of the track's edits (movie timescale, what `tkhd` records)
when there is an edit list. Fragmented files, and edit lists that add up
to nothing or overflow a Tick, keep the media duration.
Tests: a unit test with crafted edit lists (skip, empty edit, hostile
sizes) and an ffmpeg oracle test (libx264 with B-frames: 60 frames,
was 61).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem (#73): a 30 fps WebM played and exported with every third frame
missing and the frame before it shown twice (20 unique frames a second).
The same happens at 29.97, 60, 59.94, 24 and 23.976 fps, and with MP4
files whose video track has a 1 ms timescale. 25 and 50 fps were fine.
Cause: Matroska stores block timestamps in TimestampScale units (usually
1 ms), rounded to the nearest unit, so at 30 fps frame 2 (66.67 ms) is
stored as 67. MkvSource::video_frame converted the requested time to track
units with a floor (66), and the lookup (last sample with pts <= target)
returned frame 1. Every frame whose timestamp was rounded up was skipped.
Mp4Source::video_at, and the web build's WebCodecs MP4 path, floored the
same way against the track timescale; normal MP4 timescales hold frame
times exactly, so only coarse (1000) timescales were hit.
Fix: new Tick::to_rational_round (nearest unit, halves up, total for zero
and negative timebases), used for the video lookup target and the
catch-up threshold in mkv.rs, mp4.rs and webcodecs.rs. A requested time
now matches a frame whose stored timestamp is within half a unit of it,
the same precision the timestamps have. Truncating muxers (stamps rounded
down) still work. Times on exact timebases (90 kHz, 15360, 30000/1001…)
convert exactly as before.
Test: crates/codecs/src/rounded_pts_tests.rs builds synthetic Motion-JPEG
streams in the test (each picture shows its index as black/white blocks),
muxed as WebM with 1 ms timestamps and as MOV with a 1000 timescale, at
30, 29.97, 60, 59.94, 24, 23.976 and 25 fps with nearest and truncated
stamps. Requesting every frame's start over 3 s must show every frame
exactly once, and every frame's midpoint must show that frame. Before the
fix both tests failed ("30 fps, Nearest ms stamps: 30 of 90 frames never
shown [2, 5, 8, 11]…, 30 shown twice [1, 4, 7, 10]…"). The time crate
tests cover the rounding, ties and hostile timebases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trim_decoders dropped the decoders it took from idle caches while the requesting cache's lock was held. Dropping a decoder can block (a VideoToolbox session waits for its frames in flight), so the render path paid that latency under the lock. trim_decoders now returns the released decoders and frame_late keeps them in a local declared before its lock guard, so they drop after the guard on every return path; the cap is enforced exactly as before.
New tests/pool_bitexact.rs: primes the frame pool with 0xAA-filled plane buffers, then decodes an H.264 fixture into them; every frame must match ffmpeg bit for bit, and the pool must report the reuse.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: a clip with AAC audio in MP4 whose sample rate differs from the
sequence's (48 kHz clip in a 44.1 kHz sequence) exported with loud
low-frequency thumps, several times full scale, recurring through the clip.
Every exported container was affected (AAC clips them into thumps, PCM
clips at full scale); a WAV export had a single one at the clip's start.
Cause: Mp4Source::audio decodes a window starting at s0, the requested
presentation sample minus the edit-list offset, then resamples it. The
resampler derived its sub-sample phase from s0, so the edit-list offset
(-1024 samples of AAC priming) was applied a second time: the read
position started about 1024 samples before the window, was clamped to
index 0 with a large negative fraction, and linear interpolation
extrapolated a ramp over the first ~940 samples of every request. Export
asks for audio one batch of video frames at a time, so the ramp repeated
at each request. Sources at the sequence rate take the copy path and
were unaffected, as were tracks without an edit list.
Fix: take the phase from the presentation sample the window starts at
(before the edit-list offset is subtracted).
Test: aac_mp4_resampled_reads_follow_the_edit_list
(crates/codecs/tests/mp4_oracle.rs, with `require_ffmpeg!` and the
shared fixture helper as docs/testing.md asks of new oracle tests) reads
an ffmpeg AAC fixture at 44.1 kHz in frame-sized pieces and compares it
with the 48 kHz decode interpolated at the same times (peak was 7.35
against a source level of 0.15 before the fix). docs/testing.md lists it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With MEM1-MEM3 twelve clips on a timeline still took 5.1 GB, only 2.5 GB of
it live: each decoded picture allocated three planes, each eviction freed
them, and the system allocator kept the freed pages.
filmcraft_frame::pool keeps the planes of frames the GOP caches evict (up to
192 MiB per sample type; a frame something else still holds is left alone).
The H.264 decoder, through a plane allocator hook that keeps the crate free
of dependencies, and the VideoToolbox path take their output planes from it.
perf.stats reports decode.planePoolMB and decode.planesReused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The desktop app called filmcraft_platform::register() as an argument of
log::info!. A log macro does not evaluate its arguments while no logger takes
its level, and the app installs none, so VideoToolbox was never registered:
perf.stats showed hardware decoding enabled with 0 sessions, 0 declined and
every frame decoded in software. The CLI and the bench register directly and
were not affected.
Registration is now a statement of its own, with a start-up test that runs
without a logger.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each GOP cache keeps up to 384 MB of decoded frames (at least 64), and
nothing bounded the total: twelve clips on a timeline held 3.8 GB of frames
nobody was looking at, and no perf.stats counter showed it.
The pool now counts the frames of every cache against a 1 GiB budget. Over
it, caches idle for 3 s give up frames, least recently used cache first.
Caches in use keep their own budget, since a frame evicted before it is
shown costs a re-decode from the keyframe.
perf.stats reports decode.cacheMB and decode.cacheBudgetMB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every source kept its decoder for good once it had decoded a frame, with its
reference pictures and the pictures its frame threads hold in flight: about
290 MB per H.264 clip, 3.5 GB for twelve clips on a timeline.
The GOP caches now share a pool. Beyond four live decoders, the least
recently used caches that have been idle for 3 s drop theirs and make a new
one when asked again (a restart at the sync sample, as after a seek). Caches
in use keep their decoder whatever the count, so a composite of many layers
never restarts decoders per frame. Other caches are only try-locked, so two
caches trimming each other cannot deadlock.
perf.stats reports decode.liveDecoders.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New crates/platform (filmcraft-platform, L5): OS media FFI only. The one
crate allowed `unsafe` (own [lints] with unsafe_code = "deny",
#[allow(unsafe_code)] only on the videotoolbox module, SAFETY comment on
every block, safe Result-returning public API). AGENTS.md §0.3 names the
exception; docs/adr/0001-platform-ffi.md records why and the rules.
- VideoToolbox decoder for avcC / hvcC, 8/10-bit 4:2:0 and 4:2:2: session from
the sample entry's parameter sets with hardware required, asynchronous
decode (2 in flight), biplanar CVPixelBuffer copied to planar Yuv8/Yuv16,
reorder buffer of the stream's own depth, RASL skipped after a starting CRA.
Output callback under catch_unwind.
- HybridDecoder: transparent mid-stream fallback to the software decoder
(replays samples since the last restart point, drops already-returned
pictures). New codecs::software_video_decoder (built-in factories only) and
codecs::hw (switch, counters, NalStreamInfo shared with software decoders;
vui_color / sar_par helpers now used by both).
- Settings ▸ Playback ▸ Hardware decoding: Auto / Off (engine applies it to
the codecs switch); desktop app, CLI/MCP and bench call register().
- perf.stats decode.framesDecoded and decode.hardware counters.
- Tests: bit-exact parity with the software decoders (H.264 High, HEVC Main
open GOP, Main 10; resets, reseeks, flushes), forced-failure fallback,
mutation fuzz of samples and parameter sets, cross-platform fallback with a
stand-in decoder, setting Off, HEVC 4:2:2 vs ffmpeg.
- Bench --hw auto|off; results in docs/performance.md (4K CPU/frame
~86-119 ms -> ~4 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rust/clippy 1.99 added `chunks_exact_to_as_chunks` and tightened
`needless_late_init` and `drain_collect`, failing `clippy -D warnings`
on main.
- `chunks_exact(N)` / `chunks_exact_mut(N)` with a constant N become
`as_chunks::<N>()` / `as_chunks_mut::<N>()` (fixed-size arrays, so
several byte reads drop their per-element indexing). relink's
fingerprint uses the `(chunks, remainder)` split; its hash output is
unchanged.
- Late-initialised `let`s in ac3 bit allocation and opus CELT
bands/rate become `let x = if/match …`.
- `dpb.drain(..).collect()` becomes `std::mem::take(&mut dpb)`.
No behaviour change; `cargo xtask ci` passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mutation fuzzing (mutated ffmpeg-made files through `open_bytes`) found three import crashes:
- an MP4/MOV track with an empty sample-description list: `entries[0]` index out of bounds
(video and audio paths);
- an MP4/MOV track with timescale 0: division by zero in the duration;
- an MPEG-TS stream whose PTS span gave a ~0 fps rate: division by zero in `tick_of`.
Tracks now need samples, a sample description and a timescale to count as playable; frame rates
that round to zero fall back to the default; the TS/PS rate choice is a small function that never
returns a zero rate. Regression tests: `mp4::tests::damaged_track_headers_never_panic`,
`mpeg::tests::damaged_timestamps_never_give_a_zero_rate`.
Also: GOP cache, MKV/MP4/Ogg/MPEG audio paths, MXF and VP9 conversion without
`expect`/`unreachable!`; the crate denies the panicking lints outside tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AIFF is listed as an importable audio format and FilmCraft exports AIFF,
but the symphonia build had only the wav reader enabled, so every .aif /
.aiff import (including FilmCraft's own AIFF exports) failed with
"unsupported media: end of stream". Enable symphonia's aiff reader (the
same MPL-2.0 symphonia-format-riff crate already used for WAV; Cargo.lock
is unchanged). The new test decodes a synthetic 16-bit stereo AIFF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phones record portrait video as landscape pictures plus a rotation in the
container: the MP4/MOV tkhd matrix (ISO/IEC 14496-12 §8.3.2) or the
Matroska Projection pose roll (RFC 9559 §5.1.4.1.28.46). The matrix was
parsed but never used, so every portrait iPhone clip imported sideways
at its landscape size.
- isobmff: Track::display_rotation() reads 0/90/180/270° from the matrix
signs (mirrored / sheared matrices are left alone).
- matroska: parse Projection (type, yaw, pitch, roll);
VideoInfo::display_rotation() for rectangular projections.
- frame: VideoFrame::rotated() turns every plane at its own resolution
(4:2:2 chroma is widened to 4:4:4 for quarter turns) and swaps the PAR.
- codecs: Mp4Source / MkvSource report the displayed size and PAR, and
GopCache rotates decoded frames before caching them.
- web: the WebCodecs source configures the decoder with the coded size
and rotates its output the same way.
Tests: a one-frame ProRes MOV built in the test with 0/90/180/270°
matrices (reported size + which half ends up on top), plane rotation for
RGBA / 4:2:0 / 4:2:2, and a synthetic Matroska file with a -90° roll.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Lumetri grades PQ / HLG working spaces in their own signal normalised to
HDR White (Basic Correction) / HDR Range (Curves) in cd/m²
(filmcraft_color::GradeSpace): exposure in stops of light, highlights above
HDR White kept, HDR Specular scales them; Rec. 709 unchanged.
- HDR static metadata (mdcv/clli, Matroska MasteringMetadata/MaxCLL/MaxFALL)
on VideoStreamInfo::hdr; PQ media use MaxCLL / mastering peak as the
BT.2390 source peak; media.colorInfo reports it.
- HSL Secondary Refine: Denoise (median of the key) and Blur (Gaussian).
- Lumetri Presets (18 original looks in 4 folders) with thumbnails rendered by
our Lumetri on a procedural preview picture; Effects panel rows and a
thumbnail grid when wide; lumetri.presets / applyPreset / presetThumbnails.
- Color Match: Comparison View toggle; Apply Match uses its reference frame.
- Goldens lumetri_hdr_pq and lumetri_presets_grid; D-Log M stays unsupported
(no published formula).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sub-pixel filter passes run over fixed block widths; frame / region planes are recycled
through a bounded pool (zeroed on reuse) instead of fresh calloc / munmap per frame; the
palette mode-info arrays (40 bytes per 4x4) exist only when screen content tools are allowed;
CDF adaptation is branch-free. Draft mode skips deblocking, CDEF and loop restoration of shown
frames that refresh no reference slot. Media-stack test: VP9 / HEVC / AV1 draft playback only
reaches draft requests, and an export afterwards is bit-exact with ffmpeg.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Interpolation keeps its reference windows in reusable scratch (no 2 x 10 KB zeroing per block)
and filters with taps outer / outputs inner over fixed block widths; SAO edge and band offsets
select the offset per lane instead of a table lookup; uni / bi output clips with min / max; runs of
bypass bins decode with one division; context-coded bins are branch-free. Draft mode skips
deblocking and SAO of sub-layer non-reference pictures of the top sub-layer. Bit-exact with 1, 3
and all threads; draft output differs only on flagged pictures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frames are published one superblock row at a time once loop filtered; a post stage per frame
(own thread) assembles bands from the tile strips, filters them in raster order and builds the
output picture, while the next frame's tile columns decode and wait per band for the reference
rows they read. Bit-exact with 1, 3 and all threads. Draft mode skips the loop filter of frames
that refresh no reference slot. av1dec / hevcdec bench knobs and VP9 / AV1 4K playback scenarios
in the bench.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Settings > Playback > Draft decoding (off by default): the Program monitor
asks for draft frames only while playing at 1/2 resolution or lower
(FrameKey::draft, filmcraft_media::cancel::with_draft). The GOP cache runs
the decoder in draft mode for those requests and serves draft frames to draft
requests only, so paused frames, renders and exports always get exact
pictures (codecs test: exports after draft playback match ffmpeg). Draft
plans hand the GPU box-decimated YUV planes at the drawn size. perf.stats
reports draft frames, H.264 threads and the preference; the bench gains
2160p quarter and draft playback cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- HEVC inverse transform: sum basis rows of non-zero coefficients with a
const-generic size (bit-exact, tested against the direct product);
SAO edge offset gets a check-free interior loop. HEVC CPU/frame -60%.
- H.264/HEVC/VP9 planes are moved into VideoFrames, not copied.
- Catch-up decoding: frame jobs tell sources which frames are late
(filmcraft_media::cancel::with_catch_up); the GOP cache skips
non-reference samples of late frames (H.264 nal_ref_idc 0, HEVC
sub-layer non-reference pictures). Scrubs keep the last 2 s decoded.
- perf.stats: engine query + UI/control method (decode counters,
playback shown/dropped, frame-worker decode/render ms, cache use).
- Timeline waveform source peak cached; bench seek section and --only.
- docs/performance.md with before/after results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tiles decode in parallel into private region buffers (Plane / MiInfo with a
frame-coordinate origin) that are merged afterwards; intra block copy frames
keep whole-frame buffers. Post-filter row bands run on a rayon pool (par.rs,
behind the `threads` feature, sequential on wasm32). Frames become jobs that
wait on the reference slots they read and run on frame worker threads, so
independent GOP layers decode concurrently; output stays in order and each
Picture carries the pts of its temporal unit (Decoder::decode_pts, flush,
flush_result). codecs::Av1Decoder passes pts through and drains on flush.
Oracle tests now compare 1 thread, many threads and libdav1d (bit-exact),
with new multi-tile fixtures and an ignored 1080p set. README records fps
before/after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
