Problem: MP4 and QuickTime name MPEG audio without its layer (ffmpeg
stores MP2 as `mp4a` + esds object type 0x6B), and we always made the
layer III decoder for it. symphonia's decoder reads only the layer it
was made for, so every MP2 frame was rejected and the track played as
silence; MOV's `.mp2` sample entry was not recognised at all.
Fix: the MPEG audio decoder for MP4 / MOV is made on the first packet,
for the layer its frame header names (ISO/IEC 11172-3 2.4.2.3);
`.mp2` maps to the MPEG audio codec like `.mp3`; the track is labelled
"MPEG Audio" for layers I/II, as Matroska's is. Decoded tracks now match
ffmpeg sample-exactly (edit list trims the 481-sample encoder delay).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: files with E-AC-3 audio (MP4 ec-3, MOV, Matroska A_EAC3, MPEG-TS)
imported silent: filmcraft-ac3 refused bsid 11-16.
Fix: clean-room E-AC-3 decoding from ATSC A/52:2012 Annex E in filmcraft-ac3:
bsi and audfrm syntax, 1/2/3/6-block syncframes, reduced sample rates,
frame-based exponent strategies (exponents reused across short syncframes),
SNR offset strategies, default band structures, spectral extension
(translation, notch filter, noise blending). Independent substream 0 is
decoded; dependent substreams and other programs are skipped (7.1 plays as
its 5.1 core). AHT and enhanced coupling are refused as unsupported. The
audio blocks share the AC-3 path; AC-3 output is unchanged. filmcraft-codecs
routes E-AC-3 in MP4/MOV, Matroska and TS to the decoder; the TS splitter
joins dependent substream frames to their access unit.
Tests: ffmpeg oracle for raw E-AC-3 (64.8-91.0 dB SNR) and in MP4, MOV,
Matroska and TS (79.5-91.3 dB, random access identical), mutation fuzz,
synthetic frames for headers, substream skipping, refusals and exponent
reuse across 1-block syncframes, spectral extension unit tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In an open GOP (iPhone HEVC; x265 `open-gop`) each CRA after the first is
followed in the file by RASL pictures shown before it, predicted from the
GOP before. GopCache started a decoding run at `sync_before(i)`, the sync
sample at or before the wanted sample in decoding order: for a RASL
picture that is the CRA after it, and the decoders (software and
VideoToolbox) rightly leave a run's leading RASL pictures out. The frame
was never produced, so the request failed ("frame not produced") and the
clip's layer was left out: a black frame (or the track below) wherever a
seek landed on one of the 3-in-24 RASL frames of iPhone video, such as a
clip trimmed to start on one, the parked playhead, and parallel export.
- `decode_start`: a sample shown before its sync sample starts the run at
the sync sample before (the MPEG and MXF readers already did this in
their own `sync_before`; this covers MP4/MOV and Matroska too).
- A running decoder that started after that point is not continued for
it: it left the picture out.
Tests: a synthetic open-GOP stream in gop.rs, and an x265 open-GOP oracle
test (MP4 and Matroska, every frame from a fresh source and each CRA
before its leading pictures) against ffmpeg. Both fail without the fix.
Problem: a file that opens but whose picture FilmCraft cannot decode
(H.264 High 10 / 4:2:2 / 4:4:4, interlaced, a codec without a decoder)
rendered black with no sign of why. The H.264 and HEVC decoders'
"unsupported" errors reached the media source as generic decode errors,
and the renderer drops frames that fail.
Fix: the H.264 / HEVC wrappers report unsupported features as
CodecError::Unsupported. The media pool wraps every file source it opens
so a frame failing as unsupported renders the existing MEDIA UNREADABLE
slate and the item is marked Unreadable with the decoder's reason (as
for files that fail to open). Decodable frames, damaged frames and the
sound are untouched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: MkvSource used the raw Segment timestamps as media time and the
absolute Info/Duration endpoint as the clip length. A remux that keeps its
source's first PTS (ffmpeg -copyts, 1.48 s in the report) repeated its
opening picture until then and made a 3.48 s clip out of 2 s of media.
Fix: after opening, every track moves back by the earliest start among the
playable tracks (an audio track's start before CodecDelay, never below
zero), so a delay between picture and sound survives. The duration is the
Segment end minus that origin, never less than the main track's samples.
Zero-origin files are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: our AAC decoder outputs channels in AAC element order (C, L, R,
Ls, Rs, LFE for channel configuration 6) and PacketDecoder passed that
through unchanged, while the mixer, panner, loudness and export all use
WAV/SMPTE order (L, R, C, LFE, Ls, Rs). A 5.1 AAC import therefore put
centre in L, L in R, R in C, the surrounds one slot early and LFE in Rs.
The export path already reorders for the encoder; the decode path did not.
Fix: PacketDecoder reorders AAC output for channel configurations 3-6
into WAV/SMPTE order, like the AC-3 decoder and Opus mapping family 1
already do. The Opus reorder now shares the same bounds-checked helper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: a video track with no edit list (x264 MP4 written without
elst, fragmented MP4) kept its raw first PTS, the B-frame composition
delay, as its origin. Media time zero repeated the first picture for
that delay, and since the clip lasts the media duration, the last
pictures never appeared in a sequence made from it.
Fix: when the video track has no edit list, move every track back by
the earliest start among the video and audio tracks, as ffmpeg does
with an input's start time. Video-only files now start at their first
picture; a file whose audio starts at zero keeps its A/V sync, and
explicit edit lists (empty edits, delays) are untouched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FrameStreamInfo::parameters_changed compared the in-band AV1 sequence header with
av1C's byte for byte. SVT-AV1 writes a provisional av1C header whose coding tool
flags (CDEF, restoration, warped motion...) differ from the in-band one, so every
such MP4 dropped to the software decoder from its first frame, on every hardware
backend. Headers now compare by what the picture format depends on: profile,
maximum size and colour configuration. A header that does not parse still counts
as a change. Regression test with an SVT-AV1 720p 10-bit av1C / in-band pair.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A GopCache decodes with its lock held, and that decoding starts rayon work
(rotating iPhone frames, cropping, slice-threaded decoders). On the global pool
a worker that took part of it could, waiting inside it, pick up another render
job asking the same cache for a frame and block on the lock, while the lock
holder waited for the work on that worker's stack: exports of rotated iPhone
HEVC clips stopped making frames in about half the runs.
The locked decoding now runs on a separate decode pool, whose threads never wait
for a cache's lock; a caller that is a rayon worker waits on a plain thread so it
does no other rayon work while holding the lock. The cancel/catch-up/draft/
background hints travel with it (filmcraft_media::cancel::Context). Intra-only
streams crop and rotate before taking the lock again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop-frame counting never emits the first base/15 frame labels of a minute
except each tenth (frames 00-01 at 29.97, 00-03 at 59.94). Accepting them let
a damaged 00:01:00;00 normalize to a different start time, so fields_to_frames
now rejects those labels and the start timecode is left unset.
Address review of #460:
- Read only the fixed-size rtmd header (18 bytes) instead of read_sample,
which allocated the whole untrusted stsz size (a corrupt multi-GiB size
could exhaust memory during import).
- Validate hours/minutes/seconds/frames against the frame base and drop the
start timecode when they are out of range, instead of normalizing damaged
metadata into a plausible but wrong offset.
Sony XAVC cameras such as the FX3 and a6400 keep the clip's start timecode
in an rtmd (real-time metadata) data track rather than a QuickTime tmcd
track, so probe reported start_timecode: null and timecode sync could not
use the footage. Read the hours/minutes/seconds/drop-frame/frames block from
the first rtmd sample and expose it as MediaInfo::start_timecode, falling
back only when no tmcd track is present (#460).
A WebM VP9 track with AlphaMode 1 stores its alpha as a second, independently coded VP9
stream in each block's BlockAdditional (BlockAddID 1). The Matroska reader detected
AlphaMode (has_alpha was true) but never read the additions, so transparent areas
rendered black.
- matroska: BlockGroups' BlockAdditions are parsed; the BlockAdditional with ID 1 of an
unlaced block is kept as Sample::addition, read with MkvFile::read_sample_addition.
- codecs: a WebM VP9 track with AlphaMode and additions gets an AlphaDecoder
(webm_alpha.rs): it decodes each sample's alpha frame with a second software VP9
decoder and attaches its luma plane (full range, converted to the picture's depth)
as the frame's alpha. After a seek the alpha decoder resynchronises from the nearest
alpha key frame. Colour pictures are held until their alpha picture is out, since the
two decoders may return pictures after different calls. A missing or undecodable alpha
frame leaves that picture opaque; it never fails the colour decode.
Tests: vp9_webm_alpha_layer_matches_libvpx (frames out of order and forward, colour and
alpha exact against ffmpeg's libvpx decode; fails before: no alpha plane; passed 200 of
200 runs with four in parallel), set_alpha unit tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qq8BK1LmohzE1ga4NgAdWM
The MP4 source reported the sample entry's size, only swapped for the display rotation, so two
crops were lost: the codec's own (H.264 SPS frame cropping, HEVC conformance window), which the
decoder applies but the reported size and sequences made from the clip didn't, and the clean
aperture (`clap`), parsed but never used. An iPhone clip (1920x1440 coded, 1744x1308 clean
aperture, -90 degree display matrix) showed 1440x1920 instead of 1308x1744.
H.264 / HEVC sizes now come from the stream's parameter sets; the clean aperture is a rectangle
inside that picture (ignored when it makes no sense), reported as the size and cut from every
frame by the GOP cache before the display rotation, through the new VideoFrame::cropped.
Tests: bit-exact against ffmpeg (which applies both crops) for an H.264 SPS crop, an HEVC
conformance window, centred and off-centre clean apertures and a clean aperture with a rotation;
clean apertures that make no sense are ignored without a panic; VideoFrame::cropped on every pixel
format with odd sizes and misaligned corners.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg writes MP4 without a `colr` box unless asked (`+write_colr`), and
color_from only read `colr` (or vpcC / apvC), so every such file fell back
to BT.709 limited: an HEVC PQ file was detected as rec709 / hdr false, a
full-range H.264 file as limited range, and BT.601 SD with BT.709
primaries. Decoded frames were right (the decoders read the VUI); the
media info, HDR detection and Interpret Footage defaults were not.
Matroska without a `Colour` element had the same gap.
New `stream_color` reads the colour a codec configuration carries in its
bitstream with the decoders' own parsers: the VUI of the SPS in avcC /
hvcC and the color_config of the AV1 sequence header in av1C. Precedence:
`colr` (nclx / nclc) or Matroska `Colour`, then vpcC / apvC / the
bitstream, field by field (what the container leaves unspecified falls
through), then the existing size defaults (BT.601 matrix for SD, BT.709
otherwise). nclc has no range flag, so the bitstream's range now stands;
an ICC `colr` no longer hides the bitstream's code points. A truncated or
corrupt parameter set gives no colour and the defaults.
Tests build avcC / hvcC / av1C bit by bit: HEVC PQ → rec2100-pq, full-range
H.264 → Full, BT.601 SD (NTSC and PAL) primaries, unspecified VUI keeps the
defaults, colr wins, Matroska without Colour, and every truncation and
single-bit flip of each configuration under catch_unwind. An ffmpeg-oracle
test checks real x264 / x265 files against their `+write_colr` remux.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps #359's contiguous_starts audio timing, now per stream in Mp4Audio /
MkvAudio (starts and preroll in source sample frames). MP4 audio streams and
their decoders are built from the same (track, info) pairs so they stay
index-aligned. Adapts new main code to audio_streams / TrackItem::audio_stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Right after importing many 4K clips the Project panel and the timeline ask
every clip for a thumbnail. Up to six frame workers decoded them at once,
each with a frame-threaded software H.264 decoder (16 threads and 18
pictures in flight here, about 0.5 GB at 4K), and the decoders stayed: 42
4K clips with hardware decoding Off took the process past 6 GB during import.
Thumbnail jobs (item frames at the Media Browser / thumbnail priorities) now
run under filmcraft_media::cancel::with_background: H.264 decoders made for
them use 2 threads (thread_limited), parallelism comes from the workers
decoding different files, and the decoders of caches only thumbnails used
are released after each thumbnail (the clip served last keeps its own). A
monitor or playback request on such a cache replaces the small decoder with
a full one. Sequence frames are never background (the scopes ask for the
program frame at 40 while playing).
42 4K clips (Windows), peak private bytes: hardware Off 6.1-6.8 GB (killed)
-> 3.1-3.2 GB; hardware Auto 2.7-2.8 GB (#381) -> 2.3-2.4 GB. Thumbnails of
12 / 42 clips ready in 0.6 / 2.2 s instead of 2.2 s / not finishing.
Example crates/ui-egui/examples/thumb_burst.rs reproduces the burst.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cache asked for a frame within IDLE (3 s) counted as in use, and the caps
of #41 (4 live decoders, 1 GiB of frames) only bound caches idle for longer,
so moving the playhead across 42 4K clips kept 42 hardware decoders (6.5 GB
of VRAM) and up to 3.9 GB of frames. Now only caches touched within RECENT
(500 ms) are protected: caches idle 0.5-3 s give up decoders beyond
HARD_DECODERS (8) and frames beyond the shared budget, least recently used
first; the 3 s rule down to 4 decoders is unchanged. store() refreshes
last_used.
Measured with a 42-clip 4K reproduction (Windows, Media Foundation): peak
private bytes 6.6 -> 4.0 GB, frame cache 3.9 -> 1.1 GB, live decoders
42 -> 8-14, GPU memory 6.5 -> 4.3-4.6 GB, 0 dropped frames before and after.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Update MediaInfo to store all discovered audio streams in audio_streams
- Retain independent stream metadata and decoder state for MP4/MOV and Matroska/WebM
- Update audio_placement_specs to allocate successive tracks via ensure_audio_tracks
- Isolate clip rendering to read assigned audio stream rather than replaying stream 0
- Bump project schema to v13 with backwards compatibility for legacy v12 projects
Resolves#312Fixes#200Fixes#315
Matroska timestamps are whole milliseconds and a 1024-sample AAC frame at 48 kHz lasts
21.33 ms, so frames are stamped at 0, 1008, 2064, 3072… samples; OBS's remux to MP4 carries
that rounding into the sample durations (1008, 1008, 1056…). The MKV and MP4 readers put each
decoded packet at its own timestamp, overwriting the end of one packet and leaving a hole of
silence after another: a click every few packets, crackling throughout playback and export.
Packets of codecs with a fixed frame length (AAC, MPEG audio, AC-3) now run on from one
another and resynchronise to the timestamp only across gaps of more than half a packet, the
rule the MPEG-TS/PS reader already uses. Laced Matroska frames without a duration (which
repeat the block's timestamp) follow on too, and MKV Opus uses the same helper. MP4 audio
packet starts are kept in samples instead of track timescale units.
Regression tests mux our own AAC into MKV (millisecond stamps, plain and laced, a real gap)
and into an MP4 with OBS-remux durations: block-by-block playback and seeks must equal the
packets decoded back to back (all fail before the fix). An ffmpeg oracle test plays
ffmpeg-made MKVs with AAC, MP3, MP2 and AC-3 and requires them as smooth as ffmpeg's decode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extends the Windows backend (#141) to VP9 (profiles 0 and 2) and AV1 (main, 8/10-bit), 4:2:0.
- codecs::hw: FrameStreamInfo (vpcC / av1C): profile, bit depth, chroma, picture size and colour as
the software decoders report them (vp9_color / av1_color are now shared with them), random-access
samples, in-band format changes; StreamInfo lets HybridDecoder handle NAL and frame-coded streams.
- platform: media_foundation::stream (what each codec needs: DXVA profile, MFT request, input
format), MfDecoder generalised; VP9 frames / AV1 temporal units go in as they are; the MFT's
declared picture size (display aperture) is checked against the bitstream's.
- Declined up front: VP9 profiles 1/3, AV1 profiles 1/2, 4:2:2/4:4:4, mono, 12-bit, no codec
extension, no DXVA profile. A key frame of another size or a different sequence header falls back
to the software decoder.
- Tests: bit-exact parity (360p / 1080p / 2160p, seeks, resets, flush, hidden frames), forced
failure, damaged samples and av1C, format change fallback, WebM / Matroska through the media stack.
- Bench: AV1 fixtures with libaom when ffmpeg lacks SVT-AV1.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
- codecs: a resampled MP4 audio read scales the edit-list offset to the source rate in i128 and
saturates the window arithmetic; a hostile media_time overflowed i64.
- engine: media_duration follows a subclip's parent chain in a loop bounded at 16 hops instead of
recursing; a cyclic or very deep chain in a damaged project overflowed the stack.
Each has a regression test that failed before (overflow panic / stack overflow).
Fixes#66
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New backend in crates/platform (media_foundation/): a Direct3D-aware decoder MFT is driven at the
level of single access units (Annex B), given the process's D3D11 video device through an
IMFDXGIDeviceManager so it decodes with DXVA, and its NV12 / P010 textures are read back through a
staging texture into planar Yuv8 / Yuv16. H.264 Baseline/Main/High and HEVC Main / Main 10, 4:2:0.
- Hardware is verified: non-D3D-aware or async MFTs, streams the GPU's DXVA decoder does not list,
and system-memory output (Microsoft's MFT software fallback) are declined or fail the stream so
HybridDecoder continues with our decoder.
- mfplat.dll is loaded at run time so Windows N editions still start.
- register() / registered() / hardware_decoder_for() cover Windows; perf.stats reports the backend.
- Tests (Windows): bit-exact parity with the software decoders at 640x360, 1080p and 2160p
(resets, seeks, flush), forced mid-stream failure, damaged input, concurrency, repeated init.
- Bench: runs on Windows (ffmpeg on PATH, process/thread CPU clocks), HEVC Main 10 2160p row,
hardware counters per row.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
- aac: the decoder notes SBR data in fill elements (EXT_SBR_DATA /
EXT_SBR_DATA_CRC), so implicitly signalled HE-AAC is recognised;
Decoder::sbr() covers explicit signalling too.
- codecs: aac_output_rate() gives an AAC stream's output rate (the
extension rate, or twice a <= 24 kHz core when SBR data is present); the
MP4 reader reports it instead of the core rate, so sequences made from an
HE-AAC clip get 44.1 kHz instead of 22.05 kHz. Our decoder now also takes
explicitly signalled HE-AAC (it decodes the AAC-LC core) and, when the
stream rate is twice the core rate, interpolates the core to it with a
streaming half-band upsampler. The SBR high band is not reconstructed yet.
- export: the AAC bitrate is capped at 6 bits per sample and channel
(ISO/IEC 14496-3 frame limit) instead of failing with "bitrate out of
range" (320 kbps stereo at 22.05 kHz).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The probe guessed the pixel format from the codec profile alone: HEVC
reported nothing, H.264 High 4:2:2 had no bit depth, and ProRes 4444
claimed alpha because of its fourcc.
- HEVC: chroma format and bit depth from hvcC.
- H.264: from the avcC High-profile extension (ISO/IEC 14496-15 §5.3.3),
else from the first SPS; unknown (empty) if neither can be read.
- ProRes: the first frame header gives the chroma format and alpha
(alpha_channel_type), as the first DNx frame header already does.
Checked against ffprobe on ffmpeg-made HEVC 4:2:0/4:2:2 8/10-bit,
H.264 4:2:0/4:2:2 8/10-bit and ProRes 422 HQ / 4444 with and without
alpha. Unit tests cover hvcC, avcC (extension and SPS fallback) and
ProRes 4444 with and without alpha.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The clip duration came from the media duration (`mdhd`), which runs on
the decode timeline. ffmpeg / x264 MP4s with B-frames skip the reorder
delay with an edit, so every one imported a frame too long (a 120-frame
file became 121 frames, the last a held repeat). A video track delayed
by a leading empty edit was a frame short, and its last frame could not
be reached.
Use the sum of the track's edits (movie timescale, what `tkhd` records)
when there is an edit list. Fragmented files, and edit lists that add up
to nothing or overflow a Tick, keep the media duration.
Tests: a unit test with crafted edit lists (skip, empty edit, hostile
sizes) and an ffmpeg oracle test (libx264 with B-frames: 60 frames,
was 61).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem (#73): a 30 fps WebM played and exported with every third frame
missing and the frame before it shown twice (20 unique frames a second).
The same happens at 29.97, 60, 59.94, 24 and 23.976 fps, and with MP4
files whose video track has a 1 ms timescale. 25 and 50 fps were fine.
Cause: Matroska stores block timestamps in TimestampScale units (usually
1 ms), rounded to the nearest unit, so at 30 fps frame 2 (66.67 ms) is
stored as 67. MkvSource::video_frame converted the requested time to track
units with a floor (66), and the lookup (last sample with pts <= target)
returned frame 1. Every frame whose timestamp was rounded up was skipped.
Mp4Source::video_at, and the web build's WebCodecs MP4 path, floored the
same way against the track timescale; normal MP4 timescales hold frame
times exactly, so only coarse (1000) timescales were hit.
Fix: new Tick::to_rational_round (nearest unit, halves up, total for zero
and negative timebases), used for the video lookup target and the
catch-up threshold in mkv.rs, mp4.rs and webcodecs.rs. A requested time
now matches a frame whose stored timestamp is within half a unit of it,
the same precision the timestamps have. Truncating muxers (stamps rounded
down) still work. Times on exact timebases (90 kHz, 15360, 30000/1001…)
convert exactly as before.
Test: crates/codecs/src/rounded_pts_tests.rs builds synthetic Motion-JPEG
streams in the test (each picture shows its index as black/white blocks),
muxed as WebM with 1 ms timestamps and as MOV with a 1000 timescale, at
30, 29.97, 60, 59.94, 24, 23.976 and 25 fps with nearest and truncated
stamps. Requesting every frame's start over 3 s must show every frame
exactly once, and every frame's midpoint must show that frame. Before the
fix both tests failed ("30 fps, Nearest ms stamps: 30 of 90 frames never
shown [2, 5, 8, 11]…, 30 shown twice [1, 4, 7, 10]…"). The time crate
tests cover the rounding, ties and hostile timebases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
