Commit Graph
80 Commits
Author SHA1 Message Date
Brandon ThomasandClaude Opus 5.5 d8a7b28b2e Merge origin/main into #492; ffmpeg oracle: decode unaligned crops, skip clap cases on ffmpeg without clap
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 05:45:04 -04:00
Brandon Thomas 303b1518f6 Merge pull request #446 from LloydNicholson/fix/mp4-color-from-bitstream
Issue #552: MP4 and Matroska colour from the bitstream when the container has none
2026-10-10 05:03:11 -04:00
MrCloudy2andClaude Opus 5.5 e90508f16f #288: MP4 / MOV pictures are cropped as the file says (H.264 / HEVC cropping, clean aperture)
The MP4 source reported the sample entry's size, only swapped for the display rotation, so two
crops were lost: the codec's own (H.264 SPS frame cropping, HEVC conformance window), which the
decoder applies but the reported size and sequences made from the clip didn't, and the clean
aperture (`clap`), parsed but never used. An iPhone clip (1920x1440 coded, 1744x1308 clean
aperture, -90 degree display matrix) showed 1440x1920 instead of 1308x1744.

H.264 / HEVC sizes now come from the stream's parameter sets; the clean aperture is a rectangle
inside that picture (ignored when it makes no sense), reported as the size and cut from every
frame by the GOP cache before the display rotation, through the new VideoFrame::cropped.

Tests: bit-exact against ffmpeg (which applies both crops) for an H.264 SPS crop, an HEVC
conformance window, centred and off-centre clean apertures and a clean aperture with a rotation;
clean apertures that make no sense are ignored without a panic; VideoFrame::cropped on every pixel
format with odd sizes and misaligned corners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 00:50:03 +02:00
lloydandClaude Opus 5.5 d7b7027f42 M8: MP4 and Matroska colour from the bitstream when the container has none
ffmpeg writes MP4 without a `colr` box unless asked (`+write_colr`), and
color_from only read `colr` (or vpcC / apvC), so every such file fell back
to BT.709 limited: an HEVC PQ file was detected as rec709 / hdr false, a
full-range H.264 file as limited range, and BT.601 SD with BT.709
primaries. Decoded frames were right (the decoders read the VUI); the
media info, HDR detection and Interpret Footage defaults were not.
Matroska without a `Colour` element had the same gap.

New `stream_color` reads the colour a codec configuration carries in its
bitstream with the decoders' own parsers: the VUI of the SPS in avcC /
hvcC and the color_config of the AV1 sequence header in av1C. Precedence:
`colr` (nclx / nclc) or Matroska `Colour`, then vpcC / apvC / the
bitstream, field by field (what the container leaves unspecified falls
through), then the existing size defaults (BT.601 matrix for SD, BT.709
otherwise). nclc has no range flag, so the bitstream's range now stands;
an ICC `colr` no longer hides the bitstream's code points. A truncated or
corrupt parameter set gives no colour and the defaults.

Tests build avcC / hvcC / av1C bit by bit: HEVC PQ → rec2100-pq, full-range
H.264 → Full, BT.601 SD (NTSC and PAL) primaries, unspecified VUI keeps the
defaults, colr wins, Matroska without Colour, and every truncation and
single-bit flip of each configuration under catch_unwind. An ffmpeg-oracle
test checks real x264 / x265 files against their `+write_colr` remux.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 17:58:49 +02:00
Brandon ThomasandClaude Opus 5.5 701d7c7111 Merge origin/main into multi-stream audio
Keeps #359's contiguous_starts audio timing, now per stream in Mp4Audio /
MkvAudio (starts and preroll in source sample frames). MP4 audio streams and
their decoders are built from the same (track, info) pairs so they stay
index-aligned. Adapts new main code to audio_streams / TrackItem::audio_stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:08:09 -04:00
Brandon Thomas 6e80bab6b8 Merge remote-tracking branch 'origin/main' into fix-359
# Conflicts:
#	contributors/people.toml
2026-10-09 09:26:08 -04:00
Davi AndreiandClaude Sonnet 5.5 b485e2c180 Thumbnails decode with small decoders that are not kept
Right after importing many 4K clips the Project panel and the timeline ask
every clip for a thumbnail. Up to six frame workers decoded them at once,
each with a frame-threaded software H.264 decoder (16 threads and 18
pictures in flight here, about 0.5 GB at 4K), and the decoders stayed: 42
4K clips with hardware decoding Off took the process past 6 GB during import.

Thumbnail jobs (item frames at the Media Browser / thumbnail priorities) now
run under filmcraft_media::cancel::with_background: H.264 decoders made for
them use 2 threads (thread_limited), parallelism comes from the workers
decoding different files, and the decoders of caches only thumbnails used
are released after each thumbnail (the clip served last keeps its own). A
monitor or playback request on such a cache replaces the small decoder with
a full one. Sequence frames are never background (the scopes ask for the
program frame at 40 while playing).

42 4K clips (Windows), peak private bytes: hardware Off 6.1-6.8 GB (killed)
-> 3.1-3.2 GB; hardware Auto 2.7-2.8 GB (#381) -> 2.3-2.4 GB. Thumbnails of
12 / 42 clips ready in 0.6 / 2.2 s instead of 2.2 s / not finishing.
Example crates/ui-egui/examples/thumb_burst.rs reproduces the burst.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 04:32:43 -03:00
Davi AndreiandClaude Sonnet 5.5 0c64e41b53 Decode caches: navigation bursts no longer keep every clip's decoder and frames (#55)
A cache asked for a frame within IDLE (3 s) counted as in use, and the caps
of #41 (4 live decoders, 1 GiB of frames) only bound caches idle for longer,
so moving the playhead across 42 4K clips kept 42 hardware decoders (6.5 GB
of VRAM) and up to 3.9 GB of frames. Now only caches touched within RECENT
(500 ms) are protected: caches idle 0.5-3 s give up decoders beyond
HARD_DECODERS (8) and frames beyond the shared budget, least recently used
first; the 3 s rule down to 4 decoders is unchanged. store() refreshes
last_used.

Measured with a 42-clip 4K reproduction (Windows, Media Foundation): peak
private bytes 6.6 -> 4.0 GB, frame cache 3.9 -> 1.1 GB, live decoders
42 -> 8-14, GPU memory 6.5 -> 4.3-4.6 GB, 0 dropped frames before and after.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:13:42 -03:00
Idiotic f146d7ba9f feat(media,engine): implement multi-stream container audio support and schema v13
- Update MediaInfo to store all discovered audio streams in audio_streams

- Retain independent stream metadata and decoder state for MP4/MOV and Matroska/WebM

- Update audio_placement_specs to allocate successive tracks via ensure_audio_tracks

- Isolate clip rendering to read assigned audio stream rather than replaying stream 0

- Bump project schema to v13 with backwards compatibility for legacy v12 projects

Resolves #312

Fixes #200

Fixes #315
2026-10-08 22:36:45 -04:00
ippsecandClaude Opus 5.5 d51945c807 #236: audio in MKV and OBS-remuxed MP4 plays without crackling
Matroska timestamps are whole milliseconds and a 1024-sample AAC frame at 48 kHz lasts
21.33 ms, so frames are stamped at 0, 1008, 2064, 3072… samples; OBS's remux to MP4 carries
that rounding into the sample durations (1008, 1008, 1056…). The MKV and MP4 readers put each
decoded packet at its own timestamp, overwriting the end of one packet and leaving a hole of
silence after another: a click every few packets, crackling throughout playback and export.

Packets of codecs with a fixed frame length (AAC, MPEG audio, AC-3) now run on from one
another and resynchronise to the timestamp only across gaps of more than half a packet, the
rule the MPEG-TS/PS reader already uses. Laced Matroska frames without a duration (which
repeat the block's timestamp) follow on too, and MKV Opus uses the same helper. MP4 audio
packet starts are kept in samples instead of track timescale units.

Regression tests mux our own AAC into MKV (millisecond stamps, plain and laced, a real gap)
and into an MP4 with OBS-remux durations: block-by-block playback and seeks must equal the
packets decoded back to back (all fail before the fix). An ffmpeg oracle test plays
ffmpeg-made MKVs with AAC, MP3, MP2 and AC-3 and requires them as smooth as ffmpeg's decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:29:23 -04:00
Brandon Thomas 71483312b6 Merge pull request #162 from SaymonMaster14/feat/windows-vp9-av1-hwdecode
Windows: VP9 + AV1 hardware decode (Media Foundation / DXVA)

# Conflicts:
#	ROADMAP.md
#	crates/platform/README.md
2026-10-07 17:58:12 -04:00
Brandon Thomas fcf1c45583 Merge pull request #141 from SaymonMaster14/feat/windows-media-foundation-hwdecode
Windows: Media Foundation/D3D11 hardware H.264 + HEVC decode

# Conflicts:
#	README.md
#	ROADMAP.md
#	crates/platform/Cargo.toml
#	crates/platform/src/lib.rs
#	docs/architecture.md
#	docs/performance.md
2026-10-07 17:48:06 -04:00
Davi AndreiandClaude Sonnet 5.5 79cb1d0f8d Windows: VP9 + AV1 hardware decode through Media Foundation / DXVA (#30)
Extends the Windows backend (#141) to VP9 (profiles 0 and 2) and AV1 (main, 8/10-bit), 4:2:0.

- codecs::hw: FrameStreamInfo (vpcC / av1C): profile, bit depth, chroma, picture size and colour as
  the software decoders report them (vp9_color / av1_color are now shared with them), random-access
  samples, in-band format changes; StreamInfo lets HybridDecoder handle NAL and frame-coded streams.
- platform: media_foundation::stream (what each codec needs: DXVA profile, MFT request, input
  format), MfDecoder generalised; VP9 frames / AV1 temporal units go in as they are; the MFT's
  declared picture size (display aperture) is checked against the bitstream's.
- Declined up front: VP9 profiles 1/3, AV1 profiles 1/2, 4:2:2/4:4:4, mono, 12-bit, no codec
  extension, no DXVA profile. A key frame of another size or a different sequence header falls back
  to the software decoder.
- Tests: bit-exact parity (360p / 1080p / 2160p, seeks, resets, flush, hidden frames), forced
  failure, damaged samples and av1C, format change fallback, WebM / Matroska through the media stack.
- Bench: AV1 fixtures with libaom when ffmpeg lacks SVT-AV1.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
2026-10-07 12:39:45 -03:00
Brandon ThomasandClaude Opus 5.5 0dd66d8aa3 Hardening: hostile MP4 edit lists and cyclic subclip chains (#66)
- codecs: a resampled MP4 audio read scales the edit-list offset to the source rate in i128 and
  saturates the window arithmetic; a hostile media_time overflowed i64.
- engine: media_duration follows a subclip's parent chain in a loop bounded at 16 hops instead of
  recursing; a cyclic or very deep chain in a damaged project overflowed the stack.

Each has a regression test that failed before (overflow panic / stack overflow).

Fixes #66

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 09:44:45 -04:00
Davi AndreiandClaude Sonnet 5.5 5e440a6607 Windows: Media Foundation / Direct3D 11 hardware H.264 + HEVC decode (#30)
New backend in crates/platform (media_foundation/): a Direct3D-aware decoder MFT is driven at the
level of single access units (Annex B), given the process's D3D11 video device through an
IMFDXGIDeviceManager so it decodes with DXVA, and its NV12 / P010 textures are read back through a
staging texture into planar Yuv8 / Yuv16. H.264 Baseline/Main/High and HEVC Main / Main 10, 4:2:0.

- Hardware is verified: non-D3D-aware or async MFTs, streams the GPU's DXVA decoder does not list,
  and system-memory output (Microsoft's MFT software fallback) are declined or fail the stream so
  HybridDecoder continues with our decoder.
- mfplat.dll is loaded at run time so Windows N editions still start.
- register() / registered() / hardware_decoder_for() cover Windows; perf.stats reports the backend.
- Tests (Windows): bit-exact parity with the software decoders at 640x360, 1080p and 2160p
  (resets, seeks, flush), forced mid-stream failure, damaged input, concurrency, repeated init.
- Bench: runs on Windows (ffmpeg on PATH, process/thread CPU clocks), HEVC Main 10 2160p row,
  hardware counters per row.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMorBHwct72udS2LiC3ZHp
2026-10-07 10:07:40 -03:00
Brandon Thomas d4e438e932 Merge remote-tracking branch 'origin/main' into apv-105 2026-10-07 07:08:58 -04:00
Brandon Thomas 566164cf7d Merge pull request #129 from carlitosteso3/fix/he-aac-108
HE-AAC plays at its output rate; AAC export caps the bitrate (#108, parts 1–2)
2026-10-07 07:08:25 -04:00
Brandon ThomasandClaude Opus 5.5 b94c81b5d1 HE-AAC: probing reads at most one access unit's worth of bytes per sample
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 06:34:19 -04:00
Brandon Thomas 005745d9ec Merge remote-tracking branch 'origin/main' into apv-105
# Conflicts:
#	crates/codecs/src/mp4.rs
2026-10-07 06:01:14 -04:00
Brandon Thomas 36d8114d2f Merge pull request #118 from HEOJUNFO/fix/probe-pixfmt-from-config
codecs: MP4/MOV pixel format and alpha come from hvcC, avcC and the ProRes frame header (#103)
2026-10-07 05:57:10 -04:00
Brandon Thomas 79057f9c36 Merge pull request #102 from EthanSuquet/fix/mp4-duration-edit-list
codecs: an MP4/MOV clip lasts as long as its edit list plays it
2026-10-07 05:57:06 -04:00
carlitosteso3andClaude Opus 5.5 463a59c1b4 codecs: HE-AAC plays at its output rate; AAC export caps the bitrate (#108)
- aac: the decoder notes SBR data in fill elements (EXT_SBR_DATA /
  EXT_SBR_DATA_CRC), so implicitly signalled HE-AAC is recognised;
  Decoder::sbr() covers explicit signalling too.
- codecs: aac_output_rate() gives an AAC stream's output rate (the
  extension rate, or twice a <= 24 kHz core when SBR data is present); the
  MP4 reader reports it instead of the core rate, so sequences made from an
  HE-AAC clip get 44.1 kHz instead of 22.05 kHz. Our decoder now also takes
  explicitly signalled HE-AAC (it decodes the AAC-LC core) and, when the
  stream rate is twice the core rate, interpolates the core to it with a
  streaming half-band upsampler. The SBR high band is not reconstructed yet.
- export: the AAC bitrate is capped at 6 bits per sample and channel
  (ISO/IEC 14496-3 frame limit) instead of failing with "bitrate out of
  range" (320 kbps stereo at 22.05 kHz).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 22:28:30 -05:00
heojunfoandClaude Opus 5.5 f761a299a7 codecs: MP4/MOV pixel format and alpha come from hvcC, avcC and the ProRes frame header (#103)
The probe guessed the pixel format from the codec profile alone: HEVC
reported nothing, H.264 High 4:2:2 had no bit depth, and ProRes 4444
claimed alpha because of its fourcc.

- HEVC: chroma format and bit depth from hvcC.
- H.264: from the avcC High-profile extension (ISO/IEC 14496-15 §5.3.3),
  else from the first SPS; unknown (empty) if neither can be read.
- ProRes: the first frame header gives the chroma format and alpha
  (alpha_channel_type), as the first DNx frame header already does.

Checked against ffprobe on ffmpeg-made HEVC 4:2:0/4:2:2 8/10-bit,
H.264 4:2:0/4:2:2 8/10-bit and ProRes 422 HQ / 4444 with and without
alpha. Unit tests cover hvcC, avcC (extension and SPS fallback) and
ProRes 4444 with and without alpha.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 10:44:38 +09:00
mweinbach 15787f7fe8 Codecs: add clean-room APV (RFC 9924) decoder, encoder, container support, and export presets 2026-10-06 17:24:34 -04:00
EthanSuquetandClaude Opus 5.5 1f2f20a700 codecs: an MP4/MOV clip lasts as long as its edit list plays it
The clip duration came from the media duration (`mdhd`), which runs on
the decode timeline. ffmpeg / x264 MP4s with B-frames skip the reorder
delay with an edit, so every one imported a frame too long (a 120-frame
file became 121 frames, the last a held repeat). A video track delayed
by a leading empty edit was a frame short, and its last frame could not
be reached.

Use the sum of the track's edits (movie timescale, what `tkhd` records)
when there is an edit list. Fragmented files, and edit lists that add up
to nothing or overflow a Tick, keep the media duration.

Tests: a unit test with crafted edit lists (skip, empty edit, hostile
sizes) and an ffmpeg oracle test (libx264 with B-frames: 60 frames,
was 61).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:37:06 -04:00
EthanSuquetandClaude Opus 5.5 2b49201e84 codecs: video frames are looked up by the nearest timestamp unit, so a 1 ms WebM shows every frame
Problem (#73): a 30 fps WebM played and exported with every third frame
missing and the frame before it shown twice (20 unique frames a second).
The same happens at 29.97, 60, 59.94, 24 and 23.976 fps, and with MP4
files whose video track has a 1 ms timescale. 25 and 50 fps were fine.

Cause: Matroska stores block timestamps in TimestampScale units (usually
1 ms), rounded to the nearest unit, so at 30 fps frame 2 (66.67 ms) is
stored as 67. MkvSource::video_frame converted the requested time to track
units with a floor (66), and the lookup (last sample with pts <= target)
returned frame 1. Every frame whose timestamp was rounded up was skipped.
Mp4Source::video_at, and the web build's WebCodecs MP4 path, floored the
same way against the track timescale; normal MP4 timescales hold frame
times exactly, so only coarse (1000) timescales were hit.

Fix: new Tick::to_rational_round (nearest unit, halves up, total for zero
and negative timebases), used for the video lookup target and the
catch-up threshold in mkv.rs, mp4.rs and webcodecs.rs. A requested time
now matches a frame whose stored timestamp is within half a unit of it,
the same precision the timestamps have. Truncating muxers (stamps rounded
down) still work. Times on exact timebases (90 kHz, 15360, 30000/1001…)
convert exactly as before.

Test: crates/codecs/src/rounded_pts_tests.rs builds synthetic Motion-JPEG
streams in the test (each picture shows its index as black/white blocks),
muxed as WebM with 1 ms timestamps and as MOV with a 1000 timescale, at
30, 29.97, 60, 59.94, 24, 23.976 and 25 fps with nearest and truncated
stamps. Requesting every frame's start over 3 s must show every frame
exactly once, and every frame's midpoint must show that frame. Before the
fix both tests failed ("30 fps, Nearest ms stamps: 30 of 90 frames never
shown [2, 5, 8, 11]…, 30 shown twice [1, 4, 7, 10]…"). The time crate
tests cover the rounding, ties and hostile timebases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:16:37 -04:00
Brandon Thomas 5c8471c9f4 Merge pull request #41 from justjohn12345/pr/decode-memory
MEM1–MEM4 + HW1: clip memory on a timeline from 12 GB to 1.9 GB; hardware decode actually on
2026-10-06 11:38:03 -04:00
Brandon ThomasandClaude Opus 5.5 6a9fdce2ef codecs: released decoders are dropped off the cache lock; dirty-pool bit-exact test
trim_decoders dropped the decoders it took from idle caches while the requesting cache's lock was held. Dropping a decoder can block (a VideoToolbox session waits for its frames in flight), so the render path paid that latency under the lock. trim_decoders now returns the released decoders and frame_late keeps them in a local declared before its lock guard, so they drop after the guard on every return path; the cap is enforced exactly as before.

New tests/pool_bitexact.rs: primes the frame pool with 0xAA-filled plane buffers, then decodes an H.264 fixture into them; every frame must match ffmpeg bit for bit, and the pool must report the reuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:16:09 -04:00
punkhopandClaude Opus 5.5 5bf2203bb8 codecs: resampled MP4 audio reads apply the edit list once
Problem: a clip with AAC audio in MP4 whose sample rate differs from the
sequence's (48 kHz clip in a 44.1 kHz sequence) exported with loud
low-frequency thumps, several times full scale, recurring through the clip.
Every exported container was affected (AAC clips them into thumps, PCM
clips at full scale); a WAV export had a single one at the clip's start.

Cause: Mp4Source::audio decodes a window starting at s0, the requested
presentation sample minus the edit-list offset, then resamples it. The
resampler derived its sub-sample phase from s0, so the edit-list offset
(-1024 samples of AAC priming) was applied a second time: the read
position started about 1024 samples before the window, was clamped to
index 0 with a large negative fraction, and linear interpolation
extrapolated a ramp over the first ~940 samples of every request. Export
asks for audio one batch of video frames at a time, so the ramp repeated
at each request. Sources at the sequence rate take the copy path and
were unaffected, as were tracks without an edit list.

Fix: take the phase from the presentation sample the window starts at
(before the edit-list offset is subtracted).

Test: aac_mp4_resampled_reads_follow_the_edit_list
(crates/codecs/tests/mp4_oracle.rs, with `require_ffmpeg!` and the
shared fixture helper as docs/testing.md asks of new oracle tests) reads
an ffmpeg AAC fixture at 44.1 kHz in frame-sized pieces and compares it
with the 48 kHz decode interpolated at the same times (peak was 7.35
against a source level of 0.15 before the fix). docs/testing.md lists it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:29:26 -07:00
justjohn12345andClaude Opus 5.5 4f3e119ac2 MEM4: decode into the planes of evicted frames
With MEM1-MEM3 twelve clips on a timeline still took 5.1 GB, only 2.5 GB of
it live: each decoded picture allocated three planes, each eviction freed
them, and the system allocator kept the freed pages.

filmcraft_frame::pool keeps the planes of frames the GOP caches evict (up to
192 MiB per sample type; a frame something else still holds is left alone).
The H.264 decoder, through a plane allocator hook that keeps the crate free
of dependencies, and the VideoToolbox path take their output planes from it.

perf.stats reports decode.planePoolMB and decode.planesReused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:57:16 -04:00
justjohn12345andClaude Opus 5.5 b999b69f07 HW1: register the hardware decoders in every desktop run
The desktop app called filmcraft_platform::register() as an argument of
log::info!. A log macro does not evaluate its arguments while no logger takes
its level, and the app installs none, so VideoToolbox was never registered:
perf.stats showed hardware decoding enabled with 0 sessions, 0 declined and
every frame decoded in software. The CLI and the bench register directly and
were not affected.

Registration is now a statement of its own, with a start-up test that runs
without a logger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:42:28 -04:00
justjohn12345andClaude Opus 5.5 4486652f07 MEM3: one frame budget shared by all sources
Each GOP cache keeps up to 384 MB of decoded frames (at least 64), and
nothing bounded the total: twelve clips on a timeline held 3.8 GB of frames
nobody was looking at, and no perf.stats counter showed it.

The pool now counts the frames of every cache against a 1 GiB budget. Over
it, caches idle for 3 s give up frames, least recently used cache first.
Caches in use keep their own budget, since a frame evicted before it is
shown costs a re-decode from the keyframe.

perf.stats reports decode.cacheMB and decode.cacheBudgetMB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:39:48 -04:00
justjohn12345andClaude Opus 5.5 517b6b041b MEM2: idle sources give up their decoders beyond a cap
Every source kept its decoder for good once it had decoded a frame, with its
reference pictures and the pictures its frame threads hold in flight: about
290 MB per H.264 clip, 3.5 GB for twelve clips on a timeline.

The GOP caches now share a pool. Beyond four live decoders, the least
recently used caches that have been idle for 3 s drop theirs and make a new
one when asked again (a restart at the sync sample, as after a seek). Caches
in use keep their decoder whatever the count, so a composite of many layers
never restarts decoders per frame. Other caches are only try-locked, so two
caches trimming each other cannot deadlock.

perf.stats reports decode.liveDecoders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:39:48 -04:00
Brandon ThomasandClaude Opus 5.5 c3ec0e85d3 HW1: platform crate + VideoToolbox H.264/HEVC hardware decode (#30)
- New crates/platform (filmcraft-platform, L5): OS media FFI only. The one
  crate allowed `unsafe` (own [lints] with unsafe_code = "deny",
  #[allow(unsafe_code)] only on the videotoolbox module, SAFETY comment on
  every block, safe Result-returning public API). AGENTS.md §0.3 names the
  exception; docs/adr/0001-platform-ffi.md records why and the rules.
- VideoToolbox decoder for avcC / hvcC, 8/10-bit 4:2:0 and 4:2:2: session from
  the sample entry's parameter sets with hardware required, asynchronous
  decode (2 in flight), biplanar CVPixelBuffer copied to planar Yuv8/Yuv16,
  reorder buffer of the stream's own depth, RASL skipped after a starting CRA.
  Output callback under catch_unwind.
- HybridDecoder: transparent mid-stream fallback to the software decoder
  (replays samples since the last restart point, drops already-returned
  pictures). New codecs::software_video_decoder (built-in factories only) and
  codecs::hw (switch, counters, NalStreamInfo shared with software decoders;
  vui_color / sar_par helpers now used by both).
- Settings ▸ Playback ▸ Hardware decoding: Auto / Off (engine applies it to
  the codecs switch); desktop app, CLI/MCP and bench call register().
- perf.stats decode.framesDecoded and decode.hardware counters.
- Tests: bit-exact parity with the software decoders (H.264 High, HEVC Main
  open GOP, Main 10; resets, reseeks, flushes), forced-failure fallback,
  mutation fuzz of samples and parameter sets, cross-platform fallback with a
  stand-in decoder, setting Off, HEVC 4:2:2 vs ffmpeg.
- Bench --hw auto|off; results in docs/performance.md (4K CPU/frame
  ~86-119 ms -> ~4 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 04:53:28 -04:00
Brandon Thomas de57e91c80 Merge pull request #18 from storytold/pr/never-crash-codecs
codecs: never crash (damaged MP4/TS headers; no expect/unreachable)
2026-10-04 16:17:14 -04:00
Brandon ThomasandClaude Opus 5.5 3a392ae8da Clippy 1.99: fix new lints so cargo xtask ci is green again
Rust/clippy 1.99 added `chunks_exact_to_as_chunks` and tightened
`needless_late_init` and `drain_collect`, failing `clippy -D warnings`
on main.

- `chunks_exact(N)` / `chunks_exact_mut(N)` with a constant N become
  `as_chunks::<N>()` / `as_chunks_mut::<N>()` (fixed-size arrays, so
  several byte reads drop their per-element indexing). relink's
  fingerprint uses the `(chunks, remainder)` split; its hash output is
  unchanged.
- Late-initialised `let`s in ac3 bit allocation and opus CELT
  bands/rate become `let x = if/match …`.
- `dpb.drain(..).collect()` becomes `std::mem::take(&mut dpb)`.

No behaviour change; `cargo xtask ci` passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 15:55:23 -04:00
Brandon ThomasandClaude Opus 5.5 6aa3c37342 codecs: never crash — damaged MP4/TS headers, no expect/unreachable
Mutation fuzzing (mutated ffmpeg-made files through `open_bytes`) found three import crashes:
- an MP4/MOV track with an empty sample-description list: `entries[0]` index out of bounds
  (video and audio paths);
- an MP4/MOV track with timescale 0: division by zero in the duration;
- an MPEG-TS stream whose PTS span gave a ~0 fps rate: division by zero in `tick_of`.

Tracks now need samples, a sample description and a timescale to count as playable; frame rates
that round to zero fall back to the default; the TS/PS rate choice is a small function that never
returns a zero rate. Regression tests: `mp4::tests::damaged_track_headers_never_panic`,
`mpeg::tests::damaged_timestamps_never_give_a_zero_rate`.

Also: GOP cache, MKV/MP4/Ogg/MPEG audio paths, MXF and VP9 conversion without
`expect`/`unreachable!`; the crate denies the panicking lints outside tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 12:20:30 -04:00
Brandon ThomasandClaude Opus 5.5 7dd3ee4d77 Merge PR #5 from storybored379
Media: AIFF files import (enable symphonia's aiff reader)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 10:50:12 -04:00
storybored379andClaude Opus 5.5 320b6eede8 Media: AIFF files import (enable symphonia's aiff reader)
AIFF is listed as an importable audio format and FilmCraft exports AIFF,
but the symphonia build had only the wav reader enabled, so every .aif /
.aiff import (including FilmCraft's own AIFF exports) failed with
"unsupported media: end of stream". Enable symphonia's aiff reader (the
same MPL-2.0 symphonia-format-riff crate already used for WAV; Cargo.lock
is unchanged). The new test decodes a synthetic 16-bit stereo AIFF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 01:17:34 -07:00
Brandon ThomasandClaude Opus 5.5 e2afbed6c2 Tests for display rotation: Matroska roll → quarter turns (incl. flips, odd and malformed angles), MP4 mirrored/scaled matrices
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 02:50:16 -04:00
storybored379andClaude Opus 5.5 27a1f78df7 Media: apply the container's display rotation (portrait phone video)
Phones record portrait video as landscape pictures plus a rotation in the
container: the MP4/MOV tkhd matrix (ISO/IEC 14496-12 §8.3.2) or the
Matroska Projection pose roll (RFC 9559 §5.1.4.1.28.46). The matrix was
parsed but never used, so every portrait iPhone clip imported sideways
at its landscape size.

- isobmff: Track::display_rotation() reads 0/90/180/270° from the matrix
  signs (mirrored / sheared matrices are left alone).
- matroska: parse Projection (type, yaw, pitch, roll);
  VideoInfo::display_rotation() for rectangular projections.
- frame: VideoFrame::rotated() turns every plane at its own resolution
  (4:2:2 chroma is widened to 4:4:4 for quarter turns) and swaps the PAR.
- codecs: Mp4Source / MkvSource report the displayed size and PAR, and
  GopCache rotates decoded frames before caching them.
- web: the WebCodecs source configures the decoder with the coded size
  and rotates its output the same way.

Tests: a one-frame ProRes MOV built in the test with 0/90/180/270°
matrices (reported size + which half ends up on top), plane rotation for
RGBA / 4:2:0 / 4:2:2, and a synthetic Matroska file with a -90° roll.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 02:50:16 -04:00
Brandon Thomas 879be680d0 Merge branch 'worktree-agent-af7195a2c4c71a574' into integrate2 2026-10-03 19:00:14 -04:00
Brandon ThomasandClaude Opus 5.5 8e6b273eba M4.10: merge main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 10:52:21 -04:00
Brandon ThomasandClaude Opus 5.5 2a199425d0 M8.10: HDR-aware Lumetri, mastering metadata tone-map peak, HSL Refine, Lumetri Presets grid, Comparison View match
- Lumetri grades PQ / HLG working spaces in their own signal normalised to
  HDR White (Basic Correction) / HDR Range (Curves) in cd/m²
  (filmcraft_color::GradeSpace): exposure in stops of light, highlights above
  HDR White kept, HDR Specular scales them; Rec. 709 unchanged.
- HDR static metadata (mdcv/clli, Matroska MasteringMetadata/MaxCLL/MaxFALL)
  on VideoStreamInfo::hdr; PQ media use MaxCLL / mastering peak as the
  BT.2390 source peak; media.colorInfo reports it.
- HSL Secondary Refine: Denoise (median of the key) and Blur (Gaussian).
- Lumetri Presets (18 original looks in 4 folders) with thumbnails rendered by
  our Lumetri on a procedural preview picture; Effects panel rows and a
  thumbnail grid when wide; lumetri.presets / applyPreset / presetThumbnails.
- Color Match: Comparison View toggle; Apply Match uses its reference frame.
- Goldens lumetri_hdr_pq and lumetri_presets_grid; D-Log M stays unsupported
  (no published formula).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 10:31:00 -04:00
Brandon ThomasandClaude Opus 5.5 7ce0766744 M4.10: AV1 fixed-width MC passes, recycled planes, lean mode info, draft mode; media-stack draft test
Sub-pixel filter passes run over fixed block widths; frame / region planes are recycled
through a bounded pool (zeroed on reuse) instead of fresh calloc / munmap per frame; the
palette mode-info arrays (40 bytes per 4x4) exist only when screen content tools are allowed;
CDF adaptation is branch-free. Draft mode skips deblocking, CDEF and loop restoration of shown
frames that refresh no reference slot. Media-stack test: VP9 / HEVC / AV1 draft playback only
reaches draft requests, and an export afterwards is bit-exact with ffmpeg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:02:43 -04:00
Brandon ThomasandClaude Opus 5.5 c8edb9daaf M4.10: HEVC vectorised MC, SAO and output, batched bypass bins, draft mode
Interpolation keeps its reference windows in reusable scratch (no 2 x 10 KB zeroing per block)
and filters with taps outer / outputs inner over fixed block widths; SAO edge and band offsets
select the offset per lane instead of a table lookup; uni / bi output clips with min / max; runs of
bypass bins decode with one division; context-coded bins are branch-free. Draft mode skips
deblocking and SAO of sub-layer non-reference pictures of the top sub-layer. Bit-exact with 1, 3
and all threads; draft output differs only on flagged pictures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 07:38:29 -04:00
Brandon ThomasandClaude Opus 5.5 7e16968b3a M9.15: MPEG-2 decoded frames carry the draft flag introduced on main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 05:50:54 -04:00
Brandon ThomasandClaude Opus 5.5 842801de41 M4.10: VP9 frame threading (band-published references, per-frame post stage) and draft mode
Frames are published one superblock row at a time once loop filtered; a post stage per frame
(own thread) assembles bands from the tile strips, filters them in raster order and builds the
output picture, while the next frame's tile columns decode and wait per band for the reference
rows they read. Bit-exact with 1, 3 and all threads. Draft mode skips the loop filter of frames
that refresh no reference slot. av1dec / hevcdec bench knobs and VP9 / AV1 4K playback scenarios
in the bench.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 05:34:39 -04:00
Brandon Thomas 72e0dda167 Merge main into M9.15 2026-10-03 04:16:41 -04:00
Brandon ThomasandClaude Opus 5.5 37008cab5e M9.15: import .vob/.mod/.tod/.m2t/.m2v/.m1v/.mpv video and standalone .mp2 audio
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 04:16:41 -04:00