mirror of https://github.com/storytold/filmcraft.git synced 2026-10-30 04:57:19 -06:00

Files

T

Brandon ThomasandClaude Opus 5.5 c3ec0e85d3 HW1: platform crate + VideoToolbox H.264/HEVC hardware decode (#30 )

- New crates/platform (filmcraft-platform, L5): OS media FFI only. The one
  crate allowed `unsafe` (own [lints] with unsafe_code = "deny",
  #[allow(unsafe_code)] only on the videotoolbox module, SAFETY comment on
  every block, safe Result-returning public API). AGENTS.md §0.3 names the
  exception; docs/adr/0001-platform-ffi.md records why and the rules.
- VideoToolbox decoder for avcC / hvcC, 8/10-bit 4:2:0 and 4:2:2: session from
  the sample entry's parameter sets with hardware required, asynchronous
  decode (2 in flight), biplanar CVPixelBuffer copied to planar Yuv8/Yuv16,
  reorder buffer of the stream's own depth, RASL skipped after a starting CRA.
  Output callback under catch_unwind.
- HybridDecoder: transparent mid-stream fallback to the software decoder
  (replays samples since the last restart point, drops already-returned
  pictures). New codecs::software_video_decoder (built-in factories only) and
  codecs::hw (switch, counters, NalStreamInfo shared with software decoders;
  vui_color / sar_par helpers now used by both).
- Settings ▸ Playback ▸ Hardware decoding: Auto / Off (engine applies it to
  the codecs switch); desktop app, CLI/MCP and bench call register().
- perf.stats decode.framesDecoded and decode.hardware counters.
- Tests: bit-exact parity with the software decoders (H.264 High, HEVC Main
  open GOP, Main 10; resets, reseeks, flushes), forced-failure fallback,
  mutation fuzz of samples and parameter sets, cross-platform fallback with a
  stand-in decoder, setting Off, HEVC 4:2:2 vs ffmpeg.
- Bench --hw auto|off; results in docs/performance.md (4K CPU/frame
  ~86-119 ms -> ~4 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-05 04:53:28 -04:00

4.9 KiB

Raw Blame History

filmcraft-platform

OS media integration for FilmCraft (layer L5): hardware video decoding through the operating system's codecs, behind filmcraft_codecs::VideoDecoder. It holds OS media FFI and nothing else. It is the one crate of the workspace allowed to contain unsafe, under the rules of ADR 0001 and AGENTS.md §0.3.

// at startup (the desktop app, filmcraft-cli, the bench)
let availability = filmcraft_platform::register(); // Available("VideoToolbox") on macOS

What it does

macOS: VideoToolbox H.264 (avcC) and HEVC (hvcC), 8- and 10-bit, 4:2:0 and 4:2:2 (videotoolbox.rs). The session is created from the sample entry's parameter sets with a hardware decoder required; samples go in as CMSampleBuffers with asynchronous decompression (two access units in flight); the output callback copies each NV12 / P010-style biplanar CVPixelBuffer into planar Yuv8 / Yuv16 (chroma deinterleaved, 10-bit samples shifted down from the high bits, cropped to the conformance window when the buffer is the coded size). A reorder buffer of the stream's own depth (max_num_reorder_frames / sps_max_num_reorder_pics) restores presentation order; a run starting at an HEVC CRA leaves out its RASL pictures, as our decoder does. Each seek (reset) starts a fresh session.
Other systems: register() does nothing and returns Availability::Unavailable.
HybridDecoder (hybrid.rs, safe code): the hardware decoder plus the means to build our software decoder for the same SampleEntry (filmcraft_codecs::software_video_decoder). On a mid-stream failure (decode error, invalidated session, changed in-band parameter sets) it replays the samples since the last restart point (IDR / IRAP; for an HEVC CRA the one before, so its RASL pictures decode) through the software decoder, drops pictures already returned, keeps the ones the hardware had decoded but not returned, and stays in software for that instance. The replay log is bounded (600 samples / 256 MB); beyond it one error is returned and the next seek restarts in software. Streams our decoders cannot decode (HEVC 4:2:2) have no fallback: the error stands.

Guarantees

Never undecodable: the factory declines (returns None, so the software decoder is used) when Settings ▸ Playback ▸ Hardware decoding is Off, for formats it does not take (field-coded H.264, bit depths other than 8 / 10, 4:4:4 or monochrome, luma / chroma depth mismatch, larger than 8192×8192) and when VideoToolbox cannot create a hardware session.
Interchangeable: colour, pixel aspect, pts, presentation order, is_random_access and is_disposable come from the software decoders' own helpers (filmcraft_codecs::hw, video::vui_color, sar_par).
Never crash: no unwrap / expect / panic! outside tests; the output callback runs under catch_unwind; every unsafe block has a // SAFETY: comment; the public API is safe.
Counted: perf.stats decode.hardware (frames, software frames, sessions, declined, fallbacks; filmcraft_codecs::hw::hw_stats).

Tests

test	what
`tests/videotoolbox.rs` (macOS)	H.264 High, HEVC Main (open GOP: CRA + RASL) and HEVC Main 10, 640×360 (coded 368: cropping) with B-frames: every picture bit-exact with our software decoder, same pts order, count, colour and aspect, also after `reset` + reseek to every later sync sample, mid-stream `flush`, and a full pass after resets; forced mid-stream failures (`VtDecoder::fail_after`) at five points continue with the software decoder's exact output; seeded mutation of samples and parameter sets (bit flips, truncation, corrupt length prefixes) never panics or hangs; HEVC 4:2:2 10-bit is bit-exact with ffmpeg's decode
`tests/fallback.rs` (every OS)	`HybridDecoder` with a stand-in hardware decoder failing after N samples (every sync sample ± a few, first / last sample, after a seek): output identical to the software decoder; in-band parameter sets identical to the sample entry's stay in hardware, different ones switch to software
`tests/setting.rs`	Hardware decoding Off gives the software decoder through `make_video_decoder` and the media stack (no hardware frames); Auto gives VideoToolbox where available

Fixtures are made with ffmpeg into target/fixtures/platform/ (generator only, never linked); tests skip without ffmpeg or without a hardware decoder.

Performance

M4 Pro, load 150–190 (cargo xtask bench --hw off|auto): CPU per decoded frame H.264 2160p 119 → 4.2 ms, HEVC 2160p 86 → 3.7 ms; decode 35 → 107 fps and 49 → 217 fps; 4K H.264 and HEVC playback with no dropped frames at Full, 1/2 and 1/4. Details in docs/performance.md.

Not yet

Zero-copy upload of CVPixelBuffers into wgpu textures; hardware encoding; Media Foundation / D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264.

4.9 KiB Raw Blame History Unescape Escape