mirror of https://github.com/storytold/filmcraft.git synced 2026-10-24 10:10:05 -06:00

Files

T

Dario CarneluttiandClaude Sonnet 5.5 790b3a0697 platform: H.265 (HEVC) export with VideoToolbox

Adds `Format::Hevc` (Export > Format > H.265 (HEVC); `"format": "hevc"` in
`file.exportMedia`): HEVC Main, 8-bit 4:2:0, in MP4 (`hvc1`) or QuickTime, with AAC audio,
encoded by the macOS hardware encoder. There is no built-in H.265 encoder (one in pure Rust
is a project of its own, and x265 is GPL), so the format exists only where the OS has one.

- `videotoolbox_encode`: the session wrapper creates HEVC sessions too (`VtProfile::HevcMain`;
  the profile also says which codec). Keyframes are the HEVC random access points (NAL types
  16 to 21); the parameter sets are VPS / SPS / PPS; the `hvcC` record in the MP4 sample
  entry is the one VideoToolbox writes (sample description extension atoms), so profile,
  level and flags are the encoder's own. `VtEncoder::uses_hardware()` asks VideoToolbox
  whether the session runs on a hardware encoder (a test checks H.264 and HEVC), so a
  requirement dropped by accident could never become a silent software encode.
- `hardware_encode`: one adapter for both codecs. Choosing the format is the opt-in (the
  `hardware_encoding` setting is for H.264). `hevc_available()` (one small hardware session,
  created once) is registered with the new `filmcraft_export::register_format_probe`, so
  `available(Format::Hevc)`, the format list of Export mode and `exportFormats` follow the
  machine. `Format::has_builtin_encoder` says which formats need such a probe.
- `export`: `Format::is_h26x` where H.264 and H.265 behave alike (MP4 or QuickTime, AAC, size
  estimate, summary line). Two-pass VBR is refused up front ("H.265 export has no two-pass
  mode") and HDR sequences export as SDR, because there is no encoder to fall back to.
- Export mode: H.265 shows H.264's multiplexer, audio, metadata and bitrate controls, without
  profile, level, two-pass or the hardware checkbox, and says "Encoder: Hardware".

Apple M1 (8 cores), 1080p25, a 9:29 timeline (camera clip, ProRes 4444 overlays, AAC,
loudness): hardware H.265 at 10 Mb/s 195 s, hardware H.264 at 20 Mb/s 204 s (CPU 1029 s and
1042 s; the same H.264 export took 193 s and 200 s on other runs). It is not faster: the codec
that compresses further is the more expensive one (raw encoders: 166 and 189 frames/s), and
the export is bound by the CPU compositor, not by the media engine.

Quality per bit, 30 s of the same timeline against a ProRes 422 HQ export (average PSNR over
the three planes, SSIM): 20 Mb/s 46.88 dB / 0.9902 (H.264) and 46.91 dB / 0.9903 (H.265);
10 Mb/s 44.96 / 0.9863 and 45.09 / 0.9863; 6 Mb/s 43.72 / 0.9833 and 44.31 / 0.9843; 4 Mb/s
42.91 / 0.9807 and 43.64 / 0.9825; 3 Mb/s 42.37 / 0.9786 and 43.05 / 0.9807. The same picture
at 10 Mb/s and above; at 3 to 6 Mb/s H.265 needs 22 to 31 % less bitrate for the same PSNR.
Both formats therefore keep the same default bitrate. Details in docs/performance.md (HW3).

Tests: tests/hardware_encode.rs gets the HEVC counterparts of the H.264 ones (what is taken
and declined, a round trip through our HEVC decoder with luma PSNR above 30 dB, keyframes, MP4
and QuickTime exports with AAC, ffmpeg / ffprobe reading `hevc` / `Main` / `hvc1` /
`yuv420p` / BT.709, exact sizes, hostile configurations, encoders dropped at any point, the
format list agreeing with the probe, sessions on the hardware encoder); settings_tests.rs
checks names, ids, defaults, the summary, two-pass refusal and the probe; export_ui.rs the
controls H.265 shows and hides.

Stacked on the hardware H.264 encoder (the first commit of this branch).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

2026-10-07 14:00:08 -06:00

5.8 KiB

Raw Blame History

ADR 0001: `unsafe` OS media FFI in `crates/platform`, and nowhere else

Status: accepted (2026-10-05, approved by the project owner; recorded in AGENTS.md §0 item 3 and the craftrules never-crash standard)
Issue: #30 (hardware acceleration)

Context

FilmCraft is pure Rust and the workspace sets unsafe_code = "forbid". Every codec has a clean-room software decoder, and those remain the reference. But decoding 4K H.264 / HEVC in software costs 85–120 ms of CPU per frame on an M4 Pro, so 4K playback drops frames under load, while every current computer has a hardware video decoder the operating system exposes (VideoToolbox on macOS; Media Foundation / D3D11 on Windows and VA-API on Linux later).

Using it needs FFI into C / Objective-C APIs, which is unsafe Rust. The bindings we use (the objc2-* crates, MIT / Apache-2.0 / Zlib) are generated declarations; nothing GPL/LGPL is linked.

Decision

Allow unsafe in exactly one crate, crates/platform (filmcraft-platform, layer L5), for one job: OS media FFI (hardware decoding now, hardware encoding later). Audio output, dialogs, menus and other OS integration do not go in it.

Containment rules:

The crate does not use lints.workspace = true. Its own [lints] table copies the workspace lints except unsafe_code = "deny" (not forbid), and adds clippy::undocumented_unsafe_blocks = "deny". Only the FFI modules (videotoolbox) carry #[allow(unsafe_code)]; the rest of the crate (the fallback logic in hybrid) has no unsafe.
Every unsafe block has a // SAFETY: comment saying why it is sound.
The public API is safe: no pub unsafe fn, no raw pointers or FFI types in public signatures; failures are Results. The crate keeps the never-crash deny(clippy::unwrap_used, …) attribute of every clean crate.
No panic crosses the FFI boundary: the VideoToolbox output callback runs under catch_unwind and reports a panic as a failed frame.
The crate compiles on every target. OS bindings are target-specific dependencies; elsewhere register() reports Unavailable and does nothing. It is not part of the wasm check (L5).

The fallback guarantee

Registering a hardware decoder never makes a file undecodable, and never changes a picture:

The factory declines (the software decoder is used) when Settings ▸ Playback ▸ Hardware decoding is Off, for formats the hardware path does not take (field-coded H.264, bit depths other than 8 / 10, 4:4:4, mismatched luma / chroma depth), and when the OS cannot create a hardware session for the stream (hardware decoding is required, so the OS's own software decoder is never used instead of ours).
A hardware decoder that fails mid-stream (decode error, session invalidated by a GPU change or sleep, in-band parameter sets that differ from the sample entry) becomes the software decoder for that stream without the caller noticing: HybridDecoder replays the samples since the last restart point through it and drops pictures already returned. It is logged and counted (perf.stats decode.hardware.fallbacks).
Pictures are interchangeable with the software decoder's: same planes (bit-exact on the H.264 High, HEVC Main and Main 10 parity fixtures, including after seeks and flushes), colour, pixel aspect, pts and presentation order. Tests compare the two on every run on macOS, and the fallback logic is tested on every platform with a stand-in hardware decoder.

Consequences

4K H.264 / HEVC decode costs ~4 ms of CPU per frame instead of 85–120 ms (performance.md).
Better patent posture for the most encumbered formats: H.264 / HEVC decoding is done by the operating system, which carries the licences, wherever hardware decoding is used.
A new review burden: changes to crates/platform need the same scrutiny as any FFI code. Anything else that wants unsafe needs a new decision, not an extension of this one.

Addendum (2026-10-06): hardware H.264 encoding

The crate's second use of the FFI is the one this decision named, hardware encoding: a VideoToolbox H.264 encoder behind filmcraft_export::VideoEncoder (videotoolbox_encode, with the same containment rules; hardware_encode is safe code). It differs from decoding in two ways, both deliberate:

It is opt-in per export (ExportSettings::hardware_encoding, Off by default). Exports from the built-in encoder are byte-identical across machines; a hardware encoder's output depends on the machine, so it must be asked for.
There is no mid-stream fallback. A decoder can replay samples through another decoder; an encoder cannot hand a half-written stream to another one. The factory declines up front (the built-in encoder is used) for everything the hardware path does not take, and for any machine where the OS cannot create a hardware session; a hardware encoder that fails during an export stops it with an error.

Everything else in this record applies unchanged: unsafe stays in crates/platform, every block has a // SAFETY: comment, no panic crosses the FFI boundary, the public API is safe, and the crate compiles everywhere (register() is a no-op off macOS).

Addendum (2026-10-07): hardware H.265 (HEVC) encoding

The same VideoToolbox session wrapper also creates HEVC sessions (VtProfile::HevcMain). The rules above hold; HEVC adds one difference. The crate has no software HEVC encoder to fall back to, and there will not be one (pure Rust, clean-room: x265 is GPL), so Format::Hevc exists only where the OS has a hardware encoder. Choosing the format is the opt-in, filmcraft_export::available asks a probe the platform crate registers, and what the hardware path does not take (two-pass, odd sizes, a machine without the encoder) is an error naming the reason instead of a different encoder.

5.8 KiB Raw Blame History Unescape Escape

ADR 0001: unsafe OS media FFI in crates/platform, and nowhere else