filmcraft/crates/vp9

filmcraft-vp9

Clean-room, pure-Rust (no unsafe) VP9 decoder, implemented from the public VP9 Bitstream & Decoding Process Specification v0.6 (WebM project). It is an L0 crate: it depends only on filmcraft-bitstream, thiserror, and optionally rayon (default feature threads), and builds for wasm32-unknown-unknown with and without threads.

The constant tables (probability defaults, scans, coefficient bands, quantizer lookups, Pareto table, interpolation filters, mv_ref_blocks...) were extracted from the specification text by tools/extract_tables.py into src/spec_tables.rs. The 1D inverse DCT / ADST butterflies are generated by tools/gen_transforms.py, which symbolically executes the step lists of spec section 8.7.1 into src/transform_gen.rs. No code was taken or ported from libvpx, FFmpeg or any other decoder; ffmpeg (with libvpx-vp9) is used only as an external fixture generator and test oracle.

Status

Area	Status
Profiles 0 / 2 (8, 10, 12-bit 4:2:0)	done, bit-exact
Profiles 1 / 3 (4:2:2, 4:4:0, 4:4:4; 8 / 10 / 12-bit)	done, bit-exact
Uncompressed + compressed headers, superframe index (Annex B)	done
Boolean decoder, forward updates (diff_update_prob, MV probs), backward adaptation (8.4)	done
Segmentation (tree / temporal prediction, all four features), loop filter (levels, sharpness, ref / mode deltas, segment levels)	done
All 10 intra modes incl. edge rules, sub-8x8 blocks	done
Inter: 8-tap regular / smooth / sharp + bilinear, compound prediction, MV prediction (incl. previous-frame MVs, sub-8x8 append), reference scaling (8.5.2.3)	done
DCT / ADST 4..32, WHT (lossless)	done
Tiles (columns and rows); tile columns decode in parallel	done
Hidden frames, show_existing_frame, intra-only frames, resolution changes, error resilient / frame parallel modes	done
Frame threading: references published per superblock row, the loop filter / output of a frame overlaps the next frames	done
Draft mode (non-reference frames skip the loop filter; reduced-resolution playback only)	done

API

use filmcraft_vp9::{Decoder, Plane};

let mut dec = Decoder::new(); // or Decoder::with_threads(n)
for (pts, chunk) in chunks {  // one IVF / WebM block / MP4 sample each (frame or superframe)
    for pic in dec.decode(chunk, pts)? {
        // pic.width / pic.height (frame size), pic.render_width / render_height (render_size),
        // pic.bit_depth, pic.subsampling_x / subsampling_y, pic.chroma_width / chroma_height,
        // pic.y / pic.u / pic.v: Plane::U8(Vec<u8>) for 8-bit, Plane::U16(Vec<u16>) above,
        // pic.y_stride / pic.uv_stride, pic.pts, pic.key, pic.intra_only,
        // pic.color { color_space (0..7 as coded), full_range }
    }
}
let rest = dec.flush(); // pictures still in flight (frame threads)
// dec.set_draft(true): draft mode (playback at reduced resolution only), dec.threads()
dec.reset(); // forget references / contexts (seek), keeps the thread pool
let stats = dec.stats(); // what the stream exercised (frame kinds, tiles, compound, scaling...)

// Random access points: does a container sample start with a key frame (and its parameters)?
let key: bool = filmcraft_vp9::is_keyframe(chunk);
let info: Option<filmcraft_vp9::KeyframeInfo> = filmcraft_vp9::keyframe_info(chunk);

filmcraft-codecs wires the decoder into Matroska / WebM (V_VP9, profile / bit depth from CodecPrivate, colour from Colour) and MP4 (vp09 + vpcC) import; its GOP cache asks is_keyframe before starting a decode at a container sync sample, so wrongly flagged samples (e.g. an MP4 without stss) still seek correctly.

A superframe produces every shown frame it contains (normally one); hidden frames produce none. With frame threads (Decoder::new(), or with_threads(n) with n > 1) a picture can come out of a later decode call or flush (each carries the pts of the chunk that showed it); with_threads(1) returns every picture from the call that completes it.

Architecture

The calling thread parses the uncompressed and compressed headers and manages the eight reference slots, the four probability contexts, segmentation map and previous-frame motion vectors. Each tile column is decoded by one TileDecoder (rayon pool) into a private strip of samples and mode info (tile rows of a column are decoded in order: they share above contexts). Counts are then merged for backward adaptation, the mode info is assembled (the next frame reads it for motion vector prediction and segment ids) and the frame goes into the reference slots at once, while its post stage runs on a thread of its own: it assembles each superblock row ("band") from the strips, loop filters it superblock by superblock in raster order (so the result is the specification's), publishes the band above it once that can no longer change (OnceLock per band; the 16-wide filter of row r reads 8 and changes 7 rows of row r - 1, which travel in 8 padding rows on top of each band) and copies it into the output picture. Motion compensation of later frames reads reference rows through Frame::row / Frame::span, which wait for exactly the bands a block reaches, so the next frames' tile columns decode while earlier frames are still being filtered. Probability adaptation needs only the counts, so even backward-adaptive streams start the next frame before the loop filter is done; frame-parallel / error-resilient streams (no backward adaptation) never wait at all. Results never depend on scheduling: the conformance suite decodes every fixture with 1, 3 and all threads. Samples are u16 internally for every bit depth.

Draft mode (Decoder::set_draft(true); FilmCraft enables it with Settings ▸ Playback ▸ Draft decoding while the Program monitor plays at 1/2 or 1/4): frames with refresh_frame_flags 0 — no reference slot receives them, so nothing predicts from their samples — skip the loop filter and are flagged Picture::draft. The loop filter changes neither mode info, segmentation map nor probabilities, so every other frame is unchanged. Such frames occur in temporally layered streams (e.g. libvpx -ts-parameters); single-layer realtime streams refresh LAST on every frame.

Modules: header (6.2 / 6.3, superframes), boolcoder (9.2), probs (frame contexts, counts, 8.4), tile (partition / mode info / MV prediction), tile::recon (residual, tokens, intra / inter prediction calls, reconstruction), intra, inter, transform, loopfilter, frame (banded frames, pools), decoder (scheduling, post stage).

Tests

cargo test --release -p filmcraft-vp9. Fixtures are generated on first use into target/fixtures/vp9/ with ffmpeg + libvpx-vp9 (tests print a message and skip when ffmpeg or libvpx is unavailable). Every frame is compared sample-exactly with ffmpeg -c:v vp9 -f rawvideo output with 1 thread, 3 threads (frame threading with a small pool) and every core; the first mismatch reports frame, plane, position and superblock. No picture may be flagged draft without draft mode.

• Unit tests: boolean decoder against a literal model of 9.2, tables / trees, inverse DCT / ADST against floating-point bases, DC-only shortcut vs. full transform, the column-parallel 2D transform against the per-column formulation on random blocks of every size and type, scaled MC path vs. unscaled (through a banded reference frame), superframe index, inv_remap_prob, probability merging.
• Draft mode (draft_mode_changes_only_flagged_non_reference_frames): on a two-layer temporal scalability stream, single- and frame-threaded, every unflagged frame is bit-exact and only flagged frames differ.
• Conformance (tests/it/conformance.rs): the fixture matrix below plus a stream concatenating two sizes. compare_detects_mismatch guards the harness. coverage_report (ignored) prints the decoder statistics for each fixture.
• Synthetic (tests/it/synthetic.rs): headers of libvpx output are rewritten to produce what ffmpeg's libvpx wrapper cannot: inter frames of a stream encoded at another size spliced after a 352x288 key frame with explicit frame sizes, i.e. scaled reference prediction (downscale 256x208, upscale 480x400, odd 301x199; a same-size splice is the control); a key frame rewritten as a hidden intra-only frame (reset_frame_context 3 / 2 / 0, various refresh masks) shown with show_existing_frame; show_existing_frame of every slot, also inside a superframe.
• Robustness (tests/it/robustness.rs): bit flips, byte replacement and truncation of real streams, truncated frames, random data and oversized frames never panic (debug and release) or hang, single- and multi-threaded.

Fixture matrix (all bit-exact, 1 thread and thread pool)

Backward adaptation is on (-frame-parallel 0) unless noted.

fixture	content	configuration
intra_only_keyframes	352x288	-g 1 (key frames only, frame parallel)
default_good	352x288, 20 frames	800 kb/s
profile2_10bit / profile2_12bit	352x288 / 176x144	profile 2, 10 / 12-bit
profile1_444 / _422 / _440	176x144	profile 1, 4:4:4 / 4:2:2 / 4:4:0
profile3_444_10bit / profile3_422_12bit	176x144	profile 3
lossless / lossless_10bit	176x144	-lossless 1 (WHT)
tiles_cols4	1280x720	4 tile columns
tiles_rows_cols / tiles_rows4	1280x720 / 640x480	tile columns + rows / 4 tile rows
altref_hidden / altref_10bit / altref_tiles	352x288 / 720p	2-pass, -auto-alt-ref 1: hidden frames, superframes, compound prediction
frame_parallel	352x288	-frame-parallel 1 (no backward adaptation)
error_resilient	352x288	-error-resilient 1
odd_size / tiny_odd	347x251 / 33x17	odd sizes, partial superblocks
row_mt	640x360	-row-mt 1, 2 tile columns
speed0 .. speed8	176x144 / 352x288	every speed (0 two-pass, 5..8 realtime)
cq_low / cq_q4 / cq_high / cq_10bit_low / cq_12bit_high		-crf 0 / 4 / 63 / 2 / 60
aq_segmentation / aq_variance / aq_complexity	352x288	-aq-mode 3 / 1 / 2 (segmentation, temporal segment-id prediction)
sharpness	352x288	-sharpness 5
fade_intra_heavy	352x288	fade-in, key frame every 8
hd_1080p	1920x1080	realtime speed 5
resize_concatenated	352x288 then 347x251	size change at a key frame
temporal_layers	352x288	realtime, 2 temporal layers (-ts-parameters): non-reference frames

Performance

cargo test --release -p filmcraft-vp9 --test it perf:: -- --ignored --nocapture (bit-exactness is verified first), or VP9_THREADS=n VP9_LOOPS=k cargo run --release -p filmcraft-vp9 --example vp9dec -- in.ivf (decodes the file k times and prints the statistics; VP9_DRAFT=1 for draft mode).

M4.10 (2026-10-03), Apple M4 Pro (14 cores) shared with other builds (load average 190-320), so the figures are CPU cycles per frame counted by the kernel (/usr/bin/time -l, all threads summed; mean of two interleaved base / after rounds) — wall clock only within a round. Fixtures: the cargo xtask bench decode clips (testsrc2 + grain, libvpx realtime speed 8, CRF 32; 1080p 4 tile columns, 2160p 8 tile columns, frame-parallel mode), bit-exact with ffmpeg.

stream	threads	Mcycles / frame before	after	wall ms / frame before → after (same load)
1080p (120 frames)	1	68.0	52.0 (−24 %)
1080p	14	100.7	57.9 (−43 %)	95, 82 → 18, 17
2160p (72 frames)	1	273.6	205.8 (−25 %)
2160p	14	391.7	219.2 (−44 %)	146, 118 → 38, 36

Before M4.10 the frame-level work was serial (tile columns of one frame, then a loop-filter wavefront, then assembly and output conversion), so 14 threads spent 40-50 % more cycles than one thread (spinning between short parallel phases) and a 4K frame still took ~120 ms of wall time at this load; now the post stage overlaps the next frames and 4K decodes at ~27 fps on the loaded machine. What changed:

• Frame threading (see Architecture).
• Loop filter (was ~50 % of single-threaded time): the lane masks were bools with clamp and branchy selects, which kept the "vectorised" group filter scalar; masks are now 0 / -1 integers with bitwise selects and min / max clipping, so it compiles to NEON.
• Inverse transforms: the column pass runs the generated butterflies (transform_gen::cols4 / cols8) on 4 / 8 columns at once.
• Motion compensation: 8-bit sub-pixel filters accumulate in 16-bit lanes (8 per vector instead of 4): every sum of the VP9 filters plus rounding lies in [-13706, 46474], so the sum modulo 2^16 identifies it (negative sums clip to 0).
• Bool decoder: branch-free update.

Single-threaded 4K time now splits into coefficient tokens (bool decoding) ~27 %, loop filter ~25 % (two thirds of it the vertical-edge transposes and per-group decisions), motion compensation ~15 %, inverse transforms ~8 %, intra prediction ~6 %.

Known gaps

• A frame's tile decoding is still a barrier: the next frame starts once every tile column of the current one is done (its post stage then runs alongside), so streams with a single tile column decode their tiles on one thread at a time; the post stage (loop filter) of one frame runs on one thread.
• Scaled references combined with compound prediction are not covered by a fixture (both are verified separately; libvpx through ffmpeg cannot change size mid-stream and the synthetic splice cannot keep the compound streams' probability contexts intact).
• Frames larger than 16384x16384 are rejected (Error::Invalid) to bound memory.
• Damaged streams are decoded as far as the data allows (the boolean decoder reads zeros past the end, as specified); there is no concealment beyond that, and inter frames whose reference slots are empty return Error::MissingReference.