mirror of
https://github.com/storytold/filmcraft.git
synced 2026-11-11 16:34:30 -06:00
vangel76andClaude Opus 5.5
1e6f42eda2
Problem: the AV1 in-loop filters ran on one thread per frame (postfilter::apply had an unused pool). On a single-tile stream they are a quarter to a third of a frame, and since references are waited for as whole frames, that serial tail sat on the critical path of every frame that references it. Fix: the three filters run in bands of 64 luma rows on the decoder's existing rayon pool, one join per filter per frame. CDEF and loop restoration write only their own rows of a separate output. The loop filter's edges never touch the same sample, and only a band's top horizontal edges reach into the band above: each band refilters the vertical edges of a copy of the 8 rows above it, runs those edges against the copy, and the samples they changed are written back. With one thread the code path is the old one. Output stays bit-exact (1 vs 16 threads vs libdav1d; new 4:2:2 / 4:4:4 libaom fixtures). perf.rs gains a low-delay 1080p and a single-tile 2160p fixture and the all-core stage breakdown. 16 threads: 1080p GOP 1.16-1.30x, low delay 1.28x, 2160p 1.14x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
139 lines
9.8 KiB
Markdown
139 lines
9.8 KiB
Markdown
# filmcraft-av1
|
||
|
||
Clean-room, pure-Rust (no `unsafe`) AV1 decoder, implemented from the public **AV1 Bitstream &
|
||
Decoding Process Specification, version 1.0.0 with Errata 1** (AOMediaCodec/av1-spec, git
|
||
commit `5e04f3f`, 2023-06-12). It is an L0 crate: it depends only on `filmcraft-bitstream`,
|
||
`thiserror` and (optional, feature `threads`) `rayon`, and builds for `wasm32-unknown-unknown`.
|
||
|
||
All constant tables (default CDFs, scan orders, quantiser lookups, filter taps, …) and the
|
||
named constants are extracted from the specification's Markdown source by
|
||
[`tools/extract_tables.py`](tools/extract_tables.py) into `src/spec_tables.rs` (every table's
|
||
shape is checked against its declared dimensions; one missing comma in the spec's
|
||
`Split_Tx_Size` initialiser is reported and repaired). No code was taken from libaom, dav1d,
|
||
SVT-AV1 or any other implementation; ffmpeg (libsvtav1 to encode fixtures, libdav1d to decode the
|
||
reference) is used only as an external test oracle.
|
||
|
||
## Conformance status
|
||
|
||
| Stage | Spec | Status |
|
||
|---|---|---|
|
||
| OBU parsing (low-overhead format), temporal delimiters, operating-point dropping | 5.3, 7.5 | done |
|
||
| Sequence header, color config, timing / decoder model info | 5.5 | done |
|
||
| Uncompressed frame header (all fields, frame size / superres / render size, tile info, quantiser, segmentation, delta q/lf, loop filter, CDEF, LR, tx mode, skip mode, global motion, film grain params) | 5.9 | done |
|
||
| Reference frame state: set_frame_refs, setup_past_independence, load_previous, reference update, show_existing_frame | 7.8, 7.20, 7.21 | done |
|
||
| Symbol decoder, CDF adaptation, init / load / save / frame-end CDF update | 8.2, 8.3 | done |
|
||
| Tiles and tile groups (uniform and explicit spacing, tile size bytes) | 5.11 | done (tiles decoded in parallel) |
|
||
| Intra frame mode info: partitions, skip, segment id, CDEF index, delta q / lf, y / uv modes, angle deltas, CfL alphas, palette (colors, cache, color index map), filter intra, tx size (incl. var-tx syntax) | 5.11 | done, bit-exact |
|
||
| Coefficients (all_zero, eob, base / br levels, Golomb, dc sign, tx type sets, scans) | 5.11.39 | done, bit-exact |
|
||
| Dequantisation incl. quantiser matrices | 7.12 | done (qmatrix not yet oracle-tested) |
|
||
| Inverse transforms: DCT 4–64, ADST 4/8/16, flip ADST, identity 4–32, WHT (lossless), rectangular | 7.13 | done, bit-exact |
|
||
| Intra prediction: DC, V/H + directional with edge filter and upsampling, smooth (3), Paeth, recursive filter intra, CfL, palette | 7.11.2, 7.11.4, 7.11.5 | done, bit-exact |
|
||
| Inter mode info: segment id prediction, skip mode, reference frames (single / compound, uni / bi), modes, DRL, MV coding, inter-intra, motion mode, compound type, interpolation filters | 5.11.18–5.11.32 | done, bit-exact |
|
||
| MV prediction: spatial / temporal candidate stacks, global MV, extra search, clamping, warp samples | 7.10 | done, bit-exact |
|
||
| Motion field estimation (projection) and motion vector storage | 7.9, 7.19 | done, bit-exact |
|
||
| Inter prediction: MV scaling (scaled references), 8-tap sub-pixel filters, global and local warp (warp estimation, shear), OBMC, wedge / difference-weighted / inter-intra masks, distance weights, averaging | 7.11.3 | done, bit-exact |
|
||
| Intra block copy | 7.11.3 | done, bit-exact |
|
||
| Output policy with scalability (highest spatial layer per temporal unit) | 7.18.1 | done, bit-exact |
|
||
| Loop filter (all filter sizes, deltas, segment / ref / mode adjustments) | 7.14 | done, bit-exact |
|
||
| CDEF | 7.15 | done, bit-exact |
|
||
| Super-resolution upscaling | 7.16 | done, bit-exact |
|
||
| Loop restoration (Wiener, self-guided, switchable; stripes) | 7.17 | done, bit-exact |
|
||
| Film grain synthesis (output pictures only; `Decoder::apply_film_grain` turns it off) | 7.18.3 | done, bit-exact |
|
||
| Large-scale tile / tile list OBUs | 7.3 | not planned |
|
||
|
||
## Accuracy
|
||
|
||
`tests/oracle_intra.rs` encodes all-intra streams with ffmpeg's libsvtav1 (with and without the
|
||
in-loop filters; 64×64 and 128×128 superblocks; intra edge filter on and off) and compares every
|
||
decoded frame with libdav1d: **bit-exact** on all 13 fixtures (8- and 10-bit 4:2:0; 128×128 to
|
||
1280×720 including odd sizes; testsrc2, mandelbrot, gradients and noise; presets 1–8).
|
||
|
||
`tests/conformance.rs` downloads libaom's conformance test vectors on first use
|
||
(storage.googleapis.com/aom-test-data) and compares with libdav1d: `av1-1-b8-02-allintra`
|
||
(39 frames, deblocking + CDEF + self-guided restoration), `05-mv`, `06-mfmv`, `24-monochrome`
|
||
and `b10-23-film_grain-50` are **bit-exact** in the default run. The ignored
|
||
`conformance_vectors_extended` test covers the wider set; bit-exact today: `01-size-16x16`,
|
||
`-66x66`, `-196x196`, `-226x226`, `00-quantizer-00/31/63` (8-bit) and `-00/40` (10-bit),
|
||
`04-cdfupdate`, `05-mv`, `06-mfmv`, `22-svc-L1T2`, `22-svc-L2T1`, `22-svc-L2T2` (spatial layers
|
||
with scaled inter-layer prediction), `24-monochrome` (8 and 10-bit) and
|
||
`16-intra_only-intrabc-extreme-dv` (1080p intra block copy), `23-film_grain-50` (8 and 10-bit).
|
||
|
||
`tests/oracle_inter.rs` also checks draft mode: every unflagged frame stays bit-exact with
|
||
libdav1d and only flagged frames differ, with 1 and 4 threads. `tests/oracle_inter.rs` encodes SVT-AV1 GOPs (hierarchical references, compound prediction,
|
||
OBMC / warped motion, motion-field projection, all loop filters; presets 3–8, 8- and 10-bit, odd
|
||
sizes), plus super-resolution (denominator 12) and film-grain streams, and compares every frame
|
||
with libdav1d: **bit-exact** on all 6 fixtures.
|
||
|
||
## Threading and performance
|
||
|
||
`Decoder::new()` uses all cores (`Decoder::with_threads(1)` decodes on the calling thread; on
|
||
wasm32, or without the default `threads` feature, everything runs on the calling thread). Headers are parsed in order; each frame is
|
||
then decoded by a frame worker once the reference frames it reads are finished, so independent
|
||
frames (hierarchical GOP layers) decode concurrently; tiles of a frame decode in parallel into
|
||
private buffers that are merged afterwards; the in-loop filters (deblocking, CDEF, loop
|
||
restoration) each run in parallel bands of 64 rows (`postfilter.rs` explains why the bands are
|
||
independent). Results never depend on scheduling (the oracle tests compare 1 thread, many threads
|
||
and libdav1d). With frame threads, pictures may be returned by a
|
||
later `decode_pts` call or by `flush`; each carries the `pts` of the temporal unit that showed it.
|
||
|
||
`tests/perf.rs` (ignored) prints single-threaded fps (thread CPU time), per-stage ms/frame
|
||
(`Decoder::stats()`) and all-core fps with its per-stage ms/frame for 1080p fixtures (key frames,
|
||
GOPs, tiles, low delay) and a single-tile 2160p GOP; `AV1_PERF_ONLY=a,b` picks fixtures. The hot loops (sub-pixel filters,
|
||
CDEF, inverse transforms, deblocking, entropy decoding) are written for auto-vectorisation.
|
||
|
||
Measured 2026-10-01 on a 14-core Apple Silicon Mac (release build, SVT-AV1 preset 8 1080p
|
||
fixtures from `tests/perf.rs`), before (`31f8b3d`: hot loops only, no threading) and after tile / frame /
|
||
post-filter threading. The machine was heavily loaded by other builds during both runs (load
|
||
average 200–400), so the all-core wall figures understate what an idle machine reaches and are
|
||
noisy; single-threaded figures use thread CPU time and are comparable.
|
||
|
||
| Fixture | 1 thread before | 1 thread after | 14 threads before | 14 threads after |
|
||
|---|---|---|---|---|
|
||
| `perf_1080p_intra` (10 key frames) | 43.3 fps | 43.0 fps | 6.5 fps | 20.8 fps |
|
||
| `perf_1080p_gop` (60 frames, crf 30) | 45.5 fps | 40.7 fps | 7.7 fps | 11.0 fps |
|
||
| `perf_1080p_gop_10bit` | 41.3 fps | 35.9 fps | 8.0 fps | 6.6 fps |
|
||
| `perf_1080p_gop_hq` (crf 18) | 19.5 fps | 19.4 fps | 2.4 fps | 3.6 fps |
|
||
| `perf_1080p_gop_tiles` (4x2 tiles) | 44.4 fps | 44.7 fps | 2.7 fps | 16.3 fps |
|
||
|
||
### M4.10 (2026-10-03)
|
||
|
||
CPU cycles per frame counted by the kernel (`/usr/bin/time -l`, all threads summed; mean of two
|
||
interleaved base / after rounds at load average 190-320) with
|
||
`AV1_THREADS=n cargo run --release -p filmcraft-av1 --example av1dec -- in.ivf` on the
|
||
`cargo xtask bench` decode clips (SVT-AV1 preset 10, CRF 35, keyint 48, testsrc2 + grain),
|
||
bit-exact with libdav1d:
|
||
|
||
| stream | threads | Mcycles / frame before | after | after, draft mode |
|
||
|---|---|---|---|---|
|
||
| 1080p (120 frames) | 1 | 60.5 | **48.3** (−20 %) | 46.2 |
|
||
| 1080p | 14 | 63.6 | **51.9** (−18 %) | 50.3 |
|
||
| 2160p (72 frames) | 1 | 212.5 | **172.4** (−19 %) | 167.9 |
|
||
| 2160p | 14 | 220.0 | **177.1** (−20 %) | 171.1 |
|
||
|
||
What changed: the sub-pixel filter passes run over fixed block widths (taps outer, columns
|
||
inner); frame and tile-region planes come from a bounded pool of recycled buffers (zeroed on
|
||
reuse) instead of a fresh `calloc` and `munmap` per frame (allocation, page faults and zeroing
|
||
were ~10 % of the time); the palette mode-info arrays (40 bytes per 4x4 unit) are only
|
||
allocated when screen content tools are allowed; CDF adaptation is branch-free (a branch-free
|
||
symbol search was tried and was slower: most symbols have 2-4 values). Single-threaded 4K now
|
||
splits into coefficient parsing ~20 %, inter prediction ~18 %, other block-level parsing
|
||
~12 %, deblocking ~11 % (still one edge at a time), inverse transforms ~12 %, CDEF ~5 %.
|
||
|
||
Draft mode (`Decoder::set_draft(true)`, `AV1_DRAFT=1`; reduced-resolution playback only): shown
|
||
frames whose `refresh_frame_flags` is 0 — no reference slot keeps them, so neither their samples,
|
||
motion vectors nor CDFs reach another frame — skip deblocking, CDEF and loop restoration
|
||
(super-resolution still runs, it sets the output size) and are flagged `Picture::draft`. In
|
||
SVT-AV1's hierarchical GOPs that is the top layer, half of the frames.
|
||
|
||
## API
|
||
|
||
```rust
|
||
let mut dec = filmcraft_av1::Decoder::new();
|
||
for sample in samples { // one temporal unit (MP4 / Matroska sample) each
|
||
for pic in dec.decode_pts(&sample, pts)? { // shown frames ready so far, planar u16
|
||
// pic.width, pic.height, pic.bit_depth, pic.planes[0..3], pic.pts
|
||
}
|
||
}
|
||
let rest = dec.flush(); // pictures still in flight at the end
|
||
```
|