filmcraft/crates/hevc/README.md

# filmcraft-hevc

Clean-room, pure-Rust (no `unsafe`) H.265 / HEVC decoder, implemented from the public ITU-T
Rec. H.265 (08/2021, ISO/IEC 23008-2) specification. It is an L0 crate: it depends only on
`filmcraft-bitstream`, `thiserror`, and optionally `rayon` (default feature `threads`), and builds
for `wasm32-unknown-unknown`.

All syntax tables (the CABAC `initValue` tables arranged per `initType` following Table 9-4, default
scaling lists, `intraPredAngle` / `invAngle`, the chroma QP table, the deblocking beta'/tC' table,
the luma / chroma interpolation filter taps and the 32x32 DCT matrix) were extracted from the
specification text by [`tools/extract_tables.py`](tools/extract_tables.py) (it only reads the text
of the ITU-T PDF, see the script header); the output is `src/spec_tables.rs`. Scan orders are
derived by the algorithms of 6.5.3-6.5.5. No code was taken or ported from FFmpeg, libde265, x265 or
the HM reference software; ffmpeg / libx265 / VideoToolbox are only used as external test oracles
and fixture generators.

## Status

| Area | Status |
|---|---|
| NAL / Annex-B / `hvcC` length-prefixed input | done |
| VPS, SPS (VUI, HRD skipped, scaling lists, st/lt RPS, PCM), PPS (tiles, WPP, deblocking control, scaling lists, list modification) | done |
| Slice segment header (all fields: st/lt RPS, list modification, pred weight table, entry points, header extension) | done |
| POC, RPS marking, missing-reference generation, RefPicList construction, DPB output / bumping (C.5.2) | done |
| CABAC: all Main / Main 10 syntax elements, WPP context sync, dependent slice context restore, tiles | done |
| Coding quadtree, CU (skip / intra / inter, AMP, PCM, transquant bypass), PU, transform tree | done |
| Intra: 35 modes, reference substitution, [1 2 1] filtering, strong intra smoothing, constrained intra | done |
| Inter: merge (spatial, temporal, combined bi-pred, zero), AMVP, TMVP, 8-tap / 4-tap interpolation, default and explicit weighted prediction | done |
| Scaling (flat and scaling lists), DCT 4..32, DST 4x4, transform skip, sign data hiding | done |
| Deblocking (bS, slice / tile boundary rules, PCM / bypass exemption), SAO (band / edge, boundary rules) | done |
| Frame-level multithreading (CTB-row granularity dependencies) | done |
| Draft mode (non-reference pictures skip deblocking / SAO; reduced-resolution playback only) | done |
| Tiles / WPP | parsed and decoded sequentially within a picture (parallelism is across pictures) |
| 4:2:2 / 4:4:4 / monochrome, range-extension tools (RExt), 12-bit+ with extended precision | not yet (`Error::Unsupported`) |
| SCC, multilayer (SHVC / MV-HEVC), 3D extensions | not supported (layers > 0 are ignored) |

## API

```rust
use filmcraft_hevc::{Decoder, Plane};

let mut dec = Decoder::new(); // or Decoder::with_threads(n), Decoder::from_hvcc(&hvcc)?
for (pts, access_unit) in access_units {
    for pic in dec.decode(access_unit, pts)? {
        // pic.width / pic.height (cropped), pic.bit_depth,
        // pic.y / pic.u / pic.v: Plane::U8(Vec<u8>) for 8-bit, Plane::U16(Vec<u16>) above,
        // pic.y_stride / pic.uv_stride, pic.pts, pic.poc, pic.key (IRAP), pic.color (VUI), pic.sar
    }
}
for pic in dec.flush() { /* ... */ }
if let Some(err) = dec.take_error() { /* error reported by a decoding thread */ }
// dec.set_draft(true): draft mode (playback at reduced resolution only)
```

`decode` expects whole access units (a buffer with several complete access units — e.g. an entire
Annex-B file — also works). RASL pictures following a CRA that starts the stream are skipped, as
ffmpeg does. `Decoder::stats()` returns counters of what was decoded (slice types, IDR/CRA, tiles,
WPP, weighted prediction, TMVP, SAO, block kinds...).

### Architecture

Same design as `filmcraft-h264`: the calling thread parses parameter sets and slice headers and does
POC / RPS / DPB bookkeeping; the CTU layer of each picture runs as a job (rayon pool). A job
reconstructs into private planes, deblocks CTB row r once row r+1 is decoded, applies SAO to row r
once row r+1 is deblocked, and publishes final rows (samples + 16x16 compressed motion for TMVP)
into the shared `Frame` (`OnceLock` per CTB row). Motion compensation of later pictures blocks per
row on exactly the reference rows it reads. Samples are stored as `u8` for 8-bit pictures (half
the memory traffic in prediction reads, in-loop filters and output) and as `u16` above 8 bits: the
slice decoder and the filters are generic over the sample type (`picture::Sample`), monomorphised
once per type.

Unavailable references keep distinct frame IDs and POCs while sharing one immutable gray backing
for the active picture format. This backing is created only when a reference is missing, replaced
on geometry or plane-depth changes, and released on flush. Jobs still reading an older format keep
its rows alive. Each backing retains the full coded picture dimensions.

Draft mode (`Decoder::set_draft(true)`; FilmCraft enables it with Settings ▸ Playback ▸ Draft
decoding while the Program monitor plays at 1/2 or 1/4): sub-layer non-reference pictures
(TRAIL_N, RASL_N, ...) of the highest temporal sub-layer — no later picture predicts from them,
and TMVP reads only reference pictures — skip deblocking and SAO and are flagged
`Picture::draft`; every other picture is unchanged. libx265's non-reference B pictures are such
pictures.

Modules: `params` (VPS/SPS/PPS/tile layout), `slice` (NAL + slice header), `dpb` (POC, RPS, lists,
bumping), `cabac`, `slicedec` (CTU syntax + reconstruction), `mvpred` (merge / AMVP / TMVP),
`intra`, `inter`, `transform`, `filter` (deblocking, SAO, row publication).

## Tests

`cargo test -p filmcraft-hevc` (fixtures are generated on first use into `target/fixtures/hevc/`
with ffmpeg; tests print a message and skip when ffmpeg is absent).

- Unit tests: CABAC engine round trip against an encoder model and context initialisation, scan
  orders, DCT matrix properties, QpC, intra predictors, transform DC / bounded-region shortcuts,
  MV scaling.
- Synthetic streams (`src/synth_tests.rs`): a small CABAC encoder writes streams with what libx265
  never produces — tiles (uniform and explicit spacing, loop filter across tiles on / off), PCM
  (with and without in-loop filtering), multiple slices, dependent slice segments (starting inside
  and at the start of a tile), long-term reference pictures, reference list modification. Expected
  output is known exactly where filters leave PCM samples alone; every stream is also compared with
  ffmpeg.
- Robustness: hvcC input with pts round trip; randomly corrupted, truncated and garbage input never
  panics or deadlocks (single- and multi-threaded).
- Conformance (`tests/it/conformance.rs`): every fixture is decoded with 1 thread, 3 threads and every
  core and compared sample-exactly with `ffmpeg -f rawvideo -pix_fmt yuv420p / yuv420p10le`; on mismatch
  the first differing frame, plane, sample, CTB and 8x8 block are reported. Output pts/POC order is
  checked too, and that no picture is flagged draft without draft mode. `compare_detects_mismatch`
  guards the harness itself. Draft mode (`draft_mode_changes_only_flagged_non_reference_pictures`):
  on B-pyramid / weighted fixtures, single- and frame-threaded, every unflagged picture is
  bit-exact and only flagged pictures differ.

### Fixture matrix (all bit-exact, single- and multi-threaded)

libx265 unless noted (x265 enables WPP by default, so most streams carry entry points).

| fixture | source / size | configuration |
|---|---|---|
| intra_main / intra_main10 | testsrc2+noise 352x288 | keyint=1, 8 / 10-bit |
| intra_nosao_nodbk | testsrc2+noise 352x288 | intra, no SAO, no deblocking |
| ultrafast / veryfast / medium / slow | 352x288 | presets (slow: rect + AMP) |
| medium_main10 / slow_main10 | 352x288 | Main 10 presets |
| p_only | 352x288 | bframes=0 |
| bframes | 352x288, 30 frames | bframes=8, b-pyramid, ref=4 |
| no_sao / no_deblock / deblock_m3_3 | 352x288 | loop filter variants |
| no_wpp | 352x288 | WPP off |
| rect_amp / min_cu16 | mandelbrot / testsrc2 | rect + AMP, min CU 16 |
| tskip | 352x288 | transform skip |
| weightp / weightp_main10 | fade-in | weightp + weightb |
| crf0 / crf51 / qp1_main10 | | quantizer extremes |
| lossless / cu_lossless | 176x144 | cu_transquant_bypass |
| slices4 | 352x288 | 4 slices per picture |
| scaling_list | 352x288 | default scaling lists |
| ctu16 / ctu32_tu4 | 352x288 | CTB 16, CTB 32 with deep TU trees, max TU 16 |
| no_signhide / constrained_intra / no_tmvp / no_strong_intra / max_merge1 | 352x288 | tool toggles |
| open_gop | 352x288, 40 frames | CRA + RASL / RADL, keyint 12 |
| keyint5_idr | 352x288 | closed GOP, IDR every 5 |
| odd_1918x1078 | 1918x1078 | conformance-window cropping |
| hd_1080p | 1920x1080 | preset fast |
| uhd_4k_main10 | 3840x2160 | Main 10 ultrafast |
| vt_main / vt_main10 | 640x360 | Apple VideoToolbox (hevc_videotoolbox), macOS only |
| synthetic (7 streams) | 96x64 | tiles, PCM, slices, dependent segments, LTR, list modification |

`cargo test -p filmcraft-hevc --test it conformance::coverage_report -- --ignored --nocapture` prints the
decoder statistics per fixture.

## Performance

`cargo test --release -p filmcraft-hevc --test it perf:: -- --ignored --nocapture` (bit-exactness is
verified first). Apple M4 Pro (14 cores), measured while the machine was heavily shared (load
average 50-100), so these are lower bounds:

| stream | 1 thread | 14 threads |
|---|---|---|
| 1080p Main, x265 medium CRF 24, 60 frames | ~75 fps | ~225 fps |
| 2160p Main 10, x265 fast CRF 26, 20 frames | ~20 fps | ~95 fps |

No intrinsics: the hot loops are written to auto-vectorise. M4.8
profiling (`cargo xtask bench`, macOS `sample`) found the inverse transform at over half of the
decoder's CPU time: it evaluated the full matrix product with strided table reads. It now sums
contiguous basis rows scaled by the non-zero coefficients only, with the block size a compile-time
constant so the loops vectorise (exact integer sums, so bit-exact; `transform::tests` checks it
against the direct product on random sparse and dense blocks). SAO edge offset has a check-free
loop for samples whose neighbours are inside the CTB. Together: ~60 % less CPU per frame
(1080p 78 → 33 ms, 2160p 320 → 129 ms of CPU per frame through the media stack, alternating A/B), see
[docs/performance.md](../../docs/performance.md).
`cargo run --release -p filmcraft-hevc --example hevcdec -- in.hevc out.yuv` decodes a file
(`HEVC_THREADS=n`, `HEVC_LOOPS=k`, `HEVC_DRAFT=1`).

M4.10 (2026-10-03), CPU cycles per frame counted by the kernel (`/usr/bin/time -l`, all threads
summed; mean of two interleaved base / after rounds at load average 190-320) on the
`cargo xtask bench` decode clips (x265 preset fast, CRF 22, testsrc2 + grain), bit-exact:

| stream | threads | Mcycles / frame before | after | after, draft mode |
|---|---|---|---|---|
| 1080p (120 frames) | 1 | 90.9 | **67.3** (−26 %) | 61.2 |
| 1080p | 14 | 100.2 | **75.1** (−25 %) | 67.3 |
| 2160p (72 frames) | 1 | 362.6 | **262.3** (−28 %) | 232.5 |
| 2160p | 14 | 398.3 | **284.8** (−28 %) | 260.1 |

What changed: motion compensation zeroed two 10 KB windows per prediction block per component
and list (11 % of the time in `memset`) and filtered with dynamic-width scalar sums; it now keeps
reusable scratch windows and runs taps outer / outputs inner over fixed block widths. SAO
looked its offset up per sample in a table (20 % of the time); the edge / band offset is now
selected per lane from the neighbour-sign sum / band index, so the runs vectorise. Uni / bi
output clips with min / max instead of `clamp` (whose bound assert kept it scalar). Runs of
bypass bins (signs, Rice suffixes, SAO / intra fields) decode with one division, and
context-coded bins are branch-free. Single-threaded 4K now splits into CABAC residual parsing
~28 %, reference window copies ~11 %, interpolation ~11 %, deblocking + SAO + row publication
~15 %, inverse transform ~8 %.

## Known gaps

- 4:2:2 (Main 4:2:2 10, RExt) is the next step: chroma geometry is 4:2:0-specific in `slicedec`
  (chroma TU placement, second chroma cbf flags, Table 8-3 intra mode mapping), `inter` (vertical
  chroma MV scaling), `filter` (chroma edge spacing, QpC = Min(qPi, 51)) and `picture`. Range
  extension tools (implicit / explicit RDPCM, cross-component prediction, extended precision,
  persistent Rice adaptation, bypass alignment, chroma QP offset lists) are rejected with
  `Error::Unsupported`.
- Tiles and WPP substreams are decoded sequentially (entry points are parsed but the decoder finds
  substream starts itself); parallelism is at the picture level.
- Error concealment is minimal: missing references are replaced by grey pictures, undecodable
  slices leave grey / partial content.
- No fixture covers the `pic_output_flag = 0` path or decoding-order-only CRA handling after EOS
  beyond the synthetic streams; libx265 never emits PCM, tiles or long-term references (covered by
  the synthetic streams only).