CI: xtask ci in the dev profile, one test binary per crate, nextest (#2860)

* Tests: one integration-test binary per crate

Each crate with several tests/*.rs files keeps them as modules of tests/it/,
so cargo links one test executable per crate instead of one per file. Files
that change process-wide state or time a workload stay binaries of their own.

* Profiles: line tables for workspace crates, no debug info for dependencies

* Windows ARM64: nightly and on release pushes instead of every push to main

* xtask ci: clippy and tests in the dev profile

The release profile's thin LTO and single codegen unit made linking the test
binaries take most of the gate. The codec and audio crates stay at opt-level 3
in the dev profile, so their tests run at release speed.

* xtask ci: nextest, doctests, no examples, one wasm check

Tests run with cargo nextest when it is installed (cargo test otherwise) and
build with --lib --bins --tests, so examples aren't linked; doctests run
separately. The wasm check keeps going past the first failing crate. VP8
fixture generation locks across processes, since nextest runs each test in
its own.

* VP9 synthetic fixtures: write aside and rename

A test in another process could read a fixture ffmpeg was still writing.

* Fuzz tests: a third of each sweep by default, all of it with FILMCRAFT_FUZZ_ITERS

Every seed, mutation kind and thread count still runs. Header bytes each get
one overwrite value and two bit flips in rotation, so every value and bit
index still occurs across the region. FILMCRAFT_FUZZ_ITERS=N runs the whole
sweep plus N random rounds per seed, as before.

* Platform fallback tests: one failure point per hybrid path

The failure-anywhere test injected a failure at four points around every
restart point; the hybrid's paths differ only in the first GOP, around the
second and last restart points, at the end and never. The replay test needs
the first two GOPs.

* Export tests: shorter ranges with the same checks

The bitrate test measures 0.5–1.5 s, where each preset reaches its own
target (on 0–3 s the simple demo left two presets at the same rate). The
every-preset test exports three frames per preset.

* Audio tests: effects in parallel, a table-driven FFT for the Opus quality metric

The extreme-settings test keeps both settings and the 40 s tail for every
effect, one thread each. The Opus quality metric's FFT builds its twiddles
once and allocates nothing per transform (same algorithm and summation order).

* CI: dev-profile gate with nextest

xtask ci no longer builds with --release, so the release-profile overrides
go; nextest is installed for the test step.

* CI: apt packages through the cached install-apt-packages action

* Review fixes: fuzz tests with a panic hook in their own binaries, loud fixture lock

The fuzz tests of av1, h264, hevc, mpeg2v, vp8 and vp9 count panics through
a process-wide hook, so a panic in another test of the same process would
fail them; they are binaries of their own again. The VP8 fixture lock fails
the test instead of skipping it silently, and the full-sweep doc comments say
how many truncation points the full run samples.

* Fuzz tests: rotate the default subset per run (FILMCRAFT_FUZZ_SEED replays one)

* platform tests: decode-switch tests in their own binaries, one NVENC lock
This commit is contained in:
aiongg
2026-10-23 13:34:16 -04:00
committed by GitHub
parent 20e159dd68
commit 0a881266c8
271 changed files with 952 additions and 373 deletions
+14
View File
@@ -0,0 +1,14 @@
# `cargo nextest run` (used by `cargo xtask ci`) runs every test in a process of its own, in parallel.
[test-groups]
# NVENC sessions are a limited resource on consumer GPUs.
nvenc = { max-threads = 1 }
[[profile.default.overrides]]
filter = "package(filmcraft-platform) and (binary(/nvenc/) or test(/nvenc/))"
test-group = "nvenc"
# A throughput gate: it measures one core's speed, so nothing else runs beside it.
[[profile.default.overrides]]
filter = "package(filmcraft-render) and binary(mixer_perf)"
threads-required = "num-test-threads"
+10 -10
View File
@@ -31,11 +31,6 @@ jobs:
CARGO_TERM_COLOR: always CARGO_TERM_COLOR: always
CARGO_INCREMENTAL: "0" CARGO_INCREMENTAL: "0"
RUST_BACKTRACE: "1" RUST_BACKTRACE: "1"
# `xtask ci` builds clippy and the tests with --release. The release profile (thin LTO, one
# codegen unit) is for shipped binaries: on a 4-core runner it makes linking the ~220 test
# binaries take over an hour. CI keeps the optimisation level but builds in parallel units.
CARGO_PROFILE_RELEASE_LTO: "false"
CARGO_PROFILE_RELEASE_CODEGEN_UNITS: "16"
steps: steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: with:
@@ -51,11 +46,11 @@ jobs:
# libasound2-dev is for cpal (audio playback). # libasound2-dev is for cpal (audio playback).
- name: Install Linux build dependencies - name: Install Linux build dependencies
run: | uses: storytold/craft-common-actions/actions/install-apt-packages@main
sudo apt-get update with:
sudo apt-get install -y --no-install-recommends \ packages: >-
pkg-config libxkbcommon-dev libwayland-dev libx11-dev libxcb1-dev libxcursor-dev \ pkg-config libxkbcommon-dev libwayland-dev libx11-dev libxcb1-dev libxcursor-dev
libxrandr-dev libxi-dev libfontconfig1-dev libfreetype-dev mesa-vulkan-drivers \ libxrandr-dev libxi-dev libfontconfig1-dev libfreetype-dev mesa-vulkan-drivers
libasound2-dev libasound2-dev
- name: Install Rust (stable + clippy, rustfmt, wasm32) - name: Install Rust (stable + clippy, rustfmt, wasm32)
@@ -64,6 +59,11 @@ jobs:
--target wasm32-unknown-unknown --target wasm32-unknown-unknown
rustup default stable rustup default stable
# `xtask ci` runs the tests with nextest when it is installed.
- uses: taiki-e/install-action@a4d3f3f761dcd95cce1b72cb61cb6d1963438177 # v2.87.30
with:
tool: cargo-nextest
- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with: with:
save-if: ${{ github.ref == 'refs/heads/main' }} save-if: ${{ github.ref == 'refs/heads/main' }}
+6 -7
View File
@@ -4,19 +4,18 @@
# out for now: PdfCraft saw an intermittent access violation in egui UI tests there.) # out for now: PdfCraft saw an intermittent access violation in egui UI tests there.)
name: Windows ARM64 name: Windows ARM64
# The package build takes about 15 minutes, so it runs nightly on main and on pushes to `release`
# rather than on every push to main: a dependency or runner-image change still shows up within a
# day. PRs that change the Windows packaging itself still run it.
on: on:
schedule:
- cron: "41 4 * * *"
push: push:
branches: [main] branches: [release]
pull_request: pull_request:
paths: paths:
- Cargo.toml
- Cargo.lock
- "**/Cargo.toml"
- "**/build.rs"
- .github/workflows/windows-arm64.yml - .github/workflows/windows-arm64.yml
- packaging/windows/** - packaging/windows/**
schedule:
- cron: "41 4 * * 1" # weekly, so drift in the runner images shows up
workflow_dispatch: workflow_dispatch:
concurrency: concurrency:
+4
View File
@@ -92,11 +92,15 @@ too_many_arguments = "allow"
needless_range_loop = "allow" needless_range_loop = "allow"
type_complexity = "allow" type_complexity = "allow"
# Line tables keep file and line in panics and backtraces at a fraction of full debug info's size;
# dependencies need none.
[profile.dev.package."*"] [profile.dev.package."*"]
opt-level = 2 opt-level = 2
debug = false
[profile.dev] [profile.dev]
opt-level = 1 opt-level = 1
debug = "line-tables-only"
# Codec hot loops: debug-build tests (RFC/oracle vectors) are minutes slower below opt-level 3. # Codec hot loops: debug-build tests (RFC/oracle vectors) are minutes slower below opt-level 3.
[profile.dev.package.filmcraft-h264] [profile.dev.package.filmcraft-h264]
+2
View File
@@ -0,0 +1,2 @@
mod cli;
mod workflows;
+1 -1
View File
@@ -1,4 +1,4 @@
//! Spacing of browser downloads (pure, so it is tested natively in `tests/download_pacing.rs`; the //! Spacing of browser downloads (pure, so it is tested natively in `tests/it/download_pacing.rs`; the
//! anchor clicks live in [`crate::fs::download`]). //! anchor clicks live in [`crate::fs::download`]).
//! //!
//! Chrome drops anchor-click downloads that start in one tight burst, so an image-sequence export //! Chrome drops anchor-click downloads that start in one tight burst, so an image-sequence export
+1 -1
View File
@@ -1,5 +1,5 @@
//! When the web recovery snapshot is written or marked clean (pure, so it is tested natively in //! When the web recovery snapshot is written or marked clean (pure, so it is tested natively in
//! `tests/recovery_policy.rs`; the OPFS side lives in [`crate::recovery`]). //! `tests/it/recovery_policy.rs`; the OPFS side lives in [`crate::recovery`]).
//! //!
//! - A dirty project is snapshotted at most every [`INTERVAL_S`] seconds, once per revision. A write //! - A dirty project is snapshotted at most every [`INTERVAL_S`] seconds, once per revision. A write
//! that fails is retried after the interval, without waiting for another edit (#197). //! that fails is retried after the interval, without waiting for another edit (#197).
@@ -1,10 +1,10 @@
//! Download spacing. It is pure, so it runs natively even though the rest of the crate is //! Download spacing. It is pure, so it runs natively even though the rest of the crate is
//! wasm32-only. //! wasm32-only.
#[path = "../src/download_pacing.rs"] #[path = "../../src/download_pacing.rs"]
mod download_pacing; mod pacing;
use download_pacing::{GAP_MS, Pacer}; use pacing::{GAP_MS, Pacer};
/// #378: 20 downloads requested in the same instant are spread out, not clicked in one burst. /// #378: 20 downloads requested in the same instant are spread out, not clicked in one burst.
#[test] #[test]
+2
View File
@@ -0,0 +1,2 @@
mod download_pacing;
mod recovery_policy;
@@ -1,10 +1,10 @@
//! The web recovery snapshot policy. It is pure, so it runs natively even though the rest of the //! The web recovery snapshot policy. It is pure, so it runs natively even though the rest of the
//! crate is wasm32-only. //! crate is wasm32-only.
#[path = "../src/recovery_policy.rs"] #[path = "../../src/recovery_policy.rs"]
mod recovery_policy; mod policy;
use recovery_policy::{Action, INTERVAL_S, Policy}; use policy::{Action, INTERVAL_S, Policy};
/// #197: a snapshot write that fails is retried after the interval without another edit. /// #197: a snapshot write that fails is retried after the interval without another edit.
#[test] #[test]
@@ -1,7 +1,7 @@
//! Decoder accuracy against ffmpeg's decoder on streams produced by ffmpeg's AAC encoder (oracle //! Decoder accuracy against ffmpeg's decoder on streams produced by ffmpeg's AAC encoder (oracle
//! only; skipped when ffmpeg is absent). //! only; skipped when ffmpeg is absent).
mod common; use crate::common;
use common::*; use common::*;
@@ -1,6 +1,6 @@
//! Encoder quality/bitrate tests using ffmpeg as an external decoding oracle (skipped if absent). //! Encoder quality/bitrate tests using ffmpeg as an external decoding oracle (skipped if absent).
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_aac::{BitrateMode, EncoderConfig, WindowShape}; use filmcraft_aac::{BitrateMode, EncoderConfig, WindowShape};
+4
View File
@@ -0,0 +1,4 @@
mod common;
mod decoder_oracle;
mod encoder_oracle;
mod robustness;
@@ -1,7 +1,7 @@
//! Fuzz-style robustness: mutated access units, configs and ADTS streams must never panic, and the //! Fuzz-style robustness: mutated access units, configs and ADTS streams must never panic, and the
//! encoder must accept hostile input (NaN, ±inf, > full scale, silence, odd chunk sizes). //! encoder must accept hostile input (NaN, ±inf, > full scale, silence, odd chunk sizes).
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_aac::{AudioSpecificConfig, Decoder, Encoder, EncoderConfig, split_adts}; use filmcraft_aac::{AudioSpecificConfig, Decoder, Encoder, EncoderConfig, split_adts};
+2 -2
View File
@@ -54,7 +54,7 @@ let apvc: Vec<u8> = enc.decoder_config_record(); // APVDecoderConfigurationReco
- **Metadata (§8)**: Parses `metadata_mdcv` (`payloadType == 5`, Mastering Display Color Volume) - **Metadata (§8)**: Parses `metadata_mdcv` (`payloadType == 5`, Mastering Display Color Volume)
and `metadata_cll` (`payloadType == 6`, Content Light-Level Information). and `metadata_cll` (`payloadType == 6`, Content Light-Level Information).
- **Robustness (§10)**: All size fields, tile coordinates, VLC codeword lengths, and coefficient - **Robustness (§10)**: All size fields, tile coordinates, VLC codeword lengths, and coefficient
ranges (`-32768..=32767`) are checked before allocation or shift. Verified by `tests/fuzz.rs` ranges (`-32768..=32767`) are checked before allocation or shift. Verified by `tests/it/fuzz.rs`
across thousands of mutated and random inputs. across thousands of mutated and random inputs.
### Encoder ### Encoder
@@ -67,7 +67,7 @@ let apvc: Vec<u8> = enc.decoder_config_record(); // APVDecoderConfigurationReco
## Accuracy (vs FFmpeg as an External Oracle) ## Accuracy (vs FFmpeg as an External Oracle)
`tests/oracle.rs` encodes synthetic test patterns across all 7 profiles, odd dimensions, multiple `tests/it/oracle.rs` encodes synthetic test patterns across all 7 profiles, odd dimensions, multiple
tiles, custom asymmetric quantization matrices, and HDR metadata PBUs, then decodes the resulting tiles, custom asymmetric quantization matrices, and HDR metadata PBUs, then decodes the resulting
`.apv` streams with `ffmpeg -v error -xerror -f apv` and compares against `filmcraft_apv::decode_frame`. `.apv` streams with `ffmpeg -v error -xerror -f apv` and compares against `filmcraft_apv::decode_frame`.
Because APV specifies an exact integer inverse transform, **`filmcraft-apv` is 100% bit-exact Because APV specifies an exact integer inverse transform, **`filmcraft-apv` is 100% bit-exact
+3
View File
@@ -0,0 +1,3 @@
mod fuzz;
mod fuzz_hostile;
mod oracle;
+30 -24
View File
@@ -271,31 +271,37 @@ mod tests {
#[test] #[test]
fn extreme_settings_stay_finite_and_denormal_free() { fn extreme_settings_stay_finite_and_denormal_free() {
for e in effects() { // One thread per effect. The tail stays 40 s long: delays, reverbs and graphic EQs still
for setting in [0.0f32, 1.0] { // produce output in its last second, some of it within 3x of the smallest normal f32.
let mut fx = (e.create)(SR, 2); std::thread::scope(|scope| {
for p in e.params { for e in effects() {
fx.set_param(p.id, if setting == 0.0 { p.min } else { p.max }); scope.spawn(move || {
} for setting in [0.0f32, 1.0] {
let mut l = noise(3, 24000, 4.0); let mut fx = (e.create)(SR, 2);
let mut r = noise(4, 24000, 4.0); for p in e.params {
l[100] = 1e6; fx.set_param(p.id, if setting == 0.0 { p.min } else { p.max });
r[200] = -1e6; }
fx.process(&mut [&mut l, &mut r]); let mut l = noise(3, 24000, 4.0);
assert!(l.iter().chain(&r).all(|v| v.is_finite()), "{} setting {setting}", e.id); let mut r = noise(4, 24000, 4.0);
// Long silence afterwards: output decays without denormals. l[100] = 1e6;
fx.reset(); r[200] = -1e6;
let mut l = noise(3, 4800, 0.5); fx.process(&mut [&mut l, &mut r]);
let mut r = noise(4, 4800, 0.5); assert!(l.iter().chain(&r).all(|v| v.is_finite()), "{} setting {setting}", e.id);
fx.process(&mut [&mut l, &mut r]); // Long silence afterwards: output decays without denormals.
for _ in 0..40 { fx.reset();
let mut l = vec![0.0f32; 48000]; let mut l = noise(3, 4800, 0.5);
let mut r = vec![0.0f32; 48000]; let mut r = noise(4, 4800, 0.5);
fx.process(&mut [&mut l, &mut r]); fx.process(&mut [&mut l, &mut r]);
assert!(l.iter().chain(&r).all(|v| v.is_finite() && !v.is_subnormal()), "{} setting {setting}: denormal/NaN in tail", e.id); for _ in 0..40 {
} let mut l = vec![0.0f32; 48000];
let mut r = vec![0.0f32; 48000];
fx.process(&mut [&mut l, &mut r]);
assert!(l.iter().chain(&r).all(|v| v.is_finite() && !v.is_subnormal()), "{} setting {setting}: denormal/NaN in tail", e.id);
}
}
});
} }
} });
} }
#[test] #[test]
+6 -6
View File
@@ -43,12 +43,12 @@ reference) is used only as an external test oracle.
## Accuracy ## Accuracy
`tests/oracle_intra.rs` encodes all-intra streams with ffmpeg's libsvtav1 (with and without the `tests/it/oracle_intra.rs` encodes all-intra streams with ffmpeg's libsvtav1 (with and without the
in-loop filters; 64×64 and 128×128 superblocks; intra edge filter on and off) and compares every in-loop filters; 64×64 and 128×128 superblocks; intra edge filter on and off) and compares every
decoded frame with libdav1d: **bit-exact** on all 13 fixtures (8- and 10-bit 4:2:0; 128×128 to decoded frame with libdav1d: **bit-exact** on all 13 fixtures (8- and 10-bit 4:2:0; 128×128 to
1280×720 including odd sizes; testsrc2, mandelbrot, gradients and noise; presets 1–8). 1280×720 including odd sizes; testsrc2, mandelbrot, gradients and noise; presets 1–8).
`tests/conformance.rs` downloads libaom's conformance test vectors on first use `tests/it/conformance.rs` downloads libaom's conformance test vectors on first use
(storage.googleapis.com/aom-test-data) and compares with libdav1d: `av1-1-b8-02-allintra` (storage.googleapis.com/aom-test-data) and compares with libdav1d: `av1-1-b8-02-allintra`
(39 frames, deblocking + CDEF + self-guided restoration), `05-mv`, `06-mfmv`, `24-monochrome` (39 frames, deblocking + CDEF + self-guided restoration), `05-mv`, `06-mfmv`, `24-monochrome`
and `b10-23-film_grain-50` are **bit-exact** in the default run. The ignored and `b10-23-film_grain-50` are **bit-exact** in the default run. The ignored
@@ -58,8 +58,8 @@ and `b10-23-film_grain-50` are **bit-exact** in the default run. The ignored
with scaled inter-layer prediction), `24-monochrome` (8 and 10-bit) and with scaled inter-layer prediction), `24-monochrome` (8 and 10-bit) and
`16-intra_only-intrabc-extreme-dv` (1080p intra block copy), `23-film_grain-50` (8 and 10-bit). `16-intra_only-intrabc-extreme-dv` (1080p intra block copy), `23-film_grain-50` (8 and 10-bit).
`tests/oracle_inter.rs` also checks draft mode: every unflagged frame stays bit-exact with `tests/it/oracle_inter.rs` also checks draft mode: every unflagged frame stays bit-exact with
libdav1d and only flagged frames differ, with 1 and 4 threads. `tests/oracle_inter.rs` encodes SVT-AV1 GOPs (hierarchical references, compound prediction, libdav1d and only flagged frames differ, with 1 and 4 threads. `tests/it/oracle_inter.rs` encodes SVT-AV1 GOPs (hierarchical references, compound prediction,
OBMC / warped motion, motion-field projection, all loop filters; presets 3–8, 8- and 10-bit, odd OBMC / warped motion, motion-field projection, all loop filters; presets 3–8, 8- and 10-bit, odd
sizes), plus super-resolution (denominator 12) and film-grain streams, and compares every frame sizes), plus super-resolution (denominator 12) and film-grain streams, and compares every frame
with libdav1d: **bit-exact** on all 6 fixtures. with libdav1d: **bit-exact** on all 6 fixtures.
@@ -76,13 +76,13 @@ independent). Results never depend on scheduling (the oracle tests compare 1 thr
and libdav1d). With frame threads, pictures may be returned by a and libdav1d). With frame threads, pictures may be returned by a
later `decode_pts` call or by `flush`; each carries the `pts` of the temporal unit that showed it. later `decode_pts` call or by `flush`; each carries the `pts` of the temporal unit that showed it.
`tests/perf.rs` (ignored) prints single-threaded fps (thread CPU time), per-stage ms/frame `tests/it/perf.rs` (ignored) prints single-threaded fps (thread CPU time), per-stage ms/frame
(`Decoder::stats()`) and all-core fps with its per-stage ms/frame for 1080p fixtures (key frames, (`Decoder::stats()`) and all-core fps with its per-stage ms/frame for 1080p fixtures (key frames,
GOPs, tiles, low delay) and a single-tile 2160p GOP; `AV1_PERF_ONLY=a,b` picks fixtures. The hot loops (sub-pixel filters, GOPs, tiles, low delay) and a single-tile 2160p GOP; `AV1_PERF_ONLY=a,b` picks fixtures. The hot loops (sub-pixel filters,
CDEF, inverse transforms, deblocking, entropy decoding) are written for auto-vectorisation. CDEF, inverse transforms, deblocking, entropy decoding) are written for auto-vectorisation.
Measured 2026-10-01 on a 14-core Apple Silicon Mac (release build, SVT-AV1 preset 8 1080p Measured 2026-10-01 on a 14-core Apple Silicon Mac (release build, SVT-AV1 preset 8 1080p
fixtures from `tests/perf.rs`), before (`31f8b3d`: hot loops only, no threading) and after tile / frame / fixtures from `tests/it/perf.rs`), before (`31f8b3d`: hot loops only, no threading) and after tile / frame /
post-filter threading. The machine was heavily loaded by other builds during both runs (load post-filter threading. The machine was heavily loaded by other builds during both runs (load
average 200–400), so the all-core wall figures understate what an idle machine reaches and are average 200–400), so the all-core wall figures understate what an idle machine reaches and are
noisy; single-threaded figures use thread CPU time and are comparable. noisy; single-threaded figures use thread CPU time and are comparable.
@@ -1,7 +1,7 @@
//! Compare an arbitrary IVF file with libdav1d: //! Compare an arbitrary IVF file with libdav1d:
//! `AV1_IVF=/path/x.ivf AV1_PIX=yuv420p cargo test -p filmcraft-av1 --test adhoc -- --ignored --nocapture` //! `AV1_IVF=/path/x.ivf AV1_PIX=yuv420p cargo test -p filmcraft-av1 --test it adhoc:: -- --ignored --nocapture`
mod common; use crate::common;
use common::*; use common::*;
#[test] #[test]
@@ -1,6 +1,6 @@
//! libaom conformance test vectors (downloaded on first use), compared with libdav1d. //! libaom conformance test vectors (downloaded on first use), compared with libdav1d.
mod common; use crate::common;
use common::*; use common::*;
/// Vectors the decoder handles today (more are added as stages land). /// Vectors the decoder handles today (more are added as stages land).
@@ -17,7 +17,7 @@ fn conformance_vectors() {
} }
/// The wider libaom set (slow: many downloads and decodes): /// The wider libaom set (slow: many downloads and decodes):
/// `cargo test -p filmcraft-av1 --test conformance -- --ignored`; `AV1_VECTORS=a,b` overrides it. /// `cargo test -p filmcraft-av1 --test it conformance:: -- --ignored`; `AV1_VECTORS=a,b` overrides it.
#[test] #[test]
#[ignore] #[ignore]
fn conformance_vectors_extended() { fn conformance_vectors_extended() {
+7
View File
@@ -0,0 +1,7 @@
mod adhoc;
mod common;
mod conformance;
mod hostile_dims;
mod oracle_inter;
mod oracle_intra;
mod perf;
@@ -2,7 +2,7 @@
//! GOPs with hierarchical references, compound prediction, OBMC / warped motion, global motion, //! GOPs with hierarchical references, compound prediction, OBMC / warped motion, global motion,
//! motion-field projection and all loop filters. //! motion-field projection and all loop filters.
mod common; use crate::common;
use common::*; use common::*;
fn specs() -> Vec<Spec> { fn specs() -> Vec<Spec> {
@@ -1,6 +1,6 @@
//! Intra-frame bit-exactness against libdav1d (run as an external ffmpeg process). //! Intra-frame bit-exactness against libdav1d (run as an external ffmpeg process).
mod common; use crate::common;
use common::*; use common::*;
const NO_FILTERS: &str = "enable-cdef=0:enable-restoration=0:enable-dlf=0:keyint=1"; const NO_FILTERS: &str = "enable-cdef=0:enable-restoration=0:enable-dlf=0:keyint=1";
@@ -1,11 +1,11 @@
//! Decode throughput and per-stage breakdown (ignored; run with //! Decode throughput and per-stage breakdown (ignored; run with
//! `cargo test --release -p filmcraft-av1 --test perf -- --ignored --nocapture`). //! `cargo test --release -p filmcraft-av1 --test it perf:: -- --ignored --nocapture`).
//! //!
//! The machine may be busy (other builds), so single-threaded work is also measured with the //! The machine may be busy (other builds), so single-threaded work is also measured with the
//! thread CPU clock, which is robust to load; wall-clock figures are printed next to the load //! thread CPU clock, which is robust to load; wall-clock figures are printed next to the load
//! average. //! average.
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_av1::{DecodeStats, Decoder, Stage}; use filmcraft_av1::{DecodeStats, Decoder, Stage};
use std::time::Instant; use std::time::Instant;
+1 -1
View File
@@ -85,7 +85,7 @@ implementation's code was consulted.
- Unit tests per module (timestamps, decoding, sloppy SRT, WebVTT blocks and voices, CEA-608 - Unit tests per module (timestamps, decoding, sloppy SRT, WebVTT blocks and voices, CEA-608
tables, PAC round trips, known SCC pop-on and roll-up streams, layout). tables, PAC round trips, known SCC pop-on and roll-up streams, layout).
- `tests/roundtrip.rs`: property tests — SRT (also with BOM + CRLF), WebVTT (ids, speakers, - `tests/it/roundtrip.rs`: property tests — SRT (also with BOM + CRLF), WebVTT (ids, speakers,
settings, STYLE blocks), SCC and MCC (drop-frame and non-drop, gaps and back-to-back captions; settings, STYLE blocks), SCC and MCC (drop-frame and non-drop, gaps and back-to-back captions;
MCC also through CEA-708 alone), EBU STL (25 and 29.97 fps, accents, extension blocks), TTML MCC also through CEA-708 alone), EBU STL (25 and 29.97 fps, accents, extension blocks), TTML
(23.976 / 25 / 29.97 / 59.94, ids, speakers) round-trip frame-exactly; DFXP round-trips after (23.976 / 25 / 29.97 / 59.94, ids, speakers) round-trip frame-exactly; DFXP round-trips after
+3
View File
@@ -0,0 +1,3 @@
mod roundtrip;
mod ttml_body_timing;
mod ttml_speakers;
@@ -3,7 +3,7 @@
//! both 7.1 (eight channels) and 11 is 6.1 (seven). Each channel carries its own tone, so a //! both 7.1 (eight channels) and 11 is 6.1 (seven). Each channel carries its own tone, so a
//! dropped or swapped channel shows up against ffmpeg's decode. //! dropped or swapped channel shows up against ffmpeg's decode.
mod common; use crate::common;
use std::io::Cursor; use std::io::Cursor;
@@ -2,7 +2,7 @@
//! off by default, approximate frames only reach draft requests, and decoding for an export after //! off by default, approximate frames only reach draft requests, and decoding for an export after
//! draft playback (same source, same GOP cache) still matches ffmpeg bit for bit. //! draft playback (same source, same GOP cache) still matches ffmpeg bit for bit.
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_media::FrameRequest; use filmcraft_media::FrameRequest;
@@ -2,7 +2,7 @@
//! and muxes the files and decodes them as the oracle. Zero-bit mantissas carry decoder-specific //! and muxes the files and decodes them as the oracle. Zero-bit mantissas carry decoder-specific
//! dither (A/52 §7.3.4), so the criterion is SNR against ffmpeg's decode, sample-aligned. //! dither (A/52 §7.3.4), so the criterion is SNR against ffmpeg's decode, sample-aligned.
mod common; use crate::common;
use common::*; use common::*;
@@ -2,7 +2,7 @@
//! MP4 without a `colr` box unless asked, so for most files the bitstream is the only place the //! MP4 without a `colr` box unless asked, so for most files the bitstream is the only place the
//! primaries are described: stream info already read them, the decoded frames reported BT.709. //! primaries are described: stream info already read them, the decoded frames reported BT.709.
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_color::{ColorInfo, ColorSpace, Matrix, Primaries, Range, Transfer}; use filmcraft_color::{ColorInfo, ColorSpace, Matrix, Primaries, Range, Transfer};
@@ -2,7 +2,7 @@
//! by default, its approximate frames only ever reach draft requests, and decoding for an export //! by default, its approximate frames only ever reach draft requests, and decoding for an export
//! after draft playback (same source, same GOP cache) still matches ffmpeg bit for bit. //! after draft playback (same source, same GOP cache) still matches ffmpeg bit for bit.
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_media::FrameRequest; use filmcraft_media::FrameRequest;
+17
View File
@@ -0,0 +1,17 @@
mod aac_channels_oracle;
mod adts_file;
mod common;
mod draft_decoders;
mod eac3_oracle;
mod frame_color_oracle;
mod h264_draft;
mod matroska_audio_oracle;
mod media_oracle;
mod mkv_mp1_oracle;
mod mp4_oracle;
mod mp4_rtmd;
mod mpeg_oracle;
mod mxf_oracle;
mod ogg_oracle;
mod rf64_metadata;
mod y4m_file;
@@ -4,7 +4,7 @@
//! reads it, a tone must stay as smooth as ffmpeg's decode of it: no holes or overlaps between //! reads it, a tone must stay as smooth as ffmpeg's decode of it: no holes or overlaps between
//! packets. //! packets.
mod common; use crate::common;
use common::*; use common::*;
@@ -4,7 +4,7 @@
//! exactly, JPEG within a few levels, high bit depths within one 8-bit step), and the BWF //! exactly, JPEG within a few levels, high bit depths within one 8-bit step), and the BWF
//! TimeReference must give the start timecode. //! TimeReference must give the start timecode.
mod common; use crate::common;
use std::path::Path; use std::path::Path;
use std::sync::Arc; use std::sync::Arc;
@@ -2,7 +2,7 @@
//! installed encoder writes Layer I, so the frames are synthetic (valid random Layer I frames, //! installed encoder writes Layer I, so the frames are synthetic (valid random Layer I frames,
//! as in `crates/mpa/tests/oracle.rs`) and ffmpeg only remuxes them into Matroska and decodes. //! as in `crates/mpa/tests/oracle.rs`) and ffmpeg only remuxes them into Matroska and decodes.
mod common; use crate::common;
use common::*; use common::*;
@@ -2,7 +2,7 @@
//! priming), and reads at another sample rate must land on the same source samples as the native //! priming), and reads at another sample rate must land on the same source samples as the native
//! decode, however the requests are cut. //! decode, however the requests are cut.
mod common; use crate::common;
use common::*; use common::*;
@@ -4,7 +4,7 @@
//! random seeks give the frames of the sequential decode. Audio: LPCM sample-exact, MPEG audio //! random seeks give the frames of the sequential decode. Audio: LPCM sample-exact, MPEG audio
//! and AAC within a float-decoder tolerance, starting at the same sample. //! and AAC within a float-decoder tolerance, starting at the same sample.
mod common; use crate::common;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@@ -2,7 +2,7 @@
//! oracle. Our decode must match frame for frame (H.264 bit-exact, VC-3 ±2, ProRes ±1 LSB), seek //! oracle. Our decode must match frame for frame (H.264 bit-exact, VC-3 ±2, ProRes ±1 LSB), seek
//! to random frames exactly, and give sample-exact PCM. //! to random frames exactly, and give sample-exact PCM.
mod common; use crate::common;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@@ -4,7 +4,7 @@
//! `filmcraft-opus`'s own oracle; Vorbis: SNR ≥ 60 dB), and any random window must equal the //! `filmcraft-opus`'s own oracle; Vorbis: SNR ≥ 60 dB), and any random window must equal the
//! continuous decode (granule-position seeking with pre-roll). //! continuous decode (granule-position seeking with pre-roll).
mod common; use crate::common;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
+1
View File
@@ -3,6 +3,7 @@
//! primed with buffers full of 0xAA before decoding, and every frame must still match ffmpeg bit //! primed with buffers full of 0xAA before decoding, and every frame must still match ffmpeg bit
//! for bit. (Its own process: the pool is global.) //! for bit. (Its own process: the pool is global.)
#[path = "it/common/mod.rs"]
mod common; mod common;
use std::sync::Arc; use std::sync::Arc;
+2 -2
View File
@@ -73,14 +73,14 @@ restart intervals that split a line, and MXF-wrapped CinemaDNG.
## Tests ## Tests
- `tests/roundtrip.rs`: container parsing in both byte orders with and without a preview IFD; - `tests/it/roundtrip.rs`: container parsing in both byte orders with and without a preview IFD;
uncompressed samples bit-exact at 8–16 bits, strips and tiles; lossless JPEG bit-exact for every uncompressed samples bit-exact at 8–16 bits, strips and tiles; lossless JPEG bit-exact for every
predictor and component count, full-swing 16-bit data (every difference category, SSSS = 16), predictor and component count, full-swing 16-bit data (every difference category, SSSS = 16),
restarts and point transform; developed values against hand-computed ones (black / white, restarts and point transform; developed values against hand-computed ones (black / white,
white balance, ForwardMatrix, BaselineExposure, ActiveArea, DefaultCrop, LinearRaw); CinemaDNG white balance, ForwardMatrix, BaselineExposure, ActiveArea, DefaultCrop, LinearRaw); CinemaDNG
FrameRate / TimeCodes / ReelName; hostile files (every truncation, 12 000 mutations under FrameRate / TimeCodes / ReelName; hostile files (every truncation, 12 000 mutations under
`catch_unwind`, cyclic IFDs, huge sizes, bad offsets, zero denominators). `catch_unwind`, cyclic IFDs, huge sizes, bad offsets, zero denominators).
- `tests/oracle_libraw.rs`: LibRaw's `unprocessed_raw` reads the same samples from our synthetic - `tests/it/oracle_libraw.rs`: LibRaw's `unprocessed_raw` reads the same samples from our synthetic
files (uncompressed 8–16-bit in both byte orders, lossless JPEG predictors 1–7 with one and two files (uncompressed 8–16-bit in both byte orders, lossless JPEG predictors 1–7 with one and two
components). Skipped when it is not installed. components). Skipped when it is not installed.
- Unit tests: matrix maths, the published sRGB matrix, McCamy temperatures, calibration - Unit tests: matrix maths, the published sRGB matrix, McCamy temperatures, calibration
+2
View File
@@ -0,0 +1,2 @@
mod oracle_libraw;
mod roundtrip;
+4 -4
View File
@@ -59,7 +59,7 @@ first frame header to label the stream.
12-bit coefficient domain is ¼ of the orthonormal DCT, as Table 14's 13-bit IDCT input range 12-bit coefficient domain is ¼ of the orthonormal DCT, as Table 14's 13-bit IDCT input range
for 12-bit samples implies) and 1 otherwise. for 12-bit samples implies) and 1 otherwise.
- Robustness: every offset and size is validated, run lengths past coefficient 63 are rejected, - Robustness: every offset and size is validated, run lengths past coefficient 63 are rejected,
and corrupted or truncated streams return `Err` without panicking (`tests/roundtrip.rs`). and corrupted or truncated streams return `Err` without panicking (`tests/it/roundtrip.rs`).
### Where we follow ffmpeg rather than the letter of the standard ### Where we follow ffmpeg rather than the letter of the standard
@@ -100,7 +100,7 @@ the prose of §7.3.1.2; we follow the prose (normative text takes precedence).
## Accuracy (vs ffmpeg as an external oracle) ## Accuracy (vs ffmpeg as an external oracle)
`tests/oracle_decode.rs` decodes 22 fixtures made by ffmpeg's VC-3 encoder: CIDs 1235, 1237, `tests/it/oracle_decode.rs` decodes 22 fixtures made by ffmpeg's VC-3 encoder: CIDs 1235, 1237,
1238, 1241, 1242, 1243, 1244, 1250, 1251, 1252, 1253, 1258, 1259 and DNxHR LB/SQ/HQ/HQX/444 1238, 1241, 1242, 1243, 1244, 1250, 1251, 1252, 1253, 1258, 1259 and DNxHR LB/SQ/HQ/HQX/444
(Y′CbCr and RGB), with testsrc2, mandelbrot and heavy noise, at 1920×1080, 1280×720, 960×720, (Y′CbCr and RGB), with testsrc2, mandelbrot and heavy noise, at 1920×1080, 1280×720, 960×720,
1440×1080 (thin raster, interlaced), 1000×562, 640×360, 256×120 and 3840×2160. Compared with 1440×1080 (thin raster, interlaced), 1000×562, 640×360, 256×120 and 3840×2160. Compared with
@@ -111,7 +111,7 @@ decoder evaluates the exact IDCT of equation 8.3, so decode is **not bit-exact,
precision** of ffmpeg. (ffmpeg wraps rather than clips 10-bit reconstructions that overflow precision** of ffmpeg. (ffmpeg wraps rather than clips 10-bit reconstructions that overflow
1023, so the noise fixtures keep their samples inside 24…231 before encoding.) 1023, so the noise fixtures keep their samples inside 24…231 before encoding.)
`tests/oracle_encode.rs` encodes 1920×1080 frames with every profile and writes them into a `tests/it/oracle_encode.rs` encodes 1920×1080 frames with every profile and writes them into a
minimal test-only MOV. ffmpeg decodes them with `-xerror`, without errors, and agrees with our minimal test-only MOV. ffmpeg decodes them with `-xerror`, without errors, and agrees with our
decoder within ±2 (90–99% identical). Luma PSNR against the source: HQ 58.8 dB (mandelbrot) and decoder within ±2 (90–99% identical). Luma PSNR against the source: HQ 58.8 dB (mandelbrot) and
66.3 dB (testsrc2), SQ 57.5 dB, LB 56.1 dB, HQX 10-bit 67.5 dB, HQX 12-bit 80.2 dB, 444 10-bit 66.3 dB (testsrc2), SQ 57.5 dB, LB 56.1 dB, HQX 10-bit 67.5 dB, HQX 12-bit 80.2 dB, 444 10-bit
@@ -119,7 +119,7 @@ decoder within ±2 (90–99% identical). Luma PSNR against the source: HQ 58.8 d
## Performance ## Performance
`cargo test --release -p filmcraft-dnx --test perf -- --ignored --nocapture` on an M4 Pro `cargo test --release -p filmcraft-dnx --test it perf:: -- --ignored --nocapture` on an M4 Pro
(14 cores) while the machine was **heavily overloaded** by other jobs (load average ≈ 190). (14 cores) while the machine was **heavily overloaded** by other jobs (load average ≈ 190).
Under the same load `filmcraft-prores`' perf test measured 30 fps threaded / 11.8 fps single Under the same load `filmcraft-prores`' perf test measured 30 fps threaded / 11.8 fps single
for a 1080p HQ frame, against 318 / 53 fps in its README on a less loaded machine, so these are for a 1080p HQ frame, against 318 / 53 fps in its README on a less loaded machine, so these are
@@ -58,27 +58,34 @@ fn run(d: &[u8], label: &str) {
/// Seconds the whole sweep may take before it counts as a hang (override: FILMCRAFT_HOSTILE_TIMEOUT_SECS). /// Seconds the whole sweep may take before it counts as a hang (override: FILMCRAFT_HOSTILE_TIMEOUT_SECS).
const DEADLINE_SECS: u64 = 600; const DEADLINE_SECS: u64 = 600;
/// CI runs a rotating share of every part of the sweep; `FILMCRAFT_FUZZ_ITERS=1500` runs all of it (see [`Depth`]).
#[test] #[test]
fn hostile_frames_never_panic() { fn hostile_frames_never_panic() {
filmcraft_testkit::with_deadline(DEADLINE_SECS, sweep); filmcraft_testkit::with_deadline(DEADLINE_SECS, sweep);
} }
fn sweep() { fn sweep() {
let depth = Depth::from_env();
let seeds = seeds(); let seeds = seeds();
assert!(!seeds.is_empty()); assert!(!seeds.is_empty());
for (si, seed) in seeds.iter().enumerate() { for (si, seed) in seeds.iter().enumerate() {
let stride = (seed.len() / 400).max(1); for n in depth.truncations(seed.len()) {
for n in (0..seed.len()).step_by(stride) {
run(&seed[..n], &format!("seed {si} truncate {n}")); run(&seed[..n], &format!("seed {si} truncate {n}"));
} }
// The header (and the start of the scan-line index): every byte to extreme values, every bit. // The header (and the start of the scan-line index): every byte to extreme values, every bit.
for p in 0..seed.len().min(0x300) { for p in 0..seed.len().min(0x300) {
for v in [0u8, 0xff, 0x80, 0x01, 0x7f] { for (i, v) in VALUES.into_iter().enumerate() {
if !depth.value(p, i) {
continue;
}
let mut d = seed.clone(); let mut d = seed.clone();
d[p] = v; d[p] = v;
run(&d, &format!("seed {si} byte {p:#x}={v:#x}")); run(&d, &format!("seed {si} byte {p:#x}={v:#x}"));
} }
for b in 0..8 { for b in 0..8 {
if !depth.bit(p, b) {
continue;
}
let mut d = seed.clone(); let mut d = seed.clone();
d[p] ^= 1 << b; d[p] ^= 1 << b;
run(&d, &format!("seed {si} flip {p:#x}.{b}")); run(&d, &format!("seed {si} flip {p:#x}.{b}"));
@@ -93,8 +100,8 @@ fn sweep() {
run(&d, &format!("seed {si} u16 {off:#x}={v:#x}")); run(&d, &format!("seed {si} u16 {off:#x}={v:#x}"));
} }
let mut rng = Rng(0x9e3779b97f4a7c15 ^ si as u64); let mut rng = Rng(0x9e3779b97f4a7c15 ^ si as u64);
let iters = std::env::var("FILMCRAFT_FUZZ_ITERS").ok().and_then(|s| s.parse().ok()).unwrap_or(1500); // Skipped rounds still draw their mutations, so round i matches round i of a full run.
for i in 0..iters { for i in 0..depth.rounds {
let mut d = seed.clone(); let mut d = seed.clone();
for _ in 0..1 + rng.below(8) { for _ in 0..1 + rng.below(8) {
let p = rng.below(d.len()); let p = rng.below(d.len());
@@ -104,7 +111,61 @@ fn sweep() {
d[p] = rng.next() as u8; d[p] = rng.next() as u8;
} }
} }
run(&d, &format!("seed {si} random {i}")); if depth.round(i) {
run(&d, &format!("seed {si} random {i}"));
}
} }
} }
} }
const VALUES: [u8; 5] = [0, 0xff, 0x80, 0x01, 0x7f];
/// How much of the sweep runs. `FILMCRAFT_FUZZ_ITERS=N` runs all of it (about 400 truncation
/// points spread over each seed, every header byte, overwrite value and bit flip) plus N random
/// rounds per seed: 1500 is the full depth, more hunts longer. Unset, CI runs a third of the
/// truncation points and of the full depth's random rounds and, for each header byte, one value
/// and two bits. A rotation offset shifts which share runs (`FILMCRAFT_FUZZ_SEED`, else the UTC
/// day number), so every case of the full sweep runs within a few days, and a random round's
/// mutations match the same round of a full run.
struct Depth {
full: bool,
truncations: usize,
rounds: usize,
rotation: usize,
}
impl Depth {
fn from_env() -> Self {
match std::env::var("FILMCRAFT_FUZZ_ITERS").ok().and_then(|s| s.parse().ok()) {
Some(rounds) => Depth { full: true, truncations: 400, rounds, rotation: 0 },
None => {
let rotation = rotation();
eprintln!("fuzz subset rotation {rotation} (FILMCRAFT_FUZZ_SEED={rotation} replays it; FILMCRAFT_FUZZ_ITERS=1500 runs everything)");
Depth { full: false, truncations: 400, rounds: 1500, rotation }
}
}
}
/// Truncation lengths for a `len`-byte input: every point of the full sweep, or every third one.
fn truncations(&self, len: usize) -> std::iter::StepBy<std::ops::Range<usize>> {
let stride = (len / self.truncations).max(1);
let (start, step) = if self.full { (0, stride) } else { ((self.rotation % 3) * stride, 3 * stride) };
(start..len).step_by(step)
}
fn value(&self, p: usize, i: usize) -> bool {
self.full || i == (p + self.rotation % VALUES.len()) % VALUES.len()
}
fn bit(&self, p: usize, b: usize) -> bool {
self.full || (b + p + self.rotation % 4).is_multiple_of(4)
}
fn round(&self, i: usize) -> bool {
self.full || i % 3 == self.rotation % 3
}
}
/// The default subset's rotation offset: `FILMCRAFT_FUZZ_SEED`, else days since the Unix epoch (UTC).
fn rotation() -> usize {
std::env::var("FILMCRAFT_FUZZ_SEED").ok().and_then(|s| s.parse().ok()).unwrap_or_else(|| {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs());
usize::try_from(secs / 86_400).unwrap_or(0)
})
}
+6
View File
@@ -0,0 +1,6 @@
mod common;
mod fuzz_hostile;
mod oracle_decode;
mod oracle_encode;
mod perf;
mod roundtrip;
@@ -1,6 +1,6 @@
//! Decode ffmpeg-made DNxHD / DNxHR fixtures and compare with ffmpeg's own decode. //! Decode ffmpeg-made DNxHD / DNxHR fixtures and compare with ffmpeg's own decode.
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_dnx::{DecodeOptions, decode_frame_with}; use filmcraft_dnx::{DecodeOptions, decode_frame_with};
@@ -1,6 +1,6 @@
//! Encode DNxHR with our encoder, decode with ffmpeg (`-xerror`) and with our decoder. //! Encode DNxHR with our encoder, decode with ffmpeg (`-xerror`) and with our decoder.
mod common; use crate::common;
use common::*; use common::*;
use filmcraft_dnx::{ChromaFormat, Encoder, EncoderConfig, Profile, decode_frame}; use filmcraft_dnx::{ChromaFormat, Encoder, EncoderConfig, Profile, decode_frame};
@@ -1,4 +1,4 @@
//! Throughput: `cargo test --release -p filmcraft-dnx --test perf -- --ignored --nocapture`. //! Throughput: `cargo test --release -p filmcraft-dnx --test it perf:: -- --ignored --nocapture`.
use filmcraft_dnx::{ChromaFormat, DecodeOptions, Encoder, EncoderConfig, Frame, Profile, decode_frame_with}; use filmcraft_dnx::{ChromaFormat, DecodeOptions, Encoder, EncoderConfig, Frame, Profile, decode_frame_with};
use std::time::Instant; use std::time::Instant;
+15 -10
View File
@@ -155,11 +155,13 @@ fn files_in(dir: &Path) -> Vec<String> {
v v
} }
/// Every built-in preset: a quarter second of the demo sequence (seven frames at 23.976) exported /// Every built-in preset: 1.0 to 1.08 s of the demo sequence (three frames at 23.976) exported
/// with `file.exportMedia {preset}`; ffprobe checks container, codec, profile, frame size, frame /// with `file.exportMedia {preset}`; ffprobe checks container, codec, profile, frame size, frame
/// rate and audio format, and the data rate where the codec fixes it. /// rate and audio format, and the data rate where the codec fixes it. Three frames are enough for
/// all of these, and each extra frame costs a 4K H.264 encode in five presets.
#[test] #[test]
fn every_builtin_preset_exports_and_ffprobe_confirms() { fn every_builtin_preset_exports_and_ffprobe_confirms() {
let (start, end) = (1.0, 1.08);
let mut s = demo(); let mut s = demo();
let presets = s.execute("export.presets.list", json!({})).unwrap()["presets"].as_array().unwrap().clone(); let presets = s.execute("export.presets.list", json!({})).unwrap()["presets"].as_array().unwrap().clone();
assert!(presets.len() >= 24); assert!(presets.len() >= 24);
@@ -169,7 +171,7 @@ fn every_builtin_preset_exports_and_ffprobe_confirms() {
let sub = dir.0.join(format!("p{}", filmcraft_export::presets::preset_key(name))); let sub = dir.0.join(format!("p{}", filmcraft_export::presets::preset_key(name)));
std::fs::create_dir_all(&sub).unwrap(); std::fs::create_dir_all(&sub).unwrap();
let path = sub.join("Demo").to_string_lossy().to_string(); let path = sub.join("Demo").to_string_lossy().to_string();
let params = json!({"preset": name, "path": path, "range": "custom", "startSeconds": 1.0, "endSeconds": 1.25, "wait": true}); let params = json!({"preset": name, "path": path, "range": "custom", "startSeconds": start, "endSeconds": end, "wait": true});
let frames = s.execute("export.resolve", params.clone()).unwrap()["frames"].as_u64().unwrap(); let frames = s.execute("export.resolve", params.clone()).unwrap()["frames"].as_u64().unwrap();
let r = s.execute("file.exportMedia", params).unwrap_or_else(|e| panic!("{name}: {e}")); let r = s.execute("file.exportMedia", params).unwrap_or_else(|e| panic!("{name}: {e}"));
let out = r["path"].as_str().unwrap().to_string(); let out = r["path"].as_str().unwrap().to_string();
@@ -182,7 +184,7 @@ fn every_builtin_preset_exports_and_ffprobe_confirms() {
if matches!(fmt.as_str(), "png" | "tiff" | "bmp" | "jpg" | "tga" | "dpx" | "exr") { if matches!(fmt.as_str(), "png" | "tiff" | "bmp" | "jpg" | "tga" | "dpx" | "exr") {
let files = files_in(&sub); let files = files_in(&sub);
let ext = Path::new(&out).extension().unwrap().to_string_lossy().to_string(); let ext = Path::new(&out).extension().unwrap().to_string_lossy().to_string();
assert_eq!(frames, 7, "frames 23..=29 at 23.976 fps"); assert_eq!(frames, 3, "frames 23..=25 at 23.976 fps");
let want: Vec<String> = (0..frames).map(|i| format!("Demo{i:03}.{ext}")).collect(); let want: Vec<String> = (0..frames).map(|i| format!("Demo{i:03}.{ext}")).collect();
assert_eq!(files, want, "{name}"); assert_eq!(files, want, "{name}");
if let Some(j) = probe(&sub.join(&want[0]).to_string_lossy()) { if let Some(j) = probe(&sub.join(&want[0]).to_string_lossy()) {
@@ -218,7 +220,7 @@ fn every_builtin_preset_exports_and_ffprobe_confirms() {
assert_eq!(a["codec_name"], codec, "{name}"); assert_eq!(a["codec_name"], codec, "{name}");
assert_eq!(a["sample_rate"], "48000", "{name}"); assert_eq!(a["sample_rate"], "48000", "{name}");
assert_eq!(a["channels"], 2, "{name}"); assert_eq!(a["channels"], 2, "{name}");
assert!((num(&j["format"]["duration"]) - 0.25).abs() < 0.01, "{name}: {}", j["format"]["duration"]); assert!((num(&j["format"]["duration"]) - (end - start)).abs() < 0.01, "{name}: {}", j["format"]["duration"]);
continue; continue;
} }
"aac" => { "aac" => {
@@ -228,7 +230,7 @@ fn every_builtin_preset_exports_and_ffprobe_confirms() {
assert_eq!(j["streams"].as_array().unwrap().len(), 1, "{name}"); assert_eq!(j["streams"].as_array().unwrap().len(), 1, "{name}");
let a = stream(&j, "audio").unwrap(); let a = stream(&j, "audio").unwrap();
assert_eq!((a["codec_name"].as_str(), a["sample_rate"].as_str(), a["channels"].as_u64()), (Some("aac"), Some("48000"), Some(2)), "{name}"); assert_eq!((a["codec_name"].as_str(), a["sample_rate"].as_str(), a["channels"].as_u64()), (Some("aac"), Some("48000"), Some(2)), "{name}");
assert!((num(&j["format"]["duration"]) - 0.25).abs() < 0.03, "{name}: {}", j["format"]["duration"]); assert!((num(&j["format"]["duration"]) - (end - start)).abs() < 0.03, "{name}: {}", j["format"]["duration"]);
continue; continue;
} }
"gif" => { "gif" => {
@@ -332,9 +334,6 @@ fn every_builtin_preset_exports_and_ffprobe_confirms() {
} }
} }
/// H.264 bitrate control over three seconds of the demo: the adaptive Match Source preset
/// targets width × height × fps × bpp, the 1080p delivery presets their fixed targets; ffprobe's
/// measured video bitrate stays within the VBR maximum and close to the target.
#[test] #[test]
fn aac_audio_keeps_a_dot_aac_path_as_adts_and_adds_m4a_otherwise() { fn aac_audio_keeps_a_dot_aac_path_as_adts_and_adds_m4a_otherwise() {
let mut s = demo(); let mut s = demo();
@@ -354,6 +353,12 @@ fn aac_audio_keeps_a_dot_aac_path_as_adts_and_adds_m4a_otherwise() {
assert_eq!(&std::fs::read(&m4a).unwrap()[4..8], b"ftyp"); assert_eq!(&std::fs::read(&m4a).unwrap()[4..8], b"ftyp");
} }
/// H.264 bitrate control over one second of the demo: the adaptive Match Source preset targets
/// width × height × fps × bpp, the 1080p delivery presets their fixed targets; ffprobe's measured
/// video bitrate stays within the VBR maximum and close to the target. 0.5 to 1.5 s holds the
/// second half of the one-second fade in from black, where every frame changes and each preset
/// spends its whole budget; the shot after it is simple enough that both 1080p presets land at the
/// same bitrate, far under their targets.
#[test] #[test]
fn h264_presets_hit_their_bitrates() { fn h264_presets_hit_their_bitrates() {
let mut s = demo(); let mut s = demo();
@@ -366,7 +371,7 @@ fn h264_presets_hit_their_bitrates() {
let path = dir.path(&format!("{}.mp4", filmcraft_export::presets::preset_key(preset))); let path = dir.path(&format!("{}.mp4", filmcraft_export::presets::preset_key(preset)));
s.execute( s.execute(
"file.exportMedia", "file.exportMedia",
json!({"preset": preset, "path": path, "range": "custom", "startSeconds": 0.0, "endSeconds": 3.0, "audio": false, "wait": true}), json!({"preset": preset, "path": path, "range": "custom", "startSeconds": 0.5, "endSeconds": 1.5, "audio": false, "wait": true}),
) )
.unwrap(); .unwrap();
let Some(j) = probe(&path) else { return }; let Some(j) = probe(&path) else { return };
+2
View File
@@ -0,0 +1,2 @@
mod pool_trim;
mod renderer_reentrancy;
@@ -143,7 +143,7 @@ fn nested_rayon_renderers_finish_every_export_path() {
} }
for case in ["png", "gif", "stepped", "overlap-audio"] { for case in ["png", "gif", "stepped", "overlap-audio"] {
let mut child = std::process::Command::new(std::env::current_exe().unwrap()) let mut child = std::process::Command::new(std::env::current_exe().unwrap())
.args(["--exact", "nested_rayon_renderers_finish_every_export_path", "--nocapture"]) .args(["--exact", "renderer_reentrancy::nested_rayon_renderers_finish_every_export_path", "--nocapture"])
.env(CHILD, case) .env(CHILD, case)
.spawn() .spawn()
.unwrap(); .unwrap();
+3 -3
View File
@@ -108,7 +108,7 @@ with ffmpeg + libx264; tests print a message and skip when `/opt/homebrew/bin/ff
explicit bi-prediction. The expected output is known exactly and is also checked against ffmpeg. explicit bi-prediction. The expected output is known exactly and is also checked against ffmpeg.
- Robustness: avcC input with pts round trip; randomly corrupted, truncated and garbage input never - Robustness: avcC input with pts round trip; randomly corrupted, truncated and garbage input never
panics or deadlocks (single- and multi-threaded). panics or deadlocks (single- and multi-threaded).
- Conformance (`tests/conformance.rs`): every fixture is decoded single-threaded, with 3 threads - Conformance (`tests/it/conformance.rs`): every fixture is decoded single-threaded, with 3 threads
and with every core, and compared byte-for-byte with `ffmpeg -f rawvideo -pix_fmt yuv420p`; on and with every core, and compared byte-for-byte with `ffmpeg -f rawvideo -pix_fmt yuv420p`; on
mismatch the first differing frame, plane, sample position and macroblock are reported. Output mismatch the first differing frame, plane, sample position and macroblock are reported. Output
pts/POC order is checked too, and that no picture is flagged draft without draft mode. Draft pts/POC order is checked too, and that no picture is flagged draft without draft mode. Draft
@@ -164,12 +164,12 @@ encoder is unavailable.
| vt_high | testsrc2+noise 640x360 | VideoToolbox, High | | vt_high | testsrc2+noise 640x360 | VideoToolbox, High |
| vt_high_1080p | testsrc2 1920x1080 | VideoToolbox, High 8 Mbit/s | | vt_high_1080p | testsrc2 1920x1080 | VideoToolbox, High 8 Mbit/s |
`cargo test --release -p filmcraft-h264 --test conformance coverage_report -- --ignored --nocapture` `cargo test --release -p filmcraft-h264 --test it conformance::coverage_report -- --ignored --nocapture`
prints which coding tools each fixture exercises. prints which coding tools each fixture exercises.
## Performance ## Performance
`cargo test --release -p filmcraft-h264 --test perf -- --ignored --nocapture` (1080p High profile, `cargo test --release -p filmcraft-h264 --test it perf:: -- --ignored --nocapture` (1080p High profile,
CABAC, B-pyramid, 120 frames, 8.8 Mbit/s; bit-exactness is verified first). Measured on an Apple M4 CABAC, B-pyramid, 120 frames, 8.8 Mbit/s; bit-exactness is verified first). Measured on an Apple M4
Pro (14 cores) while the machine was heavily shared (load average 40-180), so these are lower bounds: Pro (14 cores) while the machine was heavily shared (load average 40-180), so these are lower bounds:
+66 -8
View File
@@ -1,6 +1,7 @@
//! Hostile-input robustness: header-region sweeps (SPS/PPS/first slice headers), truncation and //! Hostile-input robustness: header-region sweeps (SPS/PPS/first slice headers), truncation and
//! random mutation of small libx264 streams must never panic or run away. Skips without ffmpeg. //! random mutation of small libx264 streams must never panic or run away. Skips without ffmpeg.
#[path = "it/common/mod.rs"]
mod common; mod common;
use std::panic::{AssertUnwindSafe, catch_unwind}; use std::panic::{AssertUnwindSafe, catch_unwind};
@@ -82,6 +83,7 @@ fn run_all(d: &[u8], label: &str) {
/// Seconds the whole sweep may take before it counts as a hang (override: FILMCRAFT_HOSTILE_TIMEOUT_SECS). /// Seconds the whole sweep may take before it counts as a hang (override: FILMCRAFT_HOSTILE_TIMEOUT_SECS).
const DEADLINE_SECS: u64 = 600; const DEADLINE_SECS: u64 = 600;
/// CI runs a rotating share of every part of the sweep; `FILMCRAFT_FUZZ_ITERS=400` runs all of it (see [`Depth`]).
#[test] #[test]
fn hostile_streams_never_panic() { fn hostile_streams_never_panic() {
filmcraft_testkit::with_deadline(DEADLINE_SECS, sweep); filmcraft_testkit::with_deadline(DEADLINE_SECS, sweep);
@@ -89,6 +91,7 @@ fn hostile_streams_never_panic() {
fn sweep() { fn sweep() {
let Some(ff) = filmcraft_testkit::ffmpeg_or_skip("h264 fuzz") else { return }; let Some(ff) = filmcraft_testkit::ffmpeg_or_skip("h264 fuzz") else { return };
let depth = Depth::from_env();
for &(name, args, pix) in SEEDS { for &(name, args, pix) in SEEDS {
if !filmcraft_testkit::ffmpeg_has_encoder(&ff, "libx264") { if !filmcraft_testkit::ffmpeg_has_encoder(&ff, "libx264") {
filmcraft_testkit::skip("h264 fuzz", &format!("this ffmpeg has no libx264 encoder (seed {name})")); filmcraft_testkit::skip("h264 fuzz", &format!("this ffmpeg has no libx264 encoder (seed {name})"));
@@ -97,25 +100,31 @@ fn sweep() {
let Some(seed) = make(&ff, name, args, pix) else { let Some(seed) = make(&ff, name, args, pix) else {
panic!("ffmpeg has libx264 but could not encode the {name} seed"); panic!("ffmpeg has libx264 but could not encode the {name} seed");
}; };
let stride = (seed.len() / 300).max(1); for n in depth.truncations(seed.len()) {
for n in (0..seed.len()).step_by(stride) {
run_all(&seed[..n], &format!("{name} truncate {n}")); run_all(&seed[..n], &format!("{name} truncate {n}"));
} }
// SPS/PPS/first slice header region: every byte to a few values, every bit // SPS/PPS/first slice header region: every byte to a few values, every bit
for p in 0..seed.len().min(120) { for p in 0..seed.len().min(120) {
for v in [0u8, 0xff, 0x80, 0x01, 0x7f] { for (i, v) in VALUES.into_iter().enumerate() {
if !depth.value(p, i) {
continue;
}
let mut d = seed.clone(); let mut d = seed.clone();
d[p] = v; d[p] = v;
run_all(&d, &format!("{name} byte {p}={v:#x}")); run_all(&d, &format!("{name} byte {p}={v:#x}"));
} }
for b in 0..8 { for b in 0..8 {
if !depth.bit(p, b) {
continue;
}
let mut d = seed.clone(); let mut d = seed.clone();
d[p] ^= 1 << b; d[p] ^= 1 << b;
run_all(&d, &format!("{name} flip {p}.{b}")); run_all(&d, &format!("{name} flip {p}.{b}"));
} }
} }
let mut rng = Rng(0x9e3779b97f4a7c15 ^ name.len() as u64); let mut rng = Rng(0x9e3779b97f4a7c15 ^ name.len() as u64);
for i in 0..iters() { // Skipped rounds still draw their mutations, so round i matches round i of a full run.
for i in 0..depth.rounds {
let mut d = seed.clone(); let mut d = seed.clone();
for _ in 0..1 + rng.below(6) { for _ in 0..1 + rng.below(6) {
let p = rng.below(d.len()); let p = rng.below(d.len());
@@ -125,12 +134,61 @@ fn sweep() {
d[p] = rng.next() as u8; d[p] = rng.next() as u8;
} }
} }
run_all(&d, &format!("{name} random {i}")); if depth.round(i) {
run_all(&d, &format!("{name} random {i}"));
}
} }
} }
} }
/// Random-mutation rounds per seed (raise with FILMCRAFT_FUZZ_ITERS for a long hunt). const VALUES: [u8; 5] = [0, 0xff, 0x80, 0x01, 0x7f];
fn iters() -> usize {
std::env::var("FILMCRAFT_FUZZ_ITERS").ok().and_then(|s| s.parse().ok()).unwrap_or(400) /// How much of the sweep runs. `FILMCRAFT_FUZZ_ITERS=N` runs all of it (about 300 truncation
/// points spread over each seed, every header byte, overwrite value and bit flip) plus N random
/// rounds per seed: 400 is the full depth, more hunts longer. Unset, CI runs a third of the
/// truncation points and of the full depth's random rounds and, for each header byte, one value
/// and two bits. A rotation offset shifts which share runs (`FILMCRAFT_FUZZ_SEED`, else the UTC
/// day number), so every case of the full sweep runs within a few days, and a random round's
/// mutations match the same round of a full run.
struct Depth {
full: bool,
truncations: usize,
rounds: usize,
rotation: usize,
}
impl Depth {
fn from_env() -> Self {
match std::env::var("FILMCRAFT_FUZZ_ITERS").ok().and_then(|s| s.parse().ok()) {
Some(rounds) => Depth { full: true, truncations: 300, rounds, rotation: 0 },
None => {
let rotation = rotation();
eprintln!("fuzz subset rotation {rotation} (FILMCRAFT_FUZZ_SEED={rotation} replays it; FILMCRAFT_FUZZ_ITERS=400 runs everything)");
Depth { full: false, truncations: 300, rounds: 400, rotation }
}
}
}
/// Truncation lengths for a `len`-byte input: every point of the full sweep, or every third one.
fn truncations(&self, len: usize) -> std::iter::StepBy<std::ops::Range<usize>> {
let stride = (len / self.truncations).max(1);
let (start, step) = if self.full { (0, stride) } else { ((self.rotation % 3) * stride, 3 * stride) };
(start..len).step_by(step)
}
fn value(&self, p: usize, i: usize) -> bool {
self.full || i == (p + self.rotation % VALUES.len()) % VALUES.len()
}
fn bit(&self, p: usize, b: usize) -> bool {
self.full || (b + p + self.rotation % 4).is_multiple_of(4)
}
fn round(&self, i: usize) -> bool {
self.full || i % 3 == self.rotation % 3
}
}
/// The default subset's rotation offset: `FILMCRAFT_FUZZ_SEED`, else days since the Unix epoch (UTC).
fn rotation() -> usize {
std::env::var("FILMCRAFT_FUZZ_SEED").ok().and_then(|s| s.parse().ok()).unwrap_or_else(|| {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs());
usize::try_from(secs / 86_400).unwrap_or(0)
})
} }
@@ -64,7 +64,7 @@ pub const FIXTURES: &[Fixture] = &[
filter: "", filter: "",
args: &["-c:v", "h264_videotoolbox", "-profile:v", "high", "-b:v", "8M"], args: &["-c:v", "h264_videotoolbox", "-profile:v", "high", "-b:v", "8M"],
}, },
// performance reference stream (see tests/perf.rs) // performance reference stream (see tests/it/perf.rs)
Fixture { Fixture {
name: "bench_1080p", name: "bench_1080p",
source: "testsrc2", source: "testsrc2",
@@ -1,6 +1,6 @@
//! Bit-exactness against ffmpeg's decoder on libx264-generated fixtures (skipped when ffmpeg is absent). //! Bit-exactness against ffmpeg's decoder on libx264-generated fixtures (skipped when ffmpeg is absent).
mod common; use crate::common;
macro_rules! fixture_tests { macro_rules! fixture_tests {
($($name:ident),* $(,)?) => { ($($name:ident),* $(,)?) => {
@@ -1,6 +1,6 @@
//! Parameter set / slice header / POC parsing over the fixture streams. //! Parameter set / slice header / POC parsing over the fixture streams.
mod common; use crate::common;
use filmcraft_bitstream::{annexb_nals, unescape_rbsp}; use filmcraft_bitstream::{annexb_nals, unescape_rbsp};
use filmcraft_h264::params::{Pps, Sps}; use filmcraft_h264::params::{Pps, Sps};
+6
View File
@@ -0,0 +1,6 @@
mod common;
mod conformance;
mod headers;
mod perf;
mod pps_validation;
mod robustness;
@@ -1,7 +1,7 @@
//! Decode speed on a 1080p High-profile stream (ignored by default; run with //! Decode speed on a 1080p High-profile stream (ignored by default; run with
//! `cargo test --release -p filmcraft-h264 --test perf -- --ignored --nocapture`). //! `cargo test --release -p filmcraft-h264 --test it perf:: -- --ignored --nocapture`).
mod common; use crate::common;
use filmcraft_h264::Decoder; use filmcraft_h264::Decoder;
use std::time::Instant; use std::time::Instant;
@@ -1,6 +1,6 @@
//! avcC / length-prefixed input and resilience against damaged streams. //! avcC / length-prefixed input and resilience against damaged streams.
mod common; use crate::common;
use filmcraft_bitstream::annexb_nals; use filmcraft_bitstream::annexb_nals;
use filmcraft_h264::Decoder; use filmcraft_h264::Decoder;
+1 -1
View File
@@ -2,7 +2,7 @@
//! //!
//! `cargo run --release -p filmcraft-h264enc --example h264enc_synth -- W H FRAMES [profile] [preset] [bframes] [rate] [slices] [src]` //! `cargo run --release -p filmcraft-h264enc --example h264enc_synth -- W H FRAMES [profile] [preset] [bframes] [rate] [slices] [src]`
//! profile: baseline|main|high, preset: speed|balanced|quality, rate: qpN | crfN | cbrN | vbrN //! profile: baseline|main|high, preset: speed|balanced|quality, rate: qpN | crfN | cbrN | vbrN
#[path = "../tests/common/mod.rs"] #[path = "../tests/it/common/mod.rs"]
mod common; mod common;
use filmcraft_h264enc::*; use filmcraft_h264enc::*;
@@ -1,7 +1,7 @@
//! Constant quality has no bitrate cap, so the level must allow what low factors spend; and every //! Constant quality has no bitrate cap, so the level must allow what low factors spend; and every
//! stream says which settings made it. //! stream says which settings made it.
mod common; use crate::common;
use common::synth; use common::synth;
use filmcraft_h264enc::*; use filmcraft_h264enc::*;
@@ -3,7 +3,7 @@
//! Every stream must decode with zero errors, with the exact frame count, and the decoder's output must be //! Every stream must decode with zero errors, with the exact frame count, and the decoder's output must be
//! bit-identical to the encoder's own reconstruction (catches prediction/transform/deblocking mismatches). //! bit-identical to the encoder's own reconstruction (catches prediction/transform/deblocking mismatches).
mod common; use crate::common;
use common::{Yuv, annexb_nals, ffmpeg, ffmpeg_decode, ffmpeg_source, ffprobe, out_dir, psnr, synth}; use common::{Yuv, annexb_nals, ffmpeg, ffmpeg_decode, ffmpeg_source, ffprobe, out_dir, psnr, synth};
use filmcraft_h264enc::*; use filmcraft_h264enc::*;
+3
View File
@@ -0,0 +1,3 @@
mod common;
mod crf_level_and_note;
mod ffmpeg_oracle;
+3 -3
View File
@@ -96,7 +96,7 @@ with ffmpeg; tests print a message and skip when ffmpeg is absent).
ffmpeg. ffmpeg.
- Robustness: hvcC input with pts round trip; randomly corrupted, truncated and garbage input never - Robustness: hvcC input with pts round trip; randomly corrupted, truncated and garbage input never
panics or deadlocks (single- and multi-threaded). panics or deadlocks (single- and multi-threaded).
- Conformance (`tests/conformance.rs`): every fixture is decoded with 1 thread, 3 threads and every - Conformance (`tests/it/conformance.rs`): every fixture is decoded with 1 thread, 3 threads and every
core and compared sample-exactly with `ffmpeg -f rawvideo -pix_fmt yuv420p / yuv420p10le`; on mismatch core and compared sample-exactly with `ffmpeg -f rawvideo -pix_fmt yuv420p / yuv420p10le`; on mismatch
the first differing frame, plane, sample, CTB and 8x8 block are reported. Output pts/POC order is the first differing frame, plane, sample, CTB and 8x8 block are reported. Output pts/POC order is
checked too, and that no picture is flagged draft without draft mode. `compare_detects_mismatch` checked too, and that no picture is flagged draft without draft mode. `compare_detects_mismatch`
@@ -135,12 +135,12 @@ libx265 unless noted (x265 enables WPP by default, so most streams carry entry p
| vt_main / vt_main10 | 640x360 | Apple VideoToolbox (hevc_videotoolbox), macOS only | | vt_main / vt_main10 | 640x360 | Apple VideoToolbox (hevc_videotoolbox), macOS only |
| synthetic (7 streams) | 96x64 | tiles, PCM, slices, dependent segments, LTR, list modification | | synthetic (7 streams) | 96x64 | tiles, PCM, slices, dependent segments, LTR, list modification |
`cargo test -p filmcraft-hevc --test conformance coverage_report -- --ignored --nocapture` prints the `cargo test -p filmcraft-hevc --test it conformance::coverage_report -- --ignored --nocapture` prints the
decoder statistics per fixture. decoder statistics per fixture.
## Performance ## Performance
`cargo test --release -p filmcraft-hevc --test perf -- --ignored --nocapture` (bit-exactness is `cargo test --release -p filmcraft-hevc --test it perf:: -- --ignored --nocapture` (bit-exactness is
verified first). Apple M4 Pro (14 cores), measured while the machine was heavily shared (load verified first). Apple M4 Pro (14 cores), measured while the machine was heavily shared (load
average 50-100), so these are lower bounds: average 50-100), so these are lower bounds:
+66 -8
View File
@@ -1,6 +1,7 @@
//! Hostile-input robustness: header-region sweeps (SPS/PPS/first slice headers), truncation and //! Hostile-input robustness: header-region sweeps (SPS/PPS/first slice headers), truncation and
//! random mutation of small libx265 streams must never panic or run away. Skips without ffmpeg. //! random mutation of small libx265 streams must never panic or run away. Skips without ffmpeg.
#[path = "it/common/mod.rs"]
mod common; mod common;
use std::panic::{AssertUnwindSafe, catch_unwind}; use std::panic::{AssertUnwindSafe, catch_unwind};
@@ -83,6 +84,7 @@ fn run_all(d: &[u8], label: &str) {
/// Seconds the whole sweep may take before it counts as a hang (override: FILMCRAFT_HOSTILE_TIMEOUT_SECS). /// Seconds the whole sweep may take before it counts as a hang (override: FILMCRAFT_HOSTILE_TIMEOUT_SECS).
const DEADLINE_SECS: u64 = 600; const DEADLINE_SECS: u64 = 600;
/// CI runs a rotating share of every part of the sweep; `FILMCRAFT_FUZZ_ITERS=400` runs all of it (see [`Depth`]).
#[test] #[test]
fn hostile_streams_never_panic() { fn hostile_streams_never_panic() {
filmcraft_testkit::with_deadline(DEADLINE_SECS, sweep); filmcraft_testkit::with_deadline(DEADLINE_SECS, sweep);
@@ -90,6 +92,7 @@ fn hostile_streams_never_panic() {
fn sweep() { fn sweep() {
let Some(ff) = filmcraft_testkit::ffmpeg_or_skip("hevc fuzz") else { return }; let Some(ff) = filmcraft_testkit::ffmpeg_or_skip("hevc fuzz") else { return };
let depth = Depth::from_env();
for &(name, enc, args, pix) in SEEDS { for &(name, enc, args, pix) in SEEDS {
if !filmcraft_testkit::ffmpeg_has_encoder(&ff, enc) { if !filmcraft_testkit::ffmpeg_has_encoder(&ff, enc) {
filmcraft_testkit::skip("hevc fuzz", &format!("this ffmpeg has no {enc} encoder (seed {name})")); filmcraft_testkit::skip("hevc fuzz", &format!("this ffmpeg has no {enc} encoder (seed {name})"));
@@ -98,25 +101,31 @@ fn sweep() {
let Some(seed) = make(&ff, name, enc, args, pix) else { let Some(seed) = make(&ff, name, enc, args, pix) else {
panic!("ffmpeg has {enc} but could not encode the {name} seed"); panic!("ffmpeg has {enc} but could not encode the {name} seed");
}; };
let stride = (seed.len() / 300).max(1); for n in depth.truncations(seed.len()) {
for n in (0..seed.len()).step_by(stride) {
run_all(&seed[..n], &format!("{name} truncate {n}")); run_all(&seed[..n], &format!("{name} truncate {n}"));
} }
// SPS/PPS/first slice header region: every byte to a few values, every bit // SPS/PPS/first slice header region: every byte to a few values, every bit
for p in 0..seed.len().min(120) { for p in 0..seed.len().min(120) {
for v in [0u8, 0xff, 0x80, 0x01, 0x7f] { for (i, v) in VALUES.into_iter().enumerate() {
if !depth.value(p, i) {
continue;
}
let mut d = seed.clone(); let mut d = seed.clone();
d[p] = v; d[p] = v;
run_all(&d, &format!("{name} byte {p}={v:#x}")); run_all(&d, &format!("{name} byte {p}={v:#x}"));
} }
for b in 0..8 { for b in 0..8 {
if !depth.bit(p, b) {
continue;
}
let mut d = seed.clone(); let mut d = seed.clone();
d[p] ^= 1 << b; d[p] ^= 1 << b;
run_all(&d, &format!("{name} flip {p}.{b}")); run_all(&d, &format!("{name} flip {p}.{b}"));
} }
} }
let mut rng = Rng(0x9e3779b97f4a7c15 ^ name.len() as u64); let mut rng = Rng(0x9e3779b97f4a7c15 ^ name.len() as u64);
for i in 0..iters() { // Skipped rounds still draw their mutations, so round i matches round i of a full run.
for i in 0..depth.rounds {
let mut d = seed.clone(); let mut d = seed.clone();
for _ in 0..1 + rng.below(6) { for _ in 0..1 + rng.below(6) {
let p = rng.below(d.len()); let p = rng.below(d.len());
@@ -126,12 +135,61 @@ fn sweep() {
d[p] = rng.next() as u8; d[p] = rng.next() as u8;
} }
} }
run_all(&d, &format!("{name} random {i}")); if depth.round(i) {
run_all(&d, &format!("{name} random {i}"));
}
} }
} }
} }
/// Random-mutation rounds per seed (raise with FILMCRAFT_FUZZ_ITERS for a long hunt). const VALUES: [u8; 5] = [0, 0xff, 0x80, 0x01, 0x7f];
fn iters() -> usize {
std::env::var("FILMCRAFT_FUZZ_ITERS").ok().and_then(|s| s.parse().ok()).unwrap_or(400) /// How much of the sweep runs. `FILMCRAFT_FUZZ_ITERS=N` runs all of it (about 300 truncation
/// points spread over each seed, every header byte, overwrite value and bit flip) plus N random
/// rounds per seed: 400 is the full depth, more hunts longer. Unset, CI runs a third of the
/// truncation points and of the full depth's random rounds and, for each header byte, one value
/// and two bits. A rotation offset shifts which share runs (`FILMCRAFT_FUZZ_SEED`, else the UTC
/// day number), so every case of the full sweep runs within a few days, and a random round's
/// mutations match the same round of a full run.
struct Depth {
full: bool,
truncations: usize,
rounds: usize,
rotation: usize,
}
impl Depth {
fn from_env() -> Self {
match std::env::var("FILMCRAFT_FUZZ_ITERS").ok().and_then(|s| s.parse().ok()) {
Some(rounds) => Depth { full: true, truncations: 300, rounds, rotation: 0 },
None => {
let rotation = rotation();
eprintln!("fuzz subset rotation {rotation} (FILMCRAFT_FUZZ_SEED={rotation} replays it; FILMCRAFT_FUZZ_ITERS=400 runs everything)");
Depth { full: false, truncations: 300, rounds: 400, rotation }
}
}
}
/// Truncation lengths for a `len`-byte input: every point of the full sweep, or every third one.
fn truncations(&self, len: usize) -> std::iter::StepBy<std::ops::Range<usize>> {
let stride = (len / self.truncations).max(1);
let (start, step) = if self.full { (0, stride) } else { ((self.rotation % 3) * stride, 3 * stride) };
(start..len).step_by(step)
}
fn value(&self, p: usize, i: usize) -> bool {
self.full || i == (p + self.rotation % VALUES.len()) % VALUES.len()
}
fn bit(&self, p: usize, b: usize) -> bool {
self.full || (b + p + self.rotation % 4).is_multiple_of(4)
}
fn round(&self, i: usize) -> bool {
self.full || i % 3 == self.rotation % 3
}
}
/// The default subset's rotation offset: `FILMCRAFT_FUZZ_SEED`, else days since the Unix epoch (UTC).
fn rotation() -> usize {
std::env::var("FILMCRAFT_FUZZ_SEED").ok().and_then(|s| s.parse().ok()).unwrap_or_else(|| {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs());
usize::try_from(secs / 86_400).unwrap_or(0)
})
} }
@@ -73,7 +73,7 @@ pub const FIXTURES: &[Fixture] = &[
fx!("odd_1918x1078", "testsrc2", 1918, 1078, 3, NOISE, 8, ["-preset", "veryfast", "-x265-params", "log-level=error"]), fx!("odd_1918x1078", "testsrc2", 1918, 1078, 3, NOISE, 8, ["-preset", "veryfast", "-x265-params", "log-level=error"]),
fx!("hd_1080p", "testsrc2", 1920, 1080, 3, "", 8, ["-preset", "fast", "-x265-params", "log-level=error"]), fx!("hd_1080p", "testsrc2", 1920, 1080, 3, "", 8, ["-preset", "fast", "-x265-params", "log-level=error"]),
fx!("uhd_4k_main10", "testsrc2", 3840, 2160, 2, "", 10, ["-preset", "ultrafast", "-x265-params", "log-level=error"]), fx!("uhd_4k_main10", "testsrc2", 3840, 2160, 2, "", 10, ["-preset", "ultrafast", "-x265-params", "log-level=error"]),
// performance reference streams (tests/perf.rs) // performance reference streams (tests/it/perf.rs)
fx!("bench_1080p", "testsrc2", 1920, 1080, 60, "noise=alls=3:allf=t", 8, ["-preset", "medium", "-crf", "24", "-x265-params", "log-level=error"]), fx!("bench_1080p", "testsrc2", 1920, 1080, 60, "noise=alls=3:allf=t", 8, ["-preset", "medium", "-crf", "24", "-x265-params", "log-level=error"]),
fx!("bench_4k", "testsrc2", 3840, 2160, 20, "noise=alls=3:allf=t", 10, ["-preset", "fast", "-crf", "26", "-x265-params", "log-level=error"]), fx!("bench_4k", "testsrc2", 3840, 2160, 20, "noise=alls=3:allf=t", 10, ["-preset", "fast", "-crf", "26", "-x265-params", "log-level=error"]),
// Apple VideoToolbox (hardware encoder, macOS only) // Apple VideoToolbox (hardware encoder, macOS only)
@@ -1,7 +1,7 @@
//! Bit-exactness against ffmpeg's decoder on libx265 / VideoToolbox fixtures (skipped when ffmpeg is //! Bit-exactness against ffmpeg's decoder on libx265 / VideoToolbox fixtures (skipped when ffmpeg is
//! absent). //! absent).
mod common; use crate::common;
macro_rules! fixture_tests { macro_rules! fixture_tests {
($($name:ident),* $(,)?) => { ($($name:ident),* $(,)?) => {
+4
View File
@@ -0,0 +1,4 @@
mod common;
mod conformance;
mod perf;
mod robustness;
@@ -1,7 +1,7 @@
//! Decode speed (ignored by default; run with //! Decode speed (ignored by default; run with
//! `cargo test --release -p filmcraft-hevc --test perf -- --ignored --nocapture`). //! `cargo test --release -p filmcraft-hevc --test it perf:: -- --ignored --nocapture`).
mod common; use crate::common;
use filmcraft_hevc::Decoder; use filmcraft_hevc::Decoder;
use std::time::Instant; use std::time::Instant;
@@ -1,6 +1,6 @@
//! hvcC / length-prefixed input and resilience against damaged streams. //! hvcC / length-prefixed input and resilience against damaged streams.
mod common; use crate::common;
use filmcraft_bitstream::annexb_nals; use filmcraft_bitstream::annexb_nals;
use filmcraft_hevc::Decoder; use filmcraft_hevc::Decoder;
+3 -3
View File
@@ -25,7 +25,7 @@ read off files it exports, never from the application itself.
The other formats' editions are listed in the module docs (`edl`, `fcp7`, `fcpxml`, `otio`, `ale`). The other formats' editions are listed in the module docs (`edl`, `fcp7`, `fcpxml`, `otio`, `ale`).
**Invariant — Unicode media URLs:** file URL prefix probes must not panic when their byte boundary **Invariant — Unicode media URLs:** file URL prefix probes must not panic when their byte boundary
falls inside a UTF-8 character. Timeline importers use the shared converter; `tests/file_urls.rs` falls inside a UTF-8 character. Timeline importers use the shared converter; `tests/it/file_urls.rs`
checks expected Unicode paths through the public converter, resolver and EDL importer. checks expected Unicode paths through the public converter, resolver and EDL importer.
## AAF and OMF ## AAF and OMF
@@ -85,7 +85,7 @@ OMF carries no video at all.
## Tests ## Tests
- `tests/aaf_omf.rs`: AAF round trips at 25, 29.97 DF, 23.976 and 59.94 DF (clips, gaps, - `tests/it/aaf_omf.rs`: AAF round trips at 25, 29.97 DF, 23.976 and 59.94 DF (clips, gaps,
two-sided and one-sided transitions, links, markers, start timecodes, constant and keyframed two-sided and one-sided transitions, links, markers, start timecodes, constant and keyframed
gain, media markers and timecode); 512-byte sectors; breakout to mono; embedded trimmed audio gain, media markers and timecode); 512-byte sectors; breakout to mono; embedded trimmed audio
(sample data and source offsets); separate consolidated files with a video mixdown; OMF round (sample data and source offsets); separate consolidated files with a video mixdown; OMF round
@@ -93,7 +93,7 @@ OMF carries no video at all.
random corruption never panic. Nested sequences: AAF round trips as compositions (nests in random corruption never panic. Nested sequences: AAF round trips as compositions (nests in
nests, a nest used twice, another frame and sample rate, breakout to mono), OMF with the nests, a nest used twice, another frame and sample rate, breakout to mono), OMF with the
rendered sound, the audio needs of nests, a sequence inside itself. rendered sound, the audio needs of nests, a sequence inside itself.
- `tests/uncarried_clips.rs`: every format's export report names each graphic clip and adjustment - `tests/it/uncarried_clips.rs`: every format's export report names each graphic clip and adjustment
layer once, as a warning; a sequence without them reports nothing; what a re-import gives back layer once, as a warning; a sequence without them reports nothing; what a re-import gives back
matches the report; the FCP7 XML import names a clip item whose file is not defined. matches the report; the FCP7 XML import names a clip item whose file is not defined.
- `src/aaf/tests.rs`: written files checked against the required properties of every class - `src/aaf/tests.rs`: written files checked against the required properties of every class
@@ -1,6 +1,6 @@
//! AAF and OMF: export → import round trips (no external oracle exists for these formats). //! AAF and OMF: export → import round trips (no external oracle exists for these formats).
mod support; use crate::support;
use filmcraft_interchange::aaf::{self, AafOptions}; use filmcraft_interchange::aaf::{self, AafOptions};
use filmcraft_interchange::essence::{AudioEssence, EssenceData, EssenceKey, MediaOptions, MixdownVideo, NeedOptions, NestNeeds, audio_needs}; use filmcraft_interchange::essence::{AudioEssence, EssenceData, EssenceKey, MediaOptions, MixdownVideo, NeedOptions, NestNeeds, audio_needs};
@@ -1,6 +1,6 @@
//! Crate API: detection, merging fragments into a project, timecode rebasing. //! Crate API: detection, merging fragments into a project, timecode rebasing.
mod support; use crate::support;
use std::collections::HashSet; use std::collections::HashSet;
@@ -1,6 +1,6 @@
//! CMX 3600 EDL: golden documents (hand-written from the public CMX 3600 conventions) and round trips. //! CMX 3600 EDL: golden documents (hand-written from the public CMX 3600 conventions) and round trips.
mod support; use crate::support;
use filmcraft_interchange::{ExportOptions, Format, ImportOptions, ReelMode, detect, export, export_edl_per_track, import, import_with}; use filmcraft_interchange::{ExportOptions, Format, ImportOptions, ReelMode, detect, export, export_edl_per_track, import, import_with};
use filmcraft_project::{ItemKind, MediaRef, TrackKind, TransitionAlign}; use filmcraft_project::{ItemKind, MediaRef, TrackKind, TransitionAlign};
@@ -1,7 +1,7 @@
//! Final Cut Pro 7 XML (xmeml): a hand-written Premiere-style document (written from the public //! Final Cut Pro 7 XML (xmeml): a hand-written Premiere-style document (written from the public
//! xmeml element descriptions) and export → import round trips. //! xmeml element descriptions) and export → import round trips.
mod support; use crate::support;
use filmcraft_interchange::{ExportOptions, Format, Imported, detect, export, import}; use filmcraft_interchange::{ExportOptions, Format, Imported, detect, export, import};
use filmcraft_media::Generator; use filmcraft_media::Generator;
@@ -1,7 +1,7 @@
//! FCPXML: a hand-written document (from the public FCPXML element reference) with connected //! FCPXML: a hand-written document (from the public FCPXML element reference) with connected
//! clips, a transition, a retimed clip, a gap and a compound clip; plus export → import round trips. //! clips, a transition, a retimed clip, a gap and a compound clip; plus export → import round trips.
mod support; use crate::support;
use filmcraft_interchange::{ExportOptions, FcpxmlVersion, Format, detect, import}; use filmcraft_interchange::{ExportOptions, FcpxmlVersion, Format, detect, import};
use filmcraft_project::{ItemKind, Keyframe, Label, Marker, MarkerId, MarkerKind, MediaRef, Param, ParamValue, Project, TrackKind, TransitionAlign}; use filmcraft_project::{ItemKind, Keyframe, Label, Marker, MarkerId, MarkerKind, MediaRef, Param, ParamValue, Project, TrackKind, TransitionAlign};

Some files were not shown because too many files have changed in this diff Show More