- New crates/platform (filmcraft-platform, L5): OS media FFI only. The one
crate allowed `unsafe` (own [lints] with unsafe_code = "deny",
#[allow(unsafe_code)] only on the videotoolbox module, SAFETY comment on
every block, safe Result-returning public API). AGENTS.md §0.3 names the
exception; docs/adr/0001-platform-ffi.md records why and the rules.
- VideoToolbox decoder for avcC / hvcC, 8/10-bit 4:2:0 and 4:2:2: session from
the sample entry's parameter sets with hardware required, asynchronous
decode (2 in flight), biplanar CVPixelBuffer copied to planar Yuv8/Yuv16,
reorder buffer of the stream's own depth, RASL skipped after a starting CRA.
Output callback under catch_unwind.
- HybridDecoder: transparent mid-stream fallback to the software decoder
(replays samples since the last restart point, drops already-returned
pictures). New codecs::software_video_decoder (built-in factories only) and
codecs::hw (switch, counters, NalStreamInfo shared with software decoders;
vui_color / sar_par helpers now used by both).
- Settings ▸ Playback ▸ Hardware decoding: Auto / Off (engine applies it to
the codecs switch); desktop app, CLI/MCP and bench call register().
- perf.stats decode.framesDecoded and decode.hardware counters.
- Tests: bit-exact parity with the software decoders (H.264 High, HEVC Main
open GOP, Main 10; resets, reseeks, flushes), forced-failure fallback,
mutation fuzz of samples and parameter sets, cross-platform fallback with a
stand-in decoder, setting Off, HEVC 4:2:2 vs ffmpeg.
- Bench --hw auto|off; results in docs/performance.md (4K CPU/frame
~86-119 ms -> ~4 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rust/clippy 1.99 added `chunks_exact_to_as_chunks` and tightened
`needless_late_init` and `drain_collect`, failing `clippy -D warnings`
on main.
- `chunks_exact(N)` / `chunks_exact_mut(N)` with a constant N become
`as_chunks::<N>()` / `as_chunks_mut::<N>()` (fixed-size arrays, so
several byte reads drop their per-element indexing). relink's
fingerprint uses the `(chunks, remainder)` split; its hash output is
unchanged.
- Late-initialised `let`s in ac3 bit allocation and opus CELT
bands/rate become `let x = if/match …`.
- `dpb.drain(..).collect()` becomes `std::mem::take(&mut dpb)`.
No behaviour change; `cargo xtask ci` passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mutation fuzzing (mutated ffmpeg-made files through `open_bytes`) found three import crashes:
- an MP4/MOV track with an empty sample-description list: `entries[0]` index out of bounds
(video and audio paths);
- an MP4/MOV track with timescale 0: division by zero in the duration;
- an MPEG-TS stream whose PTS span gave a ~0 fps rate: division by zero in `tick_of`.
Tracks now need samples, a sample description and a timescale to count as playable; frame rates
that round to zero fall back to the default; the TS/PS rate choice is a small function that never
returns a zero rate. Regression tests: `mp4::tests::damaged_track_headers_never_panic`,
`mpeg::tests::damaged_timestamps_never_give_a_zero_rate`.
Also: GOP cache, MKV/MP4/Ogg/MPEG audio paths, MXF and VP9 conversion without
`expect`/`unreachable!`; the crate denies the panicking lints outside tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AIFF is listed as an importable audio format and FilmCraft exports AIFF,
but the symphonia build had only the wav reader enabled, so every .aif /
.aiff import (including FilmCraft's own AIFF exports) failed with
"unsupported media: end of stream". Enable symphonia's aiff reader (the
same MPL-2.0 symphonia-format-riff crate already used for WAV; Cargo.lock
is unchanged). The new test decodes a synthetic 16-bit stereo AIFF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phones record portrait video as landscape pictures plus a rotation in the
container: the MP4/MOV tkhd matrix (ISO/IEC 14496-12 §8.3.2) or the
Matroska Projection pose roll (RFC 9559 §5.1.4.1.28.46). The matrix was
parsed but never used, so every portrait iPhone clip imported sideways
at its landscape size.
- isobmff: Track::display_rotation() reads 0/90/180/270° from the matrix
signs (mirrored / sheared matrices are left alone).
- matroska: parse Projection (type, yaw, pitch, roll);
VideoInfo::display_rotation() for rectangular projections.
- frame: VideoFrame::rotated() turns every plane at its own resolution
(4:2:2 chroma is widened to 4:4:4 for quarter turns) and swaps the PAR.
- codecs: Mp4Source / MkvSource report the displayed size and PAR, and
GopCache rotates decoded frames before caching them.
- web: the WebCodecs source configures the decoder with the coded size
and rotates its output the same way.
Tests: a one-frame ProRes MOV built in the test with 0/90/180/270°
matrices (reported size + which half ends up on top), plane rotation for
RGBA / 4:2:0 / 4:2:2, and a synthetic Matroska file with a -90° roll.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Lumetri grades PQ / HLG working spaces in their own signal normalised to
HDR White (Basic Correction) / HDR Range (Curves) in cd/m²
(filmcraft_color::GradeSpace): exposure in stops of light, highlights above
HDR White kept, HDR Specular scales them; Rec. 709 unchanged.
- HDR static metadata (mdcv/clli, Matroska MasteringMetadata/MaxCLL/MaxFALL)
on VideoStreamInfo::hdr; PQ media use MaxCLL / mastering peak as the
BT.2390 source peak; media.colorInfo reports it.
- HSL Secondary Refine: Denoise (median of the key) and Blur (Gaussian).
- Lumetri Presets (18 original looks in 4 folders) with thumbnails rendered by
our Lumetri on a procedural preview picture; Effects panel rows and a
thumbnail grid when wide; lumetri.presets / applyPreset / presetThumbnails.
- Color Match: Comparison View toggle; Apply Match uses its reference frame.
- Goldens lumetri_hdr_pq and lumetri_presets_grid; D-Log M stays unsupported
(no published formula).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sub-pixel filter passes run over fixed block widths; frame / region planes are recycled
through a bounded pool (zeroed on reuse) instead of fresh calloc / munmap per frame; the
palette mode-info arrays (40 bytes per 4x4) exist only when screen content tools are allowed;
CDF adaptation is branch-free. Draft mode skips deblocking, CDEF and loop restoration of shown
frames that refresh no reference slot. Media-stack test: VP9 / HEVC / AV1 draft playback only
reaches draft requests, and an export afterwards is bit-exact with ffmpeg.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Interpolation keeps its reference windows in reusable scratch (no 2 x 10 KB zeroing per block)
and filters with taps outer / outputs inner over fixed block widths; SAO edge and band offsets
select the offset per lane instead of a table lookup; uni / bi output clips with min / max; runs of
bypass bins decode with one division; context-coded bins are branch-free. Draft mode skips
deblocking and SAO of sub-layer non-reference pictures of the top sub-layer. Bit-exact with 1, 3
and all threads; draft output differs only on flagged pictures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frames are published one superblock row at a time once loop filtered; a post stage per frame
(own thread) assembles bands from the tile strips, filters them in raster order and builds the
output picture, while the next frame's tile columns decode and wait per band for the reference
rows they read. Bit-exact with 1, 3 and all threads. Draft mode skips the loop filter of frames
that refresh no reference slot. av1dec / hevcdec bench knobs and VP9 / AV1 4K playback scenarios
in the bench.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Settings > Playback > Draft decoding (off by default): the Program monitor
asks for draft frames only while playing at 1/2 resolution or lower
(FrameKey::draft, filmcraft_media::cancel::with_draft). The GOP cache runs
the decoder in draft mode for those requests and serves draft frames to draft
requests only, so paused frames, renders and exports always get exact
pictures (codecs test: exports after draft playback match ffmpeg). Draft
plans hand the GPU box-decimated YUV planes at the drawn size. perf.stats
reports draft frames, H.264 threads and the preference; the bench gains
2160p quarter and draft playback cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- HEVC inverse transform: sum basis rows of non-zero coefficients with a
const-generic size (bit-exact, tested against the direct product);
SAO edge offset gets a check-free interior loop. HEVC CPU/frame -60%.
- H.264/HEVC/VP9 planes are moved into VideoFrames, not copied.
- Catch-up decoding: frame jobs tell sources which frames are late
(filmcraft_media::cancel::with_catch_up); the GOP cache skips
non-reference samples of late frames (H.264 nal_ref_idc 0, HEVC
sub-layer non-reference pictures). Scrubs keep the last 2 s decoded.
- perf.stats: engine query + UI/control method (decode counters,
playback shown/dropped, frame-worker decode/render ms, cache use).
- Timeline waveform source peak cached; bench seek section and --only.
- docs/performance.md with before/after results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tiles decode in parallel into private region buffers (Plane / MiInfo with a
frame-coordinate origin) that are merged afterwards; intra block copy frames
keep whole-frame buffers. Post-filter row bands run on a rayon pool (par.rs,
behind the `threads` feature, sequential on wasm32). Frames become jobs that
wait on the reference slots they read and run on frame worker threads, so
independent GOP layers decode concurrently; output stays in order and each
Picture carries the pts of its temporal unit (Decoder::decode_pts, flush,
flush_result). codecs::Av1Decoder passes pts through and drains on flush.
Oracle tests now compare 1 thread, many threads and libdav1d (bit-exact),
with new multi-tile fixtures and an ignored 1080p set. README records fps
before/after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- filmcraft-media: ByteReader/ReaderOpener (open containers without reading the whole
file) and the pending flag for asynchronous (web Blob) reads.
- filmcraft-codecs: MP4/MOV and Matroska open from a ByteReader; registry of reader
openers (platform sources first).
- filmcraft-engine: Services::reader / export_in_memory; streamed imports; media pool
never caches a still-loading file as offline; Session::pump_jobs advances stepped
exports on hosts without threads.
- filmcraft-export: Exporter (one batch per step, re-renders on pending reads) used by
export(); OutputSink for in-memory output.
- filmcraft-ui-egui: FrameServer::pump (time-budgeted cooperative frame jobs on wasm,
retries jobs whose media is loading); screenshots as base64 on wasm; web-time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Av1Decoder in filmcraft-codecs: feeds the av1C configuration OBUs before the
first sample (and after every reset), converts pictures to 8/16-bit planar
YUV with the stream's colour description, and reports random-access points
from the first frame header of each temporal unit (shown key frame) so GOP
seeking ignores bogus container sync flags. Matroska V_AV1 tracks map to an
av01 sample entry; MP4 labels show the AV1 profile.
All-intra 8-bit MP4 and 10-bit WebM files seek sample-exactly against
libdav1d. Inter frames still return "unsupported" until inter prediction
lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New L0 crate filmcraft-dnx, written from SMPTE ST 2019-1:2016 + Amd 1:2023
(tables extracted from the standard's text by tools/extract_tables.py).
Decoder: every compression ID (DNxHD 1235-1260 incl. field-coded and
macroblock-adaptive interlace, thin rasters, RGB 4:4:4; DNxHR 1270-1274
LB/SQ/HQ/HQX/444), 8/10/12-bit, 4:2:0/4:2:2/4:4:4, DCT and lossless alpha,
macroblock rows in parallel. Matches ffmpeg within IDCT precision (max 2 LSB,
<=50 per million beyond 1, 87-99% identical) on 22 ffmpeg fixtures; follows
ffmpeg where it departs from the standard's text (reconstruction offset,
CID 1271 weights, 4:4:4 signalling) - documented in the README.
Encoder: DNxHR LB/SQ/HQ/HQX(10/12-bit)/444, CBR at the eq. 7.1 frame sizes,
uniform-qsf binary search + per-macroblock refinement; ffmpeg decodes every
profile with -xerror (HQ luma PSNR 58.8-66.3 dB).
Wiring: MOV AVdn/AVdh import (codec label from the CID), export
Format::DnxHr (MOV AVdh, dnxProfile lb|sq|hq|hqx) in the export dialog and
file.exportMedia.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Missing media no longer fails: opening a project scans for missing files
(`media.findMissing`, `file.open` → missingMedia). Missing clips render our own
offline slate (render::offline: red striped field, warning triangle, "MEDIA NOT
FOUND", file name, hint) with silent audio, and the Link Media dialog opens.
- Media identity: imports record the size and a fast content fingerprint (hash of
size + first/last MiB) in MediaClip::identity (optional field, older builds
ignore it, no schema bump).
- media.relink checks file name / extension / fingerprint ("Clip ID") / duration /
media start / frame size+rate+channels, refuses a mismatched fingerprint unless
forced, supports Align Timecode (Project::shift_media_time), and relinks the
others automatically by the derived folder remap (Windows/macOS separators).
- media.autoRelink {from,to} | {folder}, media.search, media.replaceFootage,
media.makeOffline {deleteFiles?}, media.offlineAll, media.status, media.linkMedia.
- Media pool caches sources per reference (path, offline flag), so relink, Make
Offline and undo take effect at once; proxy-aware resolution and a full-res
provider that export uses.
- codecs::gop no longer holds its lock while decoding: rayon decoders inside a
parallel export could steal another frame of the same source and deadlock.
- UI: Link Media dialog (rows, match options, folder search with preview, Locate…,
Link, Offline, Offline All, Cancel), Make Offline dialog, Project panel offline
markers and context menu, media thumbnails keyed by media reference.
- Tests: moved media opens offline (slate rendered) and relinks others by folder,
prefix remap and folder search, fingerprint mismatch refused, Make Offline,
headless UI test of the dialog.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- filmcraft_media::cancel: a per-job cancellation flag passed through a thread-local, so the
MediaSource API is unchanged. Frame workers run each job inside it.
- FrameServer::retain_queue (called by schedule_playback every refresh) now also cancels running
jobs for frames the playhead has passed. Before, a worker blocked on a source's decoder lock
went on to decode its frame after playback had moved 50–100 frames on, found it evicted and
restarted the decoder at the keyframe (decoder trace: "seek i=32 key=0 next=142").
- GopCache gives up (CodecError/MediaError::Cancelled) before seeking or decoding for a cancelled
job and between samples; decoder state stays consistent, so the next request carries on.
- The CPU compositor checks the flag between tracks and effects; cancelled results are never
cached.
- Stop drops queued prefetch jobs and cancels running ones (FrameServer::stop_prefetch). They
used to run to completion after Stop, e.g. 14 frames of live effects rendering competing with
a Render In to Out started right after.
bench, same-session A/B (GPU path, 8 s, load avg 35–41):
after-preview Full live play (1080p + Lumetri/Sharpen/Levels/Tint, CPU-bound):
38/154 → 98/94 shown/dropped, 297 → 267 ms CPU/frame, 84 jobs cancelled
after-preview plays from previews, h264-1080, stack3, h264-2160: unchanged within noise
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Root causes (seen with the bench's decoder counters):
- A frame-threaded H.264 decoder holds 16–32 pictures in flight. GopCache only continued the
running decoder when it was at most 16 samples past the wanted sample, so a request for a frame
that had been fed but not output yet restarted decoding at the keyframe: 3–13 extra seeks per
8 s play at 4K, each re-decoding up to a GOP. It now tracks the highest pts output since the
restart and continues while the wanted frame is still inside the decoder.
- The byte budget (384 MB) held 31 4K frames, less than the decoder's lead plus the prefetch
window, so frames were evicted before they were shown and re-decoded. The cache now always has
room for 64 frames.
- A request just past the next sync sample restarted there, skipping frames playback still
wanted, which then forced a restart at the earlier keyframe. A running decoder now carries on
through up to 48 samples into the next GOP.
- Intra-only streams (ProRes render previews, MJPEG) decoded one frame at a time under the cache
lock and re-fed up to 64 samples per miss. VideoDecoder::intra_only() lets GopCache decode them
outside the lock with a pool of decoders, one sample per frame.
Output is unchanged (same decoder, same frames); unit tests cover the in-flight, continue-through
and parallel intra cases.
bench (GPU path, 8 s, same-session A/B, load avg 70–120 on 14 cores):
h264-2160 Full: 0–12 → 71–171 shown of 192; decoder seeks 7–10 → 1; 229–355 → 175 ms CPU/frame
h264-2160 Half: 0–15 → 94–159 shown; seeks 6–13 → 1; 247–524 → 173–185 ms CPU/frame
after-preview Full, 1st play after render: 86/106 → 192/0 shown/dropped, 272 → 26 ms CPU/frame,
1716 → 182 samples decoded; 2nd play: 135/57 → 192/0
h264-1080 and stack3 unchanged (1 and 3 seeks before and after)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- crates/ui-egui/examples/bench_playback.rs drives the Program monitor's own FrameServer,
prefetch (now FrameServer::schedule_playback, shared with the monitor) and PlaybackMeter at a
simulated 60 Hz display, optionally compositing on a headless wgpu device. Scenarios:
h264-1080, stack3 (3 × 1080p, scaled/rotated/opacity), h264-2160, demo, after-preview
(live play → Render Effects In to Out → 2 plays), seek-storm. Reports latency percentiles,
on-time ratio, worker/decode wall and thread-CPU time, UI-thread present time, process CPU per
frame, decoder seeks/samples (codecs::gop_stats) and wasted jobs; --json for machine output.
Fixtures are generated with ffmpeg into target/fixtures/playback.
- FrameServer records per-job timings (thread CPU via rustix clocks) while profiling is on.
- Dropped/shown were counted per UI refresh: "shown" counted every refresh that had the frame
(322 "shown" in an 8 s play of 192 frames) and "dropped" counted a frame that arrived a few ms
after its first refresh. PlaybackMeter counts timeline frames: shown when the exact picture was
on screen while due, dropped otherwise, including frames passed over without a refresh.
Baseline (GPU path, 8 s, load avg 104–168 on 14 cores; CPU columns are load-robust):
h264-1080 Full 191/1 shown/dropped, 46 ms CPU/frame; Half 188/4
stack3 Full 189/3, 102 ms CPU/frame; Half 157/35 (3 extra decoder seeks)
h264-2160 Full 8/184, 313 ms CPU/frame, 10 seeks; demo Full 114/78, Half 191/1
after-preview: live 0/192; 1st play after render 124/67, 2nd 119/73 (UI-thread present
p50 21 ms / p95 45 ms); seek-storm: 0 of 40 seeks shown within 150 ms
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
crates/codecs decodes Opus with filmcraft-opus (48 kHz output):
- Matroska/WebM A_OPUS: CodecPrivate = OpusHead. CodecDelay removes the
pre-skip exactly (header pre-skip when CodecDelay is absent); packet starts
are accumulated from TOC durations because block timestamps are only
tick-accurate. Random access decodes max(SeekPreRoll, 320 ms) first.
- MP4/MOV Opus + dOps: edit list removes pre-skip (dOps pre-skip without
one); same pre-roll on random access.
- Family-1 surround reordered from Vorbis to WAV/SMPTE channel order;
corrupt packets are concealed instead of failing.
Tests (ffmpeg/libopus fixtures, skipped when absent): WebM stereo, MKV mono
VoIP (SILK/hybrid), MKV 5.1 multistream, MP4 stereo; sample-aligned against
libopus (52-89 dB sequential, 45-89 dB after random access).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
