Footage: reduced-resolution renders get movie frames converted at their size

Cause: at Half, Third and Quarter resolution (the viewer's Auto resolution
in most windows) every movie frame was converted from Y'CbCr to f32 RGBA at
full size and then resampled down. In the browser (wasm, one thread) that
was about 8 ms converting a 1080p frame, 6 ms resampling it and 2 ms
zeroing a 33 MB buffer, per layer and frame, for a frame shown at a
quarter or a ninth of its pixels; the footage frame cache held the
full-size frames.

Fix: FootageSource::frame_at_size asks for a frame shown at a given size
(default: the full frame, as before). The renderer asks for it where it
resampled footage (below 3/4 scale; not for proxies, fields or Interpret
Footage's pixel options, which need the full frame). MediaPool makes opaque
4:2:0 movie frames of the footage's own size there: it box-filters them to
half size in Y'CbCr before converting (each 2x2 block of luma, and the
[1 6 1]/8 neighbourhood of its chroma sample, which is the mean of the
upsampled chroma its pixels get), resamples the rest of the way and caches
the result per size. Conversion is linear until colours clip, so this is
the RGB box filter resample starts with, except in blocks where some
colours clip. Other frames convert at full size and resample as before.
In the browser the conversion and resampling take about 4 ms instead of
16. Docs (architecture).

Tests:
- convert::scaled_conversion_matches_resampling_the_converted_frame: half,
  third, quarter and odd sizes of an unclipped 4:2:0 frame are within 1e-5
  of resampling the converted frame; larger sizes, odd frame sizes and
  4:4:4 convert at full size.
- tests_footage_size::reduced_renders_take_footage_resampled_by_the_source:
  a source that resamples frames itself renders the same frames as one
  that doesn't, at 8 and 32 bpc and every resolution, and is asked only
  below 3/4 scale, and not with Invert Alpha.
- footage::h264_frames_at_reduced_sizes (ffmpeg fixture): 1080p H.264
  testsrc2 frames, whose saturated colours clip, at 960x540, 640x360,
  480x270 and 323x182 are 58-62 dB from the resampled full frames, and
  are cached per size.
- convert's ignored 1080p bench also times the conversion at a third.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
daijro
2026-10-09 16:44:49 -05:00
co-authored by Claude Opus 5.5
parent 2ccc66e73b
commit 74b151cedc
6 changed files with 298 additions and 36 deletions
+5 -1
View File
@@ -230,7 +230,11 @@ Nulls / Nulls Follow Points take pins as well as paths, and `puppet.follow` give
Position expression that trails the leader by a delay. The renderer flattens nested effect
groups into `Params` keys (`effects::flatten_params`). Commands: `paint.*`, `puppet.*`.
Half, Third and Quarter resolution render proportionally fewer pixels end to end.
Half, Third and Quarter resolution render proportionally fewer pixels end to end. Footage
starts at the render's size where the source can make it more cheaply than the full frame
(`FootageSource::frame_at_size`): movies in opaque 4:2:0 Y'CbCr are box-filtered to half size
before conversion (a quarter of the conversions; the result differs from converting first only
in blocks where colours clip) and resampled from there, as other frames are from full size.
**GPU compositor** (`crates/gpu`; Project Settings ▸ Video Rendering and Effects ▸ Mercury GPU
Acceleration, the default, or Mercury Software Only; `render.backend`). The CPU renderer is the