aleleba/effectcraft
1
0
Fork 0
mirror of https://github.com/storytold/effectcraft.git synced 2026-10-28 12:07:48 -06:00
Code Issues Packages Projects Releases Wiki Activity

Footage: reduced-resolution renders get movie frames converted at their size

Browse Source 
Cause: at Half, Third and Quarter resolution (the viewer's Auto resolution
in most windows) every movie frame was converted from Y'CbCr to f32 RGBA at
full size and then resampled down. In the browser (wasm, one thread) that
was about 8 ms converting a 1080p frame, 6 ms resampling it and 2 ms
zeroing a 33 MB buffer, per layer and frame, for a frame shown at a
quarter or a ninth of its pixels; the footage frame cache held the
full-size frames.

Fix: FootageSource::frame_at_size asks for a frame shown at a given size
(default: the full frame, as before). The renderer asks for it where it
resampled footage (below 3/4 scale; not for proxies, fields or Interpret
Footage's pixel options, which need the full frame). MediaPool makes opaque
4:2:0 movie frames of the footage's own size there: it box-filters them to
half size in Y'CbCr before converting (each 2x2 block of luma, and the
[1 6 1]/8 neighbourhood of its chroma sample, which is the mean of the
upsampled chroma its pixels get), resamples the rest of the way and caches
the result per size. Conversion is linear until colours clip, so this is
the RGB box filter resample starts with, except in blocks where some
colours clip. Other frames convert at full size and resample as before.
In the browser the conversion and resampling take about 4 ms instead of
16. Docs (architecture).

Tests:
- convert::scaled_conversion_matches_resampling_the_converted_frame: half,
  third, quarter and odd sizes of an unclipped 4:2:0 frame are within 1e-5
  of resampling the converted frame; larger sizes, odd frame sizes and
  4:4:4 convert at full size.
- tests_footage_size::reduced_renders_take_footage_resampled_by_the_source:
  a source that resamples frames itself renders the same frames as one
  that doesn't, at 8 and 32 bpc and every resolution, and is asked only
  below 3/4 scale, and not with Invert Alpha.
- footage::h264_frames_at_reduced_sizes (ffmpeg fixture): 1080p H.264
  testsrc2 frames, whose saturated colours clip, at 960x540, 640x360,
  480x270 and 323x182 are 58-62 dB from the resampled full frames, and
  are cached per size.
- convert's ignored 1080p bench also times the conversion at a third.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
daijro
2026-10-09 16:44:49 -05:00
co-authored by Claude Opus 5.5
parent 2ccc66e73b
commit 74b151cedc
6 changed files with 298 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/render/src/lib.rs
+35 -12
View File
@@ -40,6 +40,13 @@ use rayon::prelude::*;
pub trait FootageSource: Send + Sync {
/// The frame of `item` at source time `t`, straight from the file (any size).
fn frame(&self, item: ItemId, footage: &Footage, t: Tick) -> Option<Arc<Image>>;
/// [`FootageSource::frame`] for a render that shows it at `width` × `height`, at most half its
/// size: either that frame or, when the source can make it more cheaply than the whole frame
/// (movies, see `effectcraft-media`), the frame already resampled to that size as
/// [`effectcraft_raster::resample`] would.
fn frame_at_size(&self, item: ItemId, footage: &Footage, t: Tick, _width: u32, _height: u32) -> Option<Arc<Image>> {
self.frame(item, footage, t)
}
/// `frames` interleaved stereo sample frames of `item`'s audio from source time `t` at
/// `rate` Hz (`None` when unavailable).
fn audio(&self, _item: ItemId, _footage: &Footage, _t: Tick, _frames: usize, _rate: u32) -> Option<Vec<f32>> {
@@ -1104,8 +1111,10 @@ impl<'a> Renderer<'a> {
}
let lt = ctx.source_time(layer);
// The proxy is decoded instead, at its own size, and fills the footage's frame.
let pf = self.proxy_for(*item).unwrap_or(f);
let img = self.footage_frame(ctx, layer, *item, pf, lt)?;
let proxy = self.proxy_for(*item);
let pf = proxy.unwrap_or(f);
let size = self.reduced_size(f.width, f.height).filter(|_| proxy.is_none());
let img = self.footage_frame(ctx, layer, *item, pf, lt, size)?;
Some(self.footage_buf(&img, pf, f.width, f.height))
}
LayerSource::Text => Some(self.authored(text::render(ctx, layer, self.raster_scale(ctx, layer)))),
@@ -1127,12 +1136,11 @@ impl<'a> Renderer<'a> {
let s = self.opts.scale;
let k = if img.width > 0 { w.max(1) as f64 / img.width as f64 } else { 1.0 };
// Keep native pixels when downsampling is small; resample otherwise.
let mut buf = if s < 0.75 && k <= 1.0 + 1e-9 {
let rw = ((w as f64 * s).round() as u32).max(1);
let rh = ((h as f64 * s).round() as u32).max(1);
Buf { img: effectcraft_raster::resample(img, rw, rh), offset: [0.0; 2], scale: s }
} else {
Buf { img: img.clone(), offset: [0.0; 2], scale: 1.0 / k }
let mut buf = match self.reduced_size(w, h) {
// The source resampled it already (`FootageSource::frame_at_size`).
Some(size) if (img.width, img.height) == size => Buf { img: img.clone(), offset: [0.0; 2], scale: s },
Some((rw, rh)) if k <= 1.0 + 1e-9 => Buf { img: effectcraft_raster::resample(img, rw, rh), offset: [0.0; 2], scale: s },
_ => Buf { img: img.clone(), offset: [0.0; 2], scale: 1.0 / k },
};
if let Some(c) = self.pipe.media_in(f) {
color::convert(&mut buf.img, &c);
@@ -1140,6 +1148,13 @@ impl<'a> Renderer<'a> {
buf
}
/// The size of `w` × `h` layer pixels in this render when it is reduced enough that footage
/// is resampled to it (below 3/4 scale).
fn reduced_size(&self, w: u32, h: u32) -> Option<(u32, u32)> {
let s = self.opts.scale;
(s < 0.75).then(|| (((w as f64 * s).round() as u32).max(1), ((h as f64 * s).round() as u32).max(1)))
}
/// Authored colours into the working space (linear working spaces).
fn authored(&self, mut buf: Buf) -> Buf {
if let Some(c) = self.pipe.authored_in() {
@@ -1151,7 +1166,8 @@ impl<'a> Renderer<'a> {
/// A footage frame at source time `t`, frame-blended between the two nearest source frames
/// when the layer's Frame Blending switch (and the comp's Enable Frame Blending) is on and
/// `t` falls between frames (footage rate ≠ comp rate, time stretch, time remapping).
fn footage_frame(&self, ctx: &EvalCtx, layer: &Layer, item: ItemId, f: &Footage, t: Tick) -> Option<Arc<Image>> {
/// `size`: a plain frame may come resampled to it ([`FootageSource::frame_at_size`]).
fn footage_frame(&self, ctx: &EvalCtx, layer: &Layer, item: ItemId, f: &Footage, t: Tick, size: Option<(u32, u32)>) -> Option<Arc<Image>> {
// Interpret Footage ▸ Separate Fields: each field is a frame at twice the rate.
if f.fields != effectcraft_project::FieldOrder::Off && matches!(f.kind, FootageKind::Video | FootageKind::Sequence) {
let field_rate = FrameRate::new(f.frame_rate.num * 2, f.frame_rate.den);
@@ -1162,11 +1178,13 @@ impl<'a> Renderer<'a> {
let parity = if (i % 2 == 0) == dominant_upper { 0 } else { 1 };
return Some(Arc::new(interpret_pixels(&field_frame(&img, parity), f)));
}
let img = self.footage_frame_raw(ctx, layer, item, f, t)?;
// Interpretation is per pixel but not linear: it runs on the frame before resampling.
let size = size.filter(|_| !(f.invert_alpha || f.linear_light_applies()));
let img = self.footage_frame_raw(ctx, layer, item, f, t, size)?;
if f.invert_alpha || f.linear_light_applies() { Some(Arc::new(interpret_pixels(&img, f))) } else { Some(img) }
}
fn footage_frame_raw(&self, ctx: &EvalCtx, layer: &Layer, item: ItemId, f: &Footage, t: Tick) -> Option<Arc<Image>> {
fn footage_frame_raw(&self, ctx: &EvalCtx, layer: &Layer, item: ItemId, f: &Footage, t: Tick, size: Option<(u32, u32)>) -> Option<Arc<Image>> {
let mode = frame_blend_mode(ctx, layer);
if mode != FrameBlend::Off && matches!(f.kind, FootageKind::Video | FootageKind::Sequence) {
let (i, w) = frame_position(t, f.frame_rate);
@@ -1194,7 +1212,10 @@ impl<'a> Renderer<'a> {
}
}
}
self.footage.frame(item, f, t)
match size {
Some((w, h)) => self.footage.frame_at_size(item, f, t, w, h),
None => self.footage.frame(item, f, t),
}
}
/// Source → masks, clamped/quantised to the project depth.
@@ -2152,6 +2173,8 @@ mod tests_collapse;
#[cfg(test)]
mod tests_color;
#[cfg(test)]
mod tests_footage_size;
#[cfg(test)]
mod tests_frame_blend;
#[cfg(test)]
mod tests_mask_blur;
crates/render/src/tests_footage_size.rs
+83
View File
@@ -0,0 +1,83 @@
//! Footage at a reduced resolution: a source that resamples frames itself
//! ([`FootageSource::frame_at_size`], movies in `effectcraft-media`) renders what resampling the
//! full frame renders.
use std::sync::{Arc, Mutex};
use rayon::prelude::*;
use effectcraft_color::Label;
use effectcraft_project::build;
use effectcraft_project::{AlphaMode, BitDepth, Comp, Footage, FootageKind, ItemId, ItemKind, LayerSource, Project};
use effectcraft_time::{FrameRate, Tick};
use crate::{FootageSource, Image, RenderOpts, Renderer};
/// Footage with a textured frame; `sizes` (when set) resamples it itself and records the sizes
/// it was asked for.
struct Clip {
sizes: Option<Mutex<Vec<(u32, u32)>>>,
}
impl FootageSource for Clip {
fn frame(&self, _: ItemId, f: &Footage, _: Tick) -> Option<Arc<Image>> {
let mut img = Image::new(f.width, f.height);
img.rows_mut().for_each(|(y, row)| {
for (x, p) in row.iter_mut().enumerate() {
let s = (0.5 + 0.3 * (x as f32 * 0.37 + y as f32 * 0.11).sin()) * 0.9;
*p = [s, 1.0 - s, s * 0.5, 1.0];
}
});
Some(Arc::new(img))
}
fn frame_at_size(&self, item: ItemId, f: &Footage, t: Tick, w: u32, h: u32) -> Option<Arc<Image>> {
let Some(sizes) = &self.sizes else { return self.frame(item, f, t) };
sizes.lock().unwrap().push((w, h));
Some(Arc::new(effectcraft_raster::resample(&*self.frame(item, f, t)?, w, h)))
}
}
fn project(depth: BitDepth, invert_alpha: bool) -> (Project, ItemId) {
let mut p = Project::default();
p.settings.bit_depth = depth;
let comp = Comp::new(96, 54, FrameRate::FPS_30, Tick::from_seconds_f64(1.0));
let cid = p.add_item("Comp", Label::Sandstone, None, ItemKind::Comp(comp.clone().into()));
let f = Footage {
path: "clip.mp4".into(),
kind: FootageKind::Video,
width: 96,
height: 54,
pixel_aspect: 1.0,
frame_rate: FrameRate::FPS_30,
duration: Tick::from_seconds_f64(1.0),
has_video: true,
alpha: AlphaMode::Ignore,
loop_count: 1,
invert_alpha,
..Default::default()
};
let fid = p.add_item("clip", Label::Aqua, None, ItemKind::Footage(f));
let l = build::layer(&mut p, &comp, "clip", LayerSource::Footage { item: fid }, (96, 54), None);
p.comp_mut(cid).unwrap().layers.push(l);
(p, cid)
}
#[test]
fn reduced_renders_take_footage_resampled_by_the_source() {
for depth in [BitDepth::Bpc8, BitDepth::Bpc32] {
let (p, cid) = project(depth, false);
for (scale, asked) in [(1.0, None), (0.75, None), (0.5, Some((48, 27))), (1.0 / 3.0, Some((32, 18))), (0.25, Some((24, 14)))] {
let opts = RenderOpts { scale, ..Default::default() };
let plain = Renderer::new(&p, &Clip { sizes: None }, opts).comp_frame(cid, Tick::ZERO);
let sized = Clip { sizes: Some(Mutex::default()) };
let frame = Renderer::new(&p, &sized, opts).comp_frame(cid, Tick::ZERO);
assert_eq!(frame, plain, "{depth:?} at {scale}");
assert_eq!(sized.sizes.unwrap().into_inner().unwrap(), asked.into_iter().collect::<Vec<_>>(), "{depth:?} at {scale}");
}
}
// Interpreted pixels (Invert Alpha) are interpreted at full size, then resampled.
let (p, cid) = project(BitDepth::Bpc8, true);
let sized = Clip { sizes: Some(Mutex::default()) };
Renderer::new(&p, &sized, RenderOpts { scale: 0.5, ..Default::default() }).comp_frame(cid, Tick::ZERO);
assert!(sized.sizes.unwrap().into_inner().unwrap().is_empty());
}