mirror of
https://github.com/storytold/filmcraft.git
synced 2026-10-24 10:10:05 -06:00
render: composite only what overlays cover, and reuse the float images
The CPU compositor, which is what export runs, works on premultiplied linear f32 images, 33 MB each at 1080p, and every layer was one of them. A profile of a 9:29 export with a hardware H.264 encoder (so the encoder is no longer the bottleneck) put 61-64 % of the busy CPU in compositing, 12 % in zeroing and copying memory, 8-11 % each in the float -> 8-bit and RGBA -> Y'CbCr conversions and 4-7 % in decoding the ProRes overlays: a banner that is transparent except for its bottom 100 rows still cost a whole conversion and a whole mix, and the bottom layer paid for a zero-filled canvas and a pass that mixes it over that canvas. Three changes, each of which gives the bits the general path gives: - The canvas is allocated when a layer needs it, and an opaque first layer (a Y'CbCr frame without an alpha plane, opacity 100 %, Normal blend, the size of the output) becomes the canvas instead of being mixed over zeros. - A clip of plain media whose frame has an alpha plane (a ProRes 4444 banner) is converted and mixed only inside the rectangle where its alpha is not zero: `VideoFrame::alpha_region`, `to_linear_f32_region` and `blend::composite_at` (`composite` is `composite_at` at the origin; Dissolve's pattern follows the pixel's place in the output). A clip with effects, opacity masks, frame blending, colour management or a picture that is not placed one to one on the output takes the general path, as before. - `filmcraft_frame::pool` keeps idle f32 images (320 MiB at most). `take_f32_overwritten` hands one back with its old contents, which the conversions overwrite whole (so no zero-fill), and the export pipeline gives each frame's image back after the 8-bit conversion. Apple M1 (8 cores), a machine that was not idle, so the two builds alternate and the median is reported: - Synthetic 1080p frame (`cargo run --release -p filmcraft-render --example frame_bench`, 5 rounds): camera only 8.9 -> 6.5 ms, camera + banner 16.2 -> 7.8 ms, same picture checksum. - A real project (1080p25 camera clip, ProRes 4444 banner and subscribe bar), 30 s with the banner exported with the built-in encoder, which is deterministic: 46.4 -> 36.7 s, and both exports are the same file (same MD5), so every frame of the real pipeline matches. - The same project with a hardware encoder, 3 rounds: 30 s with the banner 17.4 -> 11.2 s (CPU 99 -> 56 s), 30 s of the camera alone 11.4 -> 8.8 s, 20 s with the subscribe bar 11.1 -> 7.0 s. Tests: `render/src/region_tests.rs` compares each shortcut with the general path bit for bit (an opaque bottom layer, banners over and under a camera, an 8-bit sticker, every blend mode, several opacities, moved, scaled and effect clips, a bigger source, reduced-resolution rendering), each render twice so the second starts from recycled buffers, and checks which layers take a shortcut; the frame tests check that a region is the crop of the whole conversion, that `alpha_region` is the box around what is not transparent, and that a buffer full of NaN never leaks into a conversion, for every pixel format. Breaking a shortcut on purpose (a region one pixel off, an opaque layer taken as the canvas at half opacity, a conversion that skips the alpha channel) makes them fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Dario Carnelutti
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
86f3375775
commit
d2ab33ba76
@@ -118,9 +118,13 @@ impl Pipeline {
|
||||
if lim.enabled {
|
||||
limit_f32(&mut out, lim.min_percent, lim.max_percent);
|
||||
}
|
||||
// the float image is not needed any more: its buffer serves the next frame's layers
|
||||
filmcraft_frame::pool::recycle_f32(img.px);
|
||||
return (Vec::new(), out);
|
||||
}
|
||||
let mut rgba = img.over_black_rgba8();
|
||||
// the float image is not needed any more: its buffer serves the next frame's layers
|
||||
filmcraft_frame::pool::recycle_f32(img.px);
|
||||
if lim.enabled {
|
||||
limit_rgba8(&mut rgba, lim.min_percent, lim.max_percent);
|
||||
}
|
||||
|
||||
+312
-17
@@ -66,6 +66,38 @@ pub struct VideoFrame {
|
||||
pub pts: Tick,
|
||||
}
|
||||
|
||||
/// A rectangle of pixels: `x`, `y` is its top-left corner.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub struct Region {
|
||||
pub x: usize,
|
||||
pub y: usize,
|
||||
pub w: usize,
|
||||
pub h: usize,
|
||||
}
|
||||
|
||||
impl Region {
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.w == 0 || self.h == 0
|
||||
}
|
||||
/// This rectangle cut to a `w`×`h` picture (empty when it lies outside).
|
||||
pub fn clip(self, w: usize, h: usize) -> Region {
|
||||
let x = self.x.min(w);
|
||||
let y = self.y.min(h);
|
||||
Region { x, y, w: self.w.min(w - x), h: self.h.min(h - y) }
|
||||
}
|
||||
/// Whether this rectangle is the whole `w`×`h` picture.
|
||||
pub fn is_full(&self, w: usize, h: usize) -> bool {
|
||||
(self.x, self.y, self.w, self.h) == (0, 0, w, h)
|
||||
}
|
||||
}
|
||||
|
||||
/// First and last index of the items of `row` for which `on` holds.
|
||||
fn row_extent<T>(row: &[T], on: impl Fn(&T) -> bool) -> Option<(usize, usize)> {
|
||||
let first = row.iter().position(&on)?;
|
||||
let last = row.iter().rposition(&on)?;
|
||||
Some((first, last))
|
||||
}
|
||||
|
||||
impl VideoFrame {
|
||||
pub fn rgba8(width: u32, height: u32, data: Vec<u8>) -> Self {
|
||||
debug_assert_eq!(data.len(), (width * height * 4) as usize);
|
||||
@@ -199,14 +231,55 @@ impl VideoFrame {
|
||||
/// transfer. Float frames are already linear and ignore it.
|
||||
pub fn to_linear_f32_decimated_with(&self, n: usize, decode: Option<&DecodeTable>) -> (usize, usize, Vec<f32>) {
|
||||
let n = n.max(1);
|
||||
let (ow, oh) = self.decimated_size(n);
|
||||
// every element is written below, so a recycled buffer needs no zero-fill
|
||||
let mut out = pool::take_f32_overwritten(ow * oh * 4);
|
||||
self.convert_region(n, decode, Region { x: 0, y: 0, w: ow, h: oh }, &mut out);
|
||||
(ow, oh, out)
|
||||
}
|
||||
|
||||
/// The size of the picture after `n`×`n` box decimation.
|
||||
pub fn decimated_size(&self, n: usize) -> (usize, usize) {
|
||||
let n = n.max(1);
|
||||
((self.width as usize / n).max(1), (self.height as usize / n).max(1))
|
||||
}
|
||||
|
||||
/// [`VideoFrame::to_linear_f32_decimated_with`] for one rectangle of the decimated picture
|
||||
/// only: the rectangle (clipped to the picture) and its premultiplied linear RGBA pixels, row
|
||||
/// by row. Each pixel is exactly what the full conversion gives at that position. The buffer
|
||||
/// may come from the pool ([`pool::recycle_f32`] hands it back).
|
||||
pub fn to_linear_f32_region(&self, n: usize, decode: Option<&DecodeTable>, region: Region) -> (Region, Vec<f32>) {
|
||||
let n = n.max(1);
|
||||
let (ow, oh) = self.decimated_size(n);
|
||||
let r = region.clip(ow, oh);
|
||||
let mut out = pool::take_f32_overwritten(r.w * r.h * 4);
|
||||
self.convert_region(n, decode, r, &mut out);
|
||||
(r, out)
|
||||
}
|
||||
|
||||
/// Convert the rectangle `r` (inside the decimated picture) into `out`, which must hold exactly
|
||||
/// `r.w * r.h * 4` floats; every one of them is written.
|
||||
fn convert_region(&self, n: usize, decode: Option<&DecodeTable>, r: Region, out: &mut [f32]) {
|
||||
let (w, h) = (self.width as usize, self.height as usize);
|
||||
let (ow, oh) = ((w / n).max(1), (h / n).max(1));
|
||||
let mut out = vec![0f32; ow * oh * 4];
|
||||
if r.w == 0 || r.h == 0 || out.len() != r.w * r.h * 4 {
|
||||
return;
|
||||
}
|
||||
if w == 0 || h == 0 {
|
||||
out.fill(0.0);
|
||||
return;
|
||||
}
|
||||
if let PixelData::RgbaF32(d) = &self.data
|
||||
&& n == 1
|
||||
{
|
||||
out.copy_from_slice(d);
|
||||
return (ow, oh, out);
|
||||
for (ry, row) in out.chunks_exact_mut(r.w * 4).enumerate() {
|
||||
let start = ((r.y + ry) * w + r.x) * 4;
|
||||
match d.get(start..start + r.w * 4) {
|
||||
Some(src) => row.copy_from_slice(src),
|
||||
// a frame with less data than its size says: transparent, not what the buffer held before
|
||||
None => row.fill(0.0),
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
// Encoded (0..1, quantised to 12 bits) → linear lookup for this frame's transfer.
|
||||
let info = self.color;
|
||||
@@ -218,8 +291,10 @@ impl VideoFrame {
|
||||
let inv = 1.0 / (n * n) as f32;
|
||||
match &self.data {
|
||||
PixelData::RgbaF32(d) => {
|
||||
out.par_chunks_mut(ow * 4).enumerate().for_each(|(oy, row)| {
|
||||
for ox in 0..ow {
|
||||
out.par_chunks_mut(r.w * 4).enumerate().for_each(|(ry, row)| {
|
||||
let oy = r.y + ry;
|
||||
for rx in 0..r.w {
|
||||
let ox = r.x + rx;
|
||||
let mut acc = [0f32; 4];
|
||||
for dy in 0..n {
|
||||
let y = (oy * n + dy).min(h - 1);
|
||||
@@ -232,7 +307,7 @@ impl VideoFrame {
|
||||
}
|
||||
}
|
||||
for k in 0..4 {
|
||||
row[ox * 4 + k] = acc[k] * inv;
|
||||
row[rx * 4 + k] = acc[k] * inv;
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -246,8 +321,10 @@ impl VideoFrame {
|
||||
}
|
||||
None => srgb_u8_to_linear_table(),
|
||||
};
|
||||
out.par_chunks_mut(ow * 4).enumerate().for_each(|(oy, row)| {
|
||||
for ox in 0..ow {
|
||||
out.par_chunks_mut(r.w * 4).enumerate().for_each(|(ry, row)| {
|
||||
let oy = r.y + ry;
|
||||
for rx in 0..r.w {
|
||||
let ox = r.x + rx;
|
||||
let mut acc = [0f32; 4];
|
||||
for dy in 0..n {
|
||||
let y = (oy * n + dy).min(h - 1);
|
||||
@@ -262,7 +339,7 @@ impl VideoFrame {
|
||||
}
|
||||
}
|
||||
for k in 0..4 {
|
||||
row[ox * 4 + k] = acc[k] * inv;
|
||||
row[rx * 4 + k] = acc[k] * inv;
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -276,8 +353,10 @@ impl VideoFrame {
|
||||
let kg = 1.0 - kr - kb;
|
||||
let (cr_r, cb_b) = (2.0 * (1.0 - kr), 2.0 * (1.0 - kb));
|
||||
let (cr_g, cb_g) = (cr_r * kr / kg, cb_b * kb / kg);
|
||||
out.par_chunks_mut(ow * 4).enumerate().for_each(|(oy, row)| {
|
||||
for ox in 0..ow {
|
||||
out.par_chunks_mut(r.w * 4).enumerate().for_each(|(ry, row)| {
|
||||
let oy = r.y + ry;
|
||||
for rx in 0..r.w {
|
||||
let ox = r.x + rx;
|
||||
let mut acc = [0f32; 4];
|
||||
for dy in 0..n {
|
||||
let y = (oy * n + dy).min(h - 1);
|
||||
@@ -296,7 +375,7 @@ impl VideoFrame {
|
||||
}
|
||||
}
|
||||
for k in 0..4 {
|
||||
row[ox * 4 + k] = acc[k] * inv;
|
||||
row[rx * 4 + k] = acc[k] * inv;
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -306,8 +385,10 @@ impl VideoFrame {
|
||||
let cw = w.div_ceil(1 << sx);
|
||||
let bits = *bits;
|
||||
let amax = ((1u32 << bits) - 1) as f32;
|
||||
out.par_chunks_mut(ow * 4).enumerate().for_each(|(oy, row)| {
|
||||
for ox in 0..ow {
|
||||
out.par_chunks_mut(r.w * 4).enumerate().for_each(|(ry, row)| {
|
||||
let oy = r.y + ry;
|
||||
for rx in 0..r.w {
|
||||
let ox = r.x + rx;
|
||||
let mut acc = [0f32; 4];
|
||||
for dy in 0..n {
|
||||
let y = (oy * n + dy).min(h - 1);
|
||||
@@ -327,13 +408,43 @@ impl VideoFrame {
|
||||
}
|
||||
}
|
||||
for k in 0..4 {
|
||||
row[ox * 4 + k] = acc[k] * inv;
|
||||
row[rx * 4 + k] = acc[k] * inv;
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
(ow, oh, out)
|
||||
}
|
||||
|
||||
/// Where the picture is not transparent: the smallest rectangle of the `n`×`n`-decimated
|
||||
/// picture that holds every pixel whose alpha is not zero (empty for a picture that is
|
||||
/// transparent everywhere). `None` when the frame has no alpha plane to look at (it is opaque,
|
||||
/// or its alpha is not stored as integers), so nothing can be skipped. Pixels outside the
|
||||
/// rectangle have alpha exactly 0 and, in the premultiplied conversion, colour 0 as well.
|
||||
pub fn alpha_region(&self, n: usize) -> Option<Region> {
|
||||
let n = n.max(1);
|
||||
let (w, h) = (self.width as usize, self.height as usize);
|
||||
let px = w.checked_mul(h)?;
|
||||
// the first and last x of non-transparent pixels in source row `y`
|
||||
let extent: Box<dyn Fn(usize) -> Option<(usize, usize)> + Sync + '_> = match &self.data {
|
||||
PixelData::Yuv8 { alpha: Some(a), .. } if a.len() == px => Box::new(move |y| row_extent(&a[y * w..(y + 1) * w], |v| *v != 0)),
|
||||
PixelData::Yuv16 { alpha: Some(a), .. } if a.len() == px => Box::new(move |y| row_extent(&a[y * w..(y + 1) * w], |v| *v != 0)),
|
||||
PixelData::Rgba8(d) if d.len() == px.checked_mul(4)? => {
|
||||
Box::new(move |y| row_extent(d[y * w * 4..(y + 1) * w * 4].as_chunks::<4>().0, |p: &[u8; 4]| p[3] != 0))
|
||||
}
|
||||
_ => return None,
|
||||
};
|
||||
// (first row, last row, first column, last column) over the rows that have any
|
||||
let found = (0..h)
|
||||
.into_par_iter()
|
||||
.filter_map(|y| extent(y).map(|(a, b)| (y, y, a, b)))
|
||||
.reduce_with(|p, q| (p.0.min(q.0), p.1.max(q.1), p.2.min(q.2), p.3.max(q.3)));
|
||||
let (ow, oh) = self.decimated_size(n);
|
||||
let Some((y0, y1, x0, x1)) = found else { return Some(Region { x: 0, y: 0, w: 0, h: 0 }) };
|
||||
// a decimated pixel covers source pixels [o·n, o·n + n)
|
||||
let (rx0, ry0) = (x0 / n, y0 / n);
|
||||
let (rx1, ry1) = (x1 / n + 1, y1 / n + 1);
|
||||
Some(Region { x: rx0, y: ry0, w: rx1.saturating_sub(rx0), h: ry1.saturating_sub(ry0) }.clip(ow, oh))
|
||||
}
|
||||
|
||||
/// Convert to straight-alpha sRGB RGBA8 for display (fast paths for 8-bit sources).
|
||||
@@ -631,4 +742,188 @@ mod tests {
|
||||
assert_eq!(a.channels[1][0], 0.25);
|
||||
assert_eq!(a.interleaved()[..2], [0.5, 0.25]);
|
||||
}
|
||||
|
||||
/// Frames of every pixel format with partial and zero alpha, at a size that is not a multiple
|
||||
/// of the decimation factors.
|
||||
fn sample_frames() -> Vec<(&'static str, VideoFrame)> {
|
||||
sample_frames_sized(37, 23)
|
||||
}
|
||||
|
||||
fn sample_frames_sized(w: usize, h: usize) -> Vec<(&'static str, VideoFrame)> {
|
||||
let alpha = |x: usize, y: usize| -> u8 {
|
||||
if y < 9 || x < 6 {
|
||||
0
|
||||
} else if y == 9 || x == 6 {
|
||||
77
|
||||
} else {
|
||||
255
|
||||
}
|
||||
};
|
||||
let rgba8: Vec<u8> = (0..w * h).flat_map(|i| [(i * 7 % 251) as u8, (i * 13 % 241) as u8, (i * 29 % 239) as u8, alpha(i % w, i / w)]).collect();
|
||||
let f32s: Vec<f32> =
|
||||
(0..w * h).flat_map(|i| [(i % 17) as f32 / 17.0, (i % 5) as f32 / 5.0, (i % 11) as f32 / 11.0, alpha(i % w, i / w) as f32 / 255.0]).collect();
|
||||
let (cw, ch) = (w.div_ceil(2), h.div_ceil(2));
|
||||
let y8: Vec<u8> = (0..w * h).map(|i| (16 + i * 3 % 219) as u8).collect();
|
||||
let u8p: Vec<u8> = (0..cw * ch).map(|i| (16 + i * 5 % 224) as u8).collect();
|
||||
let v8p: Vec<u8> = (0..cw * ch).map(|i| (16 + i * 11 % 224) as u8).collect();
|
||||
let a8: Vec<u8> = (0..w * h).map(|i| alpha(i % w, i / w)).collect();
|
||||
let y16: Vec<u16> = (0..w * h).map(|i| (64 + i * 3 % 876) as u16).collect();
|
||||
let c16: Vec<u16> = (0..w * h).map(|i| (64 + i * 7 % 896) as u16).collect();
|
||||
let a16: Vec<u16> = (0..w * h).map(|i| (alpha(i % w, i / w) as u16) * 4).collect();
|
||||
let c422: Vec<u16> = (0..cw * h).map(|i| (64 + i * 7 % 896) as u16).collect();
|
||||
let mk = |data| VideoFrame { width: w as u32, height: h as u32, data, color: ColorInfo::REC709, par: (1, 1), pts: Tick::ZERO };
|
||||
vec![
|
||||
("rgba8", mk(PixelData::Rgba8(Arc::new(rgba8)))),
|
||||
("rgba f32", mk(PixelData::RgbaF32(Arc::new(f32s)))),
|
||||
(
|
||||
"yuv8 4:2:0 + alpha",
|
||||
mk(PixelData::Yuv8 {
|
||||
planes: [Arc::new(y8.clone()), Arc::new(u8p.clone()), Arc::new(v8p.clone())],
|
||||
chroma: Chroma::C420,
|
||||
alpha: Some(Arc::new(a8)),
|
||||
}),
|
||||
),
|
||||
("yuv8 4:2:0", mk(PixelData::Yuv8 { planes: [Arc::new(y8), Arc::new(u8p), Arc::new(v8p)], chroma: Chroma::C420, alpha: None })),
|
||||
(
|
||||
"yuv16 4:4:4 + alpha",
|
||||
mk(PixelData::Yuv16 {
|
||||
planes: [Arc::new(y16.clone()), Arc::new(c16.clone()), Arc::new(c16.clone())],
|
||||
chroma: Chroma::C444,
|
||||
bits: 10,
|
||||
alpha: Some(Arc::new(a16.clone())),
|
||||
}),
|
||||
),
|
||||
(
|
||||
"yuv16 4:2:2 + alpha",
|
||||
mk(PixelData::Yuv16 {
|
||||
planes: [Arc::new(y16.clone()), Arc::new(c422.clone()), Arc::new(c422)],
|
||||
chroma: Chroma::C422,
|
||||
bits: 10,
|
||||
alpha: Some(Arc::new(a16)),
|
||||
}),
|
||||
),
|
||||
(
|
||||
"yuv16 4:4:4",
|
||||
mk(PixelData::Yuv16 { planes: [Arc::new(y16), Arc::new(c16.clone()), Arc::new(c16)], chroma: Chroma::C444, bits: 12, alpha: None }),
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_region_is_exactly_the_crop_of_the_full_conversion() {
|
||||
for (name, f) in sample_frames() {
|
||||
for n in [1usize, 2, 3] {
|
||||
let (ow, oh, full) = f.to_linear_f32_decimated_with(n, None);
|
||||
for want in [
|
||||
Region { x: 0, y: 0, w: ow, h: oh },
|
||||
Region { x: 0, y: 0, w: 1, h: 1 },
|
||||
Region { x: ow - 1, y: oh - 1, w: 1, h: 1 },
|
||||
Region { x: 3, y: 2, w: ow - 5, h: oh - 4 },
|
||||
Region { x: 0, y: oh / 2, w: ow, h: 1 },
|
||||
Region { x: ow / 2, y: 0, w: 1, h: oh },
|
||||
// sticks out of the picture: clipped
|
||||
Region { x: ow - 2, y: oh - 2, w: 10, h: 10 },
|
||||
] {
|
||||
let (r, px) = f.to_linear_f32_region(n, None, want);
|
||||
assert_eq!(px.len(), r.w * r.h * 4, "{name} n={n} {want:?}");
|
||||
assert_eq!(r, want.clip(ow, oh));
|
||||
for ry in 0..r.h {
|
||||
for rx in 0..r.w {
|
||||
for k in 0..4 {
|
||||
let (a, b) = (px[(ry * r.w + rx) * 4 + k], full[((r.y + ry) * ow + r.x + rx) * 4 + k]);
|
||||
assert_eq!(a.to_bits(), b.to_bits(), "{name} n={n} {r:?} at ({rx},{ry}) channel {k}: {a} vs {b}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// outside the picture altogether: nothing, and no panic
|
||||
let (r, px) = f.to_linear_f32_region(n, None, Region { x: ow + 5, y: 0, w: 4, h: 4 });
|
||||
assert!(r.is_empty() && px.is_empty(), "{name}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The conversions write every float they hand out: into a buffer that held something else
|
||||
/// (here NaN, which the pool gives back as it was) the result is the one a fresh buffer gives.
|
||||
#[test]
|
||||
fn a_recycled_buffer_never_leaks_its_old_contents() {
|
||||
// big enough for the float images to be worth keeping (64 KiB), at sizes no other test of this
|
||||
// binary converts, so the poisoned buffer is the one the next conversion takes
|
||||
let poison = |len: usize| pool::recycle_f32(vec![f32::NAN; len]);
|
||||
let same = |what: &str, fresh: &[f32], reused: &[f32]| {
|
||||
assert_eq!(fresh.len(), reused.len(), "{what}");
|
||||
if let Some(i) = fresh.iter().zip(reused).position(|(a, b)| a.to_bits() != b.to_bits()) {
|
||||
panic!("{what}: element {i} is {} in a fresh buffer and {} in a recycled one", fresh[i], reused[i]);
|
||||
}
|
||||
};
|
||||
for (name, f) in sample_frames_sized(203, 151) {
|
||||
for n in [1usize, 2] {
|
||||
let (ow, oh) = f.decimated_size(n);
|
||||
let (_, _, fresh) = f.to_linear_f32_decimated(n);
|
||||
poison(ow * oh * 4);
|
||||
let (_, _, reused) = f.to_linear_f32_decimated(n);
|
||||
same(&format!("{name} n={n}, whole picture"), &fresh, &reused);
|
||||
|
||||
let want = Region { x: 1, y: 1, w: 70, h: 60 };
|
||||
let (r, fresh) = f.to_linear_f32_region(n, None, want);
|
||||
assert_eq!(r, want, "{name} n={n}");
|
||||
poison(r.w * r.h * 4);
|
||||
let (_, reused) = f.to_linear_f32_region(n, None, want);
|
||||
same(&format!("{name} n={n}, rectangle"), &fresh, &reused);
|
||||
}
|
||||
}
|
||||
// a frame with less data than its size says (no row of it fits): transparent, never what the buffer held
|
||||
let short = VideoFrame { data: PixelData::RgbaF32(Arc::new(vec![0.5; 100])), ..VideoFrame::rgba_f32(203, 151, vec![0.0; 203 * 151 * 4]) };
|
||||
poison(203 * 151 * 4);
|
||||
let (_, _, px) = short.to_linear_f32_decimated(1);
|
||||
assert!(px.iter().all(|v| *v == 0.0), "rows without data are transparent");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn alpha_region_is_the_box_around_what_is_not_transparent() {
|
||||
for (name, f) in sample_frames() {
|
||||
for n in [1usize, 2, 4] {
|
||||
let got = f.alpha_region(n);
|
||||
let has_alpha = matches!(f.data, PixelData::Rgba8(_) | PixelData::Yuv8 { alpha: Some(_), .. } | PixelData::Yuv16 { alpha: Some(_), .. });
|
||||
assert_eq!(got.is_some(), has_alpha, "{name}");
|
||||
let Some(got) = got else { continue };
|
||||
// the same box measured on the converted picture: pixels whose alpha is not zero
|
||||
let (ow, oh, px) = f.to_linear_f32_decimated(n);
|
||||
let on: Vec<(usize, usize)> = (0..oh).flat_map(|y| (0..ow).map(move |x| (x, y))).filter(|(x, y)| px[(y * ow + x) * 4 + 3] != 0.0).collect();
|
||||
let want = match (on.iter().map(|p| p.0).min(), on.iter().map(|p| p.0).max(), on.iter().map(|p| p.1).min(), on.iter().map(|p| p.1).max()) {
|
||||
(Some(x0), Some(x1), Some(y0), Some(y1)) => Region { x: x0, y: y0, w: x1 - x0 + 1, h: y1 - y0 + 1 },
|
||||
_ => Region::default(),
|
||||
};
|
||||
assert_eq!(got, want, "{name} n={n}");
|
||||
// and everything outside it is exactly transparent black
|
||||
for y in 0..oh {
|
||||
for x in 0..ow {
|
||||
if !(x >= got.x && x < got.x + got.w && y >= got.y && y < got.y + got.h) {
|
||||
assert_eq!(&px[(y * ow + x) * 4..(y * ow + x) * 4 + 4], &[0.0; 4], "{name} n={n} at ({x},{y})");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// a picture that is transparent everywhere
|
||||
let clear = VideoFrame {
|
||||
data: PixelData::Yuv8 {
|
||||
planes: [Arc::new(vec![16; 64]), Arc::new(vec![128; 16]), Arc::new(vec![128; 16])],
|
||||
chroma: Chroma::C420,
|
||||
alpha: Some(Arc::new(vec![0; 64])),
|
||||
},
|
||||
..VideoFrame::rgba8(8, 8, vec![0; 256])
|
||||
};
|
||||
assert_eq!(clear.alpha_region(1), Some(Region::default()));
|
||||
// malformed alpha (too short): nothing can be assumed
|
||||
let short = VideoFrame {
|
||||
data: PixelData::Yuv8 {
|
||||
planes: [Arc::new(vec![16; 64]), Arc::new(vec![128; 16]), Arc::new(vec![128; 16])],
|
||||
chroma: Chroma::C420,
|
||||
alpha: Some(Arc::new(vec![255; 10])),
|
||||
},
|
||||
..VideoFrame::rgba8(8, 8, vec![0; 256])
|
||||
};
|
||||
assert_eq!(short.alpha_region(1), None);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,10 @@
|
||||
//! freed pages (measured: 2.5 GB retained next to 2.5 GB live after twelve clips). Decoders take
|
||||
//! their planes here instead ([`take_u8`], [`take_u16`]) and caches hand evicted frames back
|
||||
//! ([`recycle`]), so steady-state decoding allocates nothing.
|
||||
//!
|
||||
//! The compositor's working images (premultiplied linear `f32`, 33 MB at 1080p) get the same
|
||||
//! treatment ([`take_f32_overwritten`], [`recycle_f32`]): allocated and freed once per layer per
|
||||
//! frame they cost a zero-fill and fresh page faults every time.
|
||||
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::{Arc, Mutex, PoisonError};
|
||||
@@ -60,6 +64,13 @@ static U8: Mutex<Shelf<u8>> = Mutex::new(Shelf::new());
|
||||
static U16: Mutex<Shelf<u16>> = Mutex::new(Shelf::new());
|
||||
static REUSED: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
/// Idle float images. Unlike the plane shelves a buffer keeps its length *and its old contents*:
|
||||
/// whoever overwrites every element gets it back without a zero-fill pass over tens of megabytes.
|
||||
static F32: Mutex<Vec<Vec<f32>>> = Mutex::new(Vec::new());
|
||||
|
||||
/// Bytes of idle float images kept; a buffer that does not fit is freed.
|
||||
const F32_MAX_BYTES: usize = 320 << 20;
|
||||
|
||||
fn take<T>(shelf: &Mutex<Shelf<T>>, len: usize) -> Vec<T> {
|
||||
let found = shelf.lock().unwrap_or_else(PoisonError::into_inner).take(len);
|
||||
match found {
|
||||
@@ -88,6 +99,33 @@ pub fn take_u16(len: usize) -> Vec<u16> {
|
||||
take(&U16, len)
|
||||
}
|
||||
|
||||
/// A buffer of exactly `len` floats whose contents are unspecified (whatever it held last, or zeros
|
||||
/// when none was idle): the caller must write every element before it reads any. Handing it back
|
||||
/// ([`recycle_f32`]) is optional; a buffer that is dropped is simply freed.
|
||||
pub fn take_f32_overwritten(len: usize) -> Vec<f32> {
|
||||
if len.saturating_mul(4) >= MIN_BYTES {
|
||||
let mut idle = F32.lock().unwrap_or_else(PoisonError::into_inner);
|
||||
if let Some(k) = idle.iter().position(|b| b.len() == len) {
|
||||
REUSED.fetch_add(1, Ordering::Relaxed);
|
||||
return idle.swap_remove(k);
|
||||
}
|
||||
}
|
||||
vec![0.0; len]
|
||||
}
|
||||
|
||||
/// Keep a float image nobody needs any more for the next [`take_f32_overwritten`] of its length.
|
||||
pub fn recycle_f32(buf: Vec<f32>) {
|
||||
let bytes = buf.len().saturating_mul(4);
|
||||
if bytes < MIN_BYTES {
|
||||
return;
|
||||
}
|
||||
let mut idle = F32.lock().unwrap_or_else(PoisonError::into_inner);
|
||||
let held: usize = idle.iter().map(|b| b.len().saturating_mul(4)).sum();
|
||||
if held.saturating_add(bytes) <= F32_MAX_BYTES {
|
||||
idle.push(buf);
|
||||
}
|
||||
}
|
||||
|
||||
/// Hand back a frame nobody needs any more: its planes are kept for the next [`take_u8`] /
|
||||
/// [`take_u16`] unless something else still holds the frame or a plane.
|
||||
pub fn recycle(frame: Arc<VideoFrame>) {
|
||||
@@ -111,7 +149,8 @@ pub fn stats() -> PoolStats {
|
||||
let idle = |n: usize, m: usize| n.saturating_add(m);
|
||||
let a = U8.lock().unwrap_or_else(PoisonError::into_inner).bytes;
|
||||
let b = U16.lock().unwrap_or_else(PoisonError::into_inner).bytes;
|
||||
PoolStats { idle_bytes: idle(a, b), reused: REUSED.load(Ordering::Relaxed) }
|
||||
let c: usize = F32.lock().unwrap_or_else(PoisonError::into_inner).iter().map(|b| b.len().saturating_mul(4)).sum();
|
||||
PoolStats { idle_bytes: idle(idle(a, b), c), reused: REUSED.load(Ordering::Relaxed) }
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -119,6 +158,9 @@ mod tests {
|
||||
use super::*;
|
||||
use crate::Chroma;
|
||||
|
||||
/// The plane shelves may hold up to this much alongside the float shelf in a test binary.
|
||||
const U8_BYTES_UPPER_BOUND: usize = 2 * MAX_BYTES;
|
||||
|
||||
#[test]
|
||||
fn shelf_hands_back_a_fitting_buffer_and_stays_within_its_budget() {
|
||||
let mut s = Shelf::<u8>::new();
|
||||
@@ -139,6 +181,33 @@ mod tests {
|
||||
assert!(s.take(1 << 20).is_none(), "2 MiB wastes too much for 1 MiB");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn float_images_come_back_whole_and_unchanged_within_their_budget() {
|
||||
// a length nothing else in this test binary asks for
|
||||
let len = 1_000_003usize;
|
||||
let mut b = take_f32_overwritten(len);
|
||||
assert_eq!(b.len(), len);
|
||||
assert!(b.iter().all(|v| *v == 0.0), "a fresh buffer is zeroed");
|
||||
b[17] = 4.5;
|
||||
let at = b.as_ptr() as usize;
|
||||
recycle_f32(b);
|
||||
// the same allocation, length and contents (the caller overwrites it)
|
||||
let again = take_f32_overwritten(len);
|
||||
assert_eq!((again.as_ptr() as usize, again.len(), again[17]), (at, len, 4.5));
|
||||
// another length does not match, and is not handed the idle one
|
||||
recycle_f32(again);
|
||||
let other = take_f32_overwritten(len + 1);
|
||||
assert_eq!(other.len(), len + 1);
|
||||
assert_ne!(other.as_ptr() as usize, at);
|
||||
// small buffers are not kept, and nothing exceeds the budget
|
||||
recycle_f32(vec![0.0; 100]);
|
||||
let budget_len = F32_MAX_BYTES / 4;
|
||||
recycle_f32(vec![0.0; budget_len + 1]);
|
||||
assert!(stats().idle_bytes <= F32_MAX_BYTES + (U8_BYTES_UPPER_BOUND), "idle {}", stats().idle_bytes);
|
||||
// leave nothing behind for other tests
|
||||
drop(take_f32_overwritten(len));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recycled_frames_feed_the_next_take_unless_still_shared() {
|
||||
// sizes nothing else in this test binary asks for
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
//! Per-frame cost of the CPU compositor at 1920×1080, stage by stage, with synthetic sources:
|
||||
//! a camera-like 8-bit 4:2:0 picture and a ProRes-4444-like 10-bit 4:4:4 banner whose alpha is
|
||||
//! transparent except for its bottom 100 rows.
|
||||
//!
|
||||
//! cargo run --release -p filmcraft-render --example frame_bench
|
||||
//!
|
||||
//! Prints the median of 40 frames (after 6 warm-up frames) in milliseconds, wall time on the
|
||||
//! global rayon pool, and a checksum of the final 8-bit picture of each scenario so two builds can
|
||||
//! be compared for identical output.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
use filmcraft_frame::{AudioBuffer, Chroma, PixelData, VideoFrame};
|
||||
use filmcraft_media::generators::GeneratorSource;
|
||||
use filmcraft_media::{FrameRequest, Generator, MediaInfo, MediaSource};
|
||||
use filmcraft_project::{ItemKind, Label, MediaClip, MediaRef, Project, SequenceSettings, TrackKind};
|
||||
use filmcraft_render::blend::{self, Blend};
|
||||
use filmcraft_render::{Image, RenderOptions, SourceMap, render_sequence};
|
||||
use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick, TimeRange};
|
||||
|
||||
const W: usize = 1920;
|
||||
const H: usize = 1080;
|
||||
const FPS: FrameRate = FrameRate::FPS_25;
|
||||
|
||||
struct Still {
|
||||
info: MediaInfo,
|
||||
frame: Arc<VideoFrame>,
|
||||
}
|
||||
|
||||
impl MediaSource for Still {
|
||||
fn info(&self) -> &MediaInfo {
|
||||
&self.info
|
||||
}
|
||||
fn video_frame(&self, _req: FrameRequest) -> filmcraft_media::Result<Arc<VideoFrame>> {
|
||||
Ok(self.frame.clone())
|
||||
}
|
||||
fn audio(&self, _start: i64, frames: usize, sample_rate: u32) -> filmcraft_media::Result<AudioBuffer> {
|
||||
Ok(AudioBuffer::silence(sample_rate, 2, frames))
|
||||
}
|
||||
}
|
||||
|
||||
/// A camera picture: 8-bit 4:2:0, limited range BT.709, no alpha.
|
||||
fn camera() -> Arc<VideoFrame> {
|
||||
let y: Vec<u8> = (0..W * H).map(|i| (16 + ((i % W) * 200 / W + (i / W) * 30 / H + (i * 7919) % 5) % 219) as u8).collect();
|
||||
let (cw, ch) = (W / 2, H / 2);
|
||||
let u: Vec<u8> = (0..cw * ch).map(|i| (100 + (i % cw) * 40 / cw) as u8).collect();
|
||||
let v: Vec<u8> = (0..cw * ch).map(|i| (150 - (i / cw) * 40 / ch) as u8).collect();
|
||||
Arc::new(VideoFrame {
|
||||
width: W as u32,
|
||||
height: H as u32,
|
||||
data: PixelData::Yuv8 { planes: [Arc::new(y), Arc::new(u), Arc::new(v)], chroma: Chroma::C420, alpha: None },
|
||||
color: filmcraft_color::ColorInfo::REC709,
|
||||
par: (1, 1),
|
||||
pts: Tick::ZERO,
|
||||
})
|
||||
}
|
||||
|
||||
/// A banner like the channel's: 10-bit 4:4:4 with an alpha plane, opaque only in the bottom `bar` rows
|
||||
/// (with a soft one-row edge), transparent elsewhere.
|
||||
fn banner(bar: usize) -> Arc<VideoFrame> {
|
||||
let n = W * H;
|
||||
let top = H - bar;
|
||||
let alpha: Vec<u16> = (0..n)
|
||||
.map(|i| match i / W {
|
||||
r if r + 1 < top => 0,
|
||||
r if r < top => 512,
|
||||
_ => 1023,
|
||||
})
|
||||
.collect();
|
||||
Arc::new(VideoFrame {
|
||||
width: W as u32,
|
||||
height: H as u32,
|
||||
data: PixelData::Yuv16 {
|
||||
planes: [Arc::new(vec![300u16; n]), Arc::new((0..n).map(|i| 512 + (i % 64) as u16).collect()), Arc::new(vec![600u16; n])],
|
||||
chroma: Chroma::C444,
|
||||
bits: 10,
|
||||
alpha: Some(Arc::new(alpha)),
|
||||
},
|
||||
color: filmcraft_color::ColorInfo::REC709,
|
||||
par: (1, 1),
|
||||
pts: Tick::ZERO,
|
||||
})
|
||||
}
|
||||
|
||||
fn project(layers: &[Arc<VideoFrame>]) -> (Project, filmcraft_project::ItemId, SourceMap) {
|
||||
let mut p = Project::new("bench");
|
||||
let mut map = SourceMap::default();
|
||||
let seq = p.new_sequence("s", SequenceSettings { width: W as u32, height: H as u32, frame_rate: FPS, ..Default::default() }, layers.len().max(1), 1, None);
|
||||
for (track, frame) in layers.iter().enumerate() {
|
||||
let g = GeneratorSource::new(Generator::ColorMatte { color: [0.0, 0.0, 0.0, 1.0] }, W as u32, H as u32, FPS, Tick(60 * TICKS_PER_SECOND));
|
||||
let info = g.info().clone();
|
||||
let item = p.add_item(
|
||||
&format!("layer{track}"),
|
||||
Label::Iris,
|
||||
ItemKind::Media(MediaClip {
|
||||
media: MediaRef::Generator(g.generator.clone()),
|
||||
info: info.clone(),
|
||||
interpret: Default::default(),
|
||||
mark_in: None,
|
||||
mark_out: None,
|
||||
markers: vec![],
|
||||
offline: false,
|
||||
proxy: None,
|
||||
identity: None,
|
||||
}),
|
||||
None,
|
||||
);
|
||||
let mut clip = p.make_track_item(item, TrackKind::Video, Tick::ZERO, TimeRange::new(Tick::ZERO, FPS.tick_of(2000)), FPS).expect("clip");
|
||||
clip.scale_to_frame = true;
|
||||
if let Some(t) = p.sequence_mut(seq).and_then(|s| s.video_tracks.get_mut(track)) {
|
||||
t.items.push(clip);
|
||||
}
|
||||
map.0.insert(item, Arc::new(Still { info, frame: frame.clone() }));
|
||||
}
|
||||
(p, seq, map)
|
||||
}
|
||||
|
||||
fn median_ms(mut f: impl FnMut()) -> f64 {
|
||||
for _ in 0..6 {
|
||||
f();
|
||||
}
|
||||
let mut v: Vec<f64> = (0..40)
|
||||
.map(|_| {
|
||||
let t = Instant::now();
|
||||
f();
|
||||
t.elapsed().as_secs_f64() * 1000.0
|
||||
})
|
||||
.collect();
|
||||
v.sort_by(|a, b| a.total_cmp(b));
|
||||
v[v.len() / 2]
|
||||
}
|
||||
|
||||
fn checksum(bytes: &[u8]) -> u64 {
|
||||
bytes.iter().fold(0xcbf29ce484222325u64, |h, b| (h ^ u64::from(*b)).wrapping_mul(0x100000001b3))
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("rayon threads: {}", rayon::current_num_threads());
|
||||
let (cam, ban) = (camera(), banner(100));
|
||||
println!("\n-- stages (ms per frame, 1920x1080) --");
|
||||
println!("camera frame -> linear f32 image : {:6.2}", median_ms(|| drop(std::hint::black_box(cam.to_linear_f32_decimated(1)))));
|
||||
println!("banner frame -> linear f32 image : {:6.2}", median_ms(|| drop(std::hint::black_box(ban.to_linear_f32_decimated(1)))));
|
||||
println!("Image::new (zeroed 33 MB canvas) : {:6.2}", median_ms(|| drop(std::hint::black_box(Image::new(W, H)))));
|
||||
let (_, _, cpx) = cam.to_linear_f32_decimated(1);
|
||||
let (_, _, bpx) = ban.to_linear_f32_decimated(1);
|
||||
let (cam_img, ban_img) = (Image { w: W, h: H, px: cpx }, Image { w: W, h: H, px: bpx });
|
||||
let mut canvas = Image::new(W, H);
|
||||
println!("composite camera over empty : {:6.2}", median_ms(|| blend::composite(&mut canvas, &cam_img, 1.0, Blend::Normal)));
|
||||
println!("composite banner over camera : {:6.2}", median_ms(|| blend::composite(&mut canvas, &ban_img, 1.0, Blend::Normal)));
|
||||
println!("linear f32 -> sRGB 8-bit RGBA : {:6.2}", median_ms(|| drop(std::hint::black_box(canvas.over_black_rgba8()))));
|
||||
|
||||
// FC_RECYCLE=1 hands each frame's float image back to the pool, as the exporter does
|
||||
let recycle = std::env::var_os("FC_RECYCLE").is_some();
|
||||
println!("\n-- whole frames: render_sequence + 8-bit conversion (ms per frame){} --", if recycle { ", buffers recycled" } else { "" });
|
||||
for (name, layers) in [("camera only", vec![cam.clone()]), ("camera + banner", vec![cam.clone(), ban.clone()])] {
|
||||
let (p, seq, map) = project(&layers);
|
||||
let opts = RenderOptions::default();
|
||||
let mut last = Vec::new();
|
||||
let ms = median_ms(|| {
|
||||
let img = render_sequence(&p, seq, FPS.tick_of(10), opts, &map);
|
||||
last = img.over_black_rgba8();
|
||||
if recycle {
|
||||
filmcraft_frame::pool::recycle_f32(img.px);
|
||||
}
|
||||
});
|
||||
let render_only = median_ms(|| {
|
||||
let img = std::hint::black_box(render_sequence(&p, seq, FPS.tick_of(10), opts, &map));
|
||||
if recycle {
|
||||
filmcraft_frame::pool::recycle_f32(img.px);
|
||||
}
|
||||
});
|
||||
println!("{name:16}: render {render_only:6.2} render + 8-bit {ms:6.2} checksum {:016x}", checksum(&last));
|
||||
}
|
||||
}
|
||||
@@ -241,11 +241,25 @@ fn hash2(x: usize, y: usize) -> f32 {
|
||||
/// Composite `src` over `dst` (same size) with `opacity` and `mode`.
|
||||
pub fn composite(dst: &mut Image, src: &Image, opacity: f32, mode: Blend) {
|
||||
debug_assert_eq!((dst.w, dst.h), (src.w, src.h));
|
||||
let w = dst.w;
|
||||
dst.px.par_chunks_mut(w * 4).zip(src.px.par_chunks(w * 4)).enumerate().for_each(|(y, (d, s))| {
|
||||
for x in 0..w {
|
||||
let i = x * 4;
|
||||
let mut sp = [s[i] * opacity, s[i + 1] * opacity, s[i + 2] * opacity, s[i + 3] * opacity];
|
||||
composite_at(dst, src, 0, 0, opacity, mode);
|
||||
}
|
||||
|
||||
/// [`composite`] for a `src` that is only a rectangle of its layer, with its top-left corner at
|
||||
/// (`x0`, `y0`) of `dst`. The rest of the layer is transparent, which leaves `dst` as it is, so only
|
||||
/// the rectangle is visited; its pixels are mixed exactly as [`composite`] mixes them (the dissolve
|
||||
/// pattern follows the pixel's place in `dst`). The part of the rectangle outside `dst` is ignored.
|
||||
pub fn composite_at(dst: &mut Image, src: &Image, x0: usize, y0: usize, opacity: f32, mode: Blend) {
|
||||
let (cw, ch) = (src.w.min(dst.w.saturating_sub(x0)), src.h.min(dst.h.saturating_sub(y0)));
|
||||
if cw == 0 || ch == 0 || src.px.len() != src.w * src.h * 4 {
|
||||
return;
|
||||
}
|
||||
let dw = dst.w;
|
||||
dst.px.par_chunks_mut(dw * 4).skip(y0).take(ch).zip(src.px.par_chunks(src.w * 4)).enumerate().for_each(|(ry, (d, s))| {
|
||||
let y = y0 + ry;
|
||||
for rx in 0..cw {
|
||||
let x = x0 + rx;
|
||||
let (i, j) = (x * 4, rx * 4);
|
||||
let mut sp = [s[j] * opacity, s[j + 1] * opacity, s[j + 2] * opacity, s[j + 3] * opacity];
|
||||
let sa = sp[3];
|
||||
if sa <= 0.0 {
|
||||
continue;
|
||||
|
||||
+195
-39
@@ -35,6 +35,7 @@ pub mod vfx;
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use filmcraft_frame::{PixelData, VideoFrame, pool};
|
||||
use filmcraft_geom::{Affine, Vec2};
|
||||
use filmcraft_media::{FrameRequest, SharedSource};
|
||||
use filmcraft_project::{ItemId, ItemKind, ParamValue, Project, Sequence, TrackItem};
|
||||
@@ -96,19 +97,89 @@ fn render_seq(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, s
|
||||
render_seq_tracks(project, seq, t, opts, sources, None)
|
||||
}
|
||||
|
||||
/// The frame being composited. It stays unallocated until a layer has to be mixed into it: an
|
||||
/// opaque bottom layer simply becomes the canvas, which saves a zero-filled image (33 MB at 1080p)
|
||||
/// and the pass that mixes the layer over it.
|
||||
struct Canvas {
|
||||
img: Option<Image>,
|
||||
w: usize,
|
||||
h: usize,
|
||||
}
|
||||
|
||||
impl Canvas {
|
||||
fn new(w: usize, h: usize) -> Self {
|
||||
Self { img: None, w, h }
|
||||
}
|
||||
fn is_empty(&self) -> bool {
|
||||
self.img.is_none()
|
||||
}
|
||||
/// The image, transparent black when nothing was drawn yet.
|
||||
fn get(&mut self) -> &mut Image {
|
||||
let (w, h) = (self.w, self.h);
|
||||
self.img.get_or_insert_with(|| Image::new(w, h))
|
||||
}
|
||||
fn set(&mut self, img: Image) {
|
||||
self.img = Some(img);
|
||||
}
|
||||
/// A copy of the current image.
|
||||
fn snapshot(&self) -> Image {
|
||||
self.img.clone().unwrap_or_else(|| Image::new(self.w, self.h))
|
||||
}
|
||||
fn into_image(self) -> Image {
|
||||
let (w, h) = (self.w, self.h);
|
||||
self.img.unwrap_or_else(|| Image::new(w, h))
|
||||
}
|
||||
}
|
||||
|
||||
/// What one clip adds to the frame.
|
||||
pub(crate) enum Layer {
|
||||
/// An image the size of the output.
|
||||
Full {
|
||||
image: Image,
|
||||
opacity: f32,
|
||||
blend: Blend,
|
||||
/// Every pixel has alpha exactly 1.
|
||||
opaque: bool,
|
||||
},
|
||||
/// The only part of the layer that is not transparent: a rectangle with its top-left corner at
|
||||
/// (`x`, `y`) of the output (the rest of the layer has alpha 0, which mixes nothing in).
|
||||
Region { image: Image, x: usize, y: usize, opacity: f32, blend: Blend },
|
||||
}
|
||||
|
||||
impl Layer {
|
||||
/// The layer as an image the size of the output, with its opacity and blend mode.
|
||||
fn into_full(self, w: usize, h: usize) -> (Image, f32, Blend) {
|
||||
match self {
|
||||
Layer::Full { image, opacity, blend, .. } => (image, opacity, blend),
|
||||
Layer::Region { image, x, y, opacity, blend } => {
|
||||
let mut full = Image::new(w, h);
|
||||
// the part of the rectangle that lies inside the output
|
||||
let cols = image.w.min(w.saturating_sub(x));
|
||||
for ry in 0..image.h.min(h.saturating_sub(y)) {
|
||||
let (from, to) = (ry * image.w * 4, ((y + ry) * w + x) * 4);
|
||||
if let (Some(src), Some(dst)) = (image.px.get(from..from + cols * 4), full.px.get_mut(to..to + cols * 4)) {
|
||||
dst.copy_from_slice(src);
|
||||
}
|
||||
}
|
||||
(full, opacity, blend)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Render a sequence, or only its video track `only` (a multi-camera angle; drawn even when the
|
||||
/// track's output is off).
|
||||
pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts: RenderOptions, sources: &dyn SourceProvider, only: Option<usize>) -> Image {
|
||||
let (w, h) = output_size(seq, opts.scale);
|
||||
let mut canvas = Image::new(w, h);
|
||||
let mut canvas = Canvas::new(w, h);
|
||||
if opts.depth > MAX_NEST_DEPTH {
|
||||
return canvas;
|
||||
return canvas.into_image();
|
||||
}
|
||||
let tc = format_time(t, seq.settings.frame_rate, seq.settings.drop_frame, TimeDisplay::Timecode, seq.settings.sample_rate as i64);
|
||||
for (ti, track) in seq.video_tracks.iter().enumerate() {
|
||||
// A cancelled frame job (playback moved on) stops here; its result is discarded.
|
||||
if filmcraft_media::cancel::cancelled() {
|
||||
return canvas;
|
||||
return canvas.into_image();
|
||||
}
|
||||
if only.is_some_and(|o| o != ti) || (!track.enabled && only.is_none()) {
|
||||
continue;
|
||||
@@ -133,7 +204,7 @@ pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts
|
||||
} else {
|
||||
transitions::apply_scaled(&tr.effect, &la, &lb, p, opts.scale)
|
||||
};
|
||||
blend::composite(&mut canvas, &mixed, 1.0, Blend::Normal);
|
||||
blend::composite(canvas.get(), &mixed, 1.0, Blend::Normal);
|
||||
continue;
|
||||
}
|
||||
let Some(item) = track.item_at(t) else { continue };
|
||||
@@ -148,7 +219,7 @@ pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts
|
||||
continue;
|
||||
}
|
||||
let mt = item.effect_time_at(t);
|
||||
let mut adjusted = canvas.clone();
|
||||
let mut adjusted = canvas.snapshot();
|
||||
let cx = effects::FxCtx {
|
||||
t: mt,
|
||||
px_scale: opts.scale,
|
||||
@@ -164,7 +235,7 @@ pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts
|
||||
}
|
||||
let (op, bl) = opacity_blend(item, mt);
|
||||
// Adjustment layer opacity (and opacity masks, in sequence pixels) mix adjusted over original.
|
||||
let mut out = canvas.clone();
|
||||
let mut out = canvas.snapshot();
|
||||
let mut adj = adjusted;
|
||||
adj.scale_alpha(op);
|
||||
mask::apply_opacity_masks(&mut adj, item, mt, opts.scale);
|
||||
@@ -174,13 +245,27 @@ pub(crate) fn render_seq_tracks(project: &Project, seq: &Sequence, t: Tick, opts
|
||||
mask::scale_by(&mut adj, &cov);
|
||||
}
|
||||
blend::composite(&mut out, &adj, 1.0, bl);
|
||||
canvas = out;
|
||||
canvas.set(out);
|
||||
continue;
|
||||
}
|
||||
if let Some((layer, op, bl)) = item_layer(project, seq, item, t, opts, sources, &tc) {
|
||||
blend::composite(&mut canvas, &layer, op, bl);
|
||||
match item_layer_ex(project, seq, item, t, opts, sources, &tc, true) {
|
||||
Some(Layer::Full { image, opacity, blend: bl, opaque }) => {
|
||||
if canvas.is_empty() && opaque && opacity == 1.0 && bl == Blend::Normal && (image.w, image.h) == (w, h) {
|
||||
// mixing an opaque layer over transparent black gives the layer itself
|
||||
canvas.set(image);
|
||||
} else {
|
||||
blend::composite(canvas.get(), &image, opacity, bl);
|
||||
pool::recycle_f32(image.px);
|
||||
}
|
||||
}
|
||||
Some(Layer::Region { image, x, y, opacity, blend: bl }) => {
|
||||
blend::composite_at(canvas.get(), &image, x, y, opacity, bl);
|
||||
pool::recycle_f32(image.px);
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
}
|
||||
let mut canvas = canvas.into_image();
|
||||
if opts.depth == 0 && !opts.working_output {
|
||||
colorman::to_display(&mut canvas, &seq.settings.color);
|
||||
}
|
||||
@@ -293,6 +378,31 @@ pub(crate) fn item_layer(
|
||||
sources: &dyn SourceProvider,
|
||||
tc: &str,
|
||||
) -> Option<(Image, f32, Blend)> {
|
||||
let (w, h) = output_size(seq, opts.scale);
|
||||
item_layer_ex(project, seq, item, t, opts, sources, tc, false).map(|l| l.into_full(w, h))
|
||||
}
|
||||
|
||||
/// Whether a placement matrix leaves the layer exactly where it is.
|
||||
fn identity_placement(m: &Affine) -> bool {
|
||||
(m.a - 1.0).abs() < 1e-9 && (m.d - 1.0).abs() < 1e-9 && m.b == 0.0 && m.c == 0.0 && m.e.abs() < 1e-9 && m.f.abs() < 1e-9
|
||||
}
|
||||
|
||||
/// [`item_layer`], which may also hand back only the non-transparent rectangle of a layer
|
||||
/// (`allow_region`): for a clip of plain media (no effects, no masks, no speed blending, no colour
|
||||
/// management, placed as it is) whose frame carries an alpha plane, only the part where alpha is not
|
||||
/// zero is converted and mixed, the rest of the layer being transparent anyway; and a frame
|
||||
/// without alpha is known to be opaque.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(crate) fn item_layer_ex(
|
||||
project: &Project,
|
||||
seq: &Sequence,
|
||||
item: &TrackItem,
|
||||
t: Tick,
|
||||
opts: RenderOptions,
|
||||
sources: &dyn SourceProvider,
|
||||
tc: &str,
|
||||
allow_region: bool,
|
||||
) -> Option<Layer> {
|
||||
let (w, h) = output_size(seq, opts.scale);
|
||||
// `mt`: where effects and Motion/Opacity are evaluated (differs from the shown frame's media
|
||||
// time inside a frame hold without Hold Filters).
|
||||
@@ -309,9 +419,33 @@ pub(crate) fn item_layer(
|
||||
let mut canvas = Image::new(w, h);
|
||||
graphic_clip::render_graphic(item, mt, src_size, &m, &mut canvas);
|
||||
let (op, bl) = opacity_blend(item, mt);
|
||||
return Some((canvas, op, bl));
|
||||
return Some(Layer::Full { image: canvas, opacity: op, blend: bl, opaque: false });
|
||||
}
|
||||
let mut layer = base_layer(project, seq, item, t, opts, sources, want)?;
|
||||
// plain media: nothing but the frame itself reaches the output
|
||||
let plain_media = matches!(pi.kind, ItemKind::Media(_) | ItemKind::Subclip { .. })
|
||||
&& !(opts.effects && item.has_standard_effects())
|
||||
&& !item.effect("opacity").is_some_and(|e| e.enabled && !e.masks.is_empty());
|
||||
let mut fetched = None;
|
||||
if allow_region && plain_media {
|
||||
let (src, frame, n) = media_frame(item, t, sources, want, src_size)?;
|
||||
let (lw, lh) = frame.decimated_size(n);
|
||||
let px_scale = lw as f32 / src_size.0.max(1) as f32;
|
||||
let m =
|
||||
Affine::scale(opts.scale as f64, opts.scale as f64).then_apply(&motion).then_apply(&Affine::scale(1.0 / px_scale as f64, 1.0 / px_scale as f64));
|
||||
if (lw, lh) == (w, h) && identity_placement(&m) && interpolation_blend(item, t, src.info().frame_rate()).is_none() {
|
||||
let cs = colorman::source_space(project, item.item, &frame);
|
||||
if !colorman::needs_management(&seq.settings.color, cs, &frame) {
|
||||
let (op, bl) = opacity_blend(item, mt);
|
||||
return Some(plain_layer(&frame, n, op, bl));
|
||||
}
|
||||
}
|
||||
// anything else takes the general path below, with the frame already fetched
|
||||
fetched = Some((src, frame, n));
|
||||
}
|
||||
let mut layer = match fetched {
|
||||
Some((src, frame, n)) => media_image(project, seq, item, t, &src, &frame, n, want),
|
||||
None => base_layer(project, seq, item, t, opts, sources, want)?,
|
||||
};
|
||||
let px_scale = layer.w as f32 / src_size.0.max(1) as f32;
|
||||
// layer px → source px → sequence px → output px
|
||||
let m = Affine::scale(opts.scale as f64, opts.scale as f64).then_apply(&motion).then_apply(&Affine::scale(1.0 / px_scale as f64, 1.0 / px_scale as f64));
|
||||
@@ -335,21 +469,51 @@ pub(crate) fn item_layer(
|
||||
}
|
||||
}
|
||||
mask::apply_opacity_masks(&mut layer, item, mt, px_scale);
|
||||
let placed = if layer.w == w
|
||||
&& layer.h == h
|
||||
&& (m.a - 1.0).abs() < 1e-9
|
||||
&& (m.d - 1.0).abs() < 1e-9
|
||||
&& m.b == 0.0
|
||||
&& m.c == 0.0
|
||||
&& m.e.abs() < 1e-9
|
||||
&& m.f.abs() < 1e-9
|
||||
{
|
||||
layer
|
||||
} else {
|
||||
layer.transformed(w, h, &m)
|
||||
};
|
||||
let placed = if layer.w == w && layer.h == h && identity_placement(&m) { layer } else { layer.transformed(w, h, &m) };
|
||||
let (op, bl) = opacity_blend(item, mt);
|
||||
Some((placed, op, bl))
|
||||
Some(Layer::Full { image: placed, opacity: op, blend: bl, opaque: false })
|
||||
}
|
||||
|
||||
/// The layer of a plain media frame (see [`item_layer_ex`]): only the part with alpha when the frame
|
||||
/// has an alpha plane, the whole picture otherwise (opaque when it has no alpha at all).
|
||||
fn plain_layer(frame: &VideoFrame, n: usize, opacity: f32, blend: Blend) -> Layer {
|
||||
match frame.alpha_region(n) {
|
||||
Some(r) => {
|
||||
let (r, px) = frame.to_linear_f32_region(n, None, r);
|
||||
Layer::Region { image: Image { w: r.w, h: r.h, px }, x: r.x, y: r.y, opacity, blend }
|
||||
}
|
||||
None => {
|
||||
let (w, h, px) = frame.to_linear_f32_decimated(n);
|
||||
let opaque = matches!(frame.data, PixelData::Yuv8 { alpha: None, .. } | PixelData::Yuv16 { alpha: None, .. });
|
||||
Layer::Full { image: Image { w, h, px }, opacity, blend, opaque }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The source frame of a media clip at timeline `t` (at `want` of its size), and the decimation to
|
||||
/// convert it with.
|
||||
fn media_frame(item: &TrackItem, t: Tick, sources: &dyn SourceProvider, want: f32, src_size: (u32, u32)) -> Option<(SharedSource, Arc<VideoFrame>, usize)> {
|
||||
let src = sources.source(item.item)?;
|
||||
let frame = src.video_frame(FrameRequest { time: item.source_time_at(t), scale: want }).ok()?;
|
||||
let n = decimation(frame.width as f32, src_size.0 as f32 * want);
|
||||
Some((src, frame, n))
|
||||
}
|
||||
|
||||
/// The linear image of a media clip's `frame`, colour managed, and blended with the next frame when
|
||||
/// the clip's speed asks for it.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn media_image(project: &Project, seq: &Sequence, item: &TrackItem, t: Tick, src: &SharedSource, frame: &VideoFrame, n: usize, want: f32) -> Image {
|
||||
let img = colorman::decode(project, item.item, frame, n, &seq.settings.color);
|
||||
match interpolation_blend(item, t, src.info().frame_rate()) {
|
||||
Some((next_time, wgt)) => match src.video_frame(FrameRequest { time: next_time, scale: want }) {
|
||||
Ok(f2) if f2.width == frame.width && f2.height == frame.height => {
|
||||
let b = colorman::decode(project, item.item, &f2, n, &seq.settings.color);
|
||||
img.lerp(&b, wgt)
|
||||
}
|
||||
_ => img,
|
||||
},
|
||||
None => img,
|
||||
}
|
||||
}
|
||||
|
||||
/// A track item's picture at timeline `t` before any effect (decoded, colour managed, frame
|
||||
@@ -370,20 +534,8 @@ pub(crate) fn base_layer(
|
||||
let pi = project.item(item.item)?;
|
||||
Some(match &pi.kind {
|
||||
ItemKind::Media(_) | ItemKind::Subclip { .. } => {
|
||||
let src = sources.source(item.item)?;
|
||||
let frame = src.video_frame(FrameRequest { time: ft, scale: want }).ok()?;
|
||||
let n = decimation(frame.width as f32, src_size.0 as f32 * want);
|
||||
let img = colorman::decode(project, item.item, &frame, n, &seq.settings.color);
|
||||
match interpolation_blend(item, t, src.info().frame_rate()) {
|
||||
Some((next_time, wgt)) => match src.video_frame(FrameRequest { time: next_time, scale: want }) {
|
||||
Ok(f2) if f2.width == frame.width && f2.height == frame.height => {
|
||||
let b = colorman::decode(project, item.item, &f2, n, &seq.settings.color);
|
||||
img.lerp(&b, wgt)
|
||||
}
|
||||
_ => img,
|
||||
},
|
||||
None => img,
|
||||
}
|
||||
let (src, frame, n) = media_frame(item, t, sources, want, src_size)?;
|
||||
media_image(project, seq, item, t, &src, &frame, n, want)
|
||||
}
|
||||
ItemKind::Sequence(nested) => {
|
||||
let sub = RenderOptions { scale: want, effects: opts.effects, depth: opts.depth + 1, captions: false, working_output: true };
|
||||
@@ -514,3 +666,7 @@ mod nest_tests;
|
||||
#[cfg(test)]
|
||||
#[path = "preview_tests.rs"]
|
||||
mod preview_tests;
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "region_tests.rs"]
|
||||
mod region_tests;
|
||||
|
||||
@@ -0,0 +1,301 @@
|
||||
//! The compositor's shortcuts must give exactly what the general path gives, bit for bit: an opaque
|
||||
//! bottom layer taken as the canvas, a layer that is only its non-transparent rectangle, and the
|
||||
//! recycled float buffers. The reference is the plain loop over `item_layer` + `blend::composite`
|
||||
//! (the path every clip takes when no shortcut applies).
|
||||
|
||||
use super::*;
|
||||
use filmcraft_frame::{AudioBuffer, Chroma};
|
||||
use filmcraft_media::generators::GeneratorSource;
|
||||
use filmcraft_media::{Generator, MediaInfo, MediaSource};
|
||||
use filmcraft_project::{Label, MediaClip, MediaRef, SequenceSettings, TrackKind, find_effect};
|
||||
use filmcraft_time::{FrameRate, TICKS_PER_SECOND, TimeRange};
|
||||
|
||||
const FPS: FrameRate = FrameRate::FPS_24;
|
||||
|
||||
struct Still {
|
||||
info: MediaInfo,
|
||||
frame: Arc<VideoFrame>,
|
||||
}
|
||||
|
||||
impl MediaSource for Still {
|
||||
fn info(&self) -> &MediaInfo {
|
||||
&self.info
|
||||
}
|
||||
fn video_frame(&self, _req: FrameRequest) -> filmcraft_media::Result<Arc<VideoFrame>> {
|
||||
Ok(self.frame.clone())
|
||||
}
|
||||
fn audio(&self, _start: i64, frames: usize, sample_rate: u32) -> filmcraft_media::Result<AudioBuffer> {
|
||||
Ok(AudioBuffer::silence(sample_rate, 2, frames))
|
||||
}
|
||||
}
|
||||
|
||||
fn frame(w: usize, h: usize, data: PixelData) -> Arc<VideoFrame> {
|
||||
Arc::new(VideoFrame { width: w as u32, height: h as u32, data, color: filmcraft_color::ColorInfo::REC709, par: (1, 1), pts: Tick::ZERO })
|
||||
}
|
||||
|
||||
/// A camera-like picture: 8-bit 4:2:0, no alpha.
|
||||
fn camera(w: usize, h: usize) -> Arc<VideoFrame> {
|
||||
let (cw, ch) = (w.div_ceil(2), h.div_ceil(2));
|
||||
let y: Vec<u8> = (0..w * h).map(|i| (16 + (i * 7 + i / w * 3) % 219) as u8).collect();
|
||||
let u: Vec<u8> = (0..cw * ch).map(|i| (60 + i * 5 % 140) as u8).collect();
|
||||
let v: Vec<u8> = (0..cw * ch).map(|i| (50 + i * 11 % 150) as u8).collect();
|
||||
frame(w, h, PixelData::Yuv8 { planes: [Arc::new(y), Arc::new(u), Arc::new(v)], chroma: Chroma::C420, alpha: None })
|
||||
}
|
||||
|
||||
/// Alpha 10-bit: 0 outside `rect` (x0, y0, x1, y1), 1023 inside, with a half-transparent one-pixel rim.
|
||||
fn alpha_in(w: usize, h: usize, rect: (usize, usize, usize, usize)) -> Vec<u16> {
|
||||
let (x0, y0, x1, y1) = rect;
|
||||
(0..w * h)
|
||||
.map(|i| {
|
||||
let (x, y) = (i % w, i / w);
|
||||
if x < x0 || x >= x1 || y < y0 || y >= y1 {
|
||||
0
|
||||
} else if x == x0 || y == y0 {
|
||||
512
|
||||
} else {
|
||||
1023
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// A ProRes-4444-like picture: 10-bit 4:4:4 with alpha only inside `rect`.
|
||||
fn banner(w: usize, h: usize, rect: (usize, usize, usize, usize)) -> Arc<VideoFrame> {
|
||||
let n = w * h;
|
||||
frame(
|
||||
w,
|
||||
h,
|
||||
PixelData::Yuv16 {
|
||||
planes: [
|
||||
Arc::new((0..n).map(|i| (200 + i * 3 % 600) as u16).collect()),
|
||||
Arc::new((0..n).map(|i| (400 + i % 200) as u16).collect()),
|
||||
Arc::new((0..n).map(|i| (300 + i * 7 % 300) as u16).collect()),
|
||||
],
|
||||
chroma: Chroma::C444,
|
||||
bits: 10,
|
||||
alpha: Some(Arc::new(alpha_in(w, h, rect))),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// An 8-bit RGBA still with alpha only inside `rect`.
|
||||
fn sticker(w: usize, h: usize, rect: (usize, usize, usize, usize)) -> Arc<VideoFrame> {
|
||||
let a = alpha_in(w, h, rect);
|
||||
let px: Vec<u8> = (0..w * h).flat_map(|i| [(i * 13 % 251) as u8, (i * 7 % 241) as u8, (i * 29 % 239) as u8, (a[i] / 4) as u8]).collect();
|
||||
frame(w, h, PixelData::Rgba8(Arc::new(px)))
|
||||
}
|
||||
|
||||
struct Spec {
|
||||
frame: Arc<VideoFrame>,
|
||||
opacity: f64,
|
||||
blend: u32,
|
||||
motion_scale: Option<f64>,
|
||||
effect: Option<&'static str>,
|
||||
}
|
||||
|
||||
impl Spec {
|
||||
fn new(frame: Arc<VideoFrame>) -> Self {
|
||||
Self { frame, opacity: 100.0, blend: 0, motion_scale: None, effect: None }
|
||||
}
|
||||
}
|
||||
|
||||
const W: usize = 96;
|
||||
const H: usize = 54;
|
||||
|
||||
fn scene(specs: &[Spec]) -> (Project, ItemId, SourceMap) {
|
||||
let mut p = Project::new("regions");
|
||||
let mut map = SourceMap::default();
|
||||
let seq = p.new_sequence("s", SequenceSettings { width: W as u32, height: H as u32, frame_rate: FPS, ..Default::default() }, specs.len(), 1, None);
|
||||
for (track, spec) in specs.iter().enumerate() {
|
||||
let g =
|
||||
GeneratorSource::new(Generator::ColorMatte { color: [0.0, 0.0, 0.0, 1.0] }, spec.frame.width, spec.frame.height, FPS, Tick(10 * TICKS_PER_SECOND));
|
||||
let info = g.info().clone();
|
||||
let item = p.add_item(
|
||||
&format!("layer{track}"),
|
||||
Label::Iris,
|
||||
ItemKind::Media(MediaClip {
|
||||
media: MediaRef::Generator(g.generator.clone()),
|
||||
info: info.clone(),
|
||||
interpret: Default::default(),
|
||||
mark_in: None,
|
||||
mark_out: None,
|
||||
markers: vec![],
|
||||
offline: false,
|
||||
proxy: None,
|
||||
identity: None,
|
||||
}),
|
||||
None,
|
||||
);
|
||||
let mut clip = p.make_track_item(item, TrackKind::Video, Tick::ZERO, TimeRange::new(Tick::ZERO, FPS.tick_of(48)), FPS).expect("clip");
|
||||
clip.scale_to_frame = true;
|
||||
{
|
||||
let o = clip.effect_mut("opacity").expect("opacity");
|
||||
o.params.get_mut("opacity").expect("opacity").value = ParamValue::Float(spec.opacity);
|
||||
o.params.get_mut("blend").expect("blend").value = ParamValue::Choice(spec.blend);
|
||||
}
|
||||
if let Some(s) = spec.motion_scale {
|
||||
clip.effect_mut("motion").expect("motion").params.get_mut("scale").expect("scale").value = ParamValue::Float(s);
|
||||
}
|
||||
if let Some(id) = spec.effect {
|
||||
clip.effects.push(find_effect(id).expect("effect").instance());
|
||||
}
|
||||
if let Some(t) = p.sequence_mut(seq).and_then(|s| s.video_tracks.get_mut(track)) {
|
||||
t.items.push(clip);
|
||||
}
|
||||
map.0.insert(item, Arc::new(Still { info, frame: spec.frame.clone() }));
|
||||
}
|
||||
(p, seq, map)
|
||||
}
|
||||
|
||||
/// What every clip gets when no shortcut applies: each layer from the general path, mixed into a
|
||||
/// zeroed canvas.
|
||||
fn reference(p: &Project, seq: ItemId, t: Tick, opts: RenderOptions, map: &SourceMap) -> Image {
|
||||
let s = p.sequence(seq).expect("sequence");
|
||||
let (w, h) = output_size(s, opts.scale);
|
||||
let mut canvas = Image::new(w, h);
|
||||
let tc = format_time(t, s.settings.frame_rate, s.settings.drop_frame, TimeDisplay::Timecode, s.settings.sample_rate as i64);
|
||||
for track in &s.video_tracks {
|
||||
let Some(item) = track.item_at(t) else { continue };
|
||||
if let Some((layer, op, bl)) = item_layer(p, s, item, t, opts, map, &tc) {
|
||||
blend::composite(&mut canvas, &layer, op, bl);
|
||||
}
|
||||
}
|
||||
canvas
|
||||
}
|
||||
|
||||
fn same_bits(a: &Image, b: &Image, what: &str) {
|
||||
assert_eq!((a.w, a.h, a.px.len()), (b.w, b.h, b.px.len()), "{what}: size");
|
||||
if let Some(i) = a.px.iter().zip(&b.px).position(|(x, y)| x.to_bits() != y.to_bits()) {
|
||||
panic!("{what}: first difference at pixel {} channel {}: {} vs {}", i / 4, i % 4, a.px[i], b.px[i]);
|
||||
}
|
||||
}
|
||||
|
||||
fn check(what: &str, specs: &[Spec], scale: f32) {
|
||||
let (p, seq, map) = scene(specs);
|
||||
let opts = RenderOptions { scale, ..Default::default() };
|
||||
for f in [0, 7, 30] {
|
||||
let t = FPS.tick_of(f);
|
||||
// twice: the second render starts from recycled buffers
|
||||
for round in 0..2 {
|
||||
let got = render_sequence(&p, seq, t, opts, &map);
|
||||
same_bits(&got, &reference(&p, seq, t, opts, &map), &format!("{what} (frame {f}, render {round})"));
|
||||
pool::recycle_f32(got.px);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shortcuts_give_the_general_paths_bits() {
|
||||
let bar = (0, H - 9, W, H);
|
||||
let corner = (50, 3, 91, 20);
|
||||
let full = (0, 0, W, H);
|
||||
let cam = || Spec::new(camera(W, H));
|
||||
let ban = |rect| Spec::new(banner(W, H, rect));
|
||||
|
||||
check("camera only", &[cam()], 1.0);
|
||||
check("camera + bar banner", &[cam(), ban(bar)], 1.0);
|
||||
check("camera + corner banner", &[cam(), ban(corner)], 1.0);
|
||||
check("camera + banner covering everything", &[cam(), ban(full)], 1.0);
|
||||
check("camera + two banners", &[cam(), ban(bar), ban(corner)], 1.0);
|
||||
check("banner alone (first layer)", &[ban(bar)], 1.0);
|
||||
check("banner under the camera", &[ban(bar), cam()], 1.0);
|
||||
check("camera + sticker", &[cam(), Spec::new(sticker(W, H, corner))], 1.0);
|
||||
check("camera + transparent banner", &[cam(), ban((0, 0, 0, 0))], 1.0);
|
||||
// opacity and blend modes (every mode, over the camera and as the first layer)
|
||||
for mode in 0..Blend::ALL.len() as u32 {
|
||||
check(&format!("banner blend {mode}"), &[cam(), Spec { opacity: 70.0, blend: mode, ..ban(corner) }], 1.0);
|
||||
check(&format!("camera blend {mode} first"), &[Spec { opacity: 100.0, blend: mode, ..cam() }, ban(bar)], 1.0);
|
||||
}
|
||||
check("camera at half opacity", &[Spec { opacity: 50.0, ..cam() }, ban(bar)], 1.0);
|
||||
check("camera at zero opacity", &[Spec { opacity: 0.0, ..cam() }, ban(bar)], 1.0);
|
||||
check("banner at zero opacity", &[cam(), Spec { opacity: 0.0, ..ban(bar) }], 1.0);
|
||||
// not placed as it is, or with effects: the general path
|
||||
check("moved camera", &[Spec { motion_scale: Some(50.0), ..cam() }, ban(bar)], 1.0);
|
||||
check("scaled banner", &[cam(), Spec { motion_scale: Some(60.0), ..ban(corner) }], 1.0);
|
||||
check("camera with an effect", &[Spec { effect: Some("brightness_contrast"), ..cam() }, ban(bar)], 1.0);
|
||||
check("banner with an effect", &[cam(), Spec { effect: Some("gaussian_blur"), ..ban(bar) }], 1.0);
|
||||
// a source of another size than the sequence's
|
||||
check("bigger banner", &[cam(), Spec::new(banner(W * 2, H * 2, (0, H * 2 - 18, W * 2, H * 2)))], 1.0);
|
||||
// reduced-resolution rendering decimates every frame
|
||||
check("half scale", &[cam(), ban(bar)], 0.5);
|
||||
check("quarter scale", &[cam(), ban(corner)], 0.25);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn which_layers_take_a_shortcut() {
|
||||
let (p, seq, map) = scene(&[Spec::new(camera(W, H)), Spec::new(banner(W, H, (0, H - 9, W, H)))]);
|
||||
let s = p.sequence(seq).expect("sequence");
|
||||
let (opts, t) = (RenderOptions::default(), FPS.tick_of(3));
|
||||
let layer = |track: usize, allow: bool| item_layer_ex(&p, s, s.video_tracks[track].item_at(t).expect("clip"), t, opts, &map, "", allow);
|
||||
// a camera frame has no alpha: opaque, the whole picture
|
||||
assert!(matches!(layer(0, true), Some(Layer::Full { opaque: true, .. })));
|
||||
// a banner is only its rectangle (rows H-9.. of a 96-wide picture)
|
||||
match layer(1, true) {
|
||||
Some(Layer::Region { image, x, y, .. }) => assert_eq!((x, y, image.w, image.h), (0, H - 9, W, 9)),
|
||||
_ => panic!("the banner should be a region"),
|
||||
}
|
||||
// without permission every layer is a whole image, and none is known to be opaque
|
||||
assert!(matches!(layer(0, false), Some(Layer::Full { opaque: false, .. })));
|
||||
assert!(matches!(layer(1, false), Some(Layer::Full { opaque: false, .. })));
|
||||
// a transparent banner is an empty region
|
||||
let (p, seq, map) = scene(&[Spec::new(banner(W, H, (0, 0, 0, 0)))]);
|
||||
let s = p.sequence(seq).expect("sequence");
|
||||
match item_layer_ex(&p, s, s.video_tracks[0].item_at(t).expect("clip"), t, opts, &map, "", true) {
|
||||
Some(Layer::Region { image, .. }) => assert_eq!((image.w, image.h, image.px.len()), (0, 0, 0)),
|
||||
_ => panic!("a transparent banner is an empty region"),
|
||||
}
|
||||
// an effect or a move keeps the clip on the general path
|
||||
for spec in [Spec { effect: Some("brightness_contrast"), ..Spec::new(camera(W, H)) }, Spec { motion_scale: Some(50.0), ..Spec::new(camera(W, H)) }] {
|
||||
let (p, seq, map) = scene(&[spec]);
|
||||
let s = p.sequence(seq).expect("sequence");
|
||||
assert!(matches!(item_layer_ex(&p, s, s.video_tracks[0].item_at(t).expect("clip"), t, opts, &map, "", true), Some(Layer::Full { opaque: false, .. })));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_rectangle_mixes_like_the_padded_layer_in_every_blend_mode() {
|
||||
let (w, h) = (40usize, 30usize);
|
||||
let rect = Image { w: 11, h: 7, px: (0..11 * 7 * 4).map(|i| [0.0, 0.2, 0.9, 0.5, 0.01][i % 5] * ((i / 4 % 3) as f32 + 1.0) * 0.3).collect() };
|
||||
let base = Image { w, h, px: (0..w * h * 4).map(|i| if i % 4 == 3 { 0.8 } else { ((i * 37 % 100) as f32) / 130.0 }).collect() };
|
||||
for mode in Blend::ALL.iter().copied() {
|
||||
for (x0, y0) in [(0usize, 0usize), (13, 9), (29, 23), (35, 28)] {
|
||||
// the rectangle placed in a transparent layer of the full size
|
||||
let mut padded = Image::new(w, h);
|
||||
for ry in 0..rect.h {
|
||||
for rx in 0..rect.w {
|
||||
if x0 + rx < w && y0 + ry < h {
|
||||
let (s, d) = ((ry * rect.w + rx) * 4, ((y0 + ry) * w + x0 + rx) * 4);
|
||||
padded.px[d..d + 4].copy_from_slice(&rect.px[s..s + 4]);
|
||||
}
|
||||
}
|
||||
}
|
||||
let (mut a, mut b) = (base.clone(), base.clone());
|
||||
blend::composite(&mut a, &padded, 0.8, mode);
|
||||
blend::composite_at(&mut b, &rect, x0, y0, 0.8, mode);
|
||||
same_bits(&a, &b, &format!("{mode:?} at ({x0}, {y0})"));
|
||||
}
|
||||
}
|
||||
// a rectangle that misses the picture, or an empty one, changes nothing
|
||||
let mut c = base.clone();
|
||||
blend::composite_at(&mut c, &rect, 40, 0, 1.0, Blend::Normal);
|
||||
blend::composite_at(&mut c, &rect, 0, 30, 1.0, Blend::Normal);
|
||||
blend::composite_at(&mut c, &Image { w: 0, h: 0, px: vec![] }, 3, 3, 1.0, Blend::Normal);
|
||||
same_bits(&c, &base, "outside");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_region_layer_expands_to_the_whole_layer() {
|
||||
let image = Image { w: 3, h: 2, px: (0..24).map(|i| i as f32 + 1.0).collect() };
|
||||
let (full, op, bl) = Layer::Region { image, x: 2, y: 1, opacity: 0.5, blend: Blend::Multiply }.into_full(6, 4);
|
||||
assert_eq!((full.w, full.h, op, bl), (6, 4, 0.5, Blend::Multiply));
|
||||
assert_eq!(&full.px[((6 + 2) * 4)..((6 + 5) * 4)], &(1..=12).map(|i| i as f32).collect::<Vec<_>>()[..]);
|
||||
assert_eq!(&full.px[((2 * 6 + 2) * 4)..((2 * 6 + 5) * 4)], &(13..=24).map(|i| i as f32).collect::<Vec<_>>()[..]);
|
||||
assert_eq!(full.px.iter().filter(|v| **v != 0.0).count(), 24);
|
||||
// a region that sticks out is cut, not a panic
|
||||
let sticks = Image { w: 4, h: 3, px: vec![1.0; 48] };
|
||||
let (full, ..) = Layer::Region { image: sticks, x: 4, y: 2, opacity: 1.0, blend: Blend::Normal }.into_full(6, 4);
|
||||
assert_eq!(full.px.len(), 6 * 4 * 4);
|
||||
// only the two columns and two rows inside the output are copied, none wrapped into the next row
|
||||
assert_eq!(full.px.iter().filter(|v| **v == 1.0).count(), 2 * 2 * 4);
|
||||
assert!(full.px[..(2 * 6 + 4) * 4].iter().all(|v| *v == 0.0));
|
||||
}
|
||||
@@ -272,6 +272,19 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
|
||||
and handed to the GPU as an image (a layer image keeps its clip's blend mode), so both paths
|
||||
give the same picture. Setting `FILMCRAFT_CPU_COMPOSITE=1`
|
||||
forces the CPU path in the desktop app.
|
||||
- **CPU compositor shortcuts.** The CPU compositor, which is what export runs, works on
|
||||
premultiplied linear `f32` images, 33 MB at 1080p, so what it does not allocate, convert or mix
|
||||
is time saved. Three shortcuts, each of which gives the general path's bits (compared bit for
|
||||
bit, with every blend mode, in `render/src/region_tests.rs`): the canvas stays unallocated
|
||||
until a layer needs it, and an opaque bottom layer (an 8-bit or 16-bit Y'CbCr frame without an
|
||||
alpha plane, opacity 100 %, Normal) simply becomes the canvas; a plain media frame with an alpha
|
||||
plane (a ProRes 4444 banner) is converted and mixed only inside the rectangle where its alpha is
|
||||
not zero (`VideoFrame::alpha_region`, `to_linear_f32_region`, `blend::composite_at`), the rest
|
||||
being transparent anyway; and working images are recycled through `filmcraft_frame::pool`
|
||||
(`take_f32_overwritten` / `recycle_f32`: the buffer comes back with its old contents, so there is
|
||||
no zero-fill) by the export pipeline once a frame is converted to 8 bits. A clip with effects,
|
||||
opacity masks, frame blending, colour management, or a picture that is not placed one to one on
|
||||
the output takes the general path.
|
||||
- **Frame scheduling.** `crates/ui-egui/src/frames.rs` runs a small pool of worker threads with
|
||||
prioritised jobs: the frame on screen first, then playback prefetch, then thumbnails. The UI never
|
||||
decodes. It shows the exact frame when it is ready and holds the nearest cached frame meanwhile.
|
||||
|
||||
@@ -59,6 +59,42 @@ What remains per live decoder is unchanged: the H.264 decoder publishes each mac
|
||||
picture as its own allocations (`Frame::make_row`, five per row), which is where the 290 MB per
|
||||
decoder comes from.
|
||||
|
||||
## Results (CPU compositor: layer rectangles, an opaque bottom layer and recycled images, before → after)
|
||||
|
||||
The CPU compositor is what export runs. In a profile of a 9:29 export it took about a third of the
|
||||
busy CPU with the built-in H.264 encoder (the encoder took the other two thirds), and a faster
|
||||
encoder leaves it as the largest cost of an export. It works on premultiplied linear `f32` images,
|
||||
33 MB each at 1080p, and every layer was one of them: a banner that is transparent except for its
|
||||
bottom 100 rows still cost a full conversion and a full mix, and the bottom layer paid for a
|
||||
zero-filled canvas and a pass that mixes it over that canvas.
|
||||
|
||||
Apple M1 (8 cores: 4 performance + 4 efficiency), 2026-10-06, a machine that was not idle, so the
|
||||
two builds ran alternately in the same session and the median is reported.
|
||||
|
||||
### Per frame, synthetic 1080p sources
|
||||
|
||||
`cargo run --release -p filmcraft-render --example frame_bench`: an 8-bit 4:2:0 camera picture and a
|
||||
10-bit 4:4:4 banner with alpha only in its bottom 100 rows; `render_sequence` followed by the
|
||||
conversion to 8-bit RGBA, milliseconds per frame (median of 40 frames, 5 alternating rounds):
|
||||
|
||||
| scenario | before | after |
|
||||
|---|---|---|
|
||||
| camera only | 8.9 | 6.5 |
|
||||
| camera + banner | 16.2 | 7.8 |
|
||||
|
||||
Both builds give the same picture (same checksum).
|
||||
|
||||
### A real export
|
||||
|
||||
30 s of a 1080p25 camera clip with a ProRes 4444 banner, exported with the built-in encoder
|
||||
(deterministic, byte for byte): 46.4 s before, 36.7 s after. **The two exports are the same file**
|
||||
(same bytes, same MD5), so every frame of the real pipeline (camera decode, ProRes decode,
|
||||
compositing, conversion) matches. The encoder is most of that time; with a faster one the
|
||||
compositor's savings are a larger part of the export.
|
||||
|
||||
What is left in a frame without overlays is the Y'CbCr → linear float conversion of the camera
|
||||
picture (4.5 ms in the synthetic case) and the conversion to 8 bits (2 ms).
|
||||
|
||||
## Results (GPU2: standard effects on the GPU compositor, #30, before → after)
|
||||
|
||||
Before = this change with clips that carry standard effects sent back to the CPU layer path in
|
||||
|
||||
Reference in New Issue
Block a user