mirror of
https://github.com/storytold/filmcraft.git
synced 2026-10-24 10:10:05 -06:00
main
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dario CarneluttiandClaude Sonnet 5.5
|
389cfefe92 |
platform: ask for the hardware HEVC encoder in the background at startup
`Format::Hevc` is available only where a hardware HEVC session can be created, and `hardware_encode::hevc_available` finds out by creating a small one on the first question. That question comes from the first draw of Export mode's format list, on the UI thread, and the session takes about 0.1 s (88-100 ms in seven of eight fresh processes on an Apple M1, 222 ms in the first): the app stalls for that long the first time Export is opened. `register()` now starts the probe on a thread of its own (`warm_hevc_probe`, once however many times `register` is called). Whoever asks while it runs waits for the same `OnceLock` answer, no longer than asking first; a thread that cannot start, or a probe that panics (caught), leaves the question to the first caller as before. `hevc_probed()` says whether the answer exists, for diagnostics and tests. Test: `tests/hevc_probe.rs` registers, never calls `hevc_available`, and waits for `hevc_probed()` (it fails after 60 s without the call in `register`). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Dario CarneluttiandClaude Sonnet 5.5
|
10879f385e |
export: free the frame pool when an export ends
The export pipeline hands the float image of every frame it converts back to `filmcraft_frame::pool` (#184), which keeps up to 320 MiB of them, and the decoders' plane shelves hold up to 192 MiB of each type. An export's images are the size of its frames, which nothing else asks for, so a finished export left them idle until the next one. `pool::trim()` frees every idle buffer. `Pipeline` calls it when it is dropped, one place for every way an export ends (done, failed, cancelled, dropped half way by a host that steps it, as the web build does) and for every caller of `export` / `Exporter` (previews, proxies and AAF/OMF mixdowns share the pipeline). A job still rendering at that moment allocates a few images again: the pool is only a cache. Measured with a synthetic 3 s 1080p24 matte exported as PNG frames three times in one process (Apple M1, `footprint`, 10 s after the last export): 575 MB without the trim, the pool holding 332 MB idle, and 258 MB with it. macOS returns the freed pages over a few seconds, not at once. Tests: `crates/frame/tests/pool_trim.rs` (every shelf emptied, the pool works afterwards) and `crates/export/tests/pool_trim.rs` (an image sequence, a finished and a cancelled video, an export that could not start and a stepped export dropped half way each leave nothing idle; without the trim they fail). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Dario CarneluttiandClaude Sonnet 5.5
|
790b3a0697 |
platform: H.265 (HEVC) export with VideoToolbox
Adds `Format::Hevc` (Export > Format > H.265 (HEVC); `"format": "hevc"` in
`file.exportMedia`): HEVC Main, 8-bit 4:2:0, in MP4 (`hvc1`) or QuickTime, with AAC audio,
encoded by the macOS hardware encoder. There is no built-in H.265 encoder (one in pure Rust
is a project of its own, and x265 is GPL), so the format exists only where the OS has one.
- `videotoolbox_encode`: the session wrapper creates HEVC sessions too (`VtProfile::HevcMain`;
the profile also says which codec). Keyframes are the HEVC random access points (NAL types
16 to 21); the parameter sets are VPS / SPS / PPS; the `hvcC` record in the MP4 sample
entry is the one VideoToolbox writes (sample description extension atoms), so profile,
level and flags are the encoder's own. `VtEncoder::uses_hardware()` asks VideoToolbox
whether the session runs on a hardware encoder (a test checks H.264 and HEVC), so a
requirement dropped by accident could never become a silent software encode.
- `hardware_encode`: one adapter for both codecs. Choosing the format is the opt-in (the
`hardware_encoding` setting is for H.264). `hevc_available()` (one small hardware session,
created once) is registered with the new `filmcraft_export::register_format_probe`, so
`available(Format::Hevc)`, the format list of Export mode and `exportFormats` follow the
machine. `Format::has_builtin_encoder` says which formats need such a probe.
- `export`: `Format::is_h26x` where H.264 and H.265 behave alike (MP4 or QuickTime, AAC, size
estimate, summary line). Two-pass VBR is refused up front ("H.265 export has no two-pass
mode") and HDR sequences export as SDR, because there is no encoder to fall back to.
- Export mode: H.265 shows H.264's multiplexer, audio, metadata and bitrate controls, without
profile, level, two-pass or the hardware checkbox, and says "Encoder: Hardware".
Apple M1 (8 cores), 1080p25, a 9:29 timeline (camera clip, ProRes 4444 overlays, AAC,
loudness): hardware H.265 at 10 Mb/s 195 s, hardware H.264 at 20 Mb/s 204 s (CPU 1029 s and
1042 s; the same H.264 export took 193 s and 200 s on other runs). It is not faster: the codec
that compresses further is the more expensive one (raw encoders: 166 and 189 frames/s), and
the export is bound by the CPU compositor, not by the media engine.
Quality per bit, 30 s of the same timeline against a ProRes 422 HQ export (average PSNR over
the three planes, SSIM): 20 Mb/s 46.88 dB / 0.9902 (H.264) and 46.91 dB / 0.9903 (H.265);
10 Mb/s 44.96 / 0.9863 and 45.09 / 0.9863; 6 Mb/s 43.72 / 0.9833 and 44.31 / 0.9843; 4 Mb/s
42.91 / 0.9807 and 43.64 / 0.9825; 3 Mb/s 42.37 / 0.9786 and 43.05 / 0.9807. The same picture
at 10 Mb/s and above; at 3 to 6 Mb/s H.265 needs 22 to 31 % less bitrate for the same PSNR.
Both formats therefore keep the same default bitrate. Details in docs/performance.md (HW3).
Tests: tests/hardware_encode.rs gets the HEVC counterparts of the H.264 ones (what is taken
and declined, a round trip through our HEVC decoder with luma PSNR above 30 dB, keyframes, MP4
and QuickTime exports with AAC, ffmpeg / ffprobe reading `hevc` / `Main` / `hvc1` /
`yuv420p` / BT.709, exact sizes, hostile configurations, encoders dropped at any point, the
format list agreeing with the probe, sessions on the hardware encoder); settings_tests.rs
checks names, ids, defaults, the summary, two-pass refusal and the probe; export_ui.rs the
controls H.265 shows and hides.
Stacked on the hardware H.264 encoder (the first commit of this branch).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
||
|
Dario CarneluttiandClaude Sonnet 5.5
|
283ea80a62 |
export: show the estimated time left of a render next to its percentage
Export mode, the status bar and the Progress panel showed "Encoding 37%" and nothing about how
long the rest would take. They now say "37% · 2:05 left".
- `Progress::eta()` (the new `pace` module) is the time left at the job's speed over the last
15 seconds. Each call is also a reading, so the UI polling on every frame it draws the job
feeds it, and none of the places that advance `done` had to change. A trailing window follows
the real speed changes of an export (an intro, a banner, plain camera footage) where an
average since the start lags; a stall makes the estimate grow instead of freezing it; a long
idle start (the loudness pass) is not counted as the job's speed; nothing is shown until a
second of readings has progress, and nothing once the job is done or finished.
- `format_eta`: `45 s`, `2:05`, `1:02:05`, rounded up.
- Shown in the status bar ("Exporting... 37% · 2:05 left", and the `status.job.progress`
automation label), in the Export queue (on its own line: the column is narrow) and in the
Progress panel. The engine adds `etaSeconds` (null while unknown) to every job of `jobs.list`
and to the encoding items of `export.queue.list`; docs/agents.md says so.
Measured on a whole 9:29 export (193 s, Apple M1), asking every 3 s: the first estimate comes
after 6 s; from 30 s on, the predicted end of the export is on average 1.1 s early and never
more than 14.5 s off (7.5 % of the run). The status bar read "9% · 3:00 left" at 21 s and
"97% · 5 s left" at 184 s.
Tests: `pace` (steady job, speed change, stall, idle start, count going back, bounded history,
extreme values, a whole `Progress`), `format_eta`, the engine (`etaSeconds` of jobs and queue
items), and export_ui (status bar label and queue item).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
||
|
Dario CarneluttiandClaude Sonnet 5.5
|
d2ab33ba76 |
render: composite only what overlays cover, and reuse the float images
The CPU compositor, which is what export runs, works on premultiplied linear f32 images, 33 MB each at 1080p, and every layer was one of them. A profile of a 9:29 export with a hardware H.264 encoder (so the encoder is no longer the bottleneck) put 61-64 % of the busy CPU in compositing, 12 % in zeroing and copying memory, 8-11 % each in the float -> 8-bit and RGBA -> Y'CbCr conversions and 4-7 % in decoding the ProRes overlays: a banner that is transparent except for its bottom 100 rows still cost a whole conversion and a whole mix, and the bottom layer paid for a zero-filled canvas and a pass that mixes it over that canvas. Three changes, each of which gives the bits the general path gives: - The canvas is allocated when a layer needs it, and an opaque first layer (a Y'CbCr frame without an alpha plane, opacity 100 %, Normal blend, the size of the output) becomes the canvas instead of being mixed over zeros. - A clip of plain media whose frame has an alpha plane (a ProRes 4444 banner) is converted and mixed only inside the rectangle where its alpha is not zero: `VideoFrame::alpha_region`, `to_linear_f32_region` and `blend::composite_at` (`composite` is `composite_at` at the origin; Dissolve's pattern follows the pixel's place in the output). A clip with effects, opacity masks, frame blending, colour management or a picture that is not placed one to one on the output takes the general path, as before. - `filmcraft_frame::pool` keeps idle f32 images (320 MiB at most). `take_f32_overwritten` hands one back with its old contents, which the conversions overwrite whole (so no zero-fill), and the export pipeline gives each frame's image back after the 8-bit conversion. Apple M1 (8 cores), a machine that was not idle, so the two builds alternate and the median is reported: - Synthetic 1080p frame (`cargo run --release -p filmcraft-render --example frame_bench`, 5 rounds): camera only 8.9 -> 6.5 ms, camera + banner 16.2 -> 7.8 ms, same picture checksum. - A real project (1080p25 camera clip, ProRes 4444 banner and subscribe bar), 30 s with the banner exported with the built-in encoder, which is deterministic: 46.4 -> 36.7 s, and both exports are the same file (same MD5), so every frame of the real pipeline matches. - The same project with a hardware encoder, 3 rounds: 30 s with the banner 17.4 -> 11.2 s (CPU 99 -> 56 s), 30 s of the camera alone 11.4 -> 8.8 s, 20 s with the subscribe bar 11.1 -> 7.0 s. Tests: `render/src/region_tests.rs` compares each shortcut with the general path bit for bit (an opaque bottom layer, banners over and under a camera, an 8-bit sticker, every blend mode, several opacities, moved, scaled and effect clips, a bigger source, reduced-resolution rendering), each render twice so the second starts from recycled buffers, and checks which layers take a shortcut; the frame tests check that a region is the crop of the whole conversion, that `alpha_region` is the box around what is not transparent, and that a buffer full of NaN never leaks into a conversion, for every pixel format. Breaking a shortcut on purpose (a region one pixel off, an opaque layer taken as the canvas at half opacity, a conversion that skips the alpha channel) makes them fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Dario CarneluttiandClaude Sonnet 5.5
|
4cab6bc5da |
platform: opt-in hardware H.264 encoding with VideoToolbox
Export encodes H.264 on the CPU with the built-in encoder, which is most of the time of an export (about two thirds of the busy CPU in a profile of a 20 s export). On macOS the media engine can do it: a VideoToolbox compression session behind `filmcraft_export::VideoEncoder`, registered by `filmcraft_platform::register()`. - `ExportSettings::hardware_encoding` (`HardwareEncoding::Off` | `Auto`; Export > Video > Hardware Encoding, `"hardwareEncoding": "off|auto"` in `file.exportMedia`, a boolean is understood too), off by default: the built-in encoder's output is byte-identical on every machine, a hardware encoder's is not. - `videotoolbox_encode` (FFI, same containment as the decoder: SAFETY comments, no panic across the callback, safe API) and `hardware_encode` (safe adapter and factory). - The factory declines, so the built-in encoder is used, for other formats, MXF, two-pass VBR, HDR, odd sizes, non-square pixels and any configuration VideoToolbox cannot create a hardware session for. An encoder that fails mid-export stops it with an error: unlike a decoder, it cannot hand a half-written stream to another one. - No B-frames: on real 1080p footage the quality is the same with and without them (+-0.3 dB at equal bitrate) and the bitrate lands closer to the target, and without them every frame is presented at its decode time (no composition offsets, no edit list). - `export::register_encoder` ignores a factory registered twice; `encoder_registered`. Apple M1, 8 cores, single runs: a 9:29 timeline (camera clip, ProRes 4444 overlays with alpha, AAC, loudness) exports in 311 s instead of 793 s (2.5x faster); its densest 134 s in 84 s instead of 395 s (4.7x). The outputs compare at SSIM 0.990 / PSNR 46.5 dB. The encoder alone runs 200 fps at 1080p (docs/performance.md). Tests (tests/hardware_encode.rs): what is taken and declined, a round trip through our software decoder, an export that decodes in our decoder and in ffmpeg, the fallback for everything hardware declines, exact output sizes, an empty range, hostile configurations and encoders dropped at any point; the engine's `hardwareEncoding` parameter. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Dario CarneluttiandClaude Sonnet 5.5
|
1b92ea210b |
gpu: upload and composite the alpha plane of Y'CbCr frames
The GPU compositor uploaded only the Y, Cb and Cr planes of a Yuv8/Yuv16 frame and the shader returned alpha 1.0 for them, so a ProRes 4444 overlay (banner, lower third, outro) was drawn opaque in the Program monitor and covered the footage under it. Exports were correct because they use the CPU compositor. Frames with an alpha plane now upload it as a fourth texture (R8Unorm, or R16Float converted like the other planes), bound at binding 5 of the layer and blend layouts. A uniform slot (p3.y) carries the alpha scale (0 = no alpha plane); the shader reads the alpha at full resolution and premultiplies the linear colour by it. Adds gpu_draws_yuv_alpha_plane_like_the_cpu: an alpha ramp over an opaque layer at 8, 10 and 12 bits, GPU against the CPU compositor. It fails without the change (p99 diff 190/255) and passes with it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |