Plug-ins: external effect helpers (safe, out-of-process native effects) + plug-in API 2 (#889)

* Plug-ins: API 2 (PluginHost, padding, layer_span)

Split out of the OpenFX work so it can be reviewed on its own. Additive; the WebAssembly ABI
stays at 1.

- point3 / layer / text / hidden parameters, integer sliders, parameter groups.
- EffectPlugin::padding (transparent pixels around the buffer, at most 4096 per side).
- EffectPlugin::render_v2 with a PluginHost: the input at other times, other layers, parameters
  at other times, frame rate, per-instance key, the layer's source range.
- EffectEnv::layer_span from the renderer; MAX_PLUGINS 1024.
- plugin_error: a plug-in effect's last render error, for the UI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxacwTpUwqyfpcLgKW8yjS

* Plug-ins: external effect helpers (crates/helper-host)

Native plug-in formats run in separate helper programs, never in EffectCraft's process
(docs/adr/0001-external-effect-helpers.md). New safe-Rust crate effectcraft-helper-host (L3,
native only, unsafe forbidden): it finds helper programs, starts each as a child process and
registers its effects through plug-in API 2, so the UI and command surface need nothing new.

- Protocol v1 (docs/external-helpers.md): JSON lines over stdin / stdout (helper replies are
  prefixed so plug-in chatter is ignored); pixels go through a frame exchange file (/dev/shm on
  Linux), never through the pipe. Render-time callbacks for the input at other times, other layers
  and parameters at other times, with nested renders through the same helper.
- A helper that exits, hangs past its timeout or breaks the protocol is stopped. The render fails
  with a message, the frame is left as it was, and the helper restarts for the next render. An
  effect that stops its helper 3 times is turned off (helper_blocklist.txt).
- Opt-in: nothing starts helpers unless the user asks (Effect > Scan for Effect Helpers), or opts
  in to starting them at launch with EFFECTCRAFT_HELPERS=1, an environment variable, so writing
  to the settings folder alone can't run code. effect.helpers.scan / load are refused for
  confined agent sessions (confine.rs), like file.executeFile.
- Discovery: helpers/ next to the app (Contents/Helpers on macOS), helpers/ in the settings
  folder, EFFECTCRAFT_HELPER_PATH.
- Commands effect.helpers.list / scan / load; Effect menu entries (translated); the last render
  error of an effect shows as a warning sign after its name in Effect Controls.
- serve(): the helper side for Rust helpers. tests/it/main.rs runs the whole protocol (render,
  padding, callbacks, nesting, threads, crash, restart, block list, hang) against a fake helper,
  with nothing to install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxacwTpUwqyfpcLgKW8yjS

* Effect helpers: address review (start failures, frame file mode, nextest listing)

- A helper that can't start (not installed any more, restarting too often, no frame exchange
  file) is reported but no longer charged to the effect being rendered: it is Failure::Refused,
  not Failure::Broken, so it never counts toward turning that effect off. New check in
  tests/it: once the helper is out of restarts, renders of an effect fail without blocking it
  (fails without the fix: the effect was turned off after 3 such renders).
- The frame exchange file is created owner-only (0o600) on Unix: it holds the user's frames and
  /dev/shm is shared. Asserted in the frames unit test.
- tests/it (harness = false) speaks the libtest listing protocol cargo-nextest uses:
  `--list --format terse` prints `helper_protocol: test` (nothing with `--ignored`), and
  `helper_protocol --exact` runs the checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxacwTpUwqyfpcLgKW8yjS

* Effect helpers: bound writes to a helper, drain stderr, keep callbacks in step

- Writes to a helper go through a writer thread with the request timeout: a
  helper that stops reading its stdin (hung with a large callback answer or
  render request pending) no longer blocks the render thread forever while
  holding the helper.
- stderr is drained as bytes (lossy, in 64 KiB pieces) to the end: output that
  isn't UTF-8 stopped the reader, closing the pipe under the helper (EPIPE /
  SIGPIPE on its next write); lines were unbounded.
- Callback images that can't be written are answered with `null` instead of
  failing the render while the helper waits (which left the protocol out of
  step); a frame that can't be written before the request is sent is not
  charged to the effect.
- Mid-render calls use the render's own process: no respawn or heartbeat ping
  while the helper is waiting for an answer.
- Waiting for a busy helper is bounded (load timeout) instead of forever;
  status() no longer blocks on a running render.
- Reap the child when start-up fails after spawning; cap the block list read.
- Scripts start helpers only with Allow Scripts to Write Files and Access
  Network on (they run native code), as for File.execute().
- helper-host declares the workspace's no-panic crate lints; the desktop
  main() keeps its doc comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Effect helpers: rustfmt the fake provider, build the test session in its initializer (clippy)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Plug-in Library: offer WebAssembly API 1 releases only; a helper already stopped mid-render isn't charged again

- WasmPlugin loads only modules at the WebAssembly ABI version, which stays 1 while
  PLUGIN_API_VERSION is 2: share it as effects::plugin::WASM_API_VERSION and have the
  library's compatibility check use it, so it never installs an API 2 release that
  can't load.
- with_running: a process that is already gone was stopped by what broke it (a nested
  render, charged there), so this render fails without charging its own effect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* frames: plug-in effects count as reading other frames (plug-in API 2)

With main's narrowed frame invalidation (#2795), an edit inside a precomp under an
effect that reads its layer at other times must reach the layer's whole span.
Plug-in API 2 gives every plug-in effect the input at other times, so any effect
outside ec. counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
This commit is contained in:
Lukas L Dam
2026-11-05 11:11:57 +01:00
committed by GitHub
co-authored by Claude Opus 5.5 FredApps
parent 30d6c6e1d3
commit 611a282bea
39 changed files with 2898 additions and 62 deletions
+34 -3
View File
@@ -152,7 +152,7 @@ example of a keyframe tool.
## Effect plug-ins
EffectCraft loads third-party effects through a small, versioned plug-in API (**API version 1**).
EffectCraft loads third-party effects through a small, versioned plug-in API (**API version 2**; WebAssembly ABI 1).
A plug-in effect behaves exactly like a built-in one: it is listed in Effects & Presets (and the
Effect menu) under its category, applied by id (`effect.apply {"effect": "org.example.posterize"}`)
or display name, its parameters are ordinary properties (keyframes, expressions, Effect Controls,
@@ -270,8 +270,39 @@ impl EffectPlugin for Swap {
register_plugin(Arc::new(Swap(manifest)))?; // before the UI builds its menus
```
`PluginParams` reads values by parameter id (`f`, `b`, `point`, `color`). At most 256 plug-in
effects can be registered per process.
`PluginParams` reads values by parameter id (`f`, `b`, `point`, `color`, and in API 2 `point3`,
`layer`, `text`). At most 1024 plug-in effects can be registered per process.
## Plug-in API 2 (native plug-ins)
API 2 is for Rust plug-ins and for hosts that bring effects from elsewhere (such as
[external effect helpers](external-helpers.md)); the WebAssembly ABI stays at 1. Manifests with `"api": 1` or `2` are accepted,
`effect.plugins.list` reports `api: 2`, and up to 1024 plug-in effects can be registered.
* **New parameter types**: `point3` (`default` `[x, y, z]`; x, y arrive in buffer pixels, z as is),
`layer` (another layer of the composition; flattens to its id, -1 for none, read with
`PluginParams::layer`), `text` (a string field) and `hidden` (a string that is not shown, for
opaque plug-in data). `text` / `hidden` take no slot in `flat`; their strings are in
`PluginParams::texts`, one per such parameter in declaration order, read with
`PluginParams::text(id)`. `slider` gains `"integer": true` (shown with 0 decimals) and every
parameter an optional `"group"` path (`"Advanced/Edges"`; informational). WebAssembly modules
can't use `point3`, `layer`, `text` or `hidden`.
* **`PluginFrame`** also carries `origin` (buffer pixels of layer pixel (0, 0); padding shifts it)
and `layer_size` (layer pixels).
* **`EffectPlugin::padding(params, time, scale, layer_size, instance_key) -> u32`** asks for that
many transparent buffer pixels on every side before rendering (glows, regions of definition
larger than the input); at most 4096. `layer_size` is the layer's size in layer pixels (so
sizes relative to the frame can be measured) and `instance_key` identifies the applied effect
instance (see `PluginHost::instance_key`).
* **`EffectPlugin::render_v2(frame, params, time, host)`** (default: calls `render`) gets a
`PluginHost` with `input_at(layer_time)` (the effect's input at another time, in its own extent;
the effects above this one are applied), `layer_input(param_id, layer_time)` (a layer parameter's
layer, effects and masks applied, resampled into the frame's grid), `params_at(layer_time)`,
`frame_rate()`, `time()`, `instance_key()` (a stable id of the applied effect instance: the
same on every frame, different per layer and per application; plug-ins that keep state between
frames pool their native instances by it) and `frame_range()` (the layer's source time range in
layer-time seconds, `None` when it has none). Images are `PluginImage { width, height, pixels, origin, scale }`.
It may run on several threads at once.
### Compatibility