MCP: screenshot and render_preview draw edges as a PNG export does

ui.render, which render_preview and screenshot call in the headless
session and in the app, rendered the artboard with the canvas options:
8-bit compositing and edges smoothed shape by shape. A PNG export
composites in floating point and supersamples edges, so the preview
showed edge colors the exported file does not have, most visibly where
shapes meet edge to edge.

ui.render now renders with the options of a PNG export on white, so
its pixels equal the export's at the same scale. render_preview also
no longer enlarges an artboard smaller than max_side: with the default
max_side of 1024, a 612 x 792 page came back as an interpolated
791 x 1024 image, and it now comes back at 612 x 792. docs/mcp.md and
docs/control-protocol.md describe this.
This commit is contained in:
dotCooCoo
2026-10-24 17:29:33 +07:00
committed by bflatastic
parent b049e45b88
commit 58827952b7
7 changed files with 62 additions and 7 deletions
+4 -1
View File
@@ -7,6 +7,7 @@ use vectorcraft_engine::file_access::{self, AutomationRoots};
use vectorcraft_engine::{Session, UiRequest, ViewInfo};
use vectorcraft_geom::Point;
use vectorcraft_render::Renderer;
use vectorcraft_render::encode::{RasterExportOptions, RasterFormat};
use vectorcraft_tools::{Mods, PointerEvent, PointerKind, TOOL_GROUPS, ToolKey};
use crate::backend::Backend;
@@ -256,7 +257,9 @@ impl Headless {
let r = doc.artboards.get(idx).map(|a| a.rect).ok_or("no such artboard")?;
let scale = p.get("scale").and_then(Value::as_f64).unwrap_or(1.0).clamp(0.01, 16.0);
vectorcraft_render::raster_size(r, scale)?;
let img = self.renderer.render_region(&doc, r, scale, true);
// Rendered with the options of a PNG export on white, so its pixels are the exported file's.
let opts = RasterExportOptions { background: Some([255; 3]), ..Default::default() }.render_options(RasterFormat::Png);
let img = self.renderer.render_region_with(&doc, r, scale, &opts);
// A cancelled render stops drawing: the picture is not whole and is not written.
vectorcraft_engine::cancel::check().map_err(|e| e.to_string())?;
let png = img.to_png()?;
+27
View File
@@ -122,6 +122,33 @@ fn conventions_commands_batch_and_bounded_preview() {
}
}
/// `render_preview` and `screenshot` draw the artboard pixel for pixel as a PNG export on white
/// does: two rectangles that meet at x 40.3 and a diagonal edge. `render_preview` with the default
/// `max_side`, which is larger than the artboard, returns the export's size.
#[test]
fn render_preview_and_screenshot_match_a_png_export_on_white() {
let mut s = Server::new(Box::new(Headless::with_document()));
assert_eq!(call(&mut s, "run_command", json!({"command":"file.new","params":{"width":120,"height":80}}))["isError"], false);
for (x, w, fill) in [(0.0, 40.3, "#204080"), (40.3, 49.7, "#e0a020")] {
let r = call(&mut s, "draw_shape", json!({"shape":"rectangle","x":x,"y":10,"width":w,"height":50,"fill":fill}));
assert_eq!(r["isError"], false, "{r}");
}
let r = call(&mut s, "draw_path", json!({"points":[[10,75],[110,5],[115,30]],"closed":true,"fill":"#30a050"}));
assert_eq!(r["isError"], false, "{r}");
let pixels = |b64: &str| image::load_from_memory(&vectorcraft_format::base64_decode(b64).unwrap()).unwrap().to_rgba8();
let export = payload(&call(&mut s, "run_command", json!({"command":"document.export","params":{"format":"png","background":"white"}})));
let export = pixels(export["dataBase64"].as_str().unwrap());
assert_eq!(export.dimensions(), (120, 80));
for (name, args) in [("render_preview", json!({"max_side":120})), ("render_preview", json!({})), ("screenshot", json!({}))] {
let r = call(&mut s, name, args);
assert_eq!(r["isError"], false, "{name}: {r}");
let img = pixels(r["content"][0]["data"].as_str().unwrap());
assert_eq!(img.dimensions(), export.dimensions(), "{name}");
let differ = img.pixels().zip(export.pixels()).filter(|(a, b)| a != b).count();
assert_eq!(differ, 0, "{name}: {differ} pixels differ from the export");
}
}
struct Panics;
impl Backend for Panics {
fn call(&mut self, _: &str, _: Value) -> Result<Value, String> {
+1 -1
View File
@@ -1235,7 +1235,7 @@ fn render_preview(b: &mut dyn Backend, a: &Args) -> Result<ToolResult, String> {
let encoded = r.get("pngBase64").and_then(Value::as_str).ok_or("renderer returned no image")?;
let png = vectorcraft_format::base64_decode(encoded).ok_or("renderer returned bad base64")?;
let img = image::load_from_memory_with_format(&png, image::ImageFormat::Png).map_err(|e| format!("read preview: {e}"))?;
let img = img.thumbnail(max as u32, max as u32);
let img = if img.width() > max as u32 || img.height() > max as u32 { img.thumbnail(max as u32, max as u32) } else { img };
let mut out = std::io::Cursor::new(Vec::new());
img.write_to(&mut out, image::ImageFormat::Png).map_err(|e| format!("encode preview: {e}"))?;
Ok(image_result(out.get_ref(), json!({"width":img.width(),"height":img.height()})))
+6 -2
View File
@@ -17,7 +17,8 @@
//! - `ui.resize {width, height}`, `ui.focus`, `ui.screenshot {path?, data?}` (`data: true`: the PNG
//! comes back as `pngBase64` as well)
//! - `ui.render {path?, scale?, artboard?, data?}`: render an artboard headlessly (`artboard`: its
//! 0-based index, default the first; PNG; without a path, or with `data: true`, as `pngBase64`)
//! 0-based index, default the first; PNG with the pixels of a PNG export on white; without a
//! path, or with `data: true`, as `pngBase64`)
//! - `app.open {path}` (any readable format) / `app.save {path?, svg?: {…SVG options}}` / `app.quit`
//! (`file.close`, `file.closeAll` and `app.quit` first open a `saveChanges` dialog for each
//! modified document: `ui.dialog.confirm` saves, set `discard: true` then confirm to discard)
@@ -29,6 +30,7 @@ use std::sync::atomic::{AtomicU8, Ordering};
use std::sync::mpsc::Sender;
use serde_json::{Value, json};
use vectorcraft_render::encode::{RasterExportOptions, RasterFormat};
use vectorcraft_tools::{Mods, PointerEvent, PointerKind};
use crate::VectorcraftApp;
@@ -428,7 +430,9 @@ pub fn handle(app: &mut VectorcraftApp, ctx: &egui::Context, req: &ControlReques
if let Err(e) = vectorcraft_render::raster_size(r, scale) {
return err(e);
}
let img = app.canvas.renderer.render_region(&doc, r, scale, true);
// Rendered with the options of a PNG export on white, so its pixels are the exported file's.
let opts = RasterExportOptions { background: Some([255; 3]), ..Default::default() }.render_options(RasterFormat::Png);
let img = app.canvas.renderer.render_region_with(&doc, r, scale, &opts);
let png = match img.to_png() {
Ok(png) => png,
Err(e) => return err(e),
+21
View File
@@ -105,6 +105,27 @@ fn shortcut_sets_are_read_only_inside_the_read_root() {
let _ = std::fs::remove_dir_all(base);
}
/// `ui.render` (MCP's `render_preview` and `screenshot` in the app) draws the artboard pixel for
/// pixel as `app.export` writes a PNG on white: two rectangles that meet at x 40.3 and a diagonal
/// edge.
#[test]
fn control_render_matches_a_png_export_on_white() {
let (mut app, tx, base) = confined("render");
let svg = r##"<svg xmlns="http://www.w3.org/2000/svg" width="120" height="80"><rect x="0" y="10" width="40.3" height="50" fill="#204080"/><rect x="40.3" y="10" width="49.7" height="50" fill="#e0a020"/><path d="M10 75 L110 5 L115 30 Z" fill="#30a050"/></svg>"##;
std::fs::write(base.join("in/edges.svg"), svg).unwrap();
let r = request(&mut app, &tx, "app.open", json!({"path": s(base.join("in/edges.svg"))}));
assert_eq!(r["ok"], true, "{r}");
let pixels = |v: &Value| image::load_from_memory(&vectorcraft_format::base64_decode(v.as_str().unwrap()).unwrap()).unwrap().to_rgba8();
let r = request(&mut app, &tx, "app.export", json!({"format": "png", "background": "white"}));
let export = pixels(&r["result"]["dataBase64"]);
let r = request(&mut app, &tx, "ui.render", json!({}));
let render = pixels(&r["result"]["pngBase64"]);
assert_eq!(render.dimensions(), export.dimensions());
let differ = render.pixels().zip(export.pixels()).filter(|(a, b)| a != b).count();
assert_eq!(differ, 0, "{differ} pixels differ from the export");
let _ = std::fs::remove_dir_all(base);
}
#[test]
fn frames_with_injected_input_are_confined() {
let (mut app, tx, base) = confined("synthetic");
+1 -1
View File
@@ -103,7 +103,7 @@ the two can still win that race.
| `ui.set` | `{brightness?, panel?, rulers?, outline?, grid?, smartGuides?, boundingBox?, controlBar?}` | |
| `ui.dialog.set` / `.confirm` / `.cancel` | `{field, value}` | fill and submit the open dialog |
| `ui.screenshot` | `{path?, data?}` | capture the window (PNG), written to `path`; `data: true` sends it back as `pngBase64` too. Needs a presented frame: with the screen locked or the window minimized/covered it fails after ~8 s with an explanatory error |
| `ui.render` | `{path?, scale?, artboard?, data?}` | render an artboard headlessly (PNG; `artboard`: its 0-based index, default the first): written to `path`, else (or with `data: true` as well) sent back as `pngBase64` |
| `ui.render` | `{path?, scale?, artboard?, data?}` | render an artboard headlessly (PNG with the pixels of `app.export {format: "png", background: "white"}` at the same scale; `artboard`: its 0-based index, default the first): written to `path`, else (or with `data: true` as well) sent back as `pngBase64` |
| `ui.resize` / `ui.focus` | | |
| `app.open` / `app.save` / `app.export` / `app.quit` | `{path}` / `{path?}` / `{path?, format?, artboard?, range?, scale?, …}` | `app.open` reads every format `document.open` reads (see `document.formats`) and returns its result (`{index, title, format, warnings, …}`; `warnings` say what didn't come in as it was, such as an EPS shown as its preview image and why its PostScript couldn't be read), or null when a dialog asks first or a library loads. `app.export` encodes through the engine's `document.export` (same options; the document keeps its path) and writes `path` through the host; without `path` it returns `{dataBase64, format, bytes}`, as headless mode does. `app.quit`, `file.close` and `file.closeAll` first open a `saveChanges` dialog for each modified document (they return `{"pending": "saveChanges"}`): `ui.dialog.confirm` saves, `ui.dialog.set {field: "discard", value: true}` then confirm discards, `ui.dialog.cancel` cancels the whole close or quit |
| `app.export` with `useArtboards` | `{path, useArtboards: true, range? \| artboards?, …}` | Export As: PNG, JPEG and WebP write one file per chosen artboard (default all; SVG does so for a `range`), `<path stem>-<artboard>.<ext>`, and return `{path, files: [path…]}`; a PDF keeps them as pages of `path`. `useArtboards: false` covers the bounds of the visible art. The menu's Export As… is `file.exportAs`: the `exportAs` dialog (`format`, `useArtboards`, `all`, `range`), then a save dialog and the format's options: `pngOptions` / `jpgOptions` / `webpOptions`, `svgOptions`, or `savePdf` for a PDF with Use Artboards |
+2 -2
View File
@@ -318,7 +318,7 @@ The older names below remain listed because existing workflows use them.
| `command_run` | `id`, optional `params`; runs the command through the existing backend. An unknown key in `params` is an error that names it and the nearest key the command takes (`invalid parameters for `file.new`: unknown parameter `widht`; did you mean `width`?`), so a misspelt key no longer runs the command with its default (see [Parameter keys](#parameter-keys)) |
| `command_batch` | `steps: [{id, params?}]` (at most 256), optional `stop_on_error` (default true), `atomic` and `label`; returns `completed`, `failed`, `results: [{ok, result\|error}]` and `created` (the ids the `draw_path` / `draw_shape` steps made, in order). A step's `id` is a command id, or `draw_path` / `draw_shape` with that tool's arguments as `params` (checked as `tools/call` checks them; the step's result is the tool's, so `"$N.id"` is the new object). Each edit has its own undo step, or with `atomic: true` the batch's edits are one undo step named `label` (else after its first edit; `edit.undoGroup.begin`/`end` underneath); failures set `isError`. A param string `"$N.path"` or `"$last.path"` takes a value from an earlier step's result (see `run` below). Kept step results total at most 8 MiB; past that the batch stops with the reason. This is the MCP tool, not the engine command `command.batch` (that one takes `commands: [{command, params}]` and is what the Actions panel plays); use `command_run` with `command.batch` when replaying a recorded action |
| `doc_inspect` | Optional `depth`, `childLimit`, `ids`, `kind`, `limit`, `offset`; same summary as `inspect_document` |
| `render_preview` | Optional `max_side` (1–4096, default 1024) and `artboard` (0-based, default 0); inline PNG of that artboard without editing it. Artboards too large to render within the allocation bound return a tool error |
| `render_preview` | Optional `max_side` (1–4096, default 1024) and `artboard` (0-based, default 0); inline PNG of that artboard without editing it, drawn as `export` draws a PNG with `background: "white"` at the scale that fits `max_side` (at most 1). Artboards too large to render within the allocation bound return a tool error |
| `ui_inspect`, `ui_screenshot` | Connected desktop state/window capture; tool errors in headless mode |
Every tool declares read-only, destructive, idempotent and open-world hints. File-writing
@@ -371,7 +371,7 @@ objects' fills or strokes differ (`fillMixed` / `strokeMixed`, drawn as a "?" pr
| `type_text` | `{text}` | Remote only. |
| `invoke_menu` | `{command, params?}` | Invokes a menu item by command id. Includes UI commands such as `view.*` and `window.*` in remote mode. |
| `open_panel` | `{panel}` | Remote only. `panel` is a panel id (`layers`, `swatches`, `colorGuide`, …, as `window.panel` takes) or its display label (`"Color Guide"`), in any case. |
| `screenshot` | `{path?, scale?, artboard?, window?}` | Returns MCP image content (`image/png`, base64) plus a text block. Renders the artboard; `window:true` captures the app window (remote only). With `path` the backend (the headless session or the app) writes the PNG there too. |
| `screenshot` | `{path?, scale?, artboard?, window?}` | Returns MCP image content (`image/png`, base64) plus a text block. Renders the artboard with the pixels of a PNG `export` with `background: "white"` at the same scale; `window:true` captures the app window (remote only). With `path` the backend (the headless session or the app) writes the PNG there too. |
| `open_file` | `{path}` | Opens any readable file as a new active document: `.vectorcraft`/`.drawcraft`, `.vctemplate`, `.svg`/`.svgz`, `.pdf`/`.ai`, `.ait`, `.eps`/`.epsf`, `.ps`, `.dxf`, `.emf`, `.wmf`, PNG, JPEG, GIF, WebP, TIFF, BMP, Targa (`.tga`/`.vda`/`.icb`/`.vst`), PSD/PSB (an image opens as a document of its pixel size; a Photoshop file as its merged image). Templates (`.vctemplate`, `.ait`) open as a new untitled document. PDF, `.ai` and SVG files saved with Preserve Editing reopen as the document they carry. The reply is `document.open`'s: its `warnings` say what didn't come in as it was (an EPS whose PostScript can't be read opens as its preview, and the warning names the PostScript error, the operator and the procedure). `run_command document.formats` lists the formats. |
| `save_file` | `{path?}` | Runs `document.save`: the document's own file in its own format (native `.vectorcraft` unless it was opened from or saved as SVG, PDF or a restorable `.ai`; then `warnings` say what that format loses). A path's extension picks the format (`.vectorcraft`, `.vctemplate`, `.pdf`, `.svg`, `.svgz`, `.ai`: a PDF carrying the native document, which reopens editable). |
| `export` | `{path?, format?, scale?, artboard?, range?, selection?, outlineText?, options?}` | `svg`, `svgz`, `pdf`, `eps`, `dxf`, `emf`, `wmf`, `png`, `jpg`, `webp`, `gif`, `png8` (an indexed `.png`), `tiff`, `bmp`, `tga`, `psd` (layered), `txt` (the document's text), `vectorcraft` or `template` (a native template). The tool's `format` enum and `document.formats` list them, generated from the engine's format table. When `format` is omitted, it comes from the path's extension. PDF writes one page per artboard: all of them, or `artboard` (0-based) / `range` (`"1-3, 5"`, 1-based); the other formats write one artboard. `options` carries more format options (e.g. `{"quality": 80}` for JPEG). `selection: true` exports the selected objects cropped to their bounds (the reply adds their `bounds`, and reports `format` and the encoder's `warnings` as a whole-document export does); `outlineText: true` writes SVG text as paths. Template layers are left out, live effects are kept, and exporting `vectorcraft` never changes the document's path. Without `path` the bytes come back as `dataBase64`. Both backends run the same `document.export` call. |