figcraft/docs/control-protocol.md

# FigCraft control protocol

The desktop app can be driven by other programs (tests, scripts, `figcraft-cli mcp --attach`,
sibling Crafting Apps) through a loopback socket:

```sh
cargo run -p figcraft -- --control 7979 [file.figcraft|file.svg|image]
```

The app listens on `127.0.0.1:<port>` only. Each connection carries **newline-delimited JSON**:
one request object per line, one reply line per request, in order.

```text
request:  {"id": 1, "method": "engine.execute", "params": {...}}
reply:    {"id": 1, "ok": true, "result": ...}
          {"id": 1, "ok": false, "error": "message"}
```

Requests are queued and run on the UI thread at the start of the next frame, so replies reflect the
real UI state. A request that the UI doesn't answer within 60 s gets a timeout error. A line that is
not JSON gets `{"id": null, "ok": false, "error": "bad JSON: …"}` and ends the connection (so an
HTTP request a web page makes the browser send here never runs). The implementation is in
`crates/ui-egui/src/control.rs` (methods) and `apps/figcraft/src/control_server.rs` (socket).

A request line is limited to **32 MiB before the newline**, including whitespace. The reader stops
at the first excess byte, replies `{"id": null, "ok": false, "error": "message too large"}`, then
closes the connection without consuming the rest of the frame. The server accepts at most **8
active connections**; additional connections are closed immediately. Socket reads and writes each
have a **60 s inactivity timeout**, so idle clients and clients that stop reading release their
worker. This bounds blocked socket operations, not the total duration of a frame arriving slowly.
Clients can reconnect after an idle timeout (the attached MCP backend already does).

Both the socket queue and the UI queue hold at most **8 requests** each, including requests waiting
for another window's frame. A full socket queue replies with `"control request queue is full"`
instead of dispatching the new request. These limits apply to the opt-in loopback control channel;
requests still queued after the UI timeout are discarded, including requests deferred to a hidden
window. A command already running is not interrupted.

A minimised or covered window (the main one or another, see Windows below) draws no frames, but
still answers `engine.execute`, `engine.commands`, `document.inspect`, `node.get`, `window.list`
and `ui.render`, acting on the file that window shows, so an attached agent can edit, inspect and
save while the app is out of sight. Screenshots, input (`ui.input`,
`ui.pointer`, `ui.key`, `ui.type`) and the interface's own state wait for the window to show, and
time out after 60 s if it doesn't.

Coordinates are page units, y down. `x`/`y` in `node.set`/`node.create` are relative to the parent
frame (groups have no coordinate space of their own).

**Windows.** The desktop app can show files in several windows (`window.new`). Every method takes
an optional `window` (an id from `window.list`; `0` is the main window) and then runs in that
window, with its tabs, panels and canvas: `engine.execute` acts on the file it shows,
`ui.screenshot` captures it, `ui.input` reaches it. Without `window`, a request goes to the window
last in front (the one the user clicked last, or the last `window.new` / `window.focus`). An
unknown id is an error.

## Methods

### `engine.execute {command, params?}`

Run any command (the same ids as the menus, shortcuts, Actions palette, CLI and MCP). Programmatic
calls never open dialogs: dialog requests are dropped. Returns the command's result (creation
commands return `{"id": N}`). Paints (`fills`, `strokes`, `paint`) take a colour, the full paint
JSON or the gradient shorthand `{"type":"linearGradient","stops":["#hex","#hex"],"from"?,"to"?}`
(as in `docs/mcp.md`).

```json
{"id":1,"method":"engine.execute","params":{"command":"shape.rectangle","params":{"x":10,"y":10,"width":100,"height":50}}}
{"id":1,"ok":true,"result":{"id":2}}
{"id":2,"method":"engine.execute","params":{"command":"node.set","params":{"ids":[2],"props":{"fill":"#FF6B6B","cornerRadius":8}}}}
{"id":3,"method":"engine.execute","params":{"command":"command.batch","params":{"label":"Tidy","commands":[{"command":"align.left","params":{"ids":[2,3]}},{"command":"distribute.vertical","params":{"ids":[2,3]}}]}}}
```

Inside `command.batch`, `{"$step": n}` in a step's params stands for the `id` in step `n`'s result
(zero-based; `{"$step": n, "field": "…"}` for another field; any object with a `$step` key is read
as one, and a nested `command.batch` step keeps its own references to its own steps), so one batch
can create a frame and fill it:

```json
{"id":4,"method":"engine.execute","params":{"command":"command.batch","params":{"commands":[{"command":"node.create","params":{"type":"frame","width":200,"height":100}},{"command":"node.create","params":{"type":"rectangle","parent":{"$step":0},"width":20,"height":20}}]}}}
```

### `engine.commands {}`

Every command: `[{id, label, params, menu, shortcut, undoable}]`.

### `document.inspect {depth?, allPages?}`

The active document as JSON: name, path, dirty, currentPage, selection, pages (the current page's
layer tree to `depth`, default 8; others summarised unless `allPages`), styles, variable count,
images, the colour profile (`colorProfile`: `srgb` or `displayP3`), the text being edited and the
undo history.

### `node.get {id, depth?}`

One layer (children to `depth`, default 1). Fields: id, name, type, x, y, width, height, rotation,
absoluteBounds, visible, locked, opacity, blend, fills, strokes, strokeWeight, effects, cornerRadius,
layout (including grid `columnTracks` / `rowTracks`), sizing, `minWidth` / `maxWidth` /
`minHeight` / `maxHeight`, `absolute`, constraints (children of plain frames and absolute
children), `gridCell` (a grid auto layout child's column, row and spans), clip, text properties,
component/instance links, overrides, `slot` (a slot layer's property name), boundVariables, interactions, exports,
`devStatus` (handoff status), `devChanged` (changed since it was marked ready for dev),
`annotations` (Dev Mode notes: `[{label, properties, category?}]`), `codeLinks` (a component's
links to code: `[{language, importPath, name, template}]`), `devResources` (links for developers,
its own and inherited: `[{name, url, index?, from?}]`). The auto layout props
that `node.set` accepts are listed in [mcp.md](mcp.md#conventions-agents-must-know).

### `ui.inspect {}`

UI state: panels and tabs (`panels`: the `left` and `right` panels' widths as asked for and whether the left one is closed, `leftCollapsed`), the cursor the canvas shows (`cursor`: a CSS cursor name such as `default`, `crosshair`, `ew-resize`, `grab`, `copy`, `not-allowed`, `zoom-in`, or one FigCraft draws: `rotate <degrees>`, `pen`, `pen-close`, `pen-add`, `pen-bend`, `eyedropper`, `comment`; `null` while the pointer is elsewhere or over something with a cursor of its own, a ruler guide or a gradient handle; the engine's own hint is `tool.overlay.cursor`), current tool (`tool`: its id, overlay hints and, for a selected
ellipse, its arc handles `arcHandles`, for a selected rectangle, frame, polygon or star, its corner radius and ratio handles `shapeHandles`; see [mcp.md](mcp.md#measuring-the-scale-tool-arc-handles-and-selecting)), canvas rect, dialogs, view, timings (`frameMs`,
`renderMs`), frames drawn since start or the last `ui.set {resetFrameStats: true}` (`framesDrawn`;
an idle window draws none) with the most frequent repaint requests behind them (`repaintCauses`:
`[{at: "file:line", count}]`, every request made in the frame before each one, delayed ones such as
the autosave deadline included; egui's `context.rs` begin-pass line means input events) and what the last canvas render had to do (`canvas.renderWork`: `full`, `partial` with
the `pixels` re-rendered after a pan or an edit, or `reused`), whether the canvas shows a coarse
zoom preview or the refined image (`canvas.quality`: `preview` or `refined`) and the time from
asking for a render to its preview and its refined image being handed to egui as the canvas
texture (`canvas.previewLatencyMs`, `canvas.refineLatencyMs`, on a monotonic clock, rendering
included in the browser too; the GPU upload and presenting the frame come after; `null` until
there was one). While a prototype plays, `presenter`
(full window) and `preview` (the preview pane) hold the player's state: the `current` frame and its
`name`, `start`, `history`, open `overlays`, how many delayed mouse triggers are `waiting`, the screen's `scroll` offset and `scrollRange`, the
`device` (and whether it is turned to `landscape`, its `deviceColor`) and the `viewport` of the frame
its screen shows, the presented frame's `frameSize` and, for a responsive prototype, the screen it
was resized to (`responsive`: `[w, h]`, else `null`), the size option (`scale`:
`fit|fitWidth|fill|actual`, `deviceFrame`), the `zoom` it shows at, how
far the view is panned (`pan`, `panRange`), whether a scroll is still coasting (`flinging`), the
active screen's `hotspots` (`[{id, rect: [x0, y0, x1, y1]}]` in window points, bottom-most first),
whether a click where nothing reacts flashes them (`hotspotHints`) and they are flashing now
(`hotspotFlash`), and whether the Inspect panel shows (`inspect`) with the layer it shows
(`inspected`) (`null` when nothing plays). `presentOptions` holds the options the next presentation
starts with (`scale`, `deviceFrame`, `comments`, `hotspotHints`, `inspect`).
`textPreview` is true while a live text preview shows on the canvas (a hovered text style or named
instance, a dragged axis slider; see `text.preview` in [mcp.md](mcp.md)).
`focus` is keyboard navigation's state: the `region` with the keyboard (`toolbar`, `layers`,
`canvas` or `properties`), whether `keyboard` mode is on (F6 or Tab moved the focus since the last
pointer press, so the focus ring shows) and the name of the `control` with the keyboard (what a
screen reader announces; empty when nothing has it). `reduceMotion` is true when the preference or
the system's accessibility setting turns the interface's animations off. `live` is the live
session of the window's active file as the window shows it (sessions are per file: another tab
shows its own, or `status: "off"`): `session` (its id, null when the file isn't in one),
`sessions` (how many the app has running), `status`, `role`, the number of `peers`, who you `follow`, the
peers whose cursors are drawn on the canvas (`cursorsShown`), whether the Join and Publish to
server dialogs are open (`joinDialog`, `publishDialog`; Escape closes them), the join code the
Share popover shows (`code`, with `share: {via, access}`; null when the file isn't shared with
one), what is typed in the popover's and the home screen's join field (`joinField`), whether the
popover has Local network chosen and Advanced open (`local`, `advanced`), and why the active
document can't be edited now (`readOnly`: the "View only" or "Publishing" reason, else null); the
session itself is `engine.execute {command: "collab.peers"}` (sharing with a join code, hosting,
joining and publishing are `collab.share`, `collab.host`, `collab.join` and `collab.publish`, opening invite links `collab.registerLinks`,
see [mcp.md](mcp.md#live-sessions)). The app also joins the invite link it
was started with (`figcraft "figcraft://join?…"`). `online` is the home screen's Online section:
whether it shows (`open`), your online files as its rows read (`files: [{file, name, detail,
shared, state, hasOwnerToken}]`; never a token or a link of yours), whether the servers are being
asked (`checking`), how many couldn't say (`problems`), and the file whose removal is being
confirmed or renamed (`confirmRemove`, `rename`; Escape closes them); the list itself is
`engine.execute {command: "collab.files"}`. `tabs` lists the open files as the tab strip shows
them: `[{name, live}]`, `live` being `"shared"` (in a live session others get into), `"private"`
(a file of yours on a server, open only to you) or null.

### `ui.pointer {kind, x, y | sx, sy, to?, steps?, shift?, alt?, cmd?, ctrl?, pressure?, pressureTo?}`

A mouse gesture with the current tool. `kind`: `down`, `move`, `up`, `click`, `double`, `hover`,
`drag` (needs `to: [x, y]`; `steps` intermediate moves, default 8). Position in page coordinates
(`x`, `y`) or screen pixels (`sx`, `sy`). `pressure` (0–1) is pen pressure (the Pencil records
it); a drag's pressure goes from `pressure` to `pressureTo`. Returns `{selection, tool}`. With the Move and Scale
tools, a press on a frame's or section's name label (as the canvas last drew it, along a rotated
frame's top edge) selects and drags that frame.

```json
{"id":4,"method":"engine.execute","params":{"command":"tool.set","params":{"tool":"ellipse"}}}
{"id":5,"method":"ui.pointer","params":{"kind":"drag","x":200,"y":200,"to":[300,260],"shift":true}}
```

### `ui.key {key, shift?, alt?, cmd?, ctrl?}`

A key as if typed on the canvas (in vector edit mode the edit keys come first: `V` `P` `B` `Q` `X` `Shift+E` `Shift+W` tools, arrows nudge points, `Delete`/`Shift+Delete`, `Cmd+A`, `Cmd+J`; with the shape builder, `Enter` finishes): a menu shortcut (`{"key":"G","cmd":true}` groups), a tool key,
`Escape`, `Delete`/`Backspace`, `Enter`, number keys (opacity; two quick digits make an exact percentage), arrows (nudge; shift = big nudge; auto layout children move one place instead: along the flow, between wrapped rows, to the next grid cell, or into the neighbouring nested auto layout frame; with `cmd` they resize the selection). With the Comment
tool, `ArrowUp`/`ArrowDown` open the previous/next thread of the Comments panel's list instead
(returns `{"comment": id}`). While text is being edited, the text shortcuts (Bold, alignment,
size steps, Save…) run and other keys go to the text editor. Shortcuts follow the platform the app
runs on (`cmd` is Cmd on macOS and Ctrl elsewhere; on macOS `ctrl` is the separate Control key).
Returns `{"command": id}` for shortcuts.

### `ui.type {text}`

Type into the text being edited (start editing with a double-click, `text.edit.begin {id}`, or the
text tool).

### `ui.input {events}`

Raw window input for the next frame, as if from the mouse and keyboard: it reaches the panels,
popovers and fields (`ui.pointer` and `ui.key` drive the canvas tools and shortcuts instead).
`events` is a list of `{move: [x, y]}`, `{down: [x, y], button?: "secondary"}`,
`{up: [x, y], button?}`, `{key, shift?, alt?, cmd?}` (pressed and released; egui key names such as
`ArrowLeft`, `Escape`, `A`), `{text}` (typed text), `{scroll: [dx, dy], shift?, alt?, cmd?}` (a
mouse wheel at the pointer: the content moves by that many points, so a negative `dy` scrolls down)
`{leave: true}` (the pointer left the window), `{hold: {alt?, shift?, cmd?}}` (keep those modifier
keys down for the frames that follow, e.g. Alt while dragging a component from the Assets panel onto
an instance to swap it; all false releases them) and `{gamepad: "A" | "Gamepad A" | 0}` (a
controller button pressed in the prototype playing, the presenter's or else the preview pane's: a
short name, a key/gamepad trigger's name or a standard layout index 0–16; an error when no prototype
plays); positions are window points. Events of one call
arrive in the same frame, so send a press and its release (or a double-click's two clicks) as
separate calls. Returns `{queued}`.

```json
{"id":6,"method":"ui.input","params":{"events":[{"move":[1000,315]}]}}
{"id":7,"method":"ui.input","params":{"events":[{"down":[1000,315]}]}}
{"id":8,"method":"ui.input","params":{"events":[{"up":[1000,315]}]}}
```

### `ui.menu.invoke {id}`

Invoke a menu item exactly like clicking it (this may open a dialog). Returns `{ok, lastError}`.

### `ui.set {rightTab?, leftTab?, theme?, home?, homeSection?, homeRecentsView?, homeRecentsSort?, homeRecentsOrder?, palette?, mainMenu?, focusRegion?, colorPicker?, variables?, layerFilter?, collapseLayers?, exportPreview?, preview?, presentScroll?, colorProfileDialog?, stylePicker?, stylePickerEdit?, stylePickerView?, iconGallery?, protoEditor?, contextMenu?, contextSelectLayer?, commentOpen?, commentDraft?, commentFilter?, commentSort?, commentSearch?, commentReply?, commentEdit?, presentComments?, devCompare?, devCompareMode?, devCompareSplit?, annotationFilter?, resetFrameStats?, pluginsWindow?, pluginsTab?, pluginConfirm?, pluginForm?}`

Switch panels and modes: `rightTab` `design|prototype|inspect`, `leftTab` `file|assets`, `theme`,
`home` (bool), `homeSection` (`recents`, `online` or `templates`: which section the home screen shows, as `ui.inspect` reports it; opening
`online` asks each server once what became of your online files), `homeRecentsView` (`grid|list`), `homeRecentsSort` (`modified|name|created`) and `homeRecentsOrder` (`desc|asc`): how the home screen shows its recent files (kept in the preferences; `ui.inspect` reports them under `homeRecents` with each file and whether its picture is in), `palette` (bool: the Actions palette), `mainMenu` (bool, or a top-level menu such
as `"Plugins"` to open the main menu with it open), `pluginsWindow` (bool: Plugins ▸ Manage
plugins…), `pluginsTab` (`installed|browse`: that window's tab), `pluginConfirm` (a plugin id of the
directory listed last: its install confirmation shows, as its Install button would; `null` closes
it), `pluginForm` (`{id, command?}`: start that plugin command like its menu item, opening
its parameters form when it has parameters; `null` closes the form), `focusRegion`
(`toolbar|layers|canvas|properties`: give that region the keyboard as F6 would, on the next frame),
`iconGallery` (bool:
every UI icon at 12, 16, 24 and 32 px over the window, for reviewing the icon set), `colorPicker`
(`{fills?, index?}`: open the colour picker on the selection's paint, which notes a Display P3 colour
outside sRGB; `null` closes it), `variables` (bool,
or a collection id or name to show, or `{library, collection?}` for an enabled library's collection: the
Variables editor), `layerFilter` (string:
open the layers search with that filter; `null` closes it), `collapseLayers` (true: collapse the
layers tree), `exportPreview` (bool: the Design panel Export section's preview), `preview` (true or a
frame id: open the prototype preview pane like `proto.preview`; false or `null` closes it),
`presentScroll` (`[x, y]`: scroll the presented screen, the full window's else the pane's, clamped
to what it can scroll), `colorProfileDialog` (`srgb|displayP3`: open File ▸ Color profile with that
profile chosen; `null` closes it), `stylePicker` (`fill|stroke|effect|grid|text`: open the style
picker for the selection, `text` being the text style browser of the Typography section; `null`
closes it) with `stylePickerEdit` (a style id or name: open its editor), `stylePickerView`
(`list|grid`), `protoEditor` (`[layer id, interaction index]`: open that interaction's editor in the
Prototype panel; `null` closes it), `contextMenu` (`[sx, sy]`: right-click the canvas at that screen
point, selecting what is under it (with `cmd: true`, the nested layer) and opening the context menu;
`null` closes it), `contextSelectLayer` (bool: open the context menu's Select layer submenu, which
lists every layer under the click). Comments: `commentOpen` (a thread id opens its popover, `null`
closes it), `commentDraft` (`{x, y}`: start a new comment there), `commentFilter`
(`open|resolved|mine|unread|all`: what the Comments panel and the pins show), `commentSort`
(`date|page`), `commentSearch` (the panel's search text), `commentReply` (the open thread's reply
text, focused with the cursor at its end: an `@` there shows the mention picker), `commentEdit` (a
message index of the open thread to edit, 0 for the first; `null` stops), `presentComments` (bool:
the presenter shows the presented frame's comments, also Shift+C there). Dev Mode: `devCompare` (a
ready-for-dev frame or section id opens the Compare changes view, like `dev.compare {id, show:
true}`; `null` closes it) with `devCompareMode` (`sideBySide|overlay`) and `devCompareSplit` (the
overlay slider, 0–1), `annotationFilter` (`all`, `none` for uncategorised, or a category id or name:
the annotations the canvas shows). `resetFrameStats` (true: start `framesDrawn` and
`repaintCauses` over). Returns `ui.inspect` (which reports `stylePicker`,
`stylePickerView`, `comments`: `open`, `draft`, `filter`, `sort`, `search`, `editing` and the
`mentions` the picker offers, `devCompare` (`{id, mode, split}` or `null`), `devFocus` (the frame the
focus view shows, or `null`) and `annotationFilter`).

### `ui.screenshot {path}`

PNG of the whole window (`window` picks one), UI included (needs a visible window; an idle one draws the frame to
capture). Returns `{path, width, height}`, or an error after 60 s when the window doesn't deliver.

### `ui.render {path, scale?, ids?}`

PNG of the document without UI chrome (works without a window). Without `ids` it exports the
selection, else the page's top-level layers. Returns the export result.

### `window.new {path?, index?}`

Open a window: with open file `index` moved into it (from whichever window has it), the file at
`path`, or a new file. Returns `{window}` (its id); it comes to the front. The browser build and a
running screen reader keep one window (only the main window has an accessibility tree): the file
opens as a tab there and the reply is an error saying so. The `window.new` command does the same
for menus, the Actions palette and `engine.execute`.

### `window.list {}`

Every window: `[{id, main, title, files, active, focused}]` (`files` are its tabs' names, `active`
the one it shows, `focused` the window requests go to). `ui.inspect` reports it as `windows`, plus
`window` (the window it ran in).

### `window.focus {id}`

Bring a window to the front; requests without `window` go to it from now on. The `window.focus`
command does the same.

Closing a window closes its files, asking first when one has unsaved changes; the main window
closing while others are open takes over one of them, and closing the last window quits.

### `app.quit {}`

Close the app (every window, without asking about unsaved changes).

## Example session (shell)

```sh
port=7979
figcraft --control $port &
exec 3<>/dev/tcp/127.0.0.1/$port
echo '{"id":1,"method":"engine.execute","params":{"command":"frame.create","params":{"preset":"iPhone 16","x":0,"y":0}}}' >&3
read -r reply <&3; echo "$reply"
echo '{"id":2,"method":"ui.screenshot","params":{"path":"/tmp/figcraft.png"}}' >&3
read -r reply <&3; echo "$reply"
```

For agents, prefer the MCP server (`figcraft-cli mcp --attach 7979`, see `docs/mcp.md`): it wraps
these methods in documented tools and returns images inline.