Ten prompts measured three times against vllm-qwen36, all with the same
23-metric harness. Three earlier result files are kept but excluded from
the consolidation and named for why: one used 14% of the server's
instructions block, two predate the audit-root fix and report zero shapes
by construction.
The consolidated numbers correct the previous report, which was wrong.
Production does not fail to draw. It creates structure and text - up to 40
shapes and 24 texts on the ambiguous-brief prompt - and then fails to
colour any of it. Distinct fill colours run 0 to 1 across all thirty
measurements.
Mean score per repetition: 6.79, 8.25, 0.00. No prompt reaches 60 in any
repetition.
Reporting the veto rate separately from the score turned out to matter more
than expected, so it is now the headline number alongside it. Of thirty
measurements, 21 violate the forbidden-API veto and 15 use nothing but
Penpot's three default colours; only 6 are veto-free. The score alone
collapses those into a zero that cannot distinguish "broke an API rule"
from "made an ugly design", and the two need different fixes.
That also explains why single measurements looked stable: the score is
pinned at zero by the veto, not by model consistency. All the real variance
sits in the prompts that do not veto, where scores swing 20 to 35 points
between identical runs. So the repetitions matter more for the trained
model, which should land in that non-veto regime, than for this baseline.
The eval window at the end has to budget for three repetitions on that side
too, or the comparison is asymmetric.
Ran the full agent loop against vllm-qwen36 on port 8000 (read-only HTTP)
with the Penpot plugin live, before asking for any downtime. Without this
file "it improved" would be a claim rather than a measurement.
Result over the 8 graded prompts: mean score 14.9, zero prompts at or above
60, veto violated on 2 of 8, 66% of execute_code calls raised, and 7 of 8
prompts burned all 14 turns without producing a final message.
The plan predicted production would score near zero on distinct colours and
style richness while producing a high shape count - grey boxes. The shape
count is also zero. On a fresh page it creates nothing at all, so the
failure sits upstream of the grey boxes: the model invents a Figma-shaped
API wholesale and every call throws. From the captured turns:
penpot.currentPage() is a property, not a function
penpot.createRectangle(page, 200, 56) takes no arguments
penpot.createText(page, ...) takes one, the text
penpot.getPageById(...) lives on penpotUtils
fills = [{type:'solid', color:{r,g,b,a}}] is {fillColor, fillOpacity}
shadows = [{type:'drop', x, y, blur, ...}] is {style, offsetX, offsetY}
It then spends the remaining turns querying penpot_api_info without
recovering. So the reported symptom understated it.
Two robustness fixes the run itself forced, both after losing a completed
run to them:
- A ConnectionError does not just drop the request, it can drop the MCP
session, so retrying the same tools/call against a dead session fails
identically every time - which is exactly what the first attempt showed,
four retries and four identical ConnectionErrors. The client now redoes
the handshake before retrying, and that recovered two drops in this run.
- Results are written after every prompt. The first attempt died on prompt
4 and lost the three already measured, which is the expensive data
precisely because it requires production to be up.
aleleba