387b237ee5efbff7af63df4029c8e2012a164423
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 aleleba
|
387b237ee5
|
Phase 6.3.17: run gate 5 in batches and stop burning prompts on a degraded plugin
Measured across three consecutive runs: the Penpot plugin reliably handles 5 or 6 heavy prompts and then degrades to 30-second timeouts on createPage, always with the same shape - the first few work, the rest fail in setup without exception. That is not random flakiness. Prompts now run in batches of 4 with a clean handshake between batches, giving the server a recovery point before the deterioration sets in. A setup failure no longer burns the prompt as measured-with-score-None: it goes on a pending list, the batch stops, and the run exits 3 with the exact GATE5_ONLY line to resume. Insisting past the first timeout only spends pages and dirties the JSON, since once the plugin starts timing out the rest fail identically. On the 30-second timeout the user asked to raise: it is the MCP server's own limit on a plugin task, not a client timeout, so it cannot be raised from here. What can be done is not to approach it. generateStyle with includeChildren plus generateMarkup serialise the whole subtree and are by far the most expensive part of the audit, so above 400 nodes they are skipped and rendersOk becomes not-applicable rather than risking the entire audit - and with it the prompt's measurement - timing out. That required fixing the scoring too: a metric whose VALUE is None is now not-applicable, like one whose threshold is None. Counting "could not measure" as a failure would have penalised exactly the large designs the gate is meant to reward. |
||
|
aleleba
|
f63ff6830d
|
Phase 6.3.17: fix the audit root, add fill metrics, and stop the gate degrading the file
Three defects, the first found by inspecting the user's Penpot file live while the gate reported something else. The audit resolved the page root with penpotUtils.findShapeById(rootId). Every new Penpot page shares the same root frame id, 00000000-0000-0000-0000-000000000000, and findShapeById searches globally, so that lookup always returned the root of the FIRST page in the file - the user's empty one - rather than the page the gate had just created. That is why the baseline reported shapeCount 0 on all ten prompts while the file actually held 28 shapes and 20 texts. It resolves by pageId now, which is unique. With that fixed the baseline reproduces the reported symptom exactly rather than something worse: the button gets 4 shapes, all grey; the navbar gets 10 shapes and 6 real texts with a single distinct fill colour. Production does create structure and text. What it never manages is to apply colour - it creates the shapes in a call that works, then sets fills in a later call using Figma syntax, that call throws, and the shapes keep the #B1B2B5 default. So "creates grey boxes" was accurate and "creates nothing" was my measurement error. Two metrics now capture that directly, since neither distinct-colour counts nor placeholder-grey counts see it - #FFFFFF and #000000 are not mid greys, so a design where every fill is a default passes both. explicitFillShare is the share of filled shapes whose colour is not one of Penpot's three defaults, thresholded by difficulty (0.80 high, 0.65 medium, 0.50 low, since a small artefact's structural neutrals weigh heavily in a ratio). And onlyDefaultColors is a veto: true when the whole palette is those three. White and black stay legitimate when chosen - the distinction is co-presence, not the hex. They are only suspicious when they are all there is; alongside chosen brand colours they also count as neutrals in paletteStructured. resumen() crashed adding None scores from failed prompts. Beyond the crash, an unmeasured prompt must not average in as a zero: "could not measure" and "the model did it badly" are different, and averaging them would have made a mid-run plugin outage look like cheap quality. They are reported separately and any missing prompt makes the verdict invalid, because a baseline with 5 of 10 measured is not a baseline. The gate now empties its own page after exporting the PNG. Across three runs the plugin reliably handled 5 or 6 heavy prompts and then degraded to 30-second timeouts on createPage - that is not random flakiness, it tracks the file growing by one page per prompt, so the gate was manufacturing its own failure. The evidence that matters is the PNG plus the JSON metrics, not the live page. A page is kept only when its PNG failed, so there is something to inspect. Exit code 3 now distinguishes "plugin degraded mid-run, reload it and resume these ids" from "plugin not connected". |
||
|
aleleba
|
8094183940
|
Phase 6.3.17: fix a harness fidelity bug and add a vibrancy metric
The first baseline measured production with only 2290 of the 16392 characters of the server's instructions block - 14%. The missing 86% is exactly the API grounding: Core Shape Properties and Methods, Layout Systems, Text Elements, and The penpot and penpotUtils Objects, which is where insertChild, resize(), the layouts and penpotUtils are documented. That was worth catching, because the discrepancy had a visible signature: the measurement said production creates nothing, while the user's real Claude Code session produced grey boxes, i.e. shapes greater than zero. When a harness and reality disagree, the harness is the first suspect. In phase 5 a low max_tokens manufactured an apparent regression the same way. The gate now injects the full document, minus the trailing "You have hereby read the Penpot High-Level Overview" line, which is framing of the tool response rather than part of the instructions block and would otherwise tell the model it had already read something. The finding survives the fix. Across the five prompts measured cleanly under the corrected condition, shapeCount is still zero on every one. So the API invention is not an artefact of withholding documentation from the model - it happens with the documentation present. Also adds the vibrancy requirement the user raised as first-class scope: given an ambiguous brief the model must choose and justify a palette rather than ask or fall back to defaults. Neither distinctFillColors nor placeholderGreys distinguishes a vibrant palette from a muted but technically non-grey one, so four metrics are added: chromaticFills, meanChromaticSaturation, paletteStructured (a dominant brand hue, an accent at least 30 degrees away, and neutrals), and finalMessageListsHex, because a palette chosen in silence cannot be adjusted by the user. The saturation floor of 45 is derived, not asserted: measured over the 325 non-neutral fills of this phase's hand-authored corpus, median HSL saturation is 75, p25 is 48 and p10 is 35. A floor of 45 sits just under the first quartile and is cleared by 79% of those fills, so it is a floor the target behaviour already clears rather than an aspiration. The lightness band of 15 to 85 excludes near-blacks and near-whites, which can compute as highly saturated while reading as neutral. Gate prompt 6 becomes the user's literal failing sentence, and two ambiguous-brief prompts are added. One of them had to be re-domained after the disjointness check found it shared a 6-gram with a seed - the check fails on a single shared shingle, which is what makes it useful. Results so far are partial: prompts 1-5 measured cleanly, 6 has a timed-out audit and 7-10 hit the MCP outage, so those get re-measured. Both runs are kept, the 14% one renamed to record what it was. |
||
|
aleleba
|
5f0ddd962c
|
Phase 6.3.17: measure the gate 5 baseline against production
Ran the full agent loop against vllm-qwen36 on port 8000 (read-only HTTP)
with the Penpot plugin live, before asking for any downtime. Without this
file "it improved" would be a claim rather than a measurement.
Result over the 8 graded prompts: mean score 14.9, zero prompts at or above
60, veto violated on 2 of 8, 66% of execute_code calls raised, and 7 of 8
prompts burned all 14 turns without producing a final message.
The plan predicted production would score near zero on distinct colours and
style richness while producing a high shape count - grey boxes. The shape
count is also zero. On a fresh page it creates nothing at all, so the
failure sits upstream of the grey boxes: the model invents a Figma-shaped
API wholesale and every call throws. From the captured turns:
penpot.currentPage() is a property, not a function
penpot.createRectangle(page, 200, 56) takes no arguments
penpot.createText(page, ...) takes one, the text
penpot.getPageById(...) lives on penpotUtils
fills = [{type:'solid', color:{r,g,b,a}}] is {fillColor, fillOpacity}
shadows = [{type:'drop', x, y, blur, ...}] is {style, offsetX, offsetY}
It then spends the remaining turns querying penpot_api_info without
recovering. So the reported symptom understated it.
Two robustness fixes the run itself forced, both after losing a completed
run to them:
- A ConnectionError does not just drop the request, it can drop the MCP
session, so retrying the same tools/call against a dead session fails
identically every time - which is exactly what the first attempt showed,
four retries and four identical ConnectionErrors. The client now redoes
the handshake before retrying, and that recovered two drops in this run.
- Results are written after every prompt. The first attempt died on prompt
4 and lost the three already measured, which is the expensive data
precisely because it requires production to be up.
|
||
|
aleleba
|
c65d309719
|
Phase 6.4: make the gates fail when they cannot verify something
A code review found seven ways these gates could pass green with something actually wrong. All are the same family: a missing value was treated as OK. The rule now written into all three files is that absent is not OK, absent is "could not verify", and that either fails or is reported as an explicit SKIP - it never slips through as green. 30_eval_suite.py: - A bucket with no baseline of its own fell back to the global 0.2750 and printed it in a column headed "baseline", as if it were that bucket's number. Measured against the real eval.jsonl buckets: negativos going from 0.12 to 0.33 is a real +0.21 regression, but the computed delta was +0.055 and it PASSED; manejo_errores sitting unchanged at 0.42 produced a fabricated +0.145 FAIL that would have discarded a healthy candidate mid-downtime. Now such buckets print SKIP and the verdict reports how many went unverified. - "VEREDICTO: FAIL" exited 0, so a runbook chaining the gate into quantization would have carried on to write 24 GB. Now exits 1. - A typo in BASELINE_BUCKET_LOSSES silently matched nothing; now aborts. - The penpot exemption is labelled honestly: those 11 rows are pre-existing LoRA #1 tool-calling, not new capability, so gate 1 has no regression coverage there and the log says so. 20_merge_lora.py dry-run (merge path untouched, verified by AST diff): - adapter_config.get("use_rslora", False) meant a missing key passed AND the log printed use_rslora=False, asserting it had checked something that was never there. A different PEFT version omitting a key was enough. - lora_bias was not checked at all, only bias. They are different fields: lora_bias puts a bias inside lora_B, which W + scaling * (B @ A) ignores. - The 620 keys were printed but never asserted, so an adapter with extra tensors printed "310 + 310 = 930" and passed. - rank_pattern/alpha_pattern were not checked. They set r per module, so scaling is not uniformly alpha/r while both the dry-run and the merge apply a single 2.0 to all 310 tensors. - A missing family was invisible: swap linear_attn for 150 mlp.gate targets and the total is still 310, no norm is zero because the family is simply gone, and it passed. Now presence and per-family counts are asserted, derived from the real adapter: linear_attn 150, shared_expert 120, attention_qkvo 40, otros 0. Verified against seven synthetic adapters plus the real phase 3 one; only the correct adapter passes. 21_quantize_nvfp4.py (recipe and oneshot untouched): the calibration cache now carries a provenance.json recording the training file's sha256, the recipe numbers and the bucket distribution, and loading aborts on mismatch. This is the phase's number one risk and it had no mechanical defence: the phase 5 cache on disk has exactly 512 rows, the same as the v2 recipe, so the only existing check could not tell them apart and reusing it would have calibrated with zero design data and washed out the new capability silently. Verified: that cache now aborts. gate 5: retry transport failures against the Penpot MCP, which drops connections mid-call intermittently (seen before in phase 4's gate 4). Without it a blip on prompt 6 of 8 kills a whole run and reads like a model failure. PluginNotConnected is deliberately not retried - that is a real state of the world. Also unwrap the {"result":..., "log":...} envelope the server wraps execute_code returns in; the gate was reading keys off the outer object and rejecting a valid page setup. |
||
|
aleleba
|
8ea4572edd
|
Phase 6.3: add gate 5, design quality in Penpot
Unlike gates 2-4 this one needs a real agent loop - model, tool call, live
MCP, result, up to 14 turns - because design quality only exists after the
code executes. It talks to vLLM over the OpenAI API and to the Penpot MCP
over HTTP (initialize, notifications/initialized, tools/list, tools/call),
handling both application/json and text/event-stream responses. Endpoints
and credentials come from env with no defaults and are never printed or
stored; requests errors are reduced to the exception type because the
requests message embeds the URL.
Eight graded prompts, a fresh page per prompt named gate5/<tag>/<id>/<ts>,
and the gate never deletes anything. The 17 metrics are computed by an
audit payload the gate injects, not the model. placeholderGreys and
forbidden behaviour are veto metrics: any hit scores that prompt 0.
Two things worth calling out.
The forbidden-pattern regexes are imported from 07_lint_penpot_code.py
rather than duplicated, and the gate runs those same regexes over its own
setup and audit payloads at startup - a gate that violated the API it is
grading would be measuring its own bug.
The holdout mode had a silent failure that is exactly the kind this phase
exists to catch: with the endpoint down, all 60 generations failed, each
entered the denominator with zero forbidden patterns found, and the gate
reported 0% forbidden API and APPROVED. Since that number is the fallback
trigger, a false pass there would have launched the quantization run.
Request errors are now counted separately, never enter the denominator,
and block approval outright.
Known issue, resolved separately: gate prompt 6 is the exact production
failure ("hazme una landing page de una pizzería con colores vibrantes"),
and the flagship B6 seed was written to the same wording. Shingle overlap
measures 20%, under the 34% threshold, but the seed prompt is a literal
substring of the gate prompt - the threshold is too loose for prompts this
short. The seed's domain gets changed rather than the gate's, so gate 5
measures transfer instead of memorisation; the real pizzeria prompt still
runs in the human acceptance test, which is the criterion that decides.
|