Phase 6: train a second LoRA for real Penpot UI design capability #5
Merged
aleleba
merged 36 commits from 2026-08-04 21:36:07 -06:00
agente-fase6-lora2-penpot into master
36
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 aleleba
|
4d840553d6
|
Phase 6.4.27: re-measure g5-08-reparacion-grises to complete gate 5 full on v2b-NVFP4
g5-08 was pending since it fell mid-run when Penpot's plugin disconnected
("No Penpot plugin instances are currently connected"). Re-run in isolation
via GATE5_ONLY, per the established pattern: score 64.7, clean veto, no
placeholder greys. Gate 5 full mode on v2b-NVFP4 is now complete (10/10):
mean score 28.2 (threshold 65), clean veto 6/10, prompts>=60 2/10, pizzeria
landing 47.8 (threshold 60) -- verdict: DOES NOT APPROVE.
|
||
|
aleleba
|
66d6c6c410
|
Phase 6.4.27: gate 5 (holdout + full) on v2b-NVFP4, spec config
Holdout (60 design prompts, cheap proxy, no live Penpot needed): 5.0%
forbidden-API rate (3/60) -- right at the <=5% threshold, APPROVES.
Full mode (10 official prompts against live Penpot): 9/10 measured.
g5-08-reparacion-grises is still pending -- Penpot disconnected mid-run
("No Penpot plugin instances are currently connected", the real
disconnect signature, not the transient "transport dropped mid-call").
Per the established GATE5_ONLY pattern, this prompt gets re-measured
(not accepted as a genuine failure) once Penpot reconnects; the other
9 results are preserved as-is.
|
||
|
aleleba
|
f4564bc7a0
|
Phase 6.4.27: gates 2/3/4 on v2b-NVFP4, spec and nospec
Gate 2 (tool-calls, 200 held-out): global 95.5% (191/200) on both spec and nospec, matching v2b-bf16's 95.0%. Per-MCP breakdown diverges: spec regresses penpot specifically (91.7% vs bf16's 98.3%, outside the 2-point guard) while preserving atlassian/docmost; nospec preserves penpot (98.3%, matches bf16) but regresses atlassian/docmost instead. Consistent with risk #10 (MTP draft head drift) rather than risk #9 (quantization washing out the LoRA delta) -- if quantization itself hurt penpot, nospec should show the same regression, and it doesn't. Sampling is non-greedy (temperature 0.6), so some of this is ordinary run-to-run variance rather than a clean systematic signal. Gate 3 (21-check adherencia): spec 90.5% (19/21), nospec 71.4% (15/21). Both clear the >=61.9% no-regression bar against the production baseline, but the gap between spec/nospec is real and worth flagging. Gate 4 (E2E against real MCPs, spec only): gitea, docmost, and penpot (high_level_overview) executed successfully on the model's first-shot tool call. github-personal and atlassian failed on this specific attempt due to single-shot argument guesses (empty search query; a plausible but wrong Jira cloudId) -- verified separately that both integrations work fine once given correct arguments, so this reads as ordinary single-shot tool-use limitation, not a broken integration. |
||
|
aleleba
|
1c672520a4
|
Phase 6.4.26: point 8004/8005 eval services at v2b-NVFP4
The fallback decision landed on v2b-bf16 directly (v2-NVFP4 is skipped), so the two candidate eval services need to serve the checkpoint that's actually being quantized. Container names/ports/served-model-name stay the same, only the volume and --model path move to v2b-NVFP4. |
||
|
aleleba
|
cc54edc39d
|
Phase 6.4.25c: re-score existing v2b-bf16 gate 5 results with corrected veto
New offline script re-scans the code payloads already saved in the gate 5 result JSONs (holdout, full, generalization) using the corrected forbidden- behavior list -- no live Penpot needed, reuses scan_forbidden/score_prompt from 35_gate5_penpot_design.py as the single source of truth. Holdout result changes materially: 3 of the 5 flagged prompts (pd-008, pd-009, pd-035) were flagged only for flex.appendChild. With that pattern removed, forbidden-API rate drops from 8.3% (5/60) to 3.3% (2/60), which is now under the <=5% fallback threshold -- the holdout gate now passes. Full mode and the generalization probe are unaffected: neither run's transcripts ever triggered the flex.appendChild pattern, so their scores are identical before and after. Original result files are kept untouched; corrected results are written alongside as *-corrected.json. |
||
|
aleleba
|
7a1577b9d7
|
Phase 6.4.25c: correct flex.appendChild -- live verification shows it works
Live probing against Penpot 2.16.2 (repeated, on a dedicated scratch page, cleaned up afterwards) shows board.flex.appendChild(shape) is a real function, distinct from board.appendChild, arity 1, throws nothing, and preserves visual insertion order -- unlike board.appendChild, which still inserts at the front as documented. The prior "broken" claim traces to the MCP server's own high_level_overview() text (CRITICAL / BROKEN / NEVER use), never verified with a live throw, and possibly confused with a real doc bug that mislabels the grid 3-arg example as board.flex. Removes flex.appendChild from the lint's FORBIDDEN patterns and from the gate 5 forbidden-behavior veto, updates PENPOT_API_VERIFIED.md with the verified facts, and drops the now-obsolete pattern from the replay filters in 05_build_dataset.py and 07_build_lora2_mix.py. |
||
|
aleleba
|
3496051f72
|
Phase 6.4.25b: measure generalization probe on v2b-bf16 (ADDITIONAL, not official gate 5)
4 prompts in domains absent from both seeds and the official gate5_prompts.jsonl (ferreteria, gimnasio, tienda de mascotas, panaderia). score medio 32.0, veto limpio 3/4, 0/4 prompts >=60. gp-03-tienda-mascotas violates the veto twice over: it hits both import_image variants the model has ever tried (penpot.importImage on turn 1, penpotUtils.importImage on turn 2) plus a numeric fontSize assignment -- the uploadMediaUrl reinforcement seeds from this correction round did not transfer to an unseen domain. It is also flagged via GATE5_KNOWN_OVERLAP_IDS (shares a 6-gram with the same seeds as g5-03), so its score is a double caution case. gp-02-gimnasio hit the same transient Penpot MCP transport drop as g5-06/g5-07 on its first pass, but the harness's own automatic retry (pass 2/3) recovered cleanly within the default 240s -- no manual timeout override was needed here. |
||
|
aleleba
|
bf795bd01c
|
Phase 6.4.25b: measure gate 5 full mode on v2b-bf16 (post-correction)
10/10 prompts measured. VEREDICTO: NO APRUEBA (score medio 39.8 vs threshold 65, veto limpio 8/10, 3/10 prompts >=60). The flagship g5-06-landing-pizzeria now clears its own threshold (69.6 >= 60) once a transient Penpot MCP transport drop is ruled out (see g5-06/g5-07 rerun below) -- a real improvement over v2-bf16's 34.8 on that prompt, but the aggregate gate criteria still fail. g5-03-card-producto is flagged via GATE5_KNOWN_OVERLAP_IDS: it shares a 6-gram with seeds 134-137 already baked into v2b-bf16's training, so its result should be read with caution rather than as a clean generalization signal. g5-06/g5-07 were re-measured once with GATE5_ONLY + GATE5_PROMPT_TIMEOUT_S=420 (default 240s untouched) after their first pass hit a genuine Penpot MCP transport drop (ConnectionError + handshake rebuild) that ate into the turn budget and caused a false-negative timeout. g5-04 and g5-08 show the same 'no board created' signature but their raw turn-by-turn transcripts have zero MCP/transport/handshake evidence -- both are the known post-error tool-call-format bug (literal '<tool_call>' text instead of a structured call), so they are left as genuine failures with no rerun. |
||
|
aleleba
|
c7b1311928
|
Phase 6.4.25b: gate5 -- declare THRESHOLDS for the generalization probe prompts
gp-01..gp-04 (ferreteria, gimnasio, tienda de mascotas, panaderia) had no THRESHOLDS entry, so the probe hard-failed before running a single prompt. Each entry mirrors the closest of the 10 official gate5 prompts by structure/difficulty, so the probe's pass/fail criteria stay comparable to the official gate even though it is explicitly a separate, additional check. |
||
|
aleleba
|
cc252d70f5
|
Phase 6.4.25b: gate5 -- allow explicit ack of known train/prompt overlap (GATE5_KNOWN_OVERLAP_IDS)
The 4 uploadMediaUrl reinforcement seeds added in the corrective round (134-137) share a
6-gram with g5-03-card-producto ('foto real traida de https picsum'). v2b-bf16 is already
trained on that text -- editing the seed corpus now cannot retroactively decontaminate it.
This lets the disjoint check acknowledge that specific, already-baked-in overlap explicitly
(never silently) instead of hard-aborting gate5 full mode entirely.
|
||
|
aleleba
|
8dcd736ba5
|
Phase 6.4.25b: measure gate 5 holdout mode on v2b-bf16 (post-correction)
Tasa de API prohibida: 8.3% (5/60), baja de 15% (9/60) en v2-bf16, pero
sigue por encima del umbral de 5% -- NO APRUEBA.
Desglose por patron (v2-bf16 -> v2b-bf16):
- import_image/importImage/createImage/filePath: 4 -> 1 (-75%, el
fix que apuntaban los seeds de uploadMediaUrl)
- fontSize/fontWeight/lineHeight/letterSpacing numerico: 3 -> 1 (-67%)
- flex.appendChild: 2 -> 3 (empeoro levemente; NO fue el foco de
esta ronda de seeds correctivos, es el patron ahora dominante:
3 de los 5 vetos restantes)
Se continua con el resto del runbook (verificacion de Penpot, puerta 5
full, sonda de generalizacion) para tener el cuadro completo antes de
reportar, tal como indico el usuario -- pero este resultado ya senala
que el disparador de fallback documentado en PLAN.md (<=5%) se activa
de nuevo, esta vez por un patron distinto y no cubierto por los seeds
de esta iteracion.
|
||
|
aleleba
|
7c67e100c5
|
Phase 6.4.25b: measure gate 2 (tool-calls) on v2b-bf16 candidate
Global 190/200 (95.0%), penpot 59/60 (98.3%) vs baseline v2-bf16's penpot 60/60 (100%) -- 1.67pt regression, within the 2pt guard from the gate5-full regression-guard rule. PASS. |
||
|
aleleba
|
959404823a
|
Phase 6.4.25b: parametrize gate5 prompts path, add generalization probe
GATE5_PROMPTS_PATH env var lets the same 17-metric harness (veto system, MCP retry/page handling) run against a different prompts file. Unset behavior is unchanged -- the official gate 5 (data/gate5_prompts.jsonl, 10 frozen prompts) stays untouched. data/gate5_generalization_probe.jsonl: 4 prompts in rubros confirmed absent from the 138-seed corpus, the 10 gate5 prompts, and the 60-prompt holdout -- ferreteria, gimnasio, tienda de mascotas, panaderia. Swapped out "veterinaria" (originally planned) after finding seed 83 is a substantial design-repair seed for a vet clinic; "tienda de mascotas" verified clean instead. This is an additional check, separate from the official gate 5, meant to test whether the correction generalizes across business types rather than overfitting to the pizzeria case. |
||
|
aleleba
|
20493a99b6
|
Phase 6.4.25b: gate 1 on v2b-bf16 -- full PASS on all buckets
Global weighted loss 0.2725 (vs baseline 0.2750, previous v2-bf16 was 0.2734). All 6/6 non-penpot buckets pass the 0.10 regression threshold, including delegacion_subagentes which had narrowly failed on v2-bf16 (+0.1019 -> now +0.0882). The 13 corrective seeds didn't hurt forgetting. Also points vllm-eval at the new v2b-bf16 checkpoint for the gate 2/5 re-measurement. |
||
|
aleleba
|
52dd307c39
|
Phase 6.4.25b: version the corrective LoRA #2 adapter (out/lora-adapter-penpot-v2/)
171 steps, train_loss=0.6808, eval_loss=0.6348 (comparable to v1's 0.6306),
peak CUDA memory 74.58GB -- identical to v1's run, confirming the 13
corrective seeds didn't change the memory/length envelope. All guards
verified in adapter_config.json: use_rslora=false, use_dora=false,
lora_bias=false, modules_to_save=null, r=32, lora_alpha=64.
v1 adapter (out/lora-adapter-penpot/, commit
|
||
|
aleleba
|
18efe3fa5d
|
Phase 6.4.25b: 13 corrective seeds for gate 5 diagnosis, rebuild training mix
Diagnosis (audited gate5 full-mode transcripts, PNGs, and holdout payloads): the dominant failure mode was NOT rubro/content coverage but a tool-call formatting bug -- after diagnosing an exception, the model often writes its retry as literal <tool_call> XML text embedded in reasoning instead of a structured call, so it's silently dropped (9/10 gate5 full-mode prompts hit this on their final captured turn). Reasoning length before a tool call correlates with the failure (malformed-turn mean 1895 chars vs 364 for well-formed turns), while the training mix's own tool-calling turns never exceed 1630 chars and export_shape turns never exceed 485. Corrective seeds (group D, grupo 'D', short 1-3 sentence diagnosis + immediate well-formed retry, to avoid reinforcing long-reasoning risk): - 9 seeds across fresh rubros (lavanderia, zapateria, optica, jugueteria, papeleria, peluqueria, floreria, heladeria, cerrajeria) covering distinct real API errors: .color on Text, addFlexLayout on non-Board, findShapeById 2-arg, createText() no-arg, textAlign, typography.setFont, createBoolean null, appendChild no-arg, uploadMediaUrl network rejection. - 4 seeds reinforcing uploadMediaUrl over the exact penpot.importImage variant the model actually invents (distinct from the one existing seed, which only taught the penpotUtils.importImage variant) -- rubros: relojeria, guarderia infantil, agencia de viajes, kiosco. Deliberately avoid ferreteria/gimnasio/veterinaria/panaderia: reserved for the held-out generalization probe (separate from the frozen gate 5). Added the newly-verified error strings (t.color, penpot.importImage, addFlexLayout-on-non-Board) to penpot_errors.md -- all captured live from this phase's own gate 5 full-mode run, not fabricated. Rebuilt train_lora2.jsonl (901)/eval_lora2.jsonl (99)/calibration_v2.jsonl from the 138-seed corpus (07_build_lora2_mix.py) and validated (06_validate_dataset.py, 0 exceptions, 0 secrets, max 3271 tokens). |
||
|
aleleba
|
730e42494f
|
Phase 6.4.25: gate 5 full-mode results on v2-bf16 (verdict: FAIL)
10/10 prompts measured against the live Penpot plugin. Mean score 43.1 (threshold 65), pizzeria landing 34.8 (threshold 60), 4/10 prompts >=60, veto clean on 8/10 (card-producto forbidden-API, brief-ambiguo-escuela placeholder-grey). Verdict: NO APRUEBA. Combined with gate 1 (PASS global, 1 small bucket FAIL non-blocking), gate 2 (PASS 96%), and gate 5 holdout (FAIL 15% forbidden API), this closes the fallback-decision measurement set for step 6.4.25. |
||
|
aleleba
|
0cf8bb1df7
|
Phase 6.4.25: point vllm-eval at v2-bf16 and record gate 2/5-holdout results
Gate 2 on v2-bf16: PASS, 192/200 = 96.0% (penpot 60/60 = 100.0%). Gate 5 holdout mode (60 prompts, fallback trigger): FAIL, forbidden-API rate 15.0% (9/60) vs threshold <=5% -- model still occasionally invents penpot.importImage(...) for photo/URL prompts. |
||
|
aleleba
|
363648fb5f
|
Phase 6.4.21: version the LoRA #2 adapter (Penpot design capability)
171 steps (901 examples / 16 grad-accum * 3 epochs, ceil-rounded, not the 168 the plan estimated with floor), trained decoupled on spark against the merged bf16 checkpoint that already has LoRA #1 folded in. No OOM, no aborts: train_runtime 24370s (~6h46m, within the ~7.3h projection), eval_loss 0.6306, peak CUDA 74.58GB, trainable% 0.1220 (matches phase 3 exactly). adapter_config.json confirms the guarded hyperparameters: use_rslora and use_dora both false, lora_bias false, modules_to_save null, r=32, lora_alpha=64 -- the combination 20_merge_lora.py's scaling math depends on. Weights never touched the worktree: OUTPUT_DIR was /workspace/ft-models/lora-adapter-penpot on spark (outside the worktree, root-owned by the training container), copied out via `docker exec cat` piped to a non-root file and verified by matching sha256 (79167dfa...) before landing here. Intermediate checkpoint-*/ directories stay on spark; only the final adapter is versioned, same as phase 3's out/lora-adapter/, which this leaves untouched. .gitignore was missing the negation lines for out/lora-adapter-penpot/ despite already documenting that the final adapter should be committed -- added the same two exceptions that out/lora-adapter/ has. |
||
|
aleleba
|
2781b9eb32
|
Phase 6.4.20: fix eval OOM by pinning per_device_eval_batch_size
HF Trainer defaults per_device_eval_batch_size to 8, independent of the training batch size, and that default was never overridden. During the eval forward pass (no gradient checkpointing needed there, so none is applied) an 8-example batch of long sequences materializes full fp32 logits at once and blows the CUDA budget. That was the real cause of both OOMs hit while calibrating this run (the worst-case-32 smoke run and the 8-example probe) -- not the training forward/backward, which measured a stable ~74GB peak across every length from 2808 to 3265 tokens in three separate calibrations. Fix: new EVAL_BATCH_SIZE env (default 1) wired into per_device_eval_batch_size. Verified twice after the fix: training on the 32 longest examples in the corpus (3161-3271 tokens, worst case) plus a full eval pass over all 99 real eval_lora2.jsonl examples (up to 3243 tokens) completed with no OOM, peak 72.41GB. Also adds PerStepMemoryCallback (opt-in via PER_STEP_MEMORY_LOG=1) to print per-step CUDA peak/reset, which is what let this calibration attribute the earlier OOM to eval rather than to a specific training micro-batch under GRAD_ACCUM=16. |
||
|
aleleba
|
c24f0ab236
|
Phase 6.4.20: build a worst-case probe set for the smoke run
The 32 longest examples of the training mix, 3161 to 3271 tokens each. With batch 1 and gradient accumulation 16, a two-step smoke run consumes exactly these 32, so it trains on the worst case the real run will ever see. Risk #5 in the plan is an OOM discovered hours into the run. Phase 3 peaked around 104 GB of the 121 GB available with a longest example near 2800 tokens, and this corpus goes to 3271. Rather than argue about whether the extra 17% of sequence length fits, the smoke run measures it in ten minutes. If it OOMs it OOMs immediately and cheaply, which is the whole point of running the probe before the 169-step run rather than after. |
||
|
aleleba
|
b4b4365725
|
Phase 6.3: re-measure the gate 2 and 3 production baselines on the new holdout
Rebalancing the holdout regenerated all 200 prompts, so phase 5's 192/200 stopped being comparable and every gate2/gate3/gate4 result file in the repo was from phase 5. These had to be measured with production still up, before asking for the downtime, or it would have cost the user another window later just for this. Gate 2 over the new holdout: 191/200, 95.5% global. atlassian 97.1, docmost 94.3, gitea 85.7, github-personal 100, penpot 98.3. All five failures are missing_required and there are zero invented arguments. That penpot figure is worth keeping in view: production is at 98.3% on tool-call VALIDITY, so its failure is not in the shape of the call but in the code it puts inside execute_code. The two are measured by different gates and only gate 5 sees the second. Gate 3: 61.9%, which is 13 of 21. Not comparable to phase 5's 10/10, since that run had 10 checks and this one has 21 - the 11 new content checklists for the non-obvious conventions of the other MCPs are what risk #12 in the plan called the most likely invisible regression. Production already fails 7 of them, so they have headroom rather than being a formality. Gate 4 needs no re-baseline, verified rather than assumed: grep for holdout_prompts in 34_gate4_e2e.py returns 0, so the regenerated holdout does not reach it and its phase 5 result stands. It also needs all five MCPs live, including the Penpot plugin, which closes when downtime starts. Also trims the one seed that exceeded the token ceiling. The corpus now validates clean against the production tokenizer with preserve_thinking on: 1000 of 1000 rows, zero exceptions, zero secrets, p50 1858, p90 3021, max 3271 against a 3300 limit. That check earning its keep is the reason 06_validate_dataset.py was changed to exit non-zero instead of printing [FILTERED] and moving on. |
||
|
aleleba
|
3919b6baff
|
Phase 6.3.17: close the production baseline with three repetitions
Ten prompts measured three times against vllm-qwen36, all with the same 23-metric harness. Three earlier result files are kept but excluded from the consolidation and named for why: one used 14% of the server's instructions block, two predate the audit-root fix and report zero shapes by construction. The consolidated numbers correct the previous report, which was wrong. Production does not fail to draw. It creates structure and text - up to 40 shapes and 24 texts on the ambiguous-brief prompt - and then fails to colour any of it. Distinct fill colours run 0 to 1 across all thirty measurements. Mean score per repetition: 6.79, 8.25, 0.00. No prompt reaches 60 in any repetition. Reporting the veto rate separately from the score turned out to matter more than expected, so it is now the headline number alongside it. Of thirty measurements, 21 violate the forbidden-API veto and 15 use nothing but Penpot's three default colours; only 6 are veto-free. The score alone collapses those into a zero that cannot distinguish "broke an API rule" from "made an ugly design", and the two need different fixes. That also explains why single measurements looked stable: the score is pinned at zero by the veto, not by model consistency. All the real variance sits in the prompts that do not veto, where scores swing 20 to 35 points between identical runs. So the repetitions matter more for the trained model, which should land in that non-veto regime, than for this baseline. The eval window at the end has to budget for three repetitions on that side too, or the comparison is asymmetric. |
||
|
aleleba
|
869b00c924
|
Phase 6.3.17: make gate 5 measure all ten prompts in one resilient run
The Penpot MCP keeps going down and each window where it works is expensive - three of the user's windows were spent on runs that aborted partway. The run is now built to finish rather than to be correct about why it stopped. One prompt failing no longer ends the run. Failures are collected and retried on a later pass, up to three passes, with a clean handshake and a probe between them. Even an unexpected exception is caught per prompt, because one bug in one prompt must not take down the other nine. The opening probe no longer aborts either: a plugin that does not answer now may answer on pass two, and aborting there throws away the whole window for a transient state. Two time budgets bound it: four minutes per prompt, checked before every turn of the agent loop, and forty-five minutes of wall clock. Both are env vars. A hang used to mean waiting forever; now it costs one prompt and the run continues. Results are written after every prompt, so a crash costs at most the prompt in flight, and the JSON is only marked complete when all ten have a score. The run also inventories the accumulated gate5 pages at the end and writes the list to disk. It does not delete them: the pages this run created are already emptied after their PNG is exported, but the ones from earlier runs are in the user's own file, so that is offered rather than assumed. |
||
|
aleleba
|
ad62277f14
|
Phase 6.3: stop the gate depending on penpot.root, and stop blaming the plugin for its own bugs
The setup returned penpot.root.id and the run aborted when it came back as an empty string on the second prompt of a batch. penpot.root is the root of the ACTIVE page, and after createPage plus openPage it need not have caught up yet - a race the page-emptying code introduced, since that leaves the emptied page active. The value was never useful anyway: every page shares the same root id, so it identified nothing. Setup now returns only pageId, which is unique and stable, and both the audit and the cleanup use it. The diagnostic message is the second half of the same mistake. It printed "PLUGIN DEGRADADO, ask the user to reload the browser" whenever any prompt went unmeasured, regardless of why - so it said that for a failure that was entirely the gate's own. A message that sends the user to reload their browser when the bug is mine costs both of us time. It now classifies on the error text: task timeouts and transport drops point at the plugin, anything else points at the gate and says so explicitly. Also closes two evaluation leaks the gate's own pre-flight caught, both in the seeds rather than the gate prompts, since the prompts have to stay as a real user would write them: - A seed shared the 6-gram "la home de una escuela de" with gate prompt 9. My first fix was overwritten by a subagent still writing the file, which is why it reappeared. - A seed used the same business as gate prompt 9 - a music school - without sharing any 6-gram. Shingles cannot see that: two texts describe the same business without sharing words. Training on the domain we then evaluate inflates the result invisibly. So the gate now also checks that no seed uses any of the gate's business nouns, listed explicitly. The training mix was rebuilt: it had been assembled before the 20 ambiguous-brief seeds existed, so training on it would not have used the corpus that was audited. The ambiguous-brief class gets its own mix portion rather than being folded into design, because diluted across 76 design seeds it would be at the mercy of a ratio, and that is the class the user named as the main painpoint. 125 seeds, 168 unique payloads, 446 distinct user prompts, 901 train and 99 eval. |
||
|
aleleba
|
387b237ee5
|
Phase 6.3.17: run gate 5 in batches and stop burning prompts on a degraded plugin
Measured across three consecutive runs: the Penpot plugin reliably handles 5 or 6 heavy prompts and then degrades to 30-second timeouts on createPage, always with the same shape - the first few work, the rest fail in setup without exception. That is not random flakiness. Prompts now run in batches of 4 with a clean handshake between batches, giving the server a recovery point before the deterioration sets in. A setup failure no longer burns the prompt as measured-with-score-None: it goes on a pending list, the batch stops, and the run exits 3 with the exact GATE5_ONLY line to resume. Insisting past the first timeout only spends pages and dirties the JSON, since once the plugin starts timing out the rest fail identically. On the 30-second timeout the user asked to raise: it is the MCP server's own limit on a plugin task, not a client timeout, so it cannot be raised from here. What can be done is not to approach it. generateStyle with includeChildren plus generateMarkup serialise the whole subtree and are by far the most expensive part of the audit, so above 400 nodes they are skipped and rendersOk becomes not-applicable rather than risking the entire audit - and with it the prompt's measurement - timing out. That required fixing the scoring too: a metric whose VALUE is None is now not-applicable, like one whose threshold is None. Counting "could not measure" as a failure would have penalised exactly the large designs the gate is meant to reward. |
||
|
aleleba
|
f63ff6830d
|
Phase 6.3.17: fix the audit root, add fill metrics, and stop the gate degrading the file
Three defects, the first found by inspecting the user's Penpot file live while the gate reported something else. The audit resolved the page root with penpotUtils.findShapeById(rootId). Every new Penpot page shares the same root frame id, 00000000-0000-0000-0000-000000000000, and findShapeById searches globally, so that lookup always returned the root of the FIRST page in the file - the user's empty one - rather than the page the gate had just created. That is why the baseline reported shapeCount 0 on all ten prompts while the file actually held 28 shapes and 20 texts. It resolves by pageId now, which is unique. With that fixed the baseline reproduces the reported symptom exactly rather than something worse: the button gets 4 shapes, all grey; the navbar gets 10 shapes and 6 real texts with a single distinct fill colour. Production does create structure and text. What it never manages is to apply colour - it creates the shapes in a call that works, then sets fills in a later call using Figma syntax, that call throws, and the shapes keep the #B1B2B5 default. So "creates grey boxes" was accurate and "creates nothing" was my measurement error. Two metrics now capture that directly, since neither distinct-colour counts nor placeholder-grey counts see it - #FFFFFF and #000000 are not mid greys, so a design where every fill is a default passes both. explicitFillShare is the share of filled shapes whose colour is not one of Penpot's three defaults, thresholded by difficulty (0.80 high, 0.65 medium, 0.50 low, since a small artefact's structural neutrals weigh heavily in a ratio). And onlyDefaultColors is a veto: true when the whole palette is those three. White and black stay legitimate when chosen - the distinction is co-presence, not the hex. They are only suspicious when they are all there is; alongside chosen brand colours they also count as neutrals in paletteStructured. resumen() crashed adding None scores from failed prompts. Beyond the crash, an unmeasured prompt must not average in as a zero: "could not measure" and "the model did it badly" are different, and averaging them would have made a mid-run plugin outage look like cheap quality. They are reported separately and any missing prompt makes the verdict invalid, because a baseline with 5 of 10 measured is not a baseline. The gate now empties its own page after exporting the PNG. Across three runs the plugin reliably handled 5 or 6 heavy prompts and then degraded to 30-second timeouts on createPage - that is not random flakiness, it tracks the file growing by one page per prompt, so the gate was manufacturing its own failure. The evidence that matters is the PNG plus the JSON metrics, not the live page. A page is kept only when its PNG failed, so there is something to inspect. Exit code 3 now distinguishes "plugin degraded mid-run, reload it and resume these ids" from "plugin not connected". |
||
|
aleleba
|
8094183940
|
Phase 6.3.17: fix a harness fidelity bug and add a vibrancy metric
The first baseline measured production with only 2290 of the 16392 characters of the server's instructions block - 14%. The missing 86% is exactly the API grounding: Core Shape Properties and Methods, Layout Systems, Text Elements, and The penpot and penpotUtils Objects, which is where insertChild, resize(), the layouts and penpotUtils are documented. That was worth catching, because the discrepancy had a visible signature: the measurement said production creates nothing, while the user's real Claude Code session produced grey boxes, i.e. shapes greater than zero. When a harness and reality disagree, the harness is the first suspect. In phase 5 a low max_tokens manufactured an apparent regression the same way. The gate now injects the full document, minus the trailing "You have hereby read the Penpot High-Level Overview" line, which is framing of the tool response rather than part of the instructions block and would otherwise tell the model it had already read something. The finding survives the fix. Across the five prompts measured cleanly under the corrected condition, shapeCount is still zero on every one. So the API invention is not an artefact of withholding documentation from the model - it happens with the documentation present. Also adds the vibrancy requirement the user raised as first-class scope: given an ambiguous brief the model must choose and justify a palette rather than ask or fall back to defaults. Neither distinctFillColors nor placeholderGreys distinguishes a vibrant palette from a muted but technically non-grey one, so four metrics are added: chromaticFills, meanChromaticSaturation, paletteStructured (a dominant brand hue, an accent at least 30 degrees away, and neutrals), and finalMessageListsHex, because a palette chosen in silence cannot be adjusted by the user. The saturation floor of 45 is derived, not asserted: measured over the 325 non-neutral fills of this phase's hand-authored corpus, median HSL saturation is 75, p25 is 48 and p10 is 35. A floor of 45 sits just under the first quartile and is cleared by 79% of those fills, so it is a floor the target behaviour already clears rather than an aspiration. The lightness band of 15 to 85 excludes near-blacks and near-whites, which can compute as highly saturated while reading as neutral. Gate prompt 6 becomes the user's literal failing sentence, and two ambiguous-brief prompts are added. One of them had to be re-domained after the disjointness check found it shared a 6-gram with a seed - the check fails on a single shared shingle, which is what makes it useful. Results so far are partial: prompts 1-5 measured cleanly, 6 has a timed-out audit and 7-10 hit the MCP outage, so those get re-measured. Both runs are kept, the 14% one renamed to record what it was. |
||
|
aleleba
|
5f0ddd962c
|
Phase 6.3.17: measure the gate 5 baseline against production
Ran the full agent loop against vllm-qwen36 on port 8000 (read-only HTTP)
with the Penpot plugin live, before asking for any downtime. Without this
file "it improved" would be a claim rather than a measurement.
Result over the 8 graded prompts: mean score 14.9, zero prompts at or above
60, veto violated on 2 of 8, 66% of execute_code calls raised, and 7 of 8
prompts burned all 14 turns without producing a final message.
The plan predicted production would score near zero on distinct colours and
style richness while producing a high shape count - grey boxes. The shape
count is also zero. On a fresh page it creates nothing at all, so the
failure sits upstream of the grey boxes: the model invents a Figma-shaped
API wholesale and every call throws. From the captured turns:
penpot.currentPage() is a property, not a function
penpot.createRectangle(page, 200, 56) takes no arguments
penpot.createText(page, ...) takes one, the text
penpot.getPageById(...) lives on penpotUtils
fills = [{type:'solid', color:{r,g,b,a}}] is {fillColor, fillOpacity}
shadows = [{type:'drop', x, y, blur, ...}] is {style, offsetX, offsetY}
It then spends the remaining turns querying penpot_api_info without
recovering. So the reported symptom understated it.
Two robustness fixes the run itself forced, both after losing a completed
run to them:
- A ConnectionError does not just drop the request, it can drop the MCP
session, so retrying the same tools/call against a dead session fails
identically every time - which is exactly what the first attempt showed,
four retries and four identical ConnectionErrors. The client now redoes
the handshake before retrying, and that recovered two drops in this run.
- Results are written after every prompt. The first attempt died on prompt
4 and lost the three already measured, which is the expensive data
precisely because it requires production to be up.
|
||
|
aleleba
|
c9792c5c40
|
Phase 6.3: rewrite the Penpot seed corpus and build the LoRA #2 mix
Replaces the 41 old Penpot seeds with 105 new ones. The old set taught three API forms that do not exist - findShapeById(page, id), shape.layout, and createText() with no argument - and 21 of the 41 used the first one, so patching was never an option: keeping them would mean fine-tuning against the correction. The valuable lessons were re-founded on the real API instead (the reversed children array in flex, persisting ids in storage, never logging what you also return). 105 rather than 96 because nine multi-section compositions are split into two trajectories each: the first builds the skeleton and persists ids, palette, scale and helper functions in storage, the second recovers them and fills the sections. That was forced by the 3000-token ceiling, but it is better pedagogy anyway, and it is what execute_code's own description asks for. It also paid for itself: the helpers cost ~600 chars once instead of twice, and the skeleton call needs no export_shape, which freed the budget to carry the verbatim system block. Measured against the linter with the corpus-wide thresholds active: 143 unique code payloads (the old set had 36), 32% of seeds carrying the server's system block verbatim (the old set had none), and every coverage category met where the old set had zero addGridLayout, zero shadows, zero uploadMediaUrl and zero layoutChild. The flagship changed domain from pizzeria to an empanada shop. Gate 5's prompt 6 is the exact production failure, and the seed had been written to the same wording - a literal substring of the gate prompt, sharing two 6-gram shingles where the gate's disjointness check fails on one. Training on the prompt we then evaluate would make gate 5 measure memorisation. The real pizzeria prompt still runs in the human acceptance test. The same check also caught an onboarding seed sitting too close to gate prompt 7. Mix: 1000 examples split 90/10, giving 901 train and 99 eval. The mix is 1000 rather than 900 because 900 has to be the post-split train size: 900/16 = 56.25 steps per epoch x 3 = 168 steps, deliberately paired with phase 3's 166 so the optimiser trajectory length is comparable. Building 900 and then carving out eval would have left 811 and 152 steps, silently breaking that pairing. Two guards in the builder had to be corrected against the real corpus: - The forbidden-pattern scan now reads only `code` payloads. Scanning the serialised example flagged the corrective seeds themselves - the one that opens with the user asking "Importá esta imagen con import_image", the one quoting the overview line that mentions import_image verbatim, the one explaining that board.layout does not exist. They name the wrong API precisely in order to teach against it. - The exemption for error-recovery seeds is derived from content, not from the mix portion: a forbidden pattern is allowed only where its tool result is a real error string and a later payload does the same thing without it. Keying on the portion broke as soon as an error-recovery seed lived in group A1, where the findShapeById arity seed naturally belongs. Validated with the production tokenizer at preserve_thinking=true: p50 2466, p90 2988, max 3250 tokens over the 105 seeds. Ten sit just above 3000, so MAX_TOKENS goes to 3300 for the run and the smoke run will train on the 32 longest examples specifically, turning the peak-memory question into a ten-minute measurement instead of a risk discovered hours in. |
||
|
aleleba
|
c65d309719
|
Phase 6.4: make the gates fail when they cannot verify something
A code review found seven ways these gates could pass green with something actually wrong. All are the same family: a missing value was treated as OK. The rule now written into all three files is that absent is not OK, absent is "could not verify", and that either fails or is reported as an explicit SKIP - it never slips through as green. 30_eval_suite.py: - A bucket with no baseline of its own fell back to the global 0.2750 and printed it in a column headed "baseline", as if it were that bucket's number. Measured against the real eval.jsonl buckets: negativos going from 0.12 to 0.33 is a real +0.21 regression, but the computed delta was +0.055 and it PASSED; manejo_errores sitting unchanged at 0.42 produced a fabricated +0.145 FAIL that would have discarded a healthy candidate mid-downtime. Now such buckets print SKIP and the verdict reports how many went unverified. - "VEREDICTO: FAIL" exited 0, so a runbook chaining the gate into quantization would have carried on to write 24 GB. Now exits 1. - A typo in BASELINE_BUCKET_LOSSES silently matched nothing; now aborts. - The penpot exemption is labelled honestly: those 11 rows are pre-existing LoRA #1 tool-calling, not new capability, so gate 1 has no regression coverage there and the log says so. 20_merge_lora.py dry-run (merge path untouched, verified by AST diff): - adapter_config.get("use_rslora", False) meant a missing key passed AND the log printed use_rslora=False, asserting it had checked something that was never there. A different PEFT version omitting a key was enough. - lora_bias was not checked at all, only bias. They are different fields: lora_bias puts a bias inside lora_B, which W + scaling * (B @ A) ignores. - The 620 keys were printed but never asserted, so an adapter with extra tensors printed "310 + 310 = 930" and passed. - rank_pattern/alpha_pattern were not checked. They set r per module, so scaling is not uniformly alpha/r while both the dry-run and the merge apply a single 2.0 to all 310 tensors. - A missing family was invisible: swap linear_attn for 150 mlp.gate targets and the total is still 310, no norm is zero because the family is simply gone, and it passed. Now presence and per-family counts are asserted, derived from the real adapter: linear_attn 150, shared_expert 120, attention_qkvo 40, otros 0. Verified against seven synthetic adapters plus the real phase 3 one; only the correct adapter passes. 21_quantize_nvfp4.py (recipe and oneshot untouched): the calibration cache now carries a provenance.json recording the training file's sha256, the recipe numbers and the bucket distribution, and loading aborts on mismatch. This is the phase's number one risk and it had no mechanical defence: the phase 5 cache on disk has exactly 512 rows, the same as the v2 recipe, so the only existing check could not tell them apart and reusing it would have calibrated with zero design data and washed out the new capability silently. Verified: that cache now aborts. gate 5: retry transport failures against the Penpot MCP, which drops connections mid-call intermittently (seen before in phase 4's gate 4). Without it a blip on prompt 6 of 8 kills a whole run and reads like a model failure. PluginNotConnected is deliberately not retried - that is a real state of the world. Also unwrap the {"result":..., "log":...} envelope the server wraps execute_code returns in; the gate was reading keys off the outer object and rejecting a valid page setup. |
||
|
aleleba
|
8ea4572edd
|
Phase 6.3: add gate 5, design quality in Penpot
Unlike gates 2-4 this one needs a real agent loop - model, tool call, live
MCP, result, up to 14 turns - because design quality only exists after the
code executes. It talks to vLLM over the OpenAI API and to the Penpot MCP
over HTTP (initialize, notifications/initialized, tools/list, tools/call),
handling both application/json and text/event-stream responses. Endpoints
and credentials come from env with no defaults and are never printed or
stored; requests errors are reduced to the exception type because the
requests message embeds the URL.
Eight graded prompts, a fresh page per prompt named gate5/<tag>/<id>/<ts>,
and the gate never deletes anything. The 17 metrics are computed by an
audit payload the gate injects, not the model. placeholderGreys and
forbidden behaviour are veto metrics: any hit scores that prompt 0.
Two things worth calling out.
The forbidden-pattern regexes are imported from 07_lint_penpot_code.py
rather than duplicated, and the gate runs those same regexes over its own
setup and audit payloads at startup - a gate that violated the API it is
grading would be measuring its own bug.
The holdout mode had a silent failure that is exactly the kind this phase
exists to catch: with the endpoint down, all 60 generations failed, each
entered the denominator with zero forbidden patterns found, and the gate
reported 0% forbidden API and APPROVED. Since that number is the fallback
trigger, a false pass there would have launched the quantization run.
Request errors are now counted separately, never enter the denominator,
and block approval outright.
Known issue, resolved separately: gate prompt 6 is the exact production
failure ("hazme una landing page de una pizzería con colores vibrantes"),
and the flagship B6 seed was written to the same wording. Shingle overlap
measures 20%, under the 34% threshold, but the seed prompt is a literal
substring of the gate prompt - the threshold is too loose for prompts this
short. The seed's domain gets changed rather than the gate's, so gate 5
measures transfer instead of memorisation; the real pizzeria prompt still
runs in the human acceptance test, which is the criterion that decides.
|
||
|
aleleba
|
a60d0751cf
|
Phase 6.3: fix the augmentation, close the holdout leak, stop rewarding invented parameters
Dataset build (05, 06): perturb_value is gone. It rewrote only tool_calls.arguments and left the tool results and the final answer saying something else, which is how data/train.jsonl ended up with 30 self-contradictory examples where the call says issue_number 82 and the answer says issue #77. Variation now comes from hand-written meta.paraphrases, or from meta.variation applied atomically across every field of the example at once. Nothing is substituted unless the seed declares it: guessing which number in a string is safe to change is what produced the contradictions in the first place. Prefix injection survives only as a fallback and only where the verb form can actually be conjugated, and there is a hard assert that no user turn matches the broken "Necesito que ¿Podés..." shape that 68 v1 prompts had. The penpot bucket is exempt from substitution entirely, since its payloads are code. Also asserts the bucket cannot collapse (verified: the old seeds give 320 rows from 83 unique trajectories and the build now fails) and scans for forbidden API patterns by importing them from the linter, so there is one source of truth. 06 now actually exits 1 on over-length rows. It printed [FILTERED], incremented a counter, and left the row in the file, which 10_train.py then trained on since it has no max_seq_length and batch 1. Gate 2 (32): reject any argument key absent from the schema, as its own failure category. It only checked required fields, so an invented scale or filePath passed - the gate was actively rewarding the exact behaviour this phase removes. Verified: export_shape with scale=2 now fails as unknown_argument, while a valid call still passes. Holdout (31, 35): rebalanced to penpot 60 / 35 each, added 20 real design templates, and replaced the full-string equality check with 6-gram shingles. Measured: a light paraphrase of a train.jsonl prompt scores 43% overlap and now fails the build, where the old check let it through at "not equal". Value pools are asserted disjoint from the corpus. The "2x resolution" template stays, relabelled as an invented-argument probe now that gate 2 can detect one; the createBoolean template stays because the API is real and the new B2 seeds teach it. Also dedupes: the old holdout had 15 duplicate prompts out of 200, i.e. 15 wasted measurements. Note: rebalancing the holdout means the 192/200 gate 2 baseline from phase 5 no longer applies to it, so that baseline has to be re-measured against production on the new file before it can be compared to. Gate 3 (33): 11 content checklists for the non-obvious conventions of the other MCPs - GFM table separators in Docmost, the update_page staleness retry, commit message shape, never merging the PR, dict-not-XML tool arguments. That is the most likely regression no gate currently covers. Mix builder: added the anti-collapse guard, so 420 new-portion rows that are really 96 trajectories repeated cannot pass unnoticed. |
||
|
aleleba
|
19eb50f351
|
Phase 6.3: add the LoRA #2 mix builder and refine the linter
07_build_lora2_mix.py assembles train_lora2.jsonl (900), eval_lora2.jsonl and calibration_v2.jsonl from the new Penpot seeds plus a filtered replay sample of data/train.jsonl. It never writes data/train.jsonl or data/eval.jsonl: those are the provenance of the model in production and the gate 1 baseline, and regenerating them is not idempotent anyway, since stratified_split shuffles one RNG over the concatenated list, so touching the penpot bucket reshuffles every other bucket's split too. Two things worth flagging in the mix: The 45 "corrected penpot basics" the plan lists inside the replay portion do not come from data/train.jsonl. 21 of its 41 penpot seeds teach findShapeById(page, id) and 5 use shape.layout, so sampling that bucket would re-teach the exact bug this phase removes; the forbidden-pattern filter would drop them anyway. They come from the new corpus instead. This is a conscious deviation from the plan text and is recorded in the docstring. Variation comes only from hand-written meta.paraphrases, never from automatic value substitution. That is the v1 lesson: perturb_value rewrote only tool_calls.arguments and left the tool results and the final answer saying something else, producing 30 self-contradictory examples. A perturbed Penpot code payload is just broken code. Linter fixes, both false positives found by running it against the real corpus: - flex evidence for a bare appendChild is now scoped to the whole seed rather than the single payload. A multi-call seed builds the flex board in call one and stashes helpers in storage, so by the time call two does main.appendChild(...) neither addFlexLayout( nor .flex appears in that payload. The old scope flagged exactly the storage-persistence pattern that execute_code's own description asks for. - a grey hex is a problem when it is applied, not when it is searched for. The repair seeds have to name the greys they are about to replace, so greys are allowed in that group inside a comparison context. |
||
|
aleleba
|
63da20c031
|
Phase 6.2: pin the training environment and make 10_train.py configurable
The training container from phases 3-5 no longer exists and nothing in the repo pinned its versions, so a rebuild could silently change either the checkpoint key conversion (breaking adapter naming) or the assistant-mask behaviour (training on system/user/tool tokens). requirements.train.txt pins what matters and documents the two-phase install: llmcompressor declares torch>=2.10.0 and the NGC image ships the 2.10.0a0 pre-release, which pip's resolver reads as older, so it goes in with --no-deps. Pre-flight verified against the merged bf16 checkpoint on spark: 01_inspect_modules.py prints model.layers.0.linear_attn.*, config.json is sha256-identical to the base (93a4693f...), and the index keysets match exactly (1045 tensors, 690 under model.language_model.layers.*, 0 under model.layers.*). So PEFT will name adapter #2 the same way it named #1 and ADAPTER_TO_CHECKPOINT_PREFIX in 20_merge_lora.py applies unchanged. 10_train.py: every path and hyperparameter moves to an env var, with the phase 3 values as defaults so a bare run still reproduces phase 3 exactly. Adds three guards that each cover a specific silent failure: - abort if OUTPUT_DIR already holds an adapter, unless ALLOW_OVERWRITE=1. OUTPUT_DIR was hardcoded to out/lora-adapter, which is the provenance of the model currently in production. - MAX_TOKENS aborts rather than truncates. There was no length filter at all, so one long design trajectory would blow the memory budget hours into a run; truncating would be worse, since it would silently cut assistant targets. - assert use_rslora/use_dora/bias/modules_to_save. rsLoRA scales by alpha/sqrt(r), so an adapter trained with it would merge at 2.0 where 11.3 belongs and pass every assertion in the merge script. Also adds a config banner, a token-length histogram, and a per-bucket assistant-mask ratio report. New 07_lint_penpot_code.py hard-fails on the forbidden API patterns, placeholder greys, fabricated penpot_api_info results, toy-shaped ids and per-category coverage shortfalls. Error-recovery seeds legitimately need the wrong pattern, so the exemption is derived mechanically rather than declared by hand: a payload may contain a forbidden pattern only if its tool result is a real error string from the allow-list and a later payload in the same seed does the same thing without it. Run against the 41 existing seeds it reproduces the diagnosis exactly: 110 problems, 36 unique payloads, 0% system messages, zero coverage of addGridLayout/shadows/uploadMediaUrl/layoutChild, fabricated docs and toy ids. |
||
|
aleleba
|
d9629c44ae
|
Phase 6.1: capture verified Penpot API ground truth for the LoRA #2 dataset
The 41 existing Penpot seeds contain hand-fabricated penpot_api_info and high_level_overview tool results that assert facts the server never said, which is how the model learned an API that does not exist. This adds four schema files that make the seed corpus mechanically verifiable against the real server instead. - penpot_api_docs.md: 34 verbatim captures of high_level_overview and penpot_api_info, each headed by the exact request that produced it. Every penpot_api_info tool result in a seed must be a subset of lines of this file, in original order. Records three places where the served docs contradict the runtime (addFlexLayout/addGridLayout copy-paste in the Grid section, flex.appendChild for grid children, withChildren vs includeChildren), plus the createText() example that is the direct cause of the production failure. - penpot_system_prompt.md: the server's instructions block verbatim. Goes as a system message into ~30% of the new seeds; it is the countermeasure to the "don't pick your own colours" rule that produces the grey boxes. - penpot_errors.md: the real error strings, including a section on silent failures that raise nothing at all and are why the read-back invariant exists. - PENPOT_API_VERIFIED.md: the allow-list. No seed may reference a member absent from it. Documents the four root causes (findShapeById arity 1, no shape.layout, createText() returning null, the #B1B2B5 default fill), the twelve anti-grey-box invariants, and the forbidden-pattern list the linter checks. Live re-verification of the error strings is still pending: the Penpot plugin is not currently connected, so it is deferred to the gate 5 baseline step, which needs the live connection anyway. |