New offline script re-scans the code payloads already saved in the gate 5
result JSONs (holdout, full, generalization) using the corrected forbidden-
behavior list -- no live Penpot needed, reuses scan_forbidden/score_prompt
from 35_gate5_penpot_design.py as the single source of truth.
Holdout result changes materially: 3 of the 5 flagged prompts (pd-008,
pd-009, pd-035) were flagged only for flex.appendChild. With that pattern
removed, forbidden-API rate drops from 8.3% (5/60) to 3.3% (2/60), which is
now under the <=5% fallback threshold -- the holdout gate now passes.
Full mode and the generalization probe are unaffected: neither run's
transcripts ever triggered the flex.appendChild pattern, so their scores
are identical before and after.
Original result files are kept untouched; corrected results are written
alongside as *-corrected.json.
Live probing against Penpot 2.16.2 (repeated, on a dedicated scratch page,
cleaned up afterwards) shows board.flex.appendChild(shape) is a real
function, distinct from board.appendChild, arity 1, throws nothing, and
preserves visual insertion order -- unlike board.appendChild, which still
inserts at the front as documented. The prior "broken" claim traces to the
MCP server's own high_level_overview() text (CRITICAL / BROKEN / NEVER use),
never verified with a live throw, and possibly confused with a real doc bug
that mislabels the grid 3-arg example as board.flex.
Removes flex.appendChild from the lint's FORBIDDEN patterns and from the
gate 5 forbidden-behavior veto, updates PENPOT_API_VERIFIED.md with the
verified facts, and drops the now-obsolete pattern from the replay filters
in 05_build_dataset.py and 07_build_lora2_mix.py.
4 prompts in domains absent from both seeds and the official gate5_prompts.jsonl (ferreteria,
gimnasio, tienda de mascotas, panaderia). score medio 32.0, veto limpio 3/4, 0/4 prompts >=60.
gp-03-tienda-mascotas violates the veto twice over: it hits both import_image variants the
model has ever tried (penpot.importImage on turn 1, penpotUtils.importImage on turn 2) plus a
numeric fontSize assignment -- the uploadMediaUrl reinforcement seeds from this correction
round did not transfer to an unseen domain. It is also flagged via GATE5_KNOWN_OVERLAP_IDS
(shares a 6-gram with the same seeds as g5-03), so its score is a double caution case.
gp-02-gimnasio hit the same transient Penpot MCP transport drop as g5-06/g5-07 on its first
pass, but the harness's own automatic retry (pass 2/3) recovered cleanly within the default
240s -- no manual timeout override was needed here.
10/10 prompts measured. VEREDICTO: NO APRUEBA (score medio 39.8 vs threshold 65, veto limpio
8/10, 3/10 prompts >=60). The flagship g5-06-landing-pizzeria now clears its own threshold
(69.6 >= 60) once a transient Penpot MCP transport drop is ruled out (see g5-06/g5-07 rerun
below) -- a real improvement over v2-bf16's 34.8 on that prompt, but the aggregate gate
criteria still fail. g5-03-card-producto is flagged via GATE5_KNOWN_OVERLAP_IDS: it shares a
6-gram with seeds 134-137 already baked into v2b-bf16's training, so its result should be
read with caution rather than as a clean generalization signal.
g5-06/g5-07 were re-measured once with GATE5_ONLY + GATE5_PROMPT_TIMEOUT_S=420 (default 240s
untouched) after their first pass hit a genuine Penpot MCP transport drop (ConnectionError +
handshake rebuild) that ate into the turn budget and caused a false-negative timeout. g5-04
and g5-08 show the same 'no board created' signature but their raw turn-by-turn transcripts
have zero MCP/transport/handshake evidence -- both are the known post-error tool-call-format
bug (literal '<tool_call>' text instead of a structured call), so they are left as genuine
failures with no rerun.
Tasa de API prohibida: 8.3% (5/60), baja de 15% (9/60) en v2-bf16, pero
sigue por encima del umbral de 5% -- NO APRUEBA.
Desglose por patron (v2-bf16 -> v2b-bf16):
- import_image/importImage/createImage/filePath: 4 -> 1 (-75%, el
fix que apuntaban los seeds de uploadMediaUrl)
- fontSize/fontWeight/lineHeight/letterSpacing numerico: 3 -> 1 (-67%)
- flex.appendChild: 2 -> 3 (empeoro levemente; NO fue el foco de
esta ronda de seeds correctivos, es el patron ahora dominante:
3 de los 5 vetos restantes)
Se continua con el resto del runbook (verificacion de Penpot, puerta 5
full, sonda de generalizacion) para tener el cuadro completo antes de
reportar, tal como indico el usuario -- pero este resultado ya senala
que el disparador de fallback documentado en PLAN.md (<=5%) se activa
de nuevo, esta vez por un patron distinto y no cubierto por los seeds
de esta iteracion.
Global 190/200 (95.0%), penpot 59/60 (98.3%) vs baseline v2-bf16's
penpot 60/60 (100%) -- 1.67pt regression, within the 2pt guard from
the gate5-full regression-guard rule. PASS.
GATE5_PROMPTS_PATH env var lets the same 17-metric harness (veto system,
MCP retry/page handling) run against a different prompts file. Unset
behavior is unchanged -- the official gate 5 (data/gate5_prompts.jsonl,
10 frozen prompts) stays untouched.
data/gate5_generalization_probe.jsonl: 4 prompts in rubros confirmed
absent from the 138-seed corpus, the 10 gate5 prompts, and the 60-prompt
holdout -- ferreteria, gimnasio, tienda de mascotas, panaderia. Swapped
out "veterinaria" (originally planned) after finding seed 83 is a
substantial design-repair seed for a vet clinic; "tienda de mascotas"
verified clean instead. This is an additional check, separate from the
official gate 5, meant to test whether the correction generalizes across
business types rather than overfitting to the pizzeria case.
Global weighted loss 0.2725 (vs baseline 0.2750, previous v2-bf16 was
0.2734). All 6/6 non-penpot buckets pass the 0.10 regression threshold,
including delegacion_subagentes which had narrowly failed on v2-bf16
(+0.1019 -> now +0.0882). The 13 corrective seeds didn't hurt forgetting.
Also points vllm-eval at the new v2b-bf16 checkpoint for the gate 2/5
re-measurement.
Diagnosis (audited gate5 full-mode transcripts, PNGs, and holdout payloads):
the dominant failure mode was NOT rubro/content coverage but a tool-call
formatting bug -- after diagnosing an exception, the model often writes its
retry as literal <tool_call> XML text embedded in reasoning instead of a
structured call, so it's silently dropped (9/10 gate5 full-mode prompts hit
this on their final captured turn). Reasoning length before a tool call
correlates with the failure (malformed-turn mean 1895 chars vs 364 for
well-formed turns), while the training mix's own tool-calling turns never
exceed 1630 chars and export_shape turns never exceed 485.
Corrective seeds (group D, grupo 'D', short 1-3 sentence diagnosis +
immediate well-formed retry, to avoid reinforcing long-reasoning risk):
- 9 seeds across fresh rubros (lavanderia, zapateria, optica, jugueteria,
papeleria, peluqueria, floreria, heladeria, cerrajeria) covering distinct
real API errors: .color on Text, addFlexLayout on non-Board, findShapeById
2-arg, createText() no-arg, textAlign, typography.setFont, createBoolean
null, appendChild no-arg, uploadMediaUrl network rejection.
- 4 seeds reinforcing uploadMediaUrl over the exact penpot.importImage
variant the model actually invents (distinct from the one existing seed,
which only taught the penpotUtils.importImage variant) -- rubros:
relojeria, guarderia infantil, agencia de viajes, kiosco.
Deliberately avoid ferreteria/gimnasio/veterinaria/panaderia: reserved for
the held-out generalization probe (separate from the frozen gate 5).
Added the newly-verified error strings (t.color, penpot.importImage,
addFlexLayout-on-non-Board) to penpot_errors.md -- all captured live from
this phase's own gate 5 full-mode run, not fabricated.
Rebuilt train_lora2.jsonl (901)/eval_lora2.jsonl (99)/calibration_v2.jsonl
from the 138-seed corpus (07_build_lora2_mix.py) and validated
(06_validate_dataset.py, 0 exceptions, 0 secrets, max 3271 tokens).
10/10 prompts measured against the live Penpot plugin. Mean score
43.1 (threshold 65), pizzeria landing 34.8 (threshold 60), 4/10
prompts >=60, veto clean on 8/10 (card-producto forbidden-API,
brief-ambiguo-escuela placeholder-grey). Verdict: NO APRUEBA.
Combined with gate 1 (PASS global, 1 small bucket FAIL non-blocking),
gate 2 (PASS 96%), and gate 5 holdout (FAIL 15% forbidden API), this
closes the fallback-decision measurement set for step 6.4.25.
The 32 longest examples of the training mix, 3161 to 3271 tokens each. With
batch 1 and gradient accumulation 16, a two-step smoke run consumes exactly
these 32, so it trains on the worst case the real run will ever see.
Risk #5 in the plan is an OOM discovered hours into the run. Phase 3 peaked
around 104 GB of the 121 GB available with a longest example near 2800
tokens, and this corpus goes to 3271. Rather than argue about whether the
extra 17% of sequence length fits, the smoke run measures it in ten
minutes. If it OOMs it OOMs immediately and cheaply, which is the whole
point of running the probe before the 169-step run rather than after.
Rebalancing the holdout regenerated all 200 prompts, so phase 5's 192/200
stopped being comparable and every gate2/gate3/gate4 result file in the
repo was from phase 5. These had to be measured with production still up,
before asking for the downtime, or it would have cost the user another
window later just for this.
Gate 2 over the new holdout: 191/200, 95.5% global. atlassian 97.1,
docmost 94.3, gitea 85.7, github-personal 100, penpot 98.3. All five
failures are missing_required and there are zero invented arguments.
That penpot figure is worth keeping in view: production is at 98.3% on
tool-call VALIDITY, so its failure is not in the shape of the call but in
the code it puts inside execute_code. The two are measured by different
gates and only gate 5 sees the second.
Gate 3: 61.9%, which is 13 of 21. Not comparable to phase 5's 10/10, since
that run had 10 checks and this one has 21 - the 11 new content checklists
for the non-obvious conventions of the other MCPs are what risk #12 in the
plan called the most likely invisible regression. Production already fails
7 of them, so they have headroom rather than being a formality.
Gate 4 needs no re-baseline, verified rather than assumed: grep for
holdout_prompts in 34_gate4_e2e.py returns 0, so the regenerated holdout
does not reach it and its phase 5 result stands. It also needs all five
MCPs live, including the Penpot plugin, which closes when downtime starts.
Also trims the one seed that exceeded the token ceiling. The corpus now
validates clean against the production tokenizer with preserve_thinking on:
1000 of 1000 rows, zero exceptions, zero secrets, p50 1858, p90 3021,
max 3271 against a 3300 limit. That check earning its keep is the reason
06_validate_dataset.py was changed to exit non-zero instead of printing
[FILTERED] and moving on.
Ten prompts measured three times against vllm-qwen36, all with the same
23-metric harness. Three earlier result files are kept but excluded from
the consolidation and named for why: one used 14% of the server's
instructions block, two predate the audit-root fix and report zero shapes
by construction.
The consolidated numbers correct the previous report, which was wrong.
Production does not fail to draw. It creates structure and text - up to 40
shapes and 24 texts on the ambiguous-brief prompt - and then fails to
colour any of it. Distinct fill colours run 0 to 1 across all thirty
measurements.
Mean score per repetition: 6.79, 8.25, 0.00. No prompt reaches 60 in any
repetition.
Reporting the veto rate separately from the score turned out to matter more
than expected, so it is now the headline number alongside it. Of thirty
measurements, 21 violate the forbidden-API veto and 15 use nothing but
Penpot's three default colours; only 6 are veto-free. The score alone
collapses those into a zero that cannot distinguish "broke an API rule"
from "made an ugly design", and the two need different fixes.
That also explains why single measurements looked stable: the score is
pinned at zero by the veto, not by model consistency. All the real variance
sits in the prompts that do not veto, where scores swing 20 to 35 points
between identical runs. So the repetitions matter more for the trained
model, which should land in that non-veto regime, than for this baseline.
The eval window at the end has to budget for three repetitions on that side
too, or the comparison is asymmetric.
The setup returned penpot.root.id and the run aborted when it came back as
an empty string on the second prompt of a batch. penpot.root is the root of
the ACTIVE page, and after createPage plus openPage it need not have caught
up yet - a race the page-emptying code introduced, since that leaves the
emptied page active. The value was never useful anyway: every page shares
the same root id, so it identified nothing. Setup now returns only pageId,
which is unique and stable, and both the audit and the cleanup use it.
The diagnostic message is the second half of the same mistake. It printed
"PLUGIN DEGRADADO, ask the user to reload the browser" whenever any prompt
went unmeasured, regardless of why - so it said that for a failure that was
entirely the gate's own. A message that sends the user to reload their
browser when the bug is mine costs both of us time. It now classifies on
the error text: task timeouts and transport drops point at the plugin,
anything else points at the gate and says so explicitly.
Also closes two evaluation leaks the gate's own pre-flight caught, both in
the seeds rather than the gate prompts, since the prompts have to stay as a
real user would write them:
- A seed shared the 6-gram "la home de una escuela de" with gate prompt 9.
My first fix was overwritten by a subagent still writing the file, which
is why it reappeared.
- A seed used the same business as gate prompt 9 - a music school - without
sharing any 6-gram. Shingles cannot see that: two texts describe the same
business without sharing words. Training on the domain we then evaluate
inflates the result invisibly. So the gate now also checks that no seed
uses any of the gate's business nouns, listed explicitly.
The training mix was rebuilt: it had been assembled before the 20
ambiguous-brief seeds existed, so training on it would not have used the
corpus that was audited. The ambiguous-brief class gets its own mix portion
rather than being folded into design, because diluted across 76 design
seeds it would be at the mercy of a ratio, and that is the class the user
named as the main painpoint. 125 seeds, 168 unique payloads, 446 distinct
user prompts, 901 train and 99 eval.
The first baseline measured production with only 2290 of the 16392
characters of the server's instructions block - 14%. The missing 86% is
exactly the API grounding: Core Shape Properties and Methods, Layout
Systems, Text Elements, and The penpot and penpotUtils Objects, which is
where insertChild, resize(), the layouts and penpotUtils are documented.
That was worth catching, because the discrepancy had a visible signature:
the measurement said production creates nothing, while the user's real
Claude Code session produced grey boxes, i.e. shapes greater than zero.
When a harness and reality disagree, the harness is the first suspect. In
phase 5 a low max_tokens manufactured an apparent regression the same way.
The gate now injects the full document, minus the trailing "You have hereby
read the Penpot High-Level Overview" line, which is framing of the tool
response rather than part of the instructions block and would otherwise
tell the model it had already read something.
The finding survives the fix. Across the five prompts measured cleanly
under the corrected condition, shapeCount is still zero on every one. So
the API invention is not an artefact of withholding documentation from the
model - it happens with the documentation present.
Also adds the vibrancy requirement the user raised as first-class scope:
given an ambiguous brief the model must choose and justify a palette rather
than ask or fall back to defaults. Neither distinctFillColors nor
placeholderGreys distinguishes a vibrant palette from a muted but
technically non-grey one, so four metrics are added: chromaticFills,
meanChromaticSaturation, paletteStructured (a dominant brand hue, an accent
at least 30 degrees away, and neutrals), and finalMessageListsHex, because
a palette chosen in silence cannot be adjusted by the user.
The saturation floor of 45 is derived, not asserted: measured over the 325
non-neutral fills of this phase's hand-authored corpus, median HSL
saturation is 75, p25 is 48 and p10 is 35. A floor of 45 sits just under
the first quartile and is cleared by 79% of those fills, so it is a floor
the target behaviour already clears rather than an aspiration. The
lightness band of 15 to 85 excludes near-blacks and near-whites, which can
compute as highly saturated while reading as neutral.
Gate prompt 6 becomes the user's literal failing sentence, and two
ambiguous-brief prompts are added. One of them had to be re-domained after
the disjointness check found it shared a 6-gram with a seed - the check
fails on a single shared shingle, which is what makes it useful.
Results so far are partial: prompts 1-5 measured cleanly, 6 has a timed-out
audit and 7-10 hit the MCP outage, so those get re-measured. Both runs are
kept, the 14% one renamed to record what it was.
Ran the full agent loop against vllm-qwen36 on port 8000 (read-only HTTP)
with the Penpot plugin live, before asking for any downtime. Without this
file "it improved" would be a claim rather than a measurement.
Result over the 8 graded prompts: mean score 14.9, zero prompts at or above
60, veto violated on 2 of 8, 66% of execute_code calls raised, and 7 of 8
prompts burned all 14 turns without producing a final message.
The plan predicted production would score near zero on distinct colours and
style richness while producing a high shape count - grey boxes. The shape
count is also zero. On a fresh page it creates nothing at all, so the
failure sits upstream of the grey boxes: the model invents a Figma-shaped
API wholesale and every call throws. From the captured turns:
penpot.currentPage() is a property, not a function
penpot.createRectangle(page, 200, 56) takes no arguments
penpot.createText(page, ...) takes one, the text
penpot.getPageById(...) lives on penpotUtils
fills = [{type:'solid', color:{r,g,b,a}}] is {fillColor, fillOpacity}
shadows = [{type:'drop', x, y, blur, ...}] is {style, offsetX, offsetY}
It then spends the remaining turns querying penpot_api_info without
recovering. So the reported symptom understated it.
Two robustness fixes the run itself forced, both after losing a completed
run to them:
- A ConnectionError does not just drop the request, it can drop the MCP
session, so retrying the same tools/call against a dead session fails
identically every time - which is exactly what the first attempt showed,
four retries and four identical ConnectionErrors. The client now redoes
the handshake before retrying, and that recovered two drops in this run.
- Results are written after every prompt. The first attempt died on prompt
4 and lost the three already measured, which is the expensive data
precisely because it requires production to be up.
Replaces the 41 old Penpot seeds with 105 new ones. The old set taught
three API forms that do not exist - findShapeById(page, id), shape.layout,
and createText() with no argument - and 21 of the 41 used the first one, so
patching was never an option: keeping them would mean fine-tuning against
the correction. The valuable lessons were re-founded on the real API
instead (the reversed children array in flex, persisting ids in storage,
never logging what you also return).
105 rather than 96 because nine multi-section compositions are split into
two trajectories each: the first builds the skeleton and persists ids,
palette, scale and helper functions in storage, the second recovers them
and fills the sections. That was forced by the 3000-token ceiling, but it
is better pedagogy anyway, and it is what execute_code's own description
asks for. It also paid for itself: the helpers cost ~600 chars once instead
of twice, and the skeleton call needs no export_shape, which freed the
budget to carry the verbatim system block.
Measured against the linter with the corpus-wide thresholds active:
143 unique code payloads (the old set had 36), 32% of seeds carrying the
server's system block verbatim (the old set had none), and every coverage
category met where the old set had zero addGridLayout, zero shadows, zero
uploadMediaUrl and zero layoutChild.
The flagship changed domain from pizzeria to an empanada shop. Gate 5's
prompt 6 is the exact production failure, and the seed had been written to
the same wording - a literal substring of the gate prompt, sharing two
6-gram shingles where the gate's disjointness check fails on one. Training
on the prompt we then evaluate would make gate 5 measure memorisation. The
real pizzeria prompt still runs in the human acceptance test. The same
check also caught an onboarding seed sitting too close to gate prompt 7.
Mix: 1000 examples split 90/10, giving 901 train and 99 eval. The mix is
1000 rather than 900 because 900 has to be the post-split train size: 900/16
= 56.25 steps per epoch x 3 = 168 steps, deliberately paired with phase 3's
166 so the optimiser trajectory length is comparable. Building 900 and then
carving out eval would have left 811 and 152 steps, silently breaking that
pairing.
Two guards in the builder had to be corrected against the real corpus:
- The forbidden-pattern scan now reads only `code` payloads. Scanning the
serialised example flagged the corrective seeds themselves - the one that
opens with the user asking "Importá esta imagen con import_image", the one
quoting the overview line that mentions import_image verbatim, the one
explaining that board.layout does not exist. They name the wrong API
precisely in order to teach against it.
- The exemption for error-recovery seeds is derived from content, not from
the mix portion: a forbidden pattern is allowed only where its tool result
is a real error string and a later payload does the same thing without it.
Keying on the portion broke as soon as an error-recovery seed lived in
group A1, where the findShapeById arity seed naturally belongs.
Validated with the production tokenizer at preserve_thinking=true: p50 2466,
p90 2988, max 3250 tokens over the 105 seeds. Ten sit just above 3000, so
MAX_TOKENS goes to 3300 for the run and the smoke run will train on the 32
longest examples specifically, turning the peak-memory question into a
ten-minute measurement instead of a risk discovered hours in.
Unlike gates 2-4 this one needs a real agent loop - model, tool call, live
MCP, result, up to 14 turns - because design quality only exists after the
code executes. It talks to vLLM over the OpenAI API and to the Penpot MCP
over HTTP (initialize, notifications/initialized, tools/list, tools/call),
handling both application/json and text/event-stream responses. Endpoints
and credentials come from env with no defaults and are never printed or
stored; requests errors are reduced to the exception type because the
requests message embeds the URL.
Eight graded prompts, a fresh page per prompt named gate5/<tag>/<id>/<ts>,
and the gate never deletes anything. The 17 metrics are computed by an
audit payload the gate injects, not the model. placeholderGreys and
forbidden behaviour are veto metrics: any hit scores that prompt 0.
Two things worth calling out.
The forbidden-pattern regexes are imported from 07_lint_penpot_code.py
rather than duplicated, and the gate runs those same regexes over its own
setup and audit payloads at startup - a gate that violated the API it is
grading would be measuring its own bug.
The holdout mode had a silent failure that is exactly the kind this phase
exists to catch: with the endpoint down, all 60 generations failed, each
entered the denominator with zero forbidden patterns found, and the gate
reported 0% forbidden API and APPROVED. Since that number is the fallback
trigger, a false pass there would have launched the quantization run.
Request errors are now counted separately, never enter the denominator,
and block approval outright.
Known issue, resolved separately: gate prompt 6 is the exact production
failure ("hazme una landing page de una pizzería con colores vibrantes"),
and the flagship B6 seed was written to the same wording. Shingle overlap
measures 20%, under the 34% threshold, but the seed prompt is a literal
substring of the gate prompt - the threshold is too loose for prompts this
short. The seed's domain gets changed rather than the gate's, so gate 5
measures transfer instead of memorisation; the real pizzeria prompt still
runs in the human acceptance test, which is the criterion that decides.
Dataset build (05, 06):
perturb_value is gone. It rewrote only tool_calls.arguments and left the
tool results and the final answer saying something else, which is how
data/train.jsonl ended up with 30 self-contradictory examples where the
call says issue_number 82 and the answer says issue #77. Variation now
comes from hand-written meta.paraphrases, or from meta.variation applied
atomically across every field of the example at once. Nothing is
substituted unless the seed declares it: guessing which number in a string
is safe to change is what produced the contradictions in the first place.
Prefix injection survives only as a fallback and only where the verb form
can actually be conjugated, and there is a hard assert that no user turn
matches the broken "Necesito que ¿Podés..." shape that 68 v1 prompts had.
The penpot bucket is exempt from substitution entirely, since its payloads
are code. Also asserts the bucket cannot collapse (verified: the old seeds
give 320 rows from 83 unique trajectories and the build now fails) and
scans for forbidden API patterns by importing them from the linter, so
there is one source of truth.
06 now actually exits 1 on over-length rows. It printed [FILTERED],
incremented a counter, and left the row in the file, which 10_train.py
then trained on since it has no max_seq_length and batch 1.
Gate 2 (32): reject any argument key absent from the schema, as its own
failure category. It only checked required fields, so an invented scale or
filePath passed - the gate was actively rewarding the exact behaviour this
phase removes. Verified: export_shape with scale=2 now fails as
unknown_argument, while a valid call still passes.
Holdout (31, 35): rebalanced to penpot 60 / 35 each, added 20 real design
templates, and replaced the full-string equality check with 6-gram
shingles. Measured: a light paraphrase of a train.jsonl prompt scores 43%
overlap and now fails the build, where the old check let it through at
"not equal". Value pools are asserted disjoint from the corpus. The
"2x resolution" template stays, relabelled as an invented-argument probe
now that gate 2 can detect one; the createBoolean template stays because
the API is real and the new B2 seeds teach it. Also dedupes: the old
holdout had 15 duplicate prompts out of 200, i.e. 15 wasted measurements.
Note: rebalancing the holdout means the 192/200 gate 2 baseline from phase
5 no longer applies to it, so that baseline has to be re-measured against
production on the new file before it can be compared to.
Gate 3 (33): 11 content checklists for the non-obvious conventions of the
other MCPs - GFM table separators in Docmost, the update_page staleness
retry, commit message shape, never merging the PR, dict-not-XML tool
arguments. That is the most likely regression no gate currently covers.
Mix builder: added the anti-collapse guard, so 420 new-portion rows that
are really 96 trajectories repeated cannot pass unnoticed.
The 41 existing Penpot seeds contain hand-fabricated penpot_api_info and
high_level_overview tool results that assert facts the server never said,
which is how the model learned an API that does not exist. This adds four
schema files that make the seed corpus mechanically verifiable against the
real server instead.
- penpot_api_docs.md: 34 verbatim captures of high_level_overview and
penpot_api_info, each headed by the exact request that produced it. Every
penpot_api_info tool result in a seed must be a subset of lines of this
file, in original order. Records three places where the served docs
contradict the runtime (addFlexLayout/addGridLayout copy-paste in the Grid
section, flex.appendChild for grid children, withChildren vs
includeChildren), plus the createText() example that is the direct cause
of the production failure.
- penpot_system_prompt.md: the server's instructions block verbatim. Goes as
a system message into ~30% of the new seeds; it is the countermeasure to
the "don't pick your own colours" rule that produces the grey boxes.
- penpot_errors.md: the real error strings, including a section on silent
failures that raise nothing at all and are why the read-back invariant
exists.
- PENPOT_API_VERIFIED.md: the allow-list. No seed may reference a member
absent from it. Documents the four root causes (findShapeById arity 1,
no shape.layout, createText() returning null, the #B1B2B5 default fill),
the twelve anti-grey-box invariants, and the forbidden-pattern list the
linter checks.
Live re-verification of the error strings is still pending: the Penpot
plugin is not currently connected, so it is deferred to the gate 5 baseline
step, which needs the live connection anyway.
Re-corrida completa con el test corregido (max_tokens=2048, commit
2d2c45f), incluyendo tambien el baseline BF16 re-medido con el MISMO limite
(el original de Fase 4 se midio con max_tokens=1024) para una comparacion
justa:
- BF16 (2048 tokens): 97.5% (195/200), 1 no_tool_call, 4 invalid -- vs
98.5% (197/200), 0 no_tool_call, 3 invalid del baseline original de Fase
4 (1024 tokens). El propio BF16 varia levemente al re-medir con mas
tokens (no-determinismo de vLLM con batching dinamico + mas espacio para
"reconsiderar" casos ambiguos).
- NVFP4 mezclado (2048 tokens): 96.0% (192/200), 6 no_tool_call, 2 invalid
-- vs 95.0% (190/200) con 1024 tokens.
Con el mismo limite de tokens, la brecha real BF16 vs NVFP4 se achica de
~4pts (comparacion original, asimetrica) a ~1.5pts (comparacion justa).
Auditados con criterio humano todos los casos no_tool_call/invalid de
ambos: ninguno tiene finish_reason=length (sin truncamiento) -- son
decisiones genuinas del modelo sobre prompts ambiguos (merge condicional
sin instruccion explicita, deteccion de patron DoS en tabla de 371
columnas, falta de contexto real como pageId/spaceId) presentes en AMBOS
checkpoints, no una debilidad especifica de la cuantizacion.
Hipotesis del usuario confirmada: el "FAIL" de aleleba-pr/adherencia en las
corridas NVFP4 anteriores (90%, 9/10) era un defecto del test, no del
modelo. Con max_tokens=512 y --reasoning-parser activo, el modelo (de
razonamiento) agotaba el presupuesto de tokens pensando antes de emitir el
contenido final -- la respuesta quedaba cortada, sin ninguna de las
substrings esperadas.
Re-corrida completa contra el checkpoint NVFP4 con calibracion mezclada
(mismo checkpoint de la comparacion anterior), con el test corregido
(max_tokens=2048): 100% (10/10), IDENTICO al baseline de Fase 4 BF16. El
caso de aleleba-pr ahora pasa (hits=['commit']); el texto completo
(reasoning + content, ahora guardado en el JSON) muestra que el modelo
describe correctamente el flujo (commit con prefijo fix: seguido de
aleleba-pr para armar el PR) -- sin la alucinacion vista antes ("git push
--force", "aleleba-pr-reviewer").
Esto invalida la regresion de la puerta 3 documentada en los hallazgos
anteriores de esta fase -- era un artefacto de medicion, no una
degradacion real de calidad introducida por la cuantizacion.
Checkpoint cuantizado con calibracion mezclada (256 propias + 256
ultrachat_200k, fix de dos fases), contenedor de eval propio CON
--speculative-config real (levanto healthy, MTP compartiendo embeddings/
lm_head con el modelo target, confirmado en logs).
Puerta 2: 95.0% validos (190/200) vs 94.5% (189/200) solo-propia vs 98.5%
(197/200) Fase 4 -- mejora marginal de +0.5pt sobre solo-propia, sigue
~3.5pts debajo de Fase 4. Persiste el mismo caso de nombre de tool
alucinado (getJiraProjectIssueTypes, no existe) visto en el intento
solo-propia.
Puerta 3: 90.0% (9/10), IDENTICO a solo-propia -- mismo caso exacto falla
(aleleba-pr/adherencia). La diversidad de ultrachat NO corrigio esta
regresion especifica.
Conclusion: la hipotesis de diversidad tematica en la calibracion no
resuelve la regresion observada. El checkpoint mezclado queda como una
alternativa equivalente (no mejor, no peor de forma significativa) al de
solo-datos-propios.
Diagnostico pedido por el usuario antes de invertir tiempo en recalibrar:
mismo checkpoint NVFP4 (256 muestras), mismo contenedor de eval, unica
diferencia es remover --speculative-config (servicio vllm-eval-nvfp4-nospec,
puerto 8003).
Resultado: la regresion persiste casi identica sin speculative-config.
Puerta 2: 95.5% (191/200) sin spec vs 94.5% (189/200) con spec vs 98.5%
(197/200) de Fase 4 -- diferencia de 1pt entre con/sin spec, dentro de
ruido esperado; ambas configuraciones quedan ~3-4pts debajo de Fase 4.
Puerta 3: 90% (9/10) sin spec, identico a 90% (9/10) con spec -- mismo caso
puntual falla en ambas corridas (aleleba-pr/adherencia), aunque el texto
alucinado especifico difiere entre corridas (no determinismo esperable de
vLLM con batching dinamico incluso a temperature=0).
Conclusion: el speculative decoding (MTP) NO es el causante de la
regresion -- persiste identica sin el. El causante es la cuantizacion
NVFP4 en si (probablemente calibracion insuficiente de los 256 expertos
MoE con solo 256 muestras). Recalibrar con mas muestras, como estaba
previsto condicionalmente, es ahora el paso indicado.
Contenedor de eval propio (vllm-eval-nvfp4, puerto 8002) levanto healthy y
sirvio con --speculative-config real (mtp, num_speculative_tokens=1) sin
errores -- confirma que los tensores MTP reinyectados calzan correctamente
con vLLM (puerta 0 superada).
Puerta 2 (tool-calls, 200 prompts held-out): 94.5% validos (189/200) vs
98.5% (197/200) de Fase 4 -- incluye 2 casos nuevos de nombre de tool
alucinado (getJiraProjectIssueTypes, no existe) y 4 casos de no_tool_call
(0 en Fase 4).
Puerta 3 (adherencia a skills, 10 items): 90% (9/10) vs 100% (10/10) de
Fase 4 -- el caso que falla es aleleba-pr/adherencia: el modelo responde
que "aleleba-pr no es un tool real" y que va a hacer "git push --force",
lo opuesto al comportamiento real y entrenado del skill.
Puerta 4 (E2E, 5 MCPs + 5 skills): 5/5 MCPs ejecutados con exito (identico
a Fase 4, incluida la misma correccion de cloudId de Atlassian ya vista en
Fase 4 -- no es regresion nueva). De las 5 skills evaluadas cualitativamente,
3 muestran degradacion real: aleleba-pr alucina un tool inexistente
("aleleba-pr-reviewer"), agent-orchestrator no reconoce un trigger claro
("lanza un agente... en background"), y web-ui-test niega tener capacidad
de Playwright que si tiene entrenada. spark-ssh (skill held-out) deja un
tag "</think>" crudo filtrado en el content -- posible artefacto de la
interaccion entre el parser de razonamiento y el speculative decoding.
Regresion real y no trivial vs. Fase 4 en las 3 puertas. Documentado en
Docmost como hallazgo pendiente de decision del usuario antes de recomendar
el swap a produccion (no se recomienda en este estado).
Puerta 2 (tool-calls, 200 prompts held-out, parser real qwen3_coder de vLLM): 197/200
validos (98.5%). Los 3 invalidos son casos donde el prompt referencia un recurso por
nombre (space/repo) sin ID real -- el modelo elige la tool correcta pero omite un campo
requerido (spaceId/repo) que no puede conocer en un turno unico sin una llamada previa de
lookup; no es un fallo de sintaxis del parser.
Puerta 3 (adherencia por skill + no-activacion, 10 items sobre las 5 skills reales):
100% de aprobacion. vllm-qwen36 no estaba corriendo -- baseline de produccion documentado
como pendiente, no bloqueante.
Puerta 4 (E2E real contra los 5 MCPs, ejecutado por el agente orquestador con sus propios
MCPs conectados): 5/5 exitosos. El unico caso que requirio una segunda llamada fue
atlassian (el modelo adivino un cloudId plausible que no era el real -- se corrigio con
getAccessibleAtlassianResources y la llamada tuvo exito, comportamiento esperado en un
flujo multi-turno).
- scripts/30_eval_suite.py --gate 1: eval-loss sobre el checkpoint mergeado, agrupado por
meta.bucket (aislando replay), comparado contra eval_loss=0.275 de Fase 3.
- docker-compose.eval.yml: servicio vllm-eval propio (puerto 8001), sirviendo el
checkpoint mergeado en BF16, con tool-call-parser=qwen3_coder y reasoning-parser=qwen3.
No se pudo leer el compose real de produccion (/data/compose/43/docker-compose.yml no
existe en spark, probablemente vive en el host del servidor Portainer) -- flags basados
en la arquitectura conocida del modelo.
- scripts/31_build_holdout_prompts.py: genera data/holdout_prompts.jsonl (200 prompts,
40 por MCP, sin overlap verificado contra train.jsonl/eval.jsonl).
- scripts/32_gate2_toolcalls.py: valida tool-calls devueltas por vllm-eval (parser real
de vLLM, nunca una regex propia) contra los 200 prompts held-out.
- scripts/33_gate3_adherencia.py: checklists de adherencia por skill + no-activacion,
con baseline opcional contra vllm-qwen36 si esta corriendo.
- scripts/34_gate4_e2e.py: arma el plan de llamadas E2E contra los 5 MCPs y 5 skills via
el checkpoint mergeado, para que el agente orquestador las ejecute con sus MCPs reales.
El chat_template.jinja de produccion no tiene tags {% generation %}, por lo que
return_assistant_tokens_mask salia vacio para el 100% de los ejemplos en el primer run.
Se genero data/chat_template_train.jinja (copia exacta del template real, con {%- generation -%}
envolviendo solo el contenido/tool_calls/im_end de cada turno assistant) para el fallback
de masking ya anticipado en la Decision de diseno #4 del plan principal -- el chat_template.jinja
original no se toca, sigue siendo el que sirve produccion.
data/raw/replay.jsonl: 80 prompts semilla (conversacion general/codigo/
razonamiento) x 9 pasadas variando temperatura, contra vLLM de produccion
(vllm-qwen36), duracion real ~5h07min. Revisado: 720/720 lineas son JSON
valido con messages de 2 turnos y reasoning_content no vacio; 23 ejemplos
quedaron con content vacio por agotar el presupuesto de max_tokens durante
el razonamiento (finish_reason=length), marcados en meta para que la Fase 2
decida como tratarlos.
.gitignore: excepcion para versionar replay.jsonl pese a vivir en data/raw/,
como especifica el plan (es texto revisable, no un binario de checkpoint).
data/schemas/*.json: dump fiel de las tool definitions reales de
Penpot (4), Gitea (53), GitHub-personal (43), Docmost (17) y
Atlassian (37), obtenidas directo de las definiciones ya cargadas
en la sesion de Claude Code (no se escribio un cliente MCP nuevo,
para no arriesgar desviarse del esquema real).
PENPOT_DEPLOYMENT_NOTES.md: investigacion del codigo fuente oficial
de @penpot/mcp confirma que import_image y export_shape.filePath
estan ausentes porque este deployment corre en modo remoto/multi-
usuario (isFileSystemAccessEnabled() = !isRemoteMode()) -- documentado
tambien en una subpagina nueva de Docmost.
scripts/03_build_replay.py: genera ~700 ejemplos de replay (anti-
forgetting) contra el vLLM de produccion, 80 prompts semilla
(conversacion general/codigo/razonamiento) x 9 pasadas variando
temperatura. Mapea el campo "reasoning" de la API de vLLM a
"reasoning_content" para la convencion de chat template.
Scaffolding inicial del repo: carpetas scripts/, data/schemas/,
data/raw/, out/, .gitignore para binarios/checkpoints, y el
docker-compose.yml del contenedor de training (imagen NGC pytorch
25.12-py3, GPU reservada, bind mounts a ai-projects vía NFS y a
~/ft-models en disco local rápido de spark) sin tocar jupyter-pyt
ni el vllm de producción.
aleleba