Diagnosis (audited gate5 full-mode transcripts, PNGs, and holdout payloads):
the dominant failure mode was NOT rubro/content coverage but a tool-call
formatting bug -- after diagnosing an exception, the model often writes its
retry as literal <tool_call> XML text embedded in reasoning instead of a
structured call, so it's silently dropped (9/10 gate5 full-mode prompts hit
this on their final captured turn). Reasoning length before a tool call
correlates with the failure (malformed-turn mean 1895 chars vs 364 for
well-formed turns), while the training mix's own tool-calling turns never
exceed 1630 chars and export_shape turns never exceed 485.
Corrective seeds (group D, grupo 'D', short 1-3 sentence diagnosis +
immediate well-formed retry, to avoid reinforcing long-reasoning risk):
- 9 seeds across fresh rubros (lavanderia, zapateria, optica, jugueteria,
papeleria, peluqueria, floreria, heladeria, cerrajeria) covering distinct
real API errors: .color on Text, addFlexLayout on non-Board, findShapeById
2-arg, createText() no-arg, textAlign, typography.setFont, createBoolean
null, appendChild no-arg, uploadMediaUrl network rejection.
- 4 seeds reinforcing uploadMediaUrl over the exact penpot.importImage
variant the model actually invents (distinct from the one existing seed,
which only taught the penpotUtils.importImage variant) -- rubros:
relojeria, guarderia infantil, agencia de viajes, kiosco.
Deliberately avoid ferreteria/gimnasio/veterinaria/panaderia: reserved for
the held-out generalization probe (separate from the frozen gate 5).
Added the newly-verified error strings (t.color, penpot.importImage,
addFlexLayout-on-non-Board) to penpot_errors.md -- all captured live from
this phase's own gate 5 full-mode run, not fabricated.
Rebuilt train_lora2.jsonl (901)/eval_lora2.jsonl (99)/calibration_v2.jsonl
from the 138-seed corpus (07_build_lora2_mix.py) and validated
(06_validate_dataset.py, 0 exceptions, 0 secrets, max 3271 tokens).
Rebalancing the holdout regenerated all 200 prompts, so phase 5's 192/200
stopped being comparable and every gate2/gate3/gate4 result file in the
repo was from phase 5. These had to be measured with production still up,
before asking for the downtime, or it would have cost the user another
window later just for this.
Gate 2 over the new holdout: 191/200, 95.5% global. atlassian 97.1,
docmost 94.3, gitea 85.7, github-personal 100, penpot 98.3. All five
failures are missing_required and there are zero invented arguments.
That penpot figure is worth keeping in view: production is at 98.3% on
tool-call VALIDITY, so its failure is not in the shape of the call but in
the code it puts inside execute_code. The two are measured by different
gates and only gate 5 sees the second.
Gate 3: 61.9%, which is 13 of 21. Not comparable to phase 5's 10/10, since
that run had 10 checks and this one has 21 - the 11 new content checklists
for the non-obvious conventions of the other MCPs are what risk #12 in the
plan called the most likely invisible regression. Production already fails
7 of them, so they have headroom rather than being a formality.
Gate 4 needs no re-baseline, verified rather than assumed: grep for
holdout_prompts in 34_gate4_e2e.py returns 0, so the regenerated holdout
does not reach it and its phase 5 result stands. It also needs all five
MCPs live, including the Penpot plugin, which closes when downtime starts.
Also trims the one seed that exceeded the token ceiling. The corpus now
validates clean against the production tokenizer with preserve_thinking on:
1000 of 1000 rows, zero exceptions, zero secrets, p50 1858, p90 3021,
max 3271 against a 3300 limit. That check earning its keep is the reason
06_validate_dataset.py was changed to exit non-zero instead of printing
[FILTERED] and moving on.
The setup returned penpot.root.id and the run aborted when it came back as
an empty string on the second prompt of a batch. penpot.root is the root of
the ACTIVE page, and after createPage plus openPage it need not have caught
up yet - a race the page-emptying code introduced, since that leaves the
emptied page active. The value was never useful anyway: every page shares
the same root id, so it identified nothing. Setup now returns only pageId,
which is unique and stable, and both the audit and the cleanup use it.
The diagnostic message is the second half of the same mistake. It printed
"PLUGIN DEGRADADO, ask the user to reload the browser" whenever any prompt
went unmeasured, regardless of why - so it said that for a failure that was
entirely the gate's own. A message that sends the user to reload their
browser when the bug is mine costs both of us time. It now classifies on
the error text: task timeouts and transport drops point at the plugin,
anything else points at the gate and says so explicitly.
Also closes two evaluation leaks the gate's own pre-flight caught, both in
the seeds rather than the gate prompts, since the prompts have to stay as a
real user would write them:
- A seed shared the 6-gram "la home de una escuela de" with gate prompt 9.
My first fix was overwritten by a subagent still writing the file, which
is why it reappeared.
- A seed used the same business as gate prompt 9 - a music school - without
sharing any 6-gram. Shingles cannot see that: two texts describe the same
business without sharing words. Training on the domain we then evaluate
inflates the result invisibly. So the gate now also checks that no seed
uses any of the gate's business nouns, listed explicitly.
The training mix was rebuilt: it had been assembled before the 20
ambiguous-brief seeds existed, so training on it would not have used the
corpus that was audited. The ambiguous-brief class gets its own mix portion
rather than being folded into design, because diluted across 76 design
seeds it would be at the mercy of a ratio, and that is the class the user
named as the main painpoint. 125 seeds, 168 unique payloads, 446 distinct
user prompts, 901 train and 99 eval.
Replaces the 41 old Penpot seeds with 105 new ones. The old set taught
three API forms that do not exist - findShapeById(page, id), shape.layout,
and createText() with no argument - and 21 of the 41 used the first one, so
patching was never an option: keeping them would mean fine-tuning against
the correction. The valuable lessons were re-founded on the real API
instead (the reversed children array in flex, persisting ids in storage,
never logging what you also return).
105 rather than 96 because nine multi-section compositions are split into
two trajectories each: the first builds the skeleton and persists ids,
palette, scale and helper functions in storage, the second recovers them
and fills the sections. That was forced by the 3000-token ceiling, but it
is better pedagogy anyway, and it is what execute_code's own description
asks for. It also paid for itself: the helpers cost ~600 chars once instead
of twice, and the skeleton call needs no export_shape, which freed the
budget to carry the verbatim system block.
Measured against the linter with the corpus-wide thresholds active:
143 unique code payloads (the old set had 36), 32% of seeds carrying the
server's system block verbatim (the old set had none), and every coverage
category met where the old set had zero addGridLayout, zero shadows, zero
uploadMediaUrl and zero layoutChild.
The flagship changed domain from pizzeria to an empanada shop. Gate 5's
prompt 6 is the exact production failure, and the seed had been written to
the same wording - a literal substring of the gate prompt, sharing two
6-gram shingles where the gate's disjointness check fails on one. Training
on the prompt we then evaluate would make gate 5 measure memorisation. The
real pizzeria prompt still runs in the human acceptance test. The same
check also caught an onboarding seed sitting too close to gate prompt 7.
Mix: 1000 examples split 90/10, giving 901 train and 99 eval. The mix is
1000 rather than 900 because 900 has to be the post-split train size: 900/16
= 56.25 steps per epoch x 3 = 168 steps, deliberately paired with phase 3's
166 so the optimiser trajectory length is comparable. Building 900 and then
carving out eval would have left 811 and 152 steps, silently breaking that
pairing.
Two guards in the builder had to be corrected against the real corpus:
- The forbidden-pattern scan now reads only `code` payloads. Scanning the
serialised example flagged the corrective seeds themselves - the one that
opens with the user asking "Importá esta imagen con import_image", the one
quoting the overview line that mentions import_image verbatim, the one
explaining that board.layout does not exist. They name the wrong API
precisely in order to teach against it.
- The exemption for error-recovery seeds is derived from content, not from
the mix portion: a forbidden pattern is allowed only where its tool result
is a real error string and a later payload does the same thing without it.
Keying on the portion broke as soon as an error-recovery seed lived in
group A1, where the findShapeById arity seed naturally belongs.
Validated with the production tokenizer at preserve_thinking=true: p50 2466,
p90 2988, max 3250 tokens over the 105 seeds. Ten sit just above 3000, so
MAX_TOKENS goes to 3300 for the run and the smoke run will train on the 32
longest examples specifically, turning the peak-memory question into a
ten-minute measurement instead of a risk discovered hours in.
data/raw/replay.jsonl: 80 prompts semilla (conversacion general/codigo/
razonamiento) x 9 pasadas variando temperatura, contra vLLM de produccion
(vllm-qwen36), duracion real ~5h07min. Revisado: 720/720 lineas son JSON
valido con messages de 2 turnos y reasoning_content no vacio; 23 ejemplos
quedaron con content vacio por agotar el presupuesto de max_tokens durante
el razonamiento (finish_reason=length), marcados en meta para que la Fase 2
decida como tratarlos.
.gitignore: excepcion para versionar replay.jsonl pese a vivir en data/raw/,
como especifica el plan (es texto revisable, no un binario de checkpoint).
Scaffolding inicial del repo: carpetas scripts/, data/schemas/,
data/raw/, out/, .gitignore para binarios/checkpoints, y el
docker-compose.yml del contenedor de training (imagen NGC pytorch
25.12-py3, GPU reservada, bind mounts a ai-projects vía NFS y a
~/ft-models en disco local rápido de spark) sin tocar jupyter-pyt
ni el vllm de producción.
aleleba