Phase 6.4.25b: 13 corrective seeds for gate 5 diagnosis, rebuild training mix

Browse Source

Diagnosis (audited gate5 full-mode transcripts, PNGs, and holdout payloads):
the dominant failure mode was NOT rubro/content coverage but a tool-call
formatting bug -- after diagnosing an exception, the model often writes its
retry as literal <tool_call> XML text embedded in reasoning instead of a
structured call, so it's silently dropped (9/10 gate5 full-mode prompts hit
this on their final captured turn). Reasoning length before a tool call
correlates with the failure (malformed-turn mean 1895 chars vs 364 for
well-formed turns), while the training mix's own tool-calling turns never
exceed 1630 chars and export_shape turns never exceed 485.

Corrective seeds (group D, grupo 'D', short 1-3 sentence diagnosis +
immediate well-formed retry, to avoid reinforcing long-reasoning risk):
- 9 seeds across fresh rubros (lavanderia, zapateria, optica, jugueteria,
  papeleria, peluqueria, floreria, heladeria, cerrajeria) covering distinct
  real API errors: .color on Text, addFlexLayout on non-Board, findShapeById
  2-arg, createText() no-arg, textAlign, typography.setFont, createBoolean
  null, appendChild no-arg, uploadMediaUrl network rejection.
- 4 seeds reinforcing uploadMediaUrl over the exact penpot.importImage
  variant the model actually invents (distinct from the one existing seed,
  which only taught the penpotUtils.importImage variant) -- rubros:
  relojeria, guarderia infantil, agencia de viajes, kiosco.

Deliberately avoid ferreteria/gimnasio/veterinaria/panaderia: reserved for
the held-out generalization probe (separate from the frozen gate 5).

Added the newly-verified error strings (t.color, penpot.importImage,
addFlexLayout-on-non-Board) to penpot_errors.md -- all captured live from
this phase's own gate 5 full-mode run, not fabricated.

Rebuilt train_lora2.jsonl (901)/eval_lora2.jsonl (99)/calibration_v2.jsonl
from the 138-seed corpus (07_build_lora2_mix.py) and validated
(06_validate_dataset.py, 0 exceptions, 0 secrets, max 3271 tokens).

...

This commit is contained in:

aleleba

2026-08-03 15:44:22 +00:00

parent 730e42494f

commit 18efe3fa5d

5 changed files with 682 additions and 647 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/eval_lora2.jsonl

+50 -50

File diff suppressed because one or more lines are too long