Phase 6.3: rewrite the Penpot seed corpus and build the LoRA #2 mix

Replaces the 41 old Penpot seeds with 105 new ones. The old set taught
three API forms that do not exist - findShapeById(page, id), shape.layout,
and createText() with no argument - and 21 of the 41 used the first one, so
patching was never an option: keeping them would mean fine-tuning against
the correction. The valuable lessons were re-founded on the real API
instead (the reversed children array in flex, persisting ids in storage,
never logging what you also return).

105 rather than 96 because nine multi-section compositions are split into
two trajectories each: the first builds the skeleton and persists ids,
palette, scale and helper functions in storage, the second recovers them
and fills the sections. That was forced by the 3000-token ceiling, but it
is better pedagogy anyway, and it is what execute_code's own description
asks for. It also paid for itself: the helpers cost ~600 chars once instead
of twice, and the skeleton call needs no export_shape, which freed the
budget to carry the verbatim system block.

Measured against the linter with the corpus-wide thresholds active:
143 unique code payloads (the old set had 36), 32% of seeds carrying the
server's system block verbatim (the old set had none), and every coverage
category met where the old set had zero addGridLayout, zero shadows, zero
uploadMediaUrl and zero layoutChild.

The flagship changed domain from pizzeria to an empanada shop. Gate 5's
prompt 6 is the exact production failure, and the seed had been written to
the same wording - a literal substring of the gate prompt, sharing two
6-gram shingles where the gate's disjointness check fails on one. Training
on the prompt we then evaluate would make gate 5 measure memorisation. The
real pizzeria prompt still runs in the human acceptance test. The same
check also caught an onboarding seed sitting too close to gate prompt 7.

Mix: 1000 examples split 90/10, giving 901 train and 99 eval. The mix is
1000 rather than 900 because 900 has to be the post-split train size: 900/16
= 56.25 steps per epoch x 3 = 168 steps, deliberately paired with phase 3's
166 so the optimiser trajectory length is comparable. Building 900 and then
carving out eval would have left 811 and 152 steps, silently breaking that
pairing.

Two guards in the builder had to be corrected against the real corpus:
- The forbidden-pattern scan now reads only `code` payloads. Scanning the
  serialised example flagged the corrective seeds themselves - the one that
  opens with the user asking "Importá esta imagen con import_image", the one
  quoting the overview line that mentions import_image verbatim, the one
  explaining that board.layout does not exist. They name the wrong API
  precisely in order to teach against it.
- The exemption for error-recovery seeds is derived from content, not from
  the mix portion: a forbidden pattern is allowed only where its tool result
  is a real error string and a later payload does the same thing without it.
  Keying on the portion broke as soon as an error-recovery seed lived in
  group A1, where the findShapeById arity seed naturally belongs.

Validated with the production tokenizer at preserve_thinking=true: p50 2466,
p90 2988, max 3250 tokens over the 105 seeds. Ten sit just above 3000, so
MAX_TOKENS goes to 3300 for the run and the smoke run will train on the 32
longest examples specifically, turning the peak-memory question into a
ten-minute measurement instead of a risk discovered hours in.
This commit is contained in:
2026-07-30 17:50:30 +00:00
parent c65d309719
commit c9792c5c40
5 changed files with 1474 additions and 69 deletions
+113 -28
View File
@@ -6,7 +6,7 @@ Corre LOCALMENTE (sin GPU, sin modelo, sin tokenizer). Determinista.
Produce tres archivos NUEVOS -- nunca toca data/train.jsonl ni data/eval.jsonl:
data/train_lora2.jsonl 900 ejemplos
data/train_lora2.jsonl 900 ejemplos (mezcla de 1000, partida 90/10)
data/eval_lora2.jsonl ~100 ejemplos, estratificado por porcion
data/calibration_v2.jsonl 256 ejemplos para la calibracion NVFP4
@@ -69,7 +69,14 @@ EVAL_OUT = Path(os.environ.get("EVAL_LORA2_OUT", REPO_ROOT / "data" / "eval_lora
CALIB_OUT = Path(os.environ.get("CALIB_V2_OUT", REPO_ROOT / "data" / "calibration_v2.jsonl"))
SEED = 6006
N_TRAIN = 900
# La mezcla se arma con 1000 ejemplos y se parte 90/10, de modo que TRAIN queda en exactamente
# 900. Ese 900 no es cosmetico: 900/16 = 56.25 pasos por epoca x 3 epocas = 168 pasos, que el
# plan aparea deliberadamente con los 166 de la Fase 3 para que la longitud de la trayectoria
# del optimizador sea comparable. Armar 900 y despues sacarle el eval dejaria 811 -> 152 pasos,
# y la comparacion se rompe sin que nada avise.
N_MIX = 1000
N_TRAIN_ESPERADO = 900
EVAL_FRACTION = 0.10
N_CALIBRATION = 256
@@ -88,22 +95,24 @@ PORTION_OF_GROUP = {
"D": "api_recuperacion",
}
# Los targets son sobre la mezcla de 1000; las proporciones son las mismas que documenta el
# docstring (33.3 / 6.7 / 4.4 / 2.2 y 53.3 de replay).
PORTION_TARGETS = {
"diseno": 300,
"api_forma_correcta": 60,
"api_recuperacion": 40,
"api_preguntar": 20,
"diseno": 333,
"api_forma_correcta": 67,
"api_recuperacion": 44,
"api_preguntar": 23,
}
# Replay: bucket de origen en data/train.jsonl -> cuantos ejemplos tomar.
REPLAY_TARGETS = {
"replay": 180,
"otros_mcps": 120,
"skills_adherencia": 70,
"negativos_delegacion_errores": 65, # union de los tres buckets chicos
"replay": 200,
"otros_mcps": 133,
"skills_adherencia": 78,
"negativos_delegacion_errores": 72, # union de los tres buckets chicos
}
REPLAY_UNION_BUCKETS = ("negativos", "delegacion_subagentes", "manejo_errores")
N_PENPOT_BASICOS = 45
N_PENPOT_BASICOS = 50
# Patrones prohibidos. La fuente de verdad es scripts/07_lint_penpot_code.py; aca se re-declaran
# los que aplican al SCAN de replay (no hace falta el set completo: lo que se busca es descartar
@@ -130,20 +139,89 @@ def load_jsonl(path):
return out
def has_forbidden(example):
"""True si el ejemplo serializado contiene algun patron de la API vieja."""
blob = json.dumps(example, ensure_ascii=False)
for _, rx in FORBIDDEN_REPLAY:
if rx.search(blob):
return True
return False
def code_payloads(example):
"""Los payloads de `code` del ejemplo, que es lo unico donde un patron prohibido es un bug."""
out = []
for msg in example.get("messages", []):
for tc in msg.get("tool_calls") or []:
args = tc.get("function", {}).get("arguments")
if isinstance(args, dict) and "code" in args:
out.append(args["code"])
return out
def forbidden_hits(example):
blob = json.dumps(example, ensure_ascii=False)
"""Patrones de la API vieja presentes en el CODIGO del ejemplo.
Se escanea solo el `code`, nunca el ejemplo serializado entero. Escanear la prosa marca
como infractores justo a los seeds correctivos: el que arranca con el usuario pidiendo
"Importa esta imagen con import_image", el que cita verbatim la linea del overview que
menciona `import_image`, el que explica que `board.layout` no existe, el que aclara que
`filePath` esta eliminado del schema. Todos ellos NOMBRAN la API equivocada precisamente
para ensenar a no usarla; bloquearlos seria bloquear la correccion.
"""
blob = "\n".join(code_payloads(example))
return [name for name, rx in FORBIDDEN_REPLAY if rx.search(blob)]
def has_forbidden(example):
return bool(forbidden_hits(example))
def error_strings():
"""Los strings de error verbatim de data/schemas/penpot_errors.md."""
text = (REPO_ROOT / "data" / "schemas" / "penpot_errors.md").read_text(encoding="utf-8")
out = set()
for line in text.splitlines():
if line.startswith("|"):
for cell in line.strip().strip("|").split("|"):
out.update(re.findall(r"`([^`]+)`", cell))
return out
ERROR_STRINGS = error_strings()
def unexplained_forbidden(example):
"""Patrones de la API vieja que NO son un error intencional corregido.
Misma regla mecanica que scripts/07_lint_penpot_code.py: un payload puede traer un patron
prohibido si y solo si (a) su tool result es un string de error real de la allow-list, y
(b) un payload POSTERIOR del mismo ejemplo hace lo mismo SIN el patron. O sea: el error
tiene que haber ocurrido de verdad y tiene que haber sido corregido.
Se deriva del contenido, no de la porcion ni de una bandera en `meta`. Apoyarse en la
porcion se rompe en cuanto un seed de recuperacion de error vive en un grupo que no es el
de recuperacion -- que es exactamente el caso del seed de aridad de `findShapeById` del
grupo A1, cuyo lugar natural es la familia de busqueda.
"""
msgs = example.get("messages", [])
fallidas = {
m.get("tool_call_id")
for m in msgs
if m.get("role") == "tool" and (m.get("content") or "").strip() in ERROR_STRINGS
}
llamadas = []
for m in msgs:
for tc in m.get("tool_calls") or []:
args = tc.get("function", {}).get("arguments")
if isinstance(args, dict) and "code" in args:
llamadas.append((tc.get("id"), args["code"]))
sin_explicar = []
for nombre, rx in FORBIDDEN_REPLAY:
for idx, (cid, code) in enumerate(llamadas):
if not rx.search(code):
continue
corregido = cid in fallidas and any(
not rx.search(c) for _, c in llamadas[idx + 1:]
)
if not corregido:
sin_explicar.append(nombre)
break
return sin_explicar
def paraphrase_variants(seed_ex):
"""Expande un seed en sus variantes de prompt.
@@ -320,9 +398,9 @@ def main():
mixed.extend(sample_replay(rows, target, rng, label))
# ---- 4. verificaciones duras -----------------------------------------------------------
if len(mixed) != N_TRAIN:
if len(mixed) != N_MIX:
raise SystemExit(
f"[ABORT] la mezcla tiene {len(mixed)} ejemplos, se esperaban {N_TRAIN}. "
f"[ABORT] la mezcla tiene {len(mixed)} ejemplos, se esperaban {N_MIX}. "
f"Revisar PORTION_TARGETS / REPLAY_TARGETS / N_PENPOT_BASICOS."
)
@@ -330,19 +408,19 @@ def main():
# que se identifican mecanicamente: su porcion es 'api_recuperacion'.
offenders = []
for i, ex in enumerate(mixed):
if ex["meta"].get("porcion") == "api_recuperacion":
continue
hits = forbidden_hits(ex)
hits = unexplained_forbidden(ex)
if hits:
offenders.append((i, ex["meta"].get("porcion"), hits))
if offenders:
print(f"\n[ABORT] {len(offenders)} ejemplo(s) de la mezcla final contienen patrones de la "
f"API vieja fuera de la porcion de recuperacion de error:")
print(f"\n[ABORT] {len(offenders)} ejemplo(s) de la mezcla final usan la API vieja en su "
f"`code` sin que sea un error real corregido despues:")
for i, portion, hits in offenders[:20]:
print(f" #{i} (porcion={portion}): {hits}")
sys.exit(1)
print("[OK] cero patrones de la API vieja en la mezcla final "
"(fuera de la porcion de recuperacion de error, donde son el material didactico)")
n_intencionales = sum(1 for ex in mixed if forbidden_hits(ex))
print(f"[OK] cero usos injustificados de la API vieja en la mezcla final "
f"({n_intencionales} ejemplos la contienen como error real corregido despues, que es "
f"el material didactico de recuperacion)")
# Guarda anti-colapso, analoga a la de 05_build_dataset.py. Sin ella, la porcion nueva puede
# ser 420 ejemplos que son 96 trayectorias repetidas 4.4 veces cada una, y nada lo diria: el
@@ -384,6 +462,13 @@ def main():
# ---- 5. split y escritura ---------------------------------------------------------------
split_rng = random.Random(SEED)
train, eval_ = stratified_split(mixed, split_rng)
if len(train) != N_TRAIN_ESPERADO:
print(f"[WARN] train quedo en {len(train)} y no en {N_TRAIN_ESPERADO}: los pasos de "
f"entrenamiento van a ser {round(len(train) / 16 * 3)} en vez de 168. Es por el "
f"redondeo del split por porcion; ajustar PORTION_TARGETS/REPLAY_TARGETS si "
f"importa la comparabilidad exacta con la Fase 3.")
print(f"[INFO] pasos de entrenamiento proyectados: {len(train)}/16 x 3 epocas = "
f"{round(len(train) / 16 * 3)}")
write_jsonl(TRAIN_OUT, train)
write_jsonl(EVAL_OUT, eval_)