Phase 6.3: fix the augmentation, close the holdout leak, stop rewarding invented parameters
Dataset build (05, 06): perturb_value is gone. It rewrote only tool_calls.arguments and left the tool results and the final answer saying something else, which is how data/train.jsonl ended up with 30 self-contradictory examples where the call says issue_number 82 and the answer says issue #77. Variation now comes from hand-written meta.paraphrases, or from meta.variation applied atomically across every field of the example at once. Nothing is substituted unless the seed declares it: guessing which number in a string is safe to change is what produced the contradictions in the first place. Prefix injection survives only as a fallback and only where the verb form can actually be conjugated, and there is a hard assert that no user turn matches the broken "Necesito que ¿Podés..." shape that 68 v1 prompts had. The penpot bucket is exempt from substitution entirely, since its payloads are code. Also asserts the bucket cannot collapse (verified: the old seeds give 320 rows from 83 unique trajectories and the build now fails) and scans for forbidden API patterns by importing them from the linter, so there is one source of truth. 06 now actually exits 1 on over-length rows. It printed [FILTERED], incremented a counter, and left the row in the file, which 10_train.py then trained on since it has no max_seq_length and batch 1. Gate 2 (32): reject any argument key absent from the schema, as its own failure category. It only checked required fields, so an invented scale or filePath passed - the gate was actively rewarding the exact behaviour this phase removes. Verified: export_shape with scale=2 now fails as unknown_argument, while a valid call still passes. Holdout (31, 35): rebalanced to penpot 60 / 35 each, added 20 real design templates, and replaced the full-string equality check with 6-gram shingles. Measured: a light paraphrase of a train.jsonl prompt scores 43% overlap and now fails the build, where the old check let it through at "not equal". Value pools are asserted disjoint from the corpus. The "2x resolution" template stays, relabelled as an invented-argument probe now that gate 2 can detect one; the createBoolean template stays because the API is real and the new B2 seeds teach it. Also dedupes: the old holdout had 15 duplicate prompts out of 200, i.e. 15 wasted measurements. Note: rebalancing the holdout means the 192/200 gate 2 baseline from phase 5 no longer applies to it, so that baseline has to be re-measured against production on the new file before it can be compared to. Gate 3 (33): 11 content checklists for the non-obvious conventions of the other MCPs - GFM table separators in Docmost, the update_page staleness retry, commit message shape, never merging the PR, dict-not-XML tool arguments. That is the most likely regression no gate currently covers. Mix builder: added the anti-collapse guard, so 420 new-portion rows that are really 96 trajectories repeated cannot pass unnoticed.
This commit is contained in:
+119
-21
@@ -1,5 +1,9 @@
|
||||
"""Fase 2: valida data/train.jsonl y data/eval.jsonl contra el tokenizer/chat_template REAL
|
||||
del modelo de produccion.
|
||||
"""Fase 2 (revisado en Fase 6): valida los jsonl de dataset (por defecto data/train.jsonl y
|
||||
data/eval.jsonl) contra el tokenizer/chat_template REAL del modelo de produccion.
|
||||
|
||||
Nota sobre emails de contacto en el copy de los seeds: `example.com` ya esta en
|
||||
SAFE_EMAIL_DOMAINS, asi que una direccion como `reservas@example.com` en el texto de una
|
||||
landing pasa el gate de secretos sin cambios. Cualquier otro dominio lo hace fallar.
|
||||
|
||||
Corre DENTRO del contenedor `qwen-lora-train` en spark (necesita `transformers` con el
|
||||
chat_template.jinja real de Qwen3.6, no una version instalada localmente). Invocar via:
|
||||
@@ -23,26 +27,64 @@ usa en inferencia/produccion y no se toca.
|
||||
|
||||
Por cada ejemplo: tokenizer.apply_chat_template(messages, tools=..., tokenize=True,
|
||||
return_assistant_tokens_mask=True, return_dict=True) -- assert sin excepcion, mascara de
|
||||
assistant no vacia. Filtra (no trunca) ejemplos que excedan MAX_TOKENS. Re-corre un gate de
|
||||
secretos (regex explicitas, igual que 04_sanitize.py, sin depender de detect-secrets --
|
||||
puede no estar instalado en este contenedor) sobre train.jsonl/eval.jsonl como ultima linea
|
||||
de defensa. Reporta un resumen final por bucket.
|
||||
assistant no vacia. Re-corre un gate de secretos (regex explicitas, igual que 04_sanitize.py,
|
||||
sin depender de detect-secrets -- puede no estar instalado en este contenedor) sobre los
|
||||
archivos validados como ultima linea de defensa. Reporta un resumen final por bucket con el
|
||||
histograma de longitudes (p50/p90/p99/max).
|
||||
|
||||
PARAMETROS POR ENV
|
||||
------------------
|
||||
VALIDATE_FILES lista de jsonl separados por coma (def: data/train.jsonl,data/eval.jsonl)
|
||||
MAX_TOKENS tope duro de longitud (def: 8192)
|
||||
PRESERVE_THINKING '1' para conservar el thinking de turnos previos (def: off)
|
||||
MODEL_PATH ruta del tokenizer (o argv[1])
|
||||
|
||||
Las rutas relativas se resuelven contra la raiz del repo. Ejemplo de Fase 6:
|
||||
|
||||
VALIDATE_FILES=data/train_lora2.jsonl,data/eval_lora2.jsonl \
|
||||
MAX_TOKENS=3000 PRESERVE_THINKING=1 \
|
||||
python3 scripts/06_validate_dataset.py
|
||||
|
||||
**El tope de longitud ABORTA, no filtra.** Hasta la Fase 6 este script imprimia `[FILTERED]`
|
||||
e incrementaba un contador, pero la fila se quedaba en el archivo y `10_train.py` (que corre
|
||||
sin `max_seq_length` y con batch 1) la entrenaba igual: era un falso sentido de seguridad que
|
||||
podia reventar el presupuesto de memoria a mitad de corrida, horas adentro. Ahora un solo
|
||||
ejemplo por encima de MAX_TOKENS termina en `sys.exit(1)`.
|
||||
|
||||
**PRESERVE_THINKING tiene que coincidir con el del training.** El template condiciona el
|
||||
thinking de los turnos previos a `preserve_thinking`; si este script valida con el flag
|
||||
apagado y el training entrena con el prendido, se esta midiendo otra cosa (otra longitud y
|
||||
otra mascara).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
MODEL_PATH = sys.argv[1] if len(sys.argv) > 1 else "/workspace/ft-models/Qwen--Qwen3.6-35B-A3B"
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def _resolve(raw):
|
||||
"""Las rutas relativas cuelgan de la raiz del repo."""
|
||||
p = Path(raw.strip())
|
||||
return p if p.is_absolute() else REPO_ROOT / p
|
||||
|
||||
|
||||
MODEL_PATH = (
|
||||
sys.argv[1] if len(sys.argv) > 1
|
||||
else os.environ.get("MODEL_PATH", "/workspace/ft-models/Qwen--Qwen3.6-35B-A3B")
|
||||
)
|
||||
TRAIN_CHAT_TEMPLATE_PATH = REPO_ROOT / "data" / "chat_template_train.jinja"
|
||||
MAX_TOKENS = 8192
|
||||
MAX_TOKENS = int(os.environ.get("MAX_TOKENS", "8192"))
|
||||
PRESERVE_THINKING = os.environ.get("PRESERVE_THINKING", "").lower() in ("1", "true", "yes")
|
||||
|
||||
DATASET_FILES = [
|
||||
REPO_ROOT / "data" / "train.jsonl",
|
||||
REPO_ROOT / "data" / "eval.jsonl",
|
||||
_resolve(part)
|
||||
for part in os.environ.get("VALIDATE_FILES", "data/train.jsonl,data/eval.jsonl").split(",")
|
||||
if part.strip()
|
||||
]
|
||||
|
||||
# Mismos patrones explicitos que 04_sanitize.py (subset sin dependencia de detect-secrets,
|
||||
@@ -107,10 +149,23 @@ def strip_tools_for_check(tools):
|
||||
return tools
|
||||
|
||||
|
||||
def percentile(sorted_values, q):
|
||||
"""Percentil por el metodo del vecino mas cercano (sin numpy: este contenedor puede no
|
||||
tenerlo y no vale la pena una dependencia por esto)."""
|
||||
if not sorted_values:
|
||||
return 0
|
||||
idx = min(len(sorted_values) - 1, max(0, int(round(q * (len(sorted_values) - 1)))))
|
||||
return sorted_values[idx]
|
||||
|
||||
|
||||
def validate_example(tokenizer, example):
|
||||
messages = example["messages"]
|
||||
tools = strip_tools_for_check(example.get("tools"))
|
||||
|
||||
template_kwargs = {}
|
||||
if PRESERVE_THINKING:
|
||||
template_kwargs["preserve_thinking"] = True
|
||||
|
||||
rendered = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tools=tools,
|
||||
@@ -118,6 +173,7 @@ def validate_example(tokenizer, example):
|
||||
return_assistant_tokens_mask=True,
|
||||
return_dict=True,
|
||||
add_generation_prompt=False,
|
||||
**template_kwargs,
|
||||
)
|
||||
|
||||
input_ids = rendered["input_ids"]
|
||||
@@ -141,9 +197,13 @@ def main():
|
||||
print(f"[INFO] reemplazando chat_template por la variante de training con masking: {TRAIN_CHAT_TEMPLATE_PATH}")
|
||||
tokenizer.chat_template = TRAIN_CHAT_TEMPLATE_PATH.read_text(encoding="utf-8")
|
||||
|
||||
print(f"[INFO] archivos: {', '.join(str(p) for p in DATASET_FILES)}")
|
||||
print(f"[INFO] MAX_TOKENS={MAX_TOKENS} (aborta, no filtra) PRESERVE_THINKING={PRESERVE_THINKING}")
|
||||
|
||||
summary = {}
|
||||
lengths_by_bucket = {}
|
||||
too_long = []
|
||||
total_exceptions = 0
|
||||
total_filtered_length = 0
|
||||
total_ok = 0
|
||||
|
||||
for path in DATASET_FILES:
|
||||
@@ -156,7 +216,7 @@ def main():
|
||||
|
||||
for lineno, example in examples:
|
||||
bucket = example.get("meta", {}).get("bucket", "sin_bucket")
|
||||
stats = summary.setdefault(bucket, {"ok": 0, "exceptions": 0, "filtered_length": 0})
|
||||
stats = summary.setdefault(bucket, {"ok": 0, "exceptions": 0, "too_long": 0})
|
||||
|
||||
try:
|
||||
n_tokens, mask_sum = validate_example(tokenizer, example)
|
||||
@@ -166,34 +226,72 @@ def main():
|
||||
print(f"[EXCEPTION] {path.name}:{lineno} (bucket={bucket}): {e}")
|
||||
continue
|
||||
|
||||
lengths_by_bucket.setdefault(bucket, []).append(n_tokens)
|
||||
|
||||
if n_tokens > MAX_TOKENS:
|
||||
stats["filtered_length"] += 1
|
||||
total_filtered_length += 1
|
||||
print(f"[FILTERED] {path.name}:{lineno} (bucket={bucket}): {n_tokens} tokens > {MAX_TOKENS}")
|
||||
stats["too_long"] += 1
|
||||
too_long.append((path.name, lineno, bucket, n_tokens))
|
||||
print(f"[TOO LONG] {path.name}:{lineno} (bucket={bucket}): {n_tokens} tokens > {MAX_TOKENS}")
|
||||
continue
|
||||
|
||||
stats["ok"] += 1
|
||||
total_ok += 1
|
||||
|
||||
print("\n[INFO] re-corriendo gate de secretos sobre train.jsonl/eval.jsonl")
|
||||
print("\n[INFO] re-corriendo gate de secretos sobre los archivos validados")
|
||||
secret_problems = []
|
||||
for path in DATASET_FILES:
|
||||
secret_problems.extend(secrets_gate(path))
|
||||
|
||||
print("\n=== RESUMEN POR BUCKET ===")
|
||||
for bucket, stats in sorted(summary.items()):
|
||||
print(f" {bucket}: ok={stats['ok']} filtrados_por_longitud={stats['filtered_length']} excepciones={stats['exceptions']}")
|
||||
print(f" {bucket}: ok={stats['ok']} sobre_max_tokens={stats['too_long']} excepciones={stats['exceptions']}")
|
||||
|
||||
print(f"\n=== TOTAL: ok={total_ok} filtrados_por_longitud={total_filtered_length} excepciones={total_exceptions} ===")
|
||||
# Histograma de longitudes: es lo que permite decidir si MAX_TOKENS esta bien puesto ANTES
|
||||
# de quemar una corrida de entrenamiento. Un p99 pegado al tope significa que el proximo
|
||||
# seed largo aborta la build; un maximo muy por debajo significa que se puede bajar el tope
|
||||
# (y con el, el pico de memoria).
|
||||
print("\n=== HISTOGRAMA DE TOKENS POR BUCKET (p50 / p90 / p99 / max) ===")
|
||||
all_lengths = []
|
||||
for bucket, lengths in sorted(lengths_by_bucket.items()):
|
||||
ordered = sorted(lengths)
|
||||
all_lengths.extend(ordered)
|
||||
print(
|
||||
f" {bucket:24} n={len(ordered):5} p50={percentile(ordered, 0.50):6} "
|
||||
f"p90={percentile(ordered, 0.90):6} p99={percentile(ordered, 0.99):6} "
|
||||
f"max={ordered[-1]:6}"
|
||||
)
|
||||
if all_lengths:
|
||||
ordered = sorted(all_lengths)
|
||||
print(
|
||||
f" {'TOTAL':24} n={len(ordered):5} p50={percentile(ordered, 0.50):6} "
|
||||
f"p90={percentile(ordered, 0.90):6} p99={percentile(ordered, 0.99):6} "
|
||||
f"max={ordered[-1]:6} (MAX_TOKENS={MAX_TOKENS})"
|
||||
)
|
||||
|
||||
print(f"\n=== TOTAL: ok={total_ok} sobre_max_tokens={len(too_long)} excepciones={total_exceptions} ===")
|
||||
|
||||
if secret_problems:
|
||||
print(f"\n[GATE FAIL] {len(secret_problems)} secretos sobrevivientes en train/eval:")
|
||||
print(f"\n[GATE FAIL] {len(secret_problems)} secretos sobrevivientes:")
|
||||
for problem in secret_problems:
|
||||
print(f" - {problem}")
|
||||
else:
|
||||
print("\n[GATE OK] 0 secretos sobrevivientes en train.jsonl/eval.jsonl")
|
||||
print("\n[GATE OK] 0 secretos sobrevivientes en los archivos validados")
|
||||
|
||||
if total_exceptions > 0 or secret_problems:
|
||||
if too_long:
|
||||
print(f"\n[GATE FAIL] {len(too_long)} ejemplo(s) superan MAX_TOKENS={MAX_TOKENS}:")
|
||||
for fname, lineno, bucket, n_tokens in too_long:
|
||||
print(f" - {fname}:{lineno} (bucket={bucket}): {n_tokens} tokens")
|
||||
print(
|
||||
" Estas filas NO se filtran solas: 10_train.py corre sin max_seq_length y con\n"
|
||||
" batch 1, asi que las entrenaria enteras y podria reventar el presupuesto de\n"
|
||||
" memoria a mitad de corrida. Hay que arreglar el dataset, no el validador.\n"
|
||||
" Como arreglarlo: partir la trayectoria larga en DOS ejemplos, persistiendo los\n"
|
||||
" ids que el segundo necesita en `storage` (penpot.local_storage) al final del\n"
|
||||
" primero y leyendolos al principio del segundo. Ademas de bajar la longitud, es\n"
|
||||
" mejor pedagogia: es exactamente lo que la descripcion de execute_code pide."
|
||||
)
|
||||
|
||||
if total_exceptions > 0 or secret_problems or too_long:
|
||||
sys.exit(1)
|
||||
|
||||
sys.exit(0)
|
||||
|
||||
Reference in New Issue
Block a user