171 steps, train_loss=0.6808, eval_loss=0.6348 (comparable to v1's 0.6306),
peak CUDA memory 74.58GB -- identical to v1's run, confirming the 13
corrective seeds didn't change the memory/length envelope. All guards
verified in adapter_config.json: use_rslora=false, use_dora=false,
lora_bias=false, modules_to_save=null, r=32, lora_alpha=64.
v1 adapter (out/lora-adapter-penpot/, commit 363648f) stays untouched as
a rollback point. New OUTPUT_DIR avoided the overwrite guard.
adapter_model.safetensors sha256 verified identical before/after copying
out of the container (root-owned file, fixed ownership via chown inside
the container since this OUTPUT_DIR landed inside the worktree, unlike
v1's which was copied in from outside).
171 steps (901 examples / 16 grad-accum * 3 epochs, ceil-rounded, not the
168 the plan estimated with floor), trained decoupled on spark against
the merged bf16 checkpoint that already has LoRA #1 folded in. No OOM,
no aborts: train_runtime 24370s (~6h46m, within the ~7.3h projection),
eval_loss 0.6306, peak CUDA 74.58GB, trainable% 0.1220 (matches phase 3
exactly).
adapter_config.json confirms the guarded hyperparameters: use_rslora
and use_dora both false, lora_bias false, modules_to_save null, r=32,
lora_alpha=64 -- the combination 20_merge_lora.py's scaling math
depends on.
Weights never touched the worktree: OUTPUT_DIR was
/workspace/ft-models/lora-adapter-penpot on spark (outside the
worktree, root-owned by the training container), copied out via
`docker exec cat` piped to a non-root file and verified by matching
sha256 (79167dfa...) before landing here. Intermediate checkpoint-*/
directories stay on spark; only the final adapter is versioned, same
as phase 3's out/lora-adapter/, which this leaves untouched.
.gitignore was missing the negation lines for out/lora-adapter-penpot/
despite already documenting that the final adapter should be
committed -- added the same two exceptions that out/lora-adapter/ has.
The training container from phases 3-5 no longer exists and nothing in the
repo pinned its versions, so a rebuild could silently change either the
checkpoint key conversion (breaking adapter naming) or the assistant-mask
behaviour (training on system/user/tool tokens). requirements.train.txt
pins what matters and documents the two-phase install: llmcompressor
declares torch>=2.10.0 and the NGC image ships the 2.10.0a0 pre-release,
which pip's resolver reads as older, so it goes in with --no-deps.
Pre-flight verified against the merged bf16 checkpoint on spark:
01_inspect_modules.py prints model.layers.0.linear_attn.*, config.json is
sha256-identical to the base (93a4693f...), and the index keysets match
exactly (1045 tensors, 690 under model.language_model.layers.*, 0 under
model.layers.*). So PEFT will name adapter #2 the same way it named #1 and
ADAPTER_TO_CHECKPOINT_PREFIX in 20_merge_lora.py applies unchanged.
10_train.py: every path and hyperparameter moves to an env var, with the
phase 3 values as defaults so a bare run still reproduces phase 3 exactly.
Adds three guards that each cover a specific silent failure:
- abort if OUTPUT_DIR already holds an adapter, unless ALLOW_OVERWRITE=1.
OUTPUT_DIR was hardcoded to out/lora-adapter, which is the provenance of
the model currently in production.
- MAX_TOKENS aborts rather than truncates. There was no length filter at
all, so one long design trajectory would blow the memory budget hours
into a run; truncating would be worse, since it would silently cut
assistant targets.
- assert use_rslora/use_dora/bias/modules_to_save. rsLoRA scales by
alpha/sqrt(r), so an adapter trained with it would merge at 2.0 where
11.3 belongs and pass every assertion in the merge script.
Also adds a config banner, a token-length histogram, and a per-bucket
assistant-mask ratio report.
New 07_lint_penpot_code.py hard-fails on the forbidden API patterns,
placeholder greys, fabricated penpot_api_info results, toy-shaped ids and
per-category coverage shortfalls. Error-recovery seeds legitimately need
the wrong pattern, so the exemption is derived mechanically rather than
declared by hand: a payload may contain a forbidden pattern only if its
tool result is a real error string from the allow-list and a later payload
in the same seed does the same thing without it.
Run against the 41 existing seeds it reproduces the diagnosis exactly:
110 problems, 36 unique payloads, 0% system messages, zero coverage of
addGridLayout/shadows/uploadMediaUrl/layoutChild, fabricated docs and
toy ids.
Los checkpoints intermedios (checkpoint-100/150/166, con optimizer.pt de
resumen de training) quedan ignorados -- no aportan nada mas alla del
peso final ya versionado aqui, y triplicarian el tamano sin necesidad.
data/raw/replay.jsonl: 80 prompts semilla (conversacion general/codigo/
razonamiento) x 9 pasadas variando temperatura, contra vLLM de produccion
(vllm-qwen36), duracion real ~5h07min. Revisado: 720/720 lineas son JSON
valido con messages de 2 turnos y reasoning_content no vacio; 23 ejemplos
quedaron con content vacio por agotar el presupuesto de max_tokens durante
el razonamiento (finish_reason=length), marcados en meta para que la Fase 2
decida como tratarlos.
.gitignore: excepcion para versionar replay.jsonl pese a vivir en data/raw/,
como especifica el plan (es texto revisable, no un binario de checkpoint).
Scaffolding inicial del repo: carpetas scripts/, data/schemas/,
data/raw/, out/, .gitignore para binarios/checkpoints, y el
docker-compose.yml del contenedor de training (imagen NGC pytorch
25.12-py3, GPU reservada, bind mounts a ai-projects vía NFS y a
~/ft-models en disco local rápido de spark) sin tocar jupyter-pyt
ni el vllm de producción.
aleleba