171 steps (901 examples / 16 grad-accum * 3 epochs, ceil-rounded, not the
168 the plan estimated with floor), trained decoupled on spark against
the merged bf16 checkpoint that already has LoRA #1 folded in. No OOM,
no aborts: train_runtime 24370s (~6h46m, within the ~7.3h projection),
eval_loss 0.6306, peak CUDA 74.58GB, trainable% 0.1220 (matches phase 3
exactly).
adapter_config.json confirms the guarded hyperparameters: use_rslora
and use_dora both false, lora_bias false, modules_to_save null, r=32,
lora_alpha=64 -- the combination 20_merge_lora.py's scaling math
depends on.
Weights never touched the worktree: OUTPUT_DIR was
/workspace/ft-models/lora-adapter-penpot on spark (outside the
worktree, root-owned by the training container), copied out via
`docker exec cat` piped to a non-root file and verified by matching
sha256 (79167dfa...) before landing here. Intermediate checkpoint-*/
directories stay on spark; only the final adapter is versioned, same
as phase 3's out/lora-adapter/, which this leaves untouched.
.gitignore was missing the negation lines for out/lora-adapter-penpot/
despite already documenting that the final adapter should be
committed -- added the same two exceptions that out/lora-adapter/ has.
Los checkpoints intermedios (checkpoint-100/150/166, con optimizer.pt de
resumen de training) quedan ignorados -- no aportan nada mas alla del
peso final ya versionado aqui, y triplicarian el tamano sin necesidad.
Scaffolding inicial del repo: carpetas scripts/, data/schemas/,
data/raw/, out/, .gitignore para binarios/checkpoints, y el
docker-compose.yml del contenedor de training (imagen NGC pytorch
25.12-py3, GPU reservada, bind mounts a ai-projects vía NFS y a
~/ft-models en disco local rápido de spark) sin tocar jupyter-pyt
ni el vllm de producción.
aleleba