aleleba
363648fb5f
171 steps (901 examples / 16 grad-accum * 3 epochs, ceil-rounded, not the 168 the plan estimated with floor), trained decoupled on spark against the merged bf16 checkpoint that already has LoRA #1 folded in. No OOM, no aborts: train_runtime 24370s (~6h46m, within the ~7.3h projection), eval_loss 0.6306, peak CUDA 74.58GB, trainable% 0.1220 (matches phase 3 exactly). adapter_config.json confirms the guarded hyperparameters: use_rslora and use_dora both false, lora_bias false, modules_to_save null, r=32, lora_alpha=64 -- the combination 20_merge_lora.py's scaling math depends on. Weights never touched the worktree: OUTPUT_DIR was /workspace/ft-models/lora-adapter-penpot on spark (outside the worktree, root-owned by the training container), copied out via `docker exec cat` piped to a non-root file and verified by matching sha256 (79167dfa...) before landing here. Intermediate checkpoint-*/ directories stay on spark; only the final adapter is versioned, same as phase 3's out/lora-adapter/, which this leaves untouched. .gitignore was missing the negation lines for out/lora-adapter-penpot/ despite already documenting that the final adapter should be committed -- added the same two exceptions that out/lora-adapter/ has.
31 lines
934 B
Plaintext
31 lines
934 B
Plaintext
data/raw/*
|
|
!data/raw/.gitkeep
|
|
!data/raw/replay.jsonl
|
|
!data/raw/sanitized/
|
|
!data/raw/seeds/
|
|
data/raw/.secrets_map.json
|
|
out/*.safetensors
|
|
out/lora-adapter/checkpoint-*/
|
|
*.safetensors
|
|
*.bin
|
|
*.pt
|
|
!out/lora-adapter/adapter_model.safetensors
|
|
!out/lora-adapter/training_args.bin
|
|
!out/lora-adapter-penpot/adapter_model.safetensors
|
|
!out/lora-adapter-penpot/training_args.bin
|
|
__pycache__/
|
|
*.pyc
|
|
.ipynb_checkpoints/
|
|
.env
|
|
.worktrees/
|
|
|
|
# Fase 6: checkpoints intermedios del LoRA #2. El adapter final
|
|
# (out/lora-adapter-penpot/adapter_model.safetensors, ~169 MB) SI se commitea, igual que el de
|
|
# la Fase 3; los checkpoint-*/ del Trainer son decenas de GB y viven solo en spark.
|
|
out/lora-adapter-penpot/checkpoint-*/
|
|
out/*/checkpoint-*/
|
|
|
|
# Partes intermedias del corpus de seeds: se concatenan a data/raw/seeds/penpot.jsonl, que es
|
|
# el artefacto versionado. Mantener las partes sueltas invita a editar la copia equivocada.
|
|
data/raw/seeds/_parts/
|