HF Trainer defaults per_device_eval_batch_size to 8, independent of the
training batch size, and that default was never overridden. During the
eval forward pass (no gradient checkpointing needed there, so none is
applied) an 8-example batch of long sequences materializes full fp32
logits at once and blows the CUDA budget. That was the real cause of
both OOMs hit while calibrating this run (the worst-case-32 smoke run
and the 8-example probe) -- not the training forward/backward, which
measured a stable ~74GB peak across every length from 2808 to 3265
tokens in three separate calibrations.
Fix: new EVAL_BATCH_SIZE env (default 1) wired into
per_device_eval_batch_size. Verified twice after the fix: training on
the 32 longest examples in the corpus (3161-3271 tokens, worst case)
plus a full eval pass over all 99 real eval_lora2.jsonl examples (up
to 3243 tokens) completed with no OOM, peak 72.41GB.
Also adds PerStepMemoryCallback (opt-in via PER_STEP_MEMORY_LOG=1) to
print per-step CUDA peak/reset, which is what let this calibration
attribute the earlier OOM to eval rather than to a specific training
micro-batch under GRAD_ACCUM=16.
The training container from phases 3-5 no longer exists and nothing in the
repo pinned its versions, so a rebuild could silently change either the
checkpoint key conversion (breaking adapter naming) or the assistant-mask
behaviour (training on system/user/tool tokens). requirements.train.txt
pins what matters and documents the two-phase install: llmcompressor
declares torch>=2.10.0 and the NGC image ships the 2.10.0a0 pre-release,
which pip's resolver reads as older, so it goes in with --no-deps.
Pre-flight verified against the merged bf16 checkpoint on spark:
01_inspect_modules.py prints model.layers.0.linear_attn.*, config.json is
sha256-identical to the base (93a4693f...), and the index keysets match
exactly (1045 tensors, 690 under model.language_model.layers.*, 0 under
model.layers.*). So PEFT will name adapter #2 the same way it named #1 and
ADAPTER_TO_CHECKPOINT_PREFIX in 20_merge_lora.py applies unchanged.
10_train.py: every path and hyperparameter moves to an env var, with the
phase 3 values as defaults so a bare run still reproduces phase 3 exactly.
Adds three guards that each cover a specific silent failure:
- abort if OUTPUT_DIR already holds an adapter, unless ALLOW_OVERWRITE=1.
OUTPUT_DIR was hardcoded to out/lora-adapter, which is the provenance of
the model currently in production.
- MAX_TOKENS aborts rather than truncates. There was no length filter at
all, so one long design trajectory would blow the memory budget hours
into a run; truncating would be worse, since it would silently cut
assistant targets.
- assert use_rslora/use_dora/bias/modules_to_save. rsLoRA scales by
alpha/sqrt(r), so an adapter trained with it would merge at 2.0 where
11.3 belongs and pass every assertion in the merge script.
Also adds a config banner, a token-length histogram, and a per-bucket
assistant-mask ratio report.
New 07_lint_penpot_code.py hard-fails on the forbidden API patterns,
placeholder greys, fabricated penpot_api_info results, toy-shaped ids and
per-category coverage shortfalls. Error-recovery seeds legitimately need
the wrong pattern, so the exemption is derived mechanically rather than
declared by hand: a payload may contain a forbidden pattern only if its
tool result is a real error string from the allow-list and a later payload
in the same seed does the same thing without it.
Run against the 41 existing seeds it reproduces the diagnosis exactly:
110 problems, 36 unique payloads, 0% system messages, zero coverage of
addGridLayout/shadows/uploadMediaUrl/layoutChild, fabricated docs and
toy ids.
aleleba