aleleba
2781b9eb32
HF Trainer defaults per_device_eval_batch_size to 8, independent of the training batch size, and that default was never overridden. During the eval forward pass (no gradient checkpointing needed there, so none is applied) an 8-example batch of long sequences materializes full fp32 logits at once and blows the CUDA budget. That was the real cause of both OOMs hit while calibrating this run (the worst-case-32 smoke run and the 8-example probe) -- not the training forward/backward, which measured a stable ~74GB peak across every length from 2808 to 3265 tokens in three separate calibrations. Fix: new EVAL_BATCH_SIZE env (default 1) wired into per_device_eval_batch_size. Verified twice after the fix: training on the 32 longest examples in the corpus (3161-3271 tokens, worst case) plus a full eval pass over all 99 real eval_lora2.jsonl examples (up to 3243 tokens) completed with no OOM, peak 72.41GB. Also adds PerStepMemoryCallback (opt-in via PER_STEP_MEMORY_LOG=1) to print per-step CUDA peak/reset, which is what let this calibration attribute the earlier OOM to eval rather than to a specific training micro-batch under GRAD_ACCUM=16.