Phase 6.4.20: build a worst-case probe set for the smoke run

Browse Source

The 32 longest examples of the training mix, 3161 to 3271 tokens each. With
batch 1 and gradient accumulation 16, a two-step smoke run consumes exactly
these 32, so it trains on the worst case the real run will ever see.

Risk #5 in the plan is an OOM discovered hours into the run. Phase 3 peaked
around 104 GB of the 121 GB available with a longest example near 2800
tokens, and this corpus goes to 3271. Rather than argue about whether the
extra 17% of sequence length fits, the smoke run measures it in ten
minutes. If it OOMs it OOMs immediately and cheaply, which is the whole
point of running the probe before the 169-step run rather than after.

...

This commit is contained in:

aleleba

2026-07-30 21:27:33 +00:00

parent b4b4365725

commit c24f0ab236

1 changed files with 32 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/train_lora2_worstcase32.jsonl

+32

File diff suppressed because one or more lines are too long