The training container from phases 3-5 no longer exists and nothing in the
repo pinned its versions, so a rebuild could silently change either the
checkpoint key conversion (breaking adapter naming) or the assistant-mask
behaviour (training on system/user/tool tokens). requirements.train.txt
pins what matters and documents the two-phase install: llmcompressor
declares torch>=2.10.0 and the NGC image ships the 2.10.0a0 pre-release,
which pip's resolver reads as older, so it goes in with --no-deps.
Pre-flight verified against the merged bf16 checkpoint on spark:
01_inspect_modules.py prints model.layers.0.linear_attn.*, config.json is
sha256-identical to the base (93a4693f...), and the index keysets match
exactly (1045 tensors, 690 under model.language_model.layers.*, 0 under
model.layers.*). So PEFT will name adapter #2 the same way it named #1 and
ADAPTER_TO_CHECKPOINT_PREFIX in 20_merge_lora.py applies unchanged.
10_train.py: every path and hyperparameter moves to an env var, with the
phase 3 values as defaults so a bare run still reproduces phase 3 exactly.
Adds three guards that each cover a specific silent failure:
- abort if OUTPUT_DIR already holds an adapter, unless ALLOW_OVERWRITE=1.
OUTPUT_DIR was hardcoded to out/lora-adapter, which is the provenance of
the model currently in production.
- MAX_TOKENS aborts rather than truncates. There was no length filter at
all, so one long design trajectory would blow the memory budget hours
into a run; truncating would be worse, since it would silently cut
assistant targets.
- assert use_rslora/use_dora/bias/modules_to_save. rsLoRA scales by
alpha/sqrt(r), so an adapter trained with it would merge at 2.0 where
11.3 belongs and pass every assertion in the merge script.
Also adds a config banner, a token-length histogram, and a per-bucket
assistant-mask ratio report.
New 07_lint_penpot_code.py hard-fails on the forbidden API patterns,
placeholder greys, fabricated penpot_api_info results, toy-shaped ids and
per-category coverage shortfalls. Error-recovery seeds legitimately need
the wrong pattern, so the exemption is derived mechanically rather than
declared by hand: a payload may contain a forbidden pattern only if its
tool result is a real error string from the allow-list and a later payload
in the same seed does the same thing without it.
Run against the 41 existing seeds it reproduces the diagnosis exactly:
110 problems, 36 unique payloads, 0% system messages, zero coverage of
addGridLayout/shadows/uploadMediaUrl/layoutChild, fabricated docs and
toy ids.
