aleleba
63da20c031
The training container from phases 3-5 no longer exists and nothing in the repo pinned its versions, so a rebuild could silently change either the checkpoint key conversion (breaking adapter naming) or the assistant-mask behaviour (training on system/user/tool tokens). requirements.train.txt pins what matters and documents the two-phase install: llmcompressor declares torch>=2.10.0 and the NGC image ships the 2.10.0a0 pre-release, which pip's resolver reads as older, so it goes in with --no-deps. Pre-flight verified against the merged bf16 checkpoint on spark: 01_inspect_modules.py prints model.layers.0.linear_attn.*, config.json is sha256-identical to the base (93a4693f...), and the index keysets match exactly (1045 tensors, 690 under model.language_model.layers.*, 0 under model.layers.*). So PEFT will name adapter #2 the same way it named #1 and ADAPTER_TO_CHECKPOINT_PREFIX in 20_merge_lora.py applies unchanged. 10_train.py: every path and hyperparameter moves to an env var, with the phase 3 values as defaults so a bare run still reproduces phase 3 exactly. Adds three guards that each cover a specific silent failure: - abort if OUTPUT_DIR already holds an adapter, unless ALLOW_OVERWRITE=1. OUTPUT_DIR was hardcoded to out/lora-adapter, which is the provenance of the model currently in production. - MAX_TOKENS aborts rather than truncates. There was no length filter at all, so one long design trajectory would blow the memory budget hours into a run; truncating would be worse, since it would silently cut assistant targets. - assert use_rslora/use_dora/bias/modules_to_save. rsLoRA scales by alpha/sqrt(r), so an adapter trained with it would merge at 2.0 where 11.3 belongs and pass every assertion in the merge script. Also adds a config banner, a token-length histogram, and a per-bucket assistant-mask ratio report. New 07_lint_penpot_code.py hard-fails on the forbidden API patterns, placeholder greys, fabricated penpot_api_info results, toy-shaped ids and per-category coverage shortfalls. Error-recovery seeds legitimately need the wrong pattern, so the exemption is derived mechanically rather than declared by hand: a payload may contain a forbidden pattern only if its tool result is a real error string from the allow-list and a later payload in the same seed does the same thing without it. Run against the 41 existing seeds it reproduces the diagnosis exactly: 110 problems, 36 unique payloads, 0% system messages, zero coverage of addGridLayout/shadows/uploadMediaUrl/layoutChild, fabricated docs and toy ids.