Merge the Phase 3 LoRA adapter onto the BF16 base checkpoint via a streaming, shard-by-shard script that operates directly on raw safetensors tensors (never through AutoModelForCausalLM), so non-target tensors (mtp.*, visual.*) are never at risk of being dropped.
Key finding documented in the merge script: the adapter was trained with AutoModelForCausalLM, which exposes layers as model.layers.N..., but the raw multimodal base checkpoint stores them under model.language_model.layers.N.... The script remaps names before matching LoRA targets.
Add a full 4-gate evaluation suite served against the merged checkpoint through a dedicated vLLM eval container (docker-compose.eval.yml, service vllm-eval, port 8001, vllm/vllm-openai image, unquantized BF16), independent from the production vllm-qwen36 container.
Version the held-out prompt dataset and the results of gates 2-4.
Changes
File
Change
scripts/20_merge_lora.py
Streaming shard-by-shard LoRA merge on raw safetensors tensors, with name remapping (model.layers.* -> model.language_model.layers.*) and automatic verification (tensor counts, NaN/Inf, chat template provenance).
scripts/30_eval_suite.py
Gate 1: offline eval-loss by bucket on the merged checkpoint, with both per-example and token-weighted averaging.
docker-compose.eval.yml
Dedicated eval-only vLLM container (vllm-eval, port 8001, BF16) used for gates 2-4, separate from production.
scripts/31_build_holdout_prompts.py
Generates 200 held-out prompts with no overlap against train/eval data.
scripts/32_gate2_toolcalls.py
Gate 2: tool-call validity against vLLM's real qwen3_coder parser.
scripts/33_gate3_adherencia.py
Gate 3: skill adherence and non-activation checklists.
scripts/34_gate4_e2e.py
Gate 4: builds the call plan that the orchestrating agent executes against its own real MCPs.
Merge verification: 310/310 LoRA-target tensors merged, 1045/1045 total tensors preserved, mtp.*/visual.* byte-identical to base, no NaN/Inf, chat_template.jinja correctly sourced from the base model path (not from the training adapter).
Gate 1 (offline eval-loss by bucket): token-weighted global loss 0.256 vs Phase 3's Trainer eval_loss 0.275 (diff 0.019) — confirms the merge is correct. Per-example weighting initially showed a large diff (0.52) that looked like a merge bug but was a metric artifact: the "replay" bucket holds 87% of eval tokens, so a per-example average gives equal weight to small/hard buckets, inflating the apparent loss versus Trainer's token-weighted metric.
Gate 2 (tool-call validity against vLLM's real qwen3_coder parser, 200 held-out prompts covering the 5 MCPs): 197/200 valid (98.5%). The 3 invalid cases are single-turn limitations (missing an ID field that would require a prior lookup call), not parser/syntax failures.
Gate 3 (skill adherence + non-activation checklists, 5 skills): 100% pass rate (10/10). Production baseline (vllm-qwen36) was not running at eval time — documented as pending, not blocking for this phase.
Gate 4 (real E2E against the 5 MCPs, executed by the orchestrating session against its own connected MCPs, never simulated): 5/5 successful (gitea, github-personal, docmost, atlassian, penpot).
The eval container (vllm-eval, docker-compose.eval.yml) was brought up, used for gates 2-4, and torn down (docker compose down) afterward to free GPU/memory. Production (vllm-qwen36) was never started/stopped/modified — only read-only docker ps checks were performed.
The merged checkpoint itself (~72GB, at /home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-bf16 on spark) is not committed to git — it lives outside the repo on spark's local disk, as intended.
Test Plan
Merge script ran without exceptions, with all automatic checks green (310/310 LoRA tensors, 1045/1045 total tensors, no NaN/Inf).
Gate 1 token-weighted loss is consistent with Phase 3's Trainer eval_loss.
Eval container came up and responded on /v1/models.
Gates 2-4 ran with real (not simulated) results against the 5 MCPs and 5 skills.
Eval container was stopped at the end of the evaluation run.
Production vllm-qwen36 was never touched (read-only checks only).
## Summary
- Merge the Phase 3 LoRA adapter onto the BF16 base checkpoint via a streaming, shard-by-shard script that operates directly on raw safetensors tensors (never through `AutoModelForCausalLM`), so non-target tensors (`mtp.*`, `visual.*`) are never at risk of being dropped.
- Key finding documented in the merge script: the adapter was trained with `AutoModelForCausalLM`, which exposes layers as `model.layers.N...`, but the raw multimodal base checkpoint stores them under `model.language_model.layers.N...`. The script remaps names before matching LoRA targets.
- Add a full 4-gate evaluation suite served against the merged checkpoint through a dedicated vLLM eval container (`docker-compose.eval.yml`, service `vllm-eval`, port 8001, `vllm/vllm-openai` image, unquantized BF16), independent from the production `vllm-qwen36` container.
- Version the held-out prompt dataset and the results of gates 2-4.
## Changes
| File | Change |
|------|--------|
| `scripts/20_merge_lora.py` | Streaming shard-by-shard LoRA merge on raw safetensors tensors, with name remapping (`model.layers.*` -> `model.language_model.layers.*`) and automatic verification (tensor counts, NaN/Inf, chat template provenance). |
| `scripts/30_eval_suite.py` | Gate 1: offline eval-loss by bucket on the merged checkpoint, with both per-example and token-weighted averaging. |
| `docker-compose.eval.yml` | Dedicated eval-only vLLM container (`vllm-eval`, port 8001, BF16) used for gates 2-4, separate from production. |
| `scripts/31_build_holdout_prompts.py` | Generates 200 held-out prompts with no overlap against train/eval data. |
| `scripts/32_gate2_toolcalls.py` | Gate 2: tool-call validity against vLLM's real `qwen3_coder` parser. |
| `scripts/33_gate3_adherencia.py` | Gate 3: skill adherence and non-activation checklists. |
| `scripts/34_gate4_e2e.py` | Gate 4: builds the call plan that the orchestrating agent executes against its own real MCPs. |
| `data/gate2_results.json`, `data/gate3_results.json`, `data/gate4_results.json` | Versioned results for gates 2-4. |
| `data/holdout_prompts.jsonl` | Versioned held-out prompt dataset (200 prompts). |
## Results
- **Merge verification**: 310/310 LoRA-target tensors merged, 1045/1045 total tensors preserved, `mtp.*`/`visual.*` byte-identical to base, no NaN/Inf, `chat_template.jinja` correctly sourced from the base model path (not from the training adapter).
- **Gate 1** (offline eval-loss by bucket): token-weighted global loss 0.256 vs Phase 3's Trainer `eval_loss` 0.275 (diff 0.019) — confirms the merge is correct. Per-example weighting initially showed a large diff (0.52) that looked like a merge bug but was a metric artifact: the "replay" bucket holds 87% of eval tokens, so a per-example average gives equal weight to small/hard buckets, inflating the apparent loss versus Trainer's token-weighted metric.
- **Gate 2** (tool-call validity against vLLM's real `qwen3_coder` parser, 200 held-out prompts covering the 5 MCPs): 197/200 valid (98.5%). The 3 invalid cases are single-turn limitations (missing an ID field that would require a prior lookup call), not parser/syntax failures.
- **Gate 3** (skill adherence + non-activation checklists, 5 skills): 100% pass rate (10/10). Production baseline (`vllm-qwen36`) was not running at eval time — documented as pending, not blocking for this phase.
- **Gate 4** (real E2E against the 5 MCPs, executed by the orchestrating session against its own connected MCPs, never simulated): 5/5 successful (gitea, github-personal, docmost, atlassian, penpot).
- The eval container (`vllm-eval`, `docker-compose.eval.yml`) was brought up, used for gates 2-4, and torn down (`docker compose down`) afterward to free GPU/memory. Production (`vllm-qwen36`) was never started/stopped/modified — only read-only `docker ps` checks were performed.
- The merged checkpoint itself (~72GB, at `/home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-bf16` on spark) is **not** committed to git — it lives outside the repo on spark's local disk, as intended.
## Test Plan
- [x] Merge script ran without exceptions, with all automatic checks green (310/310 LoRA tensors, 1045/1045 total tensors, no NaN/Inf).
- [x] Gate 1 token-weighted loss is consistent with Phase 3's Trainer eval_loss.
- [x] Eval container came up and responded on `/v1/models`.
- [x] Gates 2-4 ran with real (not simulated) results against the 5 MCPs and 5 skills.
- [x] Eval container was stopped at the end of the evaluation run.
- [x] Production `vllm-qwen36` was never touched (read-only checks only).
Remapea nombres de modulo (model.layers.* del adapter -> model.language_model.layers.*
del checkpoint base multimodal), preserva tensores mtp.*/visual.* al copiarlos sin
modificar, y toma chat_template.jinja de MODEL_PATH (nunca del adapter, que tiene el
template de masking de training). Verificacion automatica en verde: 310/310 tensores
LoRA-target mergeados, 1045/1045 tensores totales preservados, sin NaN/Inf, muestra de
200 tensores no-target byte-identica al base.
- scripts/30_eval_suite.py --gate 1: eval-loss sobre el checkpoint mergeado, agrupado por
meta.bucket (aislando replay), comparado contra eval_loss=0.275 de Fase 3.
- docker-compose.eval.yml: servicio vllm-eval propio (puerto 8001), sirviendo el
checkpoint mergeado en BF16, con tool-call-parser=qwen3_coder y reasoning-parser=qwen3.
No se pudo leer el compose real de produccion (/data/compose/43/docker-compose.yml no
existe en spark, probablemente vive en el host del servidor Portainer) -- flags basados
en la arquitectura conocida del modelo.
- scripts/31_build_holdout_prompts.py: genera data/holdout_prompts.jsonl (200 prompts,
40 por MCP, sin overlap verificado contra train.jsonl/eval.jsonl).
- scripts/32_gate2_toolcalls.py: valida tool-calls devueltas por vllm-eval (parser real
de vLLM, nunca una regex propia) contra los 200 prompts held-out.
- scripts/33_gate3_adherencia.py: checklists de adherencia por skill + no-activacion,
con baseline opcional contra vllm-qwen36 si esta corriendo.
- scripts/34_gate4_e2e.py: arma el plan de llamadas E2E contra los 5 MCPs y 5 skills via
el checkpoint mergeado, para que el agente orquestador las ejecute con sus MCPs reales.
El primer resultado (promedio simple por ejemplo) daba 0.5185 vs 0.275 de Fase 3, señal de
alarma segun el propio script. La causa era metodologica, no un bug de merge: el bucket
replay concentra 112927 de los ~128849 tokens assistant del split de eval (87%), mientras
que buckets dificiles como negativos/skills_adherencia/delegacion_subagentes tienen pocos
ejemplos pero loss alto -- un promedio por ejemplo les da el mismo peso que a replay,
inflando el global. transformers.Trainer pondera por token, no por ejemplo. Con el mismo
ponderado por token: 0.2560 vs 0.275 de Fase 3 (diff=0.019, dentro del margen esperado) --
confirma que el merge es correcto.
Puerta 2 (tool-calls, 200 prompts held-out, parser real qwen3_coder de vLLM): 197/200
validos (98.5%). Los 3 invalidos son casos donde el prompt referencia un recurso por
nombre (space/repo) sin ID real -- el modelo elige la tool correcta pero omite un campo
requerido (spaceId/repo) que no puede conocer en un turno unico sin una llamada previa de
lookup; no es un fallo de sintaxis del parser.
Puerta 3 (adherencia por skill + no-activacion, 10 items sobre las 5 skills reales):
100% de aprobacion. vllm-qwen36 no estaba corriendo -- baseline de produccion documentado
como pendiente, no bloqueante.
Puerta 4 (E2E real contra los 5 MCPs, ejecutado por el agente orquestador con sus propios
MCPs conectados): 5/5 exitosos. El unico caso que requirio una segunda llamada fue
atlassian (el modelo adivino un cloudId plausible que no era el real -- se corrigio con
getAccessibleAtlassianResources y la llamada tuvo exito, comportamiento esperado en un
flujo multi-turno).
aleleba
merged commit 18d50e670f into master2026-07-29 13:20:01 -06:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
AutoModelForCausalLM), so non-target tensors (mtp.*,visual.*) are never at risk of being dropped.AutoModelForCausalLM, which exposes layers asmodel.layers.N..., but the raw multimodal base checkpoint stores them undermodel.language_model.layers.N.... The script remaps names before matching LoRA targets.docker-compose.eval.yml, servicevllm-eval, port 8001,vllm/vllm-openaiimage, unquantized BF16), independent from the productionvllm-qwen36container.Changes
scripts/20_merge_lora.pymodel.layers.*->model.language_model.layers.*) and automatic verification (tensor counts, NaN/Inf, chat template provenance).scripts/30_eval_suite.pydocker-compose.eval.ymlvllm-eval, port 8001, BF16) used for gates 2-4, separate from production.scripts/31_build_holdout_prompts.pyscripts/32_gate2_toolcalls.pyqwen3_coderparser.scripts/33_gate3_adherencia.pyscripts/34_gate4_e2e.pydata/gate2_results.json,data/gate3_results.json,data/gate4_results.jsondata/holdout_prompts.jsonlResults
mtp.*/visual.*byte-identical to base, no NaN/Inf,chat_template.jinjacorrectly sourced from the base model path (not from the training adapter).eval_loss0.275 (diff 0.019) — confirms the merge is correct. Per-example weighting initially showed a large diff (0.52) that looked like a merge bug but was a metric artifact: the "replay" bucket holds 87% of eval tokens, so a per-example average gives equal weight to small/hard buckets, inflating the apparent loss versus Trainer's token-weighted metric.qwen3_coderparser, 200 held-out prompts covering the 5 MCPs): 197/200 valid (98.5%). The 3 invalid cases are single-turn limitations (missing an ID field that would require a prior lookup call), not parser/syntax failures.vllm-qwen36) was not running at eval time — documented as pending, not blocking for this phase.vllm-eval,docker-compose.eval.yml) was brought up, used for gates 2-4, and torn down (docker compose down) afterward to free GPU/memory. Production (vllm-qwen36) was never started/stopped/modified — only read-onlydocker pschecks were performed./home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-bf16on spark) is not committed to git — it lives outside the repo on spark's local disk, as intended.Test Plan
/v1/models.vllm-qwen36was never touched (read-only checks only).