The fallback decision landed on v2b-bf16 directly (v2-NVFP4 is skipped),
so the two candidate eval services need to serve the checkpoint that's
actually being quantized. Container names/ports/served-model-name stay
the same, only the volume and --model path move to v2b-NVFP4.
Global weighted loss 0.2725 (vs baseline 0.2750, previous v2-bf16 was
0.2734). All 6/6 non-penpot buckets pass the 0.10 regression threshold,
including delegacion_subagentes which had narrowly failed on v2-bf16
(+0.1019 -> now +0.0882). The 13 corrective seeds didn't hurt forgetting.
Also points vllm-eval at the new v2b-bf16 checkpoint for the gate 2/5
re-measurement.
A code review found seven ways these gates could pass green with something
actually wrong. All are the same family: a missing value was treated as OK.
The rule now written into all three files is that absent is not OK, absent
is "could not verify", and that either fails or is reported as an explicit
SKIP - it never slips through as green.
30_eval_suite.py:
- A bucket with no baseline of its own fell back to the global 0.2750 and
printed it in a column headed "baseline", as if it were that bucket's
number. Measured against the real eval.jsonl buckets: negativos going
from 0.12 to 0.33 is a real +0.21 regression, but the computed delta was
+0.055 and it PASSED; manejo_errores sitting unchanged at 0.42 produced
a fabricated +0.145 FAIL that would have discarded a healthy candidate
mid-downtime. Now such buckets print SKIP and the verdict reports how
many went unverified.
- "VEREDICTO: FAIL" exited 0, so a runbook chaining the gate into
quantization would have carried on to write 24 GB. Now exits 1.
- A typo in BASELINE_BUCKET_LOSSES silently matched nothing; now aborts.
- The penpot exemption is labelled honestly: those 11 rows are pre-existing
LoRA #1 tool-calling, not new capability, so gate 1 has no regression
coverage there and the log says so.
20_merge_lora.py dry-run (merge path untouched, verified by AST diff):
- adapter_config.get("use_rslora", False) meant a missing key passed AND
the log printed use_rslora=False, asserting it had checked something that
was never there. A different PEFT version omitting a key was enough.
- lora_bias was not checked at all, only bias. They are different fields:
lora_bias puts a bias inside lora_B, which W + scaling * (B @ A) ignores.
- The 620 keys were printed but never asserted, so an adapter with extra
tensors printed "310 + 310 = 930" and passed.
- rank_pattern/alpha_pattern were not checked. They set r per module, so
scaling is not uniformly alpha/r while both the dry-run and the merge
apply a single 2.0 to all 310 tensors.
- A missing family was invisible: swap linear_attn for 150 mlp.gate targets
and the total is still 310, no norm is zero because the family is simply
gone, and it passed. Now presence and per-family counts are asserted,
derived from the real adapter: linear_attn 150, shared_expert 120,
attention_qkvo 40, otros 0.
Verified against seven synthetic adapters plus the real phase 3 one; only
the correct adapter passes.
21_quantize_nvfp4.py (recipe and oneshot untouched): the calibration cache
now carries a provenance.json recording the training file's sha256, the
recipe numbers and the bucket distribution, and loading aborts on mismatch.
This is the phase's number one risk and it had no mechanical defence: the
phase 5 cache on disk has exactly 512 rows, the same as the v2 recipe, so
the only existing check could not tell them apart and reusing it would have
calibrated with zero design data and washed out the new capability
silently. Verified: that cache now aborts.
gate 5: retry transport failures against the Penpot MCP, which drops
connections mid-call intermittently (seen before in phase 4's gate 4).
Without it a blip on prompt 6 of 8 kills a whole run and reads like a model
failure. PluginNotConnected is deliberately not retried - that is a real
state of the world. Also unwrap the {"result":..., "log":...} envelope the
server wraps execute_code returns in; the gate was reading keys off the
outer object and rejecting a valid page setup.
Identico a vllm-eval-nvfp4 pero sin --speculative-config, para aislar si la
regresion de calidad observada en las puertas 2-3 (vs. Fase 4) viene del
speculative decoding (MTP) o de la cuantizacion NVFP4 en si -- decision
explicita del usuario antes de invertir tiempo en recalibrar con mas
muestras de calibracion.
Clona 1:1 el docker-compose.yml real de produccion de vllm-qwen36 (citado
integro en PLAN.md): mismos flags de vLLM incluido --speculative-config
(mtp, num_speculative_tokens=1) -- la primera vez que se prueba en este
proyecto -- --quantization compressed-tensors, --moe-backend
flashinfer_cutlass, --kv-cache-dtype fp8_e4m3, --hf-overrides de rope
scaling, parsers de reasoning/tool-call, y el resto de flags identicos.
Solo cambia container_name, puerto (8002 vs 8000 de produccion y 8001 del
vllm-eval de Fase 4), volumen (checkpoint NVFP4 de Fase 5),
--model/--served-model-name, y restart: "no". Nunca toca vllm-qwen36 ni su
compose real de Portainer.
- scripts/30_eval_suite.py --gate 1: eval-loss sobre el checkpoint mergeado, agrupado por
meta.bucket (aislando replay), comparado contra eval_loss=0.275 de Fase 3.
- docker-compose.eval.yml: servicio vllm-eval propio (puerto 8001), sirviendo el
checkpoint mergeado en BF16, con tool-call-parser=qwen3_coder y reasoning-parser=qwen3.
No se pudo leer el compose real de produccion (/data/compose/43/docker-compose.yml no
existe en spark, probablemente vive en el host del servidor Portainer) -- flags basados
en la arquitectura conocida del modelo.
- scripts/31_build_holdout_prompts.py: genera data/holdout_prompts.jsonl (200 prompts,
40 por MCP, sin overlap verificado contra train.jsonl/eval.jsonl).
- scripts/32_gate2_toolcalls.py: valida tool-calls devueltas por vllm-eval (parser real
de vLLM, nunca una regex propia) contra los 200 prompts held-out.
- scripts/33_gate3_adherencia.py: checklists de adherencia por skill + no-activacion,
con baseline opcional contra vllm-qwen36 si esta corriendo.
- scripts/34_gate4_e2e.py: arma el plan de llamadas E2E contra los 5 MCPs y 5 skills via
el checkpoint mergeado, para que el agente orquestador las ejecute con sus MCPs reales.
aleleba