Phase 6.4: make the gates fail when they cannot verify something
A code review found seven ways these gates could pass green with something actually wrong. All are the same family: a missing value was treated as OK. The rule now written into all three files is that absent is not OK, absent is "could not verify", and that either fails or is reported as an explicit SKIP - it never slips through as green. 30_eval_suite.py: - A bucket with no baseline of its own fell back to the global 0.2750 and printed it in a column headed "baseline", as if it were that bucket's number. Measured against the real eval.jsonl buckets: negativos going from 0.12 to 0.33 is a real +0.21 regression, but the computed delta was +0.055 and it PASSED; manejo_errores sitting unchanged at 0.42 produced a fabricated +0.145 FAIL that would have discarded a healthy candidate mid-downtime. Now such buckets print SKIP and the verdict reports how many went unverified. - "VEREDICTO: FAIL" exited 0, so a runbook chaining the gate into quantization would have carried on to write 24 GB. Now exits 1. - A typo in BASELINE_BUCKET_LOSSES silently matched nothing; now aborts. - The penpot exemption is labelled honestly: those 11 rows are pre-existing LoRA #1 tool-calling, not new capability, so gate 1 has no regression coverage there and the log says so. 20_merge_lora.py dry-run (merge path untouched, verified by AST diff): - adapter_config.get("use_rslora", False) meant a missing key passed AND the log printed use_rslora=False, asserting it had checked something that was never there. A different PEFT version omitting a key was enough. - lora_bias was not checked at all, only bias. They are different fields: lora_bias puts a bias inside lora_B, which W + scaling * (B @ A) ignores. - The 620 keys were printed but never asserted, so an adapter with extra tensors printed "310 + 310 = 930" and passed. - rank_pattern/alpha_pattern were not checked. They set r per module, so scaling is not uniformly alpha/r while both the dry-run and the merge apply a single 2.0 to all 310 tensors. - A missing family was invisible: swap linear_attn for 150 mlp.gate targets and the total is still 310, no norm is zero because the family is simply gone, and it passed. Now presence and per-family counts are asserted, derived from the real adapter: linear_attn 150, shared_expert 120, attention_qkvo 40, otros 0. Verified against seven synthetic adapters plus the real phase 3 one; only the correct adapter passes. 21_quantize_nvfp4.py (recipe and oneshot untouched): the calibration cache now carries a provenance.json recording the training file's sha256, the recipe numbers and the bucket distribution, and loading aborts on mismatch. This is the phase's number one risk and it had no mechanical defence: the phase 5 cache on disk has exactly 512 rows, the same as the v2 recipe, so the only existing check could not tell them apart and reusing it would have calibrated with zero design data and washed out the new capability silently. Verified: that cache now aborts. gate 5: retry transport failures against the Penpot MCP, which drops connections mid-call intermittently (seen before in phase 4's gate 4). Without it a blip on prompt 6 of 8 kills a whole run and reads like a model failure. PluginNotConnected is deliberately not retried - that is a real state of the world. Also unwrap the {"result":..., "log":...} envelope the server wraps execute_code returns in; the gate was reading keys off the outer object and rejecting a valid page setup.
This commit is contained in:
@@ -81,6 +81,119 @@ services:
|
||||
retries: 5
|
||||
start_period: 600s
|
||||
|
||||
# Fase 6 -- candidato v2 (base + LoRA #1 + LoRA #2 de diseno en Penpot, mergeado
|
||||
# y recuantizado a NVFP4). Clon 1:1 de vllm-eval-nvfp4, que a su vez clona el
|
||||
# compose real de produccion, incluido --speculative-config: es el contenedor
|
||||
# contra el que se corren las puertas 2/3/4/5 del candidato, y solo sirve si
|
||||
# replica exactamente los flags con los que se va a servir. Respecto de
|
||||
# vllm-eval-nvfp4 cambian SOLO container_name, puerto (8004; 8000 produccion,
|
||||
# 8001 Fase 4, 8002/8003 Fase 5), volumen, --model y --served-model-name.
|
||||
# Nunca se toca vllm-qwen36 ni su compose real de Portainer.
|
||||
vllm-eval-nvfp4-v2:
|
||||
image: vllm/vllm-openai:cu130-nightly-aarch64
|
||||
container_name: vllm-eval-nvfp4-v2
|
||||
restart: "no"
|
||||
runtime: nvidia
|
||||
environment:
|
||||
NVIDIA_VISIBLE_DEVICES: all
|
||||
NVIDIA_DRIVER_CAPABILITIES: compute,utility
|
||||
ports:
|
||||
- "8004:8000"
|
||||
ipc: host
|
||||
ulimits:
|
||||
memlock: -1
|
||||
stack: 67108864
|
||||
volumes:
|
||||
- /home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:/models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:ro
|
||||
command: >
|
||||
--model /models/Qwen3.6-35B-A3B-mcp-v2-NVFP4
|
||||
--served-model-name qwen3.6-35b-a3b-mcp-v2-nvfp4
|
||||
--host 0.0.0.0
|
||||
--port 8000
|
||||
--tensor-parallel-size 1
|
||||
--trust-remote-code
|
||||
--quantization compressed-tensors
|
||||
--moe-backend flashinfer_cutlass
|
||||
--kv-cache-dtype fp8_e4m3
|
||||
--gpu-memory-utilization 0.45
|
||||
--max-model-len 524288
|
||||
--max-num-seqs 8
|
||||
--max-num-batched-tokens 32768
|
||||
--enable-chunked-prefill
|
||||
--enable-prefix-caching
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
|
||||
--reasoning-parser qwen3
|
||||
--tool-call-parser qwen3_coder
|
||||
--enable-auto-tool-choice
|
||||
--default-chat-template-kwargs '{"preserve_thinking":true}'
|
||||
--limit-mm-per-prompt '{"image":4,"video":0,"audio":0}'
|
||||
--generation-config vllm
|
||||
--override-generation-config '{"temperature":0.6,"top_p":0.80,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'
|
||||
--hf-overrides '{"text_config":{"rope_scaling":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":262144}}}'
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 600s
|
||||
|
||||
# Fase 6 -- diagnostico de aislamiento del drift del head de MTP (riesgo #10):
|
||||
# identico a vllm-eval-nvfp4-v2 pero SIN --speculative-config. El draft head de
|
||||
# MTP se copia del linaje base y nunca se fine-tunea, mientras que el target
|
||||
# model ya derivo DOS veces (LoRA #1 y LoRA #2), asi que la tasa de aceptacion
|
||||
# del speculative decoding puede caer y degradar la calidad servida sin que la
|
||||
# cuantizacion ni el dataset tengan nada que ver. Correr las mismas puertas en
|
||||
# 8004 (spec) y 8005 (nospec) separa las dos causas. Es un servicio de
|
||||
# diagnostico: no clona produccion 1:1 a proposito (esa es justo la variable
|
||||
# que se esta aislando) y los flags de produccion NUNCA se cambian por esto,
|
||||
# solo se reporta el hallazgo.
|
||||
vllm-eval-nvfp4-v2-nospec:
|
||||
image: vllm/vllm-openai:cu130-nightly-aarch64
|
||||
container_name: vllm-eval-nvfp4-v2-nospec
|
||||
restart: "no"
|
||||
runtime: nvidia
|
||||
environment:
|
||||
NVIDIA_VISIBLE_DEVICES: all
|
||||
NVIDIA_DRIVER_CAPABILITIES: compute,utility
|
||||
ports:
|
||||
- "8005:8000"
|
||||
ipc: host
|
||||
ulimits:
|
||||
memlock: -1
|
||||
stack: 67108864
|
||||
volumes:
|
||||
- /home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:/models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:ro
|
||||
command: >
|
||||
--model /models/Qwen3.6-35B-A3B-mcp-v2-NVFP4
|
||||
--served-model-name qwen3.6-35b-a3b-mcp-v2-nvfp4-nospec
|
||||
--host 0.0.0.0
|
||||
--port 8000
|
||||
--tensor-parallel-size 1
|
||||
--trust-remote-code
|
||||
--quantization compressed-tensors
|
||||
--moe-backend flashinfer_cutlass
|
||||
--kv-cache-dtype fp8_e4m3
|
||||
--gpu-memory-utilization 0.45
|
||||
--max-model-len 524288
|
||||
--max-num-seqs 8
|
||||
--max-num-batched-tokens 32768
|
||||
--enable-chunked-prefill
|
||||
--enable-prefix-caching
|
||||
--reasoning-parser qwen3
|
||||
--tool-call-parser qwen3_coder
|
||||
--enable-auto-tool-choice
|
||||
--default-chat-template-kwargs '{"preserve_thinking":true}'
|
||||
--limit-mm-per-prompt '{"image":4,"video":0,"audio":0}'
|
||||
--generation-config vllm
|
||||
--override-generation-config '{"temperature":0.6,"top_p":0.80,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'
|
||||
--hf-overrides '{"text_config":{"rope_scaling":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":262144}}}'
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 600s
|
||||
|
||||
# Fase 5 -- diagnostico de aislamiento: identico a vllm-eval-nvfp4 pero SIN
|
||||
# --speculative-config, para determinar si la regresion de calidad observada
|
||||
# en las puertas 2-3 (vs. Fase 4) viene del speculative decoding (MTP) o de
|
||||
|
||||
Reference in New Issue
Block a user