Phase 6.4.27: gates 2/3/4 on v2b-NVFP4, spec and nospec

Gate 2 (tool-calls, 200 held-out): global 95.5% (191/200) on both spec and
nospec, matching v2b-bf16's 95.0%. Per-MCP breakdown diverges: spec regresses
penpot specifically (91.7% vs bf16's 98.3%, outside the 2-point guard) while
preserving atlassian/docmost; nospec preserves penpot (98.3%, matches bf16)
but regresses atlassian/docmost instead. Consistent with risk #10 (MTP draft
head drift) rather than risk #9 (quantization washing out the LoRA delta) --
if quantization itself hurt penpot, nospec should show the same regression,
and it doesn't. Sampling is non-greedy (temperature 0.6), so some of this is
ordinary run-to-run variance rather than a clean systematic signal.

Gate 3 (21-check adherencia): spec 90.5% (19/21), nospec 71.4% (15/21). Both
clear the >=61.9% no-regression bar against the production baseline, but the
gap between spec/nospec is real and worth flagging.

Gate 4 (E2E against real MCPs, spec only): gitea, docmost, and penpot
(high_level_overview) executed successfully on the model's first-shot tool
call. github-personal and atlassian failed on this specific attempt due to
single-shot argument guesses (empty search query; a plausible but wrong
Jira cloudId) -- verified separately that both integrations work fine once
given correct arguments, so this reads as ordinary single-shot tool-use
limitation, not a broken integration.

This commit is contained in:

aleleba

2026-08-04 21:04:22 +00:00

parent 1c672520a4

commit f4564bc7a0

5 changed files with 6359 additions and 0 deletions

data/gate3_results_v2b-nvfp4-spec.json

+256

View File

File diff suppressed because one or more lines are too long