Phase 6.3: re-measure the gate 2 and 3 production baselines on the new holdout

Rebalancing the holdout regenerated all 200 prompts, so phase 5's 192/200
stopped being comparable and every gate2/gate3/gate4 result file in the
repo was from phase 5. These had to be measured with production still up,
before asking for the downtime, or it would have cost the user another
window later just for this.

Gate 2 over the new holdout: 191/200, 95.5% global. atlassian 97.1,
docmost 94.3, gitea 85.7, github-personal 100, penpot 98.3. All five
failures are missing_required and there are zero invented arguments.

That penpot figure is worth keeping in view: production is at 98.3% on
tool-call VALIDITY, so its failure is not in the shape of the call but in
the code it puts inside execute_code. The two are measured by different
gates and only gate 5 sees the second.

Gate 3: 61.9%, which is 13 of 21. Not comparable to phase 5's 10/10, since
that run had 10 checks and this one has 21 - the 11 new content checklists
for the non-obvious conventions of the other MCPs are what risk #12 in the
plan called the most likely invisible regression. Production already fails
7 of them, so they have headroom rather than being a formality.

Gate 4 needs no re-baseline, verified rather than assumed: grep for
holdout_prompts in 34_gate4_e2e.py returns 0, so the regenerated holdout
does not reach it and its phase 5 result stands. It also needs all five
MCPs live, including the Penpot plugin, which closes when downtime starts.

Also trims the one seed that exceeded the token ceiling. The corpus now
validates clean against the production tokenizer with preserve_thinking on:
1000 of 1000 rows, zero exceptions, zero secrets, p50 1858, p90 3021,
max 3271 against a 3300 limit. That check earning its keep is the reason
06_validate_dataset.py was changed to exit non-zero instead of printing
[FILTERED] and moving on.

This commit is contained in:

aleleba

2026-07-30 21:25:58 +00:00

parent 3919b6baff

commit b4b4365725

5 changed files with 3383 additions and 7 deletions

data/calibration_v2.jsonl

+2 -2

View File

File diff suppressed because one or more lines are too long