Phase 6.4.25c: re-score existing v2b-bf16 gate 5 results with corrected veto

Browse Source

New offline script re-scans the code payloads already saved in the gate 5
result JSONs (holdout, full, generalization) using the corrected forbidden-
behavior list -- no live Penpot needed, reuses scan_forbidden/score_prompt
from 35_gate5_penpot_design.py as the single source of truth.

Holdout result changes materially: 3 of the 5 flagged prompts (pd-008,
pd-009, pd-035) were flagged only for flex.appendChild. With that pattern
removed, forbidden-API rate drops from 8.3% (5/60) to 3.3% (2/60), which is
now under the <=5% fallback threshold -- the holdout gate now passes.

Full mode and the generalization probe are unaffected: neither run's
transcripts ever triggered the flex.appendChild pattern, so their scores
are identical before and after.

Original result files are kept untouched; corrected results are written
alongside as *-corrected.json.

...

This commit is contained in:

aleleba

2026-08-04 17:40:26 +00:00

parent 7a1577b9d7

commit cc54edc39d

4 changed files with 7619 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/gate5_holdout_v2b-bf16-holdout-corrected.json

+863

File diff suppressed because one or more lines are too long

data/gate5_results_v2b-bf16-full-corrected.json

+4469

File diff suppressed because one or more lines are too long

data/gate5_results_v2b-bf16-generalization-corrected.json

+2153

File diff suppressed because one or more lines are too long