aleleba
8ea4572edd
Unlike gates 2-4 this one needs a real agent loop - model, tool call, live
MCP, result, up to 14 turns - because design quality only exists after the
code executes. It talks to vLLM over the OpenAI API and to the Penpot MCP
over HTTP (initialize, notifications/initialized, tools/list, tools/call),
handling both application/json and text/event-stream responses. Endpoints
and credentials come from env with no defaults and are never printed or
stored; requests errors are reduced to the exception type because the
requests message embeds the URL.
Eight graded prompts, a fresh page per prompt named gate5/<tag>/<id>/<ts>,
and the gate never deletes anything. The 17 metrics are computed by an
audit payload the gate injects, not the model. placeholderGreys and
forbidden behaviour are veto metrics: any hit scores that prompt 0.
Two things worth calling out.
The forbidden-pattern regexes are imported from 07_lint_penpot_code.py
rather than duplicated, and the gate runs those same regexes over its own
setup and audit payloads at startup - a gate that violated the API it is
grading would be measuring its own bug.
The holdout mode had a silent failure that is exactly the kind this phase
exists to catch: with the endpoint down, all 60 generations failed, each
entered the denominator with zero forbidden patterns found, and the gate
reported 0% forbidden API and APPROVED. Since that number is the fallback
trigger, a false pass there would have launched the quantization run.
Request errors are now counted separately, never enter the denominator,
and block approval outright.
Known issue, resolved separately: gate prompt 6 is the exact production
failure ("hazme una landing page de una pizzería con colores vibrantes"),
and the flagship B6 seed was written to the same wording. Shingle overlap
measures 20%, under the 34% threshold, but the seed prompt is a literal
substring of the gate prompt - the threshold is too loose for prompts this
short. The seed's domain gets changed rather than the gate's, so gate 5
measures transfer instead of memorisation; the real pizzeria prompt still
runs in the human acceptance test, which is the criterion that decides.