2 Commits

Include renames

Author	SHA1	Message	Date
aleleba	8094183940	Phase 6.3.17: fix a harness fidelity bug and add a vibrancy metric ... The first baseline measured production with only 2290 of the 16392 characters of the server's instructions block - 14%. The missing 86% is exactly the API grounding: Core Shape Properties and Methods, Layout Systems, Text Elements, and The penpot and penpotUtils Objects, which is where insertChild, resize(), the layouts and penpotUtils are documented. That was worth catching, because the discrepancy had a visible signature: the measurement said production creates nothing, while the user's real Claude Code session produced grey boxes, i.e. shapes greater than zero. When a harness and reality disagree, the harness is the first suspect. In phase 5 a low max_tokens manufactured an apparent regression the same way. The gate now injects the full document, minus the trailing "You have hereby read the Penpot High-Level Overview" line, which is framing of the tool response rather than part of the instructions block and would otherwise tell the model it had already read something. The finding survives the fix. Across the five prompts measured cleanly under the corrected condition, shapeCount is still zero on every one. So the API invention is not an artefact of withholding documentation from the model - it happens with the documentation present. Also adds the vibrancy requirement the user raised as first-class scope: given an ambiguous brief the model must choose and justify a palette rather than ask or fall back to defaults. Neither distinctFillColors nor placeholderGreys distinguishes a vibrant palette from a muted but technically non-grey one, so four metrics are added: chromaticFills, meanChromaticSaturation, paletteStructured (a dominant brand hue, an accent at least 30 degrees away, and neutrals), and finalMessageListsHex, because a palette chosen in silence cannot be adjusted by the user. The saturation floor of 45 is derived, not asserted: measured over the 325 non-neutral fills of this phase's hand-authored corpus, median HSL saturation is 75, p25 is 48 and p10 is 35. A floor of 45 sits just under the first quartile and is cleared by 79% of those fills, so it is a floor the target behaviour already clears rather than an aspiration. The lightness band of 15 to 85 excludes near-blacks and near-whites, which can compute as highly saturated while reading as neutral. Gate prompt 6 becomes the user's literal failing sentence, and two ambiguous-brief prompts are added. One of them had to be re-domained after the disjointness check found it shared a 6-gram with a seed - the check fails on a single shared shingle, which is what makes it useful. Results so far are partial: prompts 1-5 measured cleanly, 6 has a timed-out audit and 7-10 hit the MCP outage, so those get re-measured. Both runs are kept, the 14% one renamed to record what it was.	2026-07-30 18:38:24 +00:00
aleleba	8ea4572edd	Phase 6.3: add gate 5, design quality in Penpot ... Unlike gates 2-4 this one needs a real agent loop - model, tool call, live MCP, result, up to 14 turns - because design quality only exists after the code executes. It talks to vLLM over the OpenAI API and to the Penpot MCP over HTTP (initialize, notifications/initialized, tools/list, tools/call), handling both application/json and text/event-stream responses. Endpoints and credentials come from env with no defaults and are never printed or stored; requests errors are reduced to the exception type because the requests message embeds the URL. Eight graded prompts, a fresh page per prompt named gate5/<tag>/<id>/<ts>, and the gate never deletes anything. The 17 metrics are computed by an audit payload the gate injects, not the model. placeholderGreys and forbidden behaviour are veto metrics: any hit scores that prompt 0. Two things worth calling out. The forbidden-pattern regexes are imported from 07_lint_penpot_code.py rather than duplicated, and the gate runs those same regexes over its own setup and audit payloads at startup - a gate that violated the API it is grading would be measuring its own bug. The holdout mode had a silent failure that is exactly the kind this phase exists to catch: with the endpoint down, all 60 generations failed, each entered the denominator with zero forbidden patterns found, and the gate reported 0% forbidden API and APPROVED. Since that number is the fallback trigger, a false pass there would have launched the quantization run. Request errors are now counted separately, never enter the denominator, and block approval outright. Known issue, resolved separately: gate prompt 6 is the exact production failure ("hazme una landing page de una pizzería con colores vibrantes"), and the flagship B6 seed was written to the same wording. Shingle overlap measures 20%, under the 34% threshold, but the seed prompt is a literal substring of the gate prompt - the threshold is too loose for prompts this short. The seed's domain gets changed rather than the gate's, so gate 5 measures transfer instead of memorisation; the real pizzeria prompt still runs in the human acceptance test, which is the criterion that decides.	2026-07-30 17:18:01 +00:00