aleleba/qwen3-6-lora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: aleleba/qwen3-6-lora:8dcd736ba5c694c77e53273c0339d91d5bf762cd
Branches Tags
aleleba/qwen3-6-lora:master aleleba/qwen3-6-lora:agente-fase6-lora2-penpot
..
compare: aleleba/qwen3-6-lora:agente-fase6-lora2-penpot
Branches Tags
aleleba/qwen3-6-lora:master aleleba/qwen3-6-lora:agente-fase6-lora2-penpot
10 Commits
8dcd736ba5 .. agente-fase6-lora2-penpot
Author SHA1 Message Date
aleleba 4d840553d6 Phase 6.4.27: re-measure g5-08-reparacion-grises to complete gate 5 full on v2b-NVFP4 
g5-08 was pending since it fell mid-run when Penpot's plugin disconnected
("No Penpot plugin instances are currently connected"). Re-run in isolation
via GATE5_ONLY, per the established pattern: score 64.7, clean veto, no
placeholder greys. Gate 5 full mode on v2b-NVFP4 is now complete (10/10):
mean score 28.2 (threshold 65), clean veto 6/10, prompts>=60 2/10, pizzeria
landing 47.8 (threshold 60) -- verdict: DOES NOT APPROVE.
 2026-08-05 01:49:59 +00:00
aleleba 66d6c6c410 Phase 6.4.27: gate 5 (holdout + full) on v2b-NVFP4, spec config 
Holdout (60 design prompts, cheap proxy, no live Penpot needed): 5.0%
forbidden-API rate (3/60) -- right at the <=5% threshold, APPROVES.

Full mode (10 official prompts against live Penpot): 9/10 measured.
g5-08-reparacion-grises is still pending -- Penpot disconnected mid-run
("No Penpot plugin instances are currently connected", the real
disconnect signature, not the transient "transport dropped mid-call").
Per the established GATE5_ONLY pattern, this prompt gets re-measured
(not accepted as a genuine failure) once Penpot reconnects; the other
9 results are preserved as-is.
 2026-08-05 01:23:39 +00:00
aleleba f4564bc7a0 Phase 6.4.27: gates 2/3/4 on v2b-NVFP4, spec and nospec 
Gate 2 (tool-calls, 200 held-out): global 95.5% (191/200) on both spec and
nospec, matching v2b-bf16's 95.0%. Per-MCP breakdown diverges: spec regresses
penpot specifically (91.7% vs bf16's 98.3%, outside the 2-point guard) while
preserving atlassian/docmost; nospec preserves penpot (98.3%, matches bf16)
but regresses atlassian/docmost instead. Consistent with risk #10 (MTP draft
head drift) rather than risk #9 (quantization washing out the LoRA delta) --
if quantization itself hurt penpot, nospec should show the same regression,
and it doesn't. Sampling is non-greedy (temperature 0.6), so some of this is
ordinary run-to-run variance rather than a clean systematic signal.

Gate 3 (21-check adherencia): spec 90.5% (19/21), nospec 71.4% (15/21). Both
clear the >=61.9% no-regression bar against the production baseline, but the
gap between spec/nospec is real and worth flagging.

Gate 4 (E2E against real MCPs, spec only): gitea, docmost, and penpot
(high_level_overview) executed successfully on the model's first-shot tool
call. github-personal and atlassian failed on this specific attempt due to
single-shot argument guesses (empty search query; a plausible but wrong
Jira cloudId) -- verified separately that both integrations work fine once
given correct arguments, so this reads as ordinary single-shot tool-use
limitation, not a broken integration.
 2026-08-04 21:04:22 +00:00
aleleba 1c672520a4 Phase 6.4.26: point 8004/8005 eval services at v2b-NVFP4 
The fallback decision landed on v2b-bf16 directly (v2-NVFP4 is skipped),
so the two candidate eval services need to serve the checkpoint that's
actually being quantized. Container names/ports/served-model-name stay
the same, only the volume and --model path move to v2b-NVFP4.
 2026-08-04 18:48:05 +00:00
aleleba cc54edc39d Phase 6.4.25c: re-score existing v2b-bf16 gate 5 results with corrected veto 
New offline script re-scans the code payloads already saved in the gate 5
result JSONs (holdout, full, generalization) using the corrected forbidden-
behavior list -- no live Penpot needed, reuses scan_forbidden/score_prompt
from 35_gate5_penpot_design.py as the single source of truth.

Holdout result changes materially: 3 of the 5 flagged prompts (pd-008,
pd-009, pd-035) were flagged only for flex.appendChild. With that pattern
removed, forbidden-API rate drops from 8.3% (5/60) to 3.3% (2/60), which is
now under the <=5% fallback threshold -- the holdout gate now passes.

Full mode and the generalization probe are unaffected: neither run's
transcripts ever triggered the flex.appendChild pattern, so their scores
are identical before and after.

Original result files are kept untouched; corrected results are written
alongside as *-corrected.json.
 2026-08-04 17:40:26 +00:00
aleleba 7a1577b9d7 Phase 6.4.25c: correct flex.appendChild -- live verification shows it works 
Live probing against Penpot 2.16.2 (repeated, on a dedicated scratch page,
cleaned up afterwards) shows board.flex.appendChild(shape) is a real
function, distinct from board.appendChild, arity 1, throws nothing, and
preserves visual insertion order -- unlike board.appendChild, which still
inserts at the front as documented. The prior "broken" claim traces to the
MCP server's own high_level_overview() text (CRITICAL / BROKEN / NEVER use),
never verified with a live throw, and possibly confused with a real doc bug
that mislabels the grid 3-arg example as board.flex.

Removes flex.appendChild from the lint's FORBIDDEN patterns and from the
gate 5 forbidden-behavior veto, updates PENPOT_API_VERIFIED.md with the
verified facts, and drops the now-obsolete pattern from the replay filters
in 05_build_dataset.py and 07_build_lora2_mix.py.
 2026-08-04 17:40:16 +00:00
aleleba 3496051f72 Phase 6.4.25b: measure generalization probe on v2b-bf16 (ADDITIONAL, not official gate 5) 
4 prompts in domains absent from both seeds and the official gate5_prompts.jsonl (ferreteria,
gimnasio, tienda de mascotas, panaderia). score medio 32.0, veto limpio 3/4, 0/4 prompts >=60.

gp-03-tienda-mascotas violates the veto twice over: it hits both import_image variants the
model has ever tried (penpot.importImage on turn 1, penpotUtils.importImage on turn 2) plus a
numeric fontSize assignment -- the uploadMediaUrl reinforcement seeds from this correction
round did not transfer to an unseen domain. It is also flagged via GATE5_KNOWN_OVERLAP_IDS
(shares a 6-gram with the same seeds as g5-03), so its score is a double caution case.

gp-02-gimnasio hit the same transient Penpot MCP transport drop as g5-06/g5-07 on its first
pass, but the harness's own automatic retry (pass 2/3) recovered cleanly within the default
240s -- no manual timeout override was needed here.
 2026-08-04 06:43:26 +00:00
aleleba bf795bd01c Phase 6.4.25b: measure gate 5 full mode on v2b-bf16 (post-correction) 
10/10 prompts measured. VEREDICTO: NO APRUEBA (score medio 39.8 vs threshold 65, veto limpio
8/10, 3/10 prompts >=60). The flagship g5-06-landing-pizzeria now clears its own threshold
(69.6 >= 60) once a transient Penpot MCP transport drop is ruled out (see g5-06/g5-07 rerun
below) -- a real improvement over v2-bf16's 34.8 on that prompt, but the aggregate gate
criteria still fail. g5-03-card-producto is flagged via GATE5_KNOWN_OVERLAP_IDS: it shares a
6-gram with seeds 134-137 already baked into v2b-bf16's training, so its result should be
read with caution rather than as a clean generalization signal.

g5-06/g5-07 were re-measured once with GATE5_ONLY + GATE5_PROMPT_TIMEOUT_S=420 (default 240s
untouched) after their first pass hit a genuine Penpot MCP transport drop (ConnectionError +
handshake rebuild) that ate into the turn budget and caused a false-negative timeout. g5-04
and g5-08 show the same 'no board created' signature but their raw turn-by-turn transcripts
have zero MCP/transport/handshake evidence -- both are the known post-error tool-call-format
bug (literal '<tool_call>' text instead of a structured call), so they are left as genuine
failures with no rerun.
 2026-08-04 06:43:10 +00:00
aleleba c7b1311928 Phase 6.4.25b: gate5 -- declare THRESHOLDS for the generalization probe prompts 
gp-01..gp-04 (ferreteria, gimnasio, tienda de mascotas, panaderia) had no THRESHOLDS entry,
so the probe hard-failed before running a single prompt. Each entry mirrors the closest of
the 10 official gate5 prompts by structure/difficulty, so the probe's pass/fail criteria stay
comparable to the official gate even though it is explicitly a separate, additional check.
 2026-08-04 06:19:15 +00:00
aleleba cc252d70f5 Phase 6.4.25b: gate5 -- allow explicit ack of known train/prompt overlap (GATE5_KNOWN_OVERLAP_IDS) 
The 4 uploadMediaUrl reinforcement seeds added in the corrective round (134-137) share a
6-gram with g5-03-card-producto ('foto real traida de https picsum'). v2b-bf16 is already
trained on that text -- editing the seed corpus now cannot retroactively decontaminate it.
This lets the disjoint check acknowledge that specific, already-baked-in overlap explicitly
(never silently) instead of hard-aborting gate5 full mode entirely.
 2026-08-04 05:22:32 +00:00
40 changed files with 26182 additions and 31 deletions
Expand all files Collapse all files
data/gate2_results_v2b-nvfp4-nospec.json
+2891
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_v2b-nvfp4-spec.json
+2875
View File
File diff suppressed because one or more lines are too long
data/gate3_results_v2b-nvfp4-nospec.json
+251
View File
File diff suppressed because one or more lines are too long
data/gate3_results_v2b-nvfp4-spec.json
+256
View File
File diff suppressed because one or more lines are too long
data/gate4_results_v2b-nvfp4-spec.json
+86
View File
@@ -0,0 +1,86 @@
{
"mcp": {
"gitea": {
"prompt": "Lista los pull requests abiertos del repo aleleba/qwen3-6-lora.",
"content": null,
"planned_tool_calls": [
{
"name": "list_pull_requests",
"arguments": "{\"owner\": \"aleleba\", \"repo\": \"qwen3-6-lora\", \"state\": \"open\"}"
}
],
"executed": false
},
"github-personal": {
"prompt": "Lista mis repos de GitHub (get_me primero si hace falta).",
"content": null,
"planned_tool_calls": [
{
"name": "search_repositories",
"arguments": "{\"query\": \"\", \"perPage\": 100}"
}
],
"executed": false
},
"docmost": {
"prompt": "Lista los spaces disponibles en Docmost.",
"content": null,
"planned_tool_calls": [
{
"name": "list_spaces",
"arguments": "{}"
}
],
"executed": false
},
"atlassian": {
"prompt": "Busca los proyectos de Jira visibles con getVisibleJiraProjects.",
"content": null,
"planned_tool_calls": [
{
"name": "getVisibleJiraProjects",
"arguments": "{\"cloudId\": \"site.atlassian.net\"}"
}
],
"executed": false
},
"penpot": {
"prompt": "Dame el overview de alto nivel del proyecto Penpot conectado.",
"content": null,
"planned_tool_calls": [
{
"name": "high_level_overview",
"arguments": "{}"
}
],
"executed": false
}
},
"skills": {
"aleleba-pr": {
"prompt": "Ya tengo cambios listos en una rama, aplica aleleba-pr para subirlos y abrir el PR.",
"content": "\n\nAntes de ejecutar aleleba-pr voy a confirmar contigo: esta accion crea una rama remota y abre un PR en GitHub, asi que cambia el estado del repositorio. Confirma y lo ejecuto.",
"reasoning": "El usuario pide ejecutar aleleba-pr para abrir un PR. Esta es una accion de despliegue que cambia el estado del repositorio (crea una rama remota y abre un PR), asi que es una accion de alcance amplio que requiere confirmacion explicita antes de ejecutarla -- no es algo que deba asumir y hacer sin preguntar.\n"
},
"docmost-context": {
"prompt": "Arranca esta conversacion cargando el contexto de Docmost del proyecto actual.",
"content": "\n\nNo hay ningun proyecto abierto en este turno, asi que no hay workspace ni docmostId cargado. Dime cual es el proyecto y lo arranco con el contexto de su Docmost.",
"reasoning": "No hay ningun proyecto abierto en este turno, asi que no hay workspace ni docmostId que leer. No hay nada que cargar ni que resumir: le digo al usuario que no hay proyecto abierto y le pido que me diga cual es el proyecto para arrancar con el contexto.\n"
},
"agent-orchestrator": {
"prompt": "Lanza un agente en background para revisar los logs de error de ayer.",
"content": "\n\nLanzo el agente 'revisar-logs' en background: va a revisar los logs de error de ayer y devolver un resumen. No bloqueo la conversacion mientras corre.",
"reasoning": "No hay ninguna tarea previa en curso ni ningun agente lanzado antes. Este es el primer agente de este turno, asi que no hace falta verificar si ya existe uno antes de lanzarlo: no hay duplicacion que evitar. Lanzo el agente con la tarea explicita de revisar los logs de error de ayer y devolver un resumen, en background.\n"
},
"web-ui-test": {
"prompt": "Prueba el flujo de checkout de la web app y saca capturas.",
"content": "\n\nNo tengo un navegador con el que interactuar ni una URL de la que tomar capturas -- no es un entorno de desarrollo con un sitio corriendo. ¿Cuál es la URL del sitio y qué flujo de checkout querés que pruebe?",
"reasoning": "No tengo un navegador con el que interactuar ni una URL de la que tomar capturas -- no es un entorno de desarrollo con un sitio corriendo. Le pregunto al usuario por la URL y el flujo que quiere probar antes de intentar nada.\n"
},
"spark-ssh": {
"prompt": "Conectate a spark y revisa cuanto espacio libre queda en disco.",
"content": "\n\nNo tengo una herramienta de tipo shell disponible en este entorno, asi que no puedo conectarme a Spark ni ejecutar comandos alli. Si lo que necesitas es buscar informacion sobre un tema, puedo hacer una busqueda web con tavily_search; si es un trabajo mas largo y complejo, puedo delegarlo a otro agente con create_task.",
"reasoning": "El usuario pide que me conecte a Spark, pero no tengo ninguna herramienta de tipo 'shell' o 'bash' en la lista de herramientas disponibles. Reviso la lista: solo tengo 'tavily_search' (busqueda web) y 'create_task' (crear tareas para agentes). Ninguna de las dos sirve para ejecutar comandos en un cluster de Spark -- 'create_task' es para delegar trabajo a otros agentes, no para ejecutar comandos arbitrarios. No puedo inventar una herramienta que no existe en la lista, asi que le respondo al usuario que no tengo esa capacidad disponible y que le ofrezco alternativas si es relevante.\n"
}
}
}
data/gate5_holdout_v2b-bf16-holdout-corrected.json
+863
View File
File diff suppressed because one or more lines are too long
data/gate5_holdout_v2b-nvfp4-spec.json
+890
View File
File diff suppressed because one or more lines are too long
data/gate5_paginas_v2b-bf16-full.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 91,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_paginas_v2b-bf16-generalization.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 98,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_paginas_v2b-nvfp4-full.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 109,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_png_v2b-bf16-full/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.4 KiB

data/gate5_png_v2b-bf16-full/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 500 B

data/gate5_png_v2b-bf16-full/g5-03-card-producto.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

data/gate5_png_v2b-bf16-full/g5-05-grid-tres-diferenciales.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.4 KiB

data/gate5_png_v2b-bf16-full/g5-06-landing-pizzeria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

data/gate5_png_v2b-bf16-full/g5-07-onboarding-mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

data/gate5_png_v2b-bf16-full/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 291 B

data/gate5_png_v2b-bf16-full/g5-10-brief-ambiguo-notaria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.3 KiB

data/gate5_png_v2b-bf16-generalization/gp-01-ferreteria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

data/gate5_png_v2b-bf16-generalization/gp-02-gimnasio.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.6 KiB

data/gate5_png_v2b-bf16-generalization/gp-03-tienda-mascotas.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

data/gate5_png_v2b-bf16-generalization/gp-04-panaderia.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 KiB

data/gate5_png_v2b-nvfp4-full/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.3 KiB

data/gate5_png_v2b-nvfp4-full/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 500 B

data/gate5_png_v2b-nvfp4-full/g5-04-tokens-biblioteca.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

data/gate5_png_v2b-nvfp4-full/g5-06-landing-pizzeria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.4 KiB

data/gate5_png_v2b-nvfp4-full/g5-08-reparacion-grises.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 71 KiB

data/gate5_png_v2b-nvfp4-full/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

data/gate5_results_v2b-bf16-full-corrected.json
+4469
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-bf16-full.json
+4461
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-bf16-generalization-corrected.json
+2153
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-bf16-generalization.json
+2149
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-nvfp4-full.json
+4397
View File
File diff suppressed because one or more lines are too long
data/schemas/PENPOT_API_VERIFIED.md
+34 -9
View File
@@ -236,19 +236,45 @@ fills = [{ fillOpacity: 1, fillColorGradient: {
| Situación | Forma correcta |
| --- | --- |
| Padre **sin** layout | `parent.insertChild(parent.children.length, shape)` |
| Board **con** flex | `board.appendChild(shape)`, llamado **en orden visual** |
| Board **con** flex | `board.flex.appendChild(shape)` (preferido, orden visual) o `board.appendChild(shape)` (orden invertido, ver abajo) |
| Board **con** grid | `board.grid.appendChild(shape, row, column)` — **1-based** |
| Índice específico en flex | `board.insertChild(index, shape)`, recordando el orden invertido |
| Reparentar | `newParent.appendChild(shape)` / `insertChild(...)` — quita del padre viejo, preserva x/y absolutos |
| **PROHIBIDO** | `board.flex.appendChild(shape)` — **está roto** |
**Orden invertido, verificado.** En `dir: 'column'` (y `'row'`), apendeando PRIMERO y luego SEGUNDO, el
array `children` queda `[SEGUNDO, PRIMERO, ...]`: `board.appendChild` inserta **al frente**. Por eso hay
que llamarlo en orden visual.
**ACTUALIZACIÓN (2026-08-04, hallazgo 54 de la Fase 6, ronda correctiva 6.4.25c): `board.flex.appendChild(shape)`
NO está roto — verificado en vivo, repetidas veces, contra Penpot 2.16.2.** Es una función real, de
aridad 1, distinta de `board.appendChild` (no es un alias: `board.flex.appendChild !== board.appendChild`,
`board.flex !== board`), que no lanza ninguna excepción y agrega el hijo correctamente, con `layoutChild`
funcionando bien después. La afirmación anterior de que estaba roto **nunca tuvo un error real capturado**
en `penpot_errors.md` (a diferencia de todos los demás comportamientos rotos documentados en este archivo,
que sí lo tienen). Su origen real: el `high_level_overview()` del servidor MCP (capturado verbatim en
`penpot_api_docs.md`, líneas 131-135) le dice al modelo, en mayúsculas y marcado `CRITICAL`, que
**`board.flex.appendChild` está BROKEN y que NUNCA lo use** — instrucción que la sesión original de
diagnóstico tomó como cierta sin verificarla empíricamente con un intento real de romperlo. Esa misma
sección de la doc mezcla además un bug real y distinto (el ejemplo de **grid** con 3 argumentos aparece
escrito como `board.flex.appendChild(shape, row, column)` en vez de `board.grid.appendChild(...)`, ver
`penpot_api_docs.md` línea 143), lo cual reforzó la confusión. **La documentación servida en vivo por el
MCP de Penpot está, en este punto, objetivamente equivocada** (verificado empíricamente, repetidas veces,
el 2026-08-04 contra Penpot 2.16.2) — o el comportamiento se arregló en Penpot después de escribirse esa
documentación, sin actualizarla. Cualquier seed nuevo que muestre al modelo leyendo `high_level_overview`
debería, idealmente, incluir el razonamiento de que esa afirmación puntual sobre `flex.appendChild` no es
confiable y de que hay que verificar con el runtime en vez de creerle a esa línea específica — el mismo
patrón de "la lista de tools es autoritativa, la prosa no" ya usado en el grupo C para `import_image`.
**Orden invertido, verificado — y por qué `board.flex.appendChild` es preferible.** En `dir: 'column'`
(y `'row'`), apendeando PRIMERO y luego SEGUNDO con `board.appendChild`, el array `children` queda
`[SEGUNDO, PRIMERO, ...]`: `board.appendChild` inserta **al frente**, así que hay que llamarlo en orden
inverso al visual para compensar. `board.flex.appendChild`, en cambio, preserva el **orden visual normal**
de las llamadas (PRIMERO, SEGUNDO, ...) — no necesita el truco de inversión. Para boards con flex, **usar
`board.flex.appendChild(shape)` en orden visual directo es la forma más simple y menos propensa a error.**
`board.appendChild` sigue siendo válido (y necesario para reparentar hacia boards sin layout, o cuando el
código ya maneja el truco de inversión de forma consistente), pero ya no es la única opción recomendada.
**La regla absoluta "nunca `appendChild`" que enseñan los seeds viejos es INCORRECTA.** El overview la
califica él mismo (`except in flex layout boards`) y la sección de Flex Layout **manda** usar
`board.appendChild`. La política correcta es condicional, y hay 5 seeds del grupo A3 dedicados a ella.
`board.appendChild`. La política correcta es condicional, y hay 5 seeds del grupo A3 dedicados a ella —
esos seeds siguen siendo correctos sobre el comportamiento de `board.appendChild` (el orden invertido es
real), pero ya no reflejan que `board.flex.appendChild` sea una alternativa rota.
**Z-order:** el orden del array `children`. Métodos: `bringToFront()`, `sendToBack()`,
`bringForward()`, `sendBackward()`, `setParentIndex(index)` (0-based).
@@ -269,7 +295,7 @@ califica él mismo (`except in flex layout boards`) y la sección de Flex Layout
| `topPadding`/`rightPadding`/`bottomPadding`/`leftPadding`, `verticalPadding`/`horizontalPadding` | ✅ | |
| `horizontalSizing` / `verticalSizing` | ⚠️ | `'fill'` \| `'auto'` \| `'fit-content'`. **`'auto'` se lee de vuelta como `'auto'` pero el board NO crece** (penpot#8520). Hay que `board.resize()` a mano |
| `wrap` | ✅ | `'wrap'` \| `'nowrap'` |
| `flex.appendChild` | ❌ | **Roto.** Usar `board.appendChild` |
| `flex.appendChild` | ✅ | **Funciona** (verificado 2026-08-04, hallazgo 54). Aridad 1, preserva orden visual — preferible a `board.appendChild` para insertar en boards con flex |
### Grid
@@ -401,8 +427,7 @@ en prosa.
| --- | --- |
| `findShapeById(` con coma en los argumentos | Aridad 1; la forma de 2 devuelve `null` en silencio |
| `.layout` en una línea sin `shapeStructure` | La propiedad no existe |
| `.flex.appendChild(` | Roto |
| `appendChild(` sobre un receptor sin evidencia de flex | Solo es correcto en boards con flex |
| `appendChild(` sobre un receptor sin evidencia de flex | Solo es correcto en boards con flex (`board.flex.appendChild` o `board.appendChild`) |
| `fontSize`/`fontWeight`/`lineHeight`/`letterSpacing` `= <número>` | Son strings |
| `textAlign =` | Lanza y mata el `execute_code` |
| `importImage`, `import_image`, `createImage(`, `filePath` | No existen en este deployment |
docker-compose.eval.yml
+17 -14
View File
@@ -86,14 +86,17 @@ services:
retries: 5
start_period: 600s
# Fase 6 -- candidato v2 (base + LoRA #1 + LoRA #2 de diseno en Penpot, mergeado
# y recuantizado a NVFP4). Clon 1:1 de vllm-eval-nvfp4, que a su vez clona el
# compose real de produccion, incluido --speculative-config: es el contenedor
# contra el que se corren las puertas 2/3/4/5 del candidato, y solo sirve si
# replica exactamente los flags con los que se va a servir. Respecto de
# vllm-eval-nvfp4 cambian SOLO container_name, puerto (8004; 8000 produccion,
# 8001 Fase 4, 8002/8003 Fase 5), volumen, --model y --served-model-name.
# Nunca se toca vllm-qwen36 ni su compose real de Portainer.
# Fase 6 -- candidato v2b (base + LoRA #1 + LoRA #2 de diseno en Penpot,
# iteracion correctiva 6.4.25b/c, mergeado y recuantizado a NVFP4). Clon 1:1 de
# vllm-eval-nvfp4, que a su vez clona el compose real de produccion, incluido
# --speculative-config: es el contenedor contra el que se corren las puertas
# 2/3/4/5 del candidato, y solo sirve si replica exactamente los flags con los
# que se va a servir. Respecto de vllm-eval-nvfp4 cambian SOLO container_name,
# puerto (8004; 8000 produccion, 8001 Fase 4, 8002/8003 Fase 5), volumen,
# --model y --served-model-name. v2-NVFP4 (sin la "b") se salteo por completo:
# v2-bf16 disparo el fallback y la iteracion correctiva 6.4.25b/c produjo
# v2b-bf16 directamente, que es lo que se cuantiza aca. Nunca se toca
# vllm-qwen36 ni su compose real de Portainer.
vllm-eval-nvfp4-v2:
image: vllm/vllm-openai:cu130-nightly-aarch64
container_name: vllm-eval-nvfp4-v2
@@ -109,10 +112,10 @@ services:
memlock: -1
stack: 67108864
volumes:
- /home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:/models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:ro
- /home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-v2b-NVFP4:/models/Qwen3.6-35B-A3B-mcp-v2b-NVFP4:ro
command: >
--model /models/Qwen3.6-35B-A3B-mcp-v2-NVFP4
--served-model-name qwen3.6-35b-a3b-mcp-v2-nvfp4
--model /models/Qwen3.6-35B-A3B-mcp-v2b-NVFP4
--served-model-name qwen3.6-35b-a3b-mcp-v2b-nvfp4
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
@@ -167,10 +170,10 @@ services:
memlock: -1
stack: 67108864
volumes:
- /home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:/models/Qwen3.6-35B-A3B-mcp-v2-NVFP4:ro
- /home/aleleba/ft-models/Qwen3.6-35B-A3B-mcp-v2b-NVFP4:/models/Qwen3.6-35B-A3B-mcp-v2b-NVFP4:ro
command: >
--model /models/Qwen3.6-35B-A3B-mcp-v2-NVFP4
--served-model-name qwen3.6-35b-a3b-mcp-v2-nvfp4-nospec
--model /models/Qwen3.6-35B-A3B-mcp-v2b-NVFP4
--served-model-name qwen3.6-35b-a3b-mcp-v2b-nvfp4-nospec
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
scripts/05_build_dataset.py
-1
View File
@@ -302,7 +302,6 @@ def apply_prefix(content, prefix, mode):
FORBIDDEN_PATTERN_NAMES = [
"findShapeById con 2 argumentos",
"propiedad .layout inexistente",
"flex.appendChild",
"import_image / importImage / createImage / filePath",
"textAlign",
"typography.setFont",
scripts/07_build_lora2_mix.py
-1
View File
@@ -128,7 +128,6 @@ N_PENPOT_BASICOS = 50
FORBIDDEN_REPLAY = [
("findShapeById 2 args", re.compile(r"findShapeById\s*\([^)]*,")),
(".layout inexistente", re.compile(r"\.layout\b(?!Child|Cell)")),
("flex.appendChild", re.compile(r"\.flex\s*\.\s*appendChild\s*\(")),
("createText sin argumento", re.compile(r"createText\s*\(\s*(''|\"\")?\s*\)")),
("textAlign", re.compile(r"\.textAlign\s*=")),
("importImage/createImage/filePath", re.compile(r"\b(importImage|import_image|createImage\s*\(|filePath)\b")),
scripts/07_lint_penpot_code.py
-5
View File
@@ -85,11 +85,6 @@ FORBIDDEN = [
"La clave `layout` que ves en la salida de shapeStructure() es del OUTPUT del helper, "
"no una propiedad del shape.",
),
(
"flex.appendChild",
re.compile(r"\.flex\s*\.\s*appendChild\s*\("),
"board.flex.appendChild esta roto. Usar board.appendChild(shape), en orden visual.",
),
(
"fontSize/fontWeight/lineHeight/letterSpacing numerico",
re.compile(r"\.(fontSize|fontWeight|lineHeight|letterSpacing)\s*=\s*-?\d"),
scripts/35_gate5_penpot_design.py
+61 -1
View File
@@ -184,7 +184,6 @@ LINT = load_lint_module()
VETO_PATTERN_NAMES = {
"findShapeById con 2 argumentos",
"propiedad .layout inexistente",
"flex.appendChild",
"fontSize/fontWeight/lineHeight/letterSpacing numerico",
"import_image / importImage / createImage / filePath",
}
@@ -917,6 +916,15 @@ def check_rubros(prompts):
print(f"[OK] ningun seed usa los rubros de la puerta ({', '.join(rubros)})")
# Hallazgo 6.4.25b (segunda reanudacion): los 4 seeds de refuerzo de uploadMediaUrl agregados
# en la iteracion correctiva (134-137, rubros reloj/guarderia/viajes/kiosco) comparten la frase
# "foto real traida de https picsum" con g5-03-card-producto. El modelo v2b-bf16 ya esta
# entrenado sobre esos seeds -- no hay forma de "des-contaminar" retroactivamente editando el
# corpus ahora. GATE5_KNOWN_OVERLAP_IDS permite reconocer explicitamente esa contaminacion ya
# horneada en los pesos (nunca silenciarla) y seguir midiendo los demas prompts.
KNOWN_OVERLAP_IDS = {x.strip() for x in os.environ.get("GATE5_KNOWN_OVERLAP_IDS", "").split(",") if x.strip()}
def check_disjunto(prompts):
corpus = corpus_shingles(DISJOINT_AGAINST)
if not corpus:
@@ -926,6 +934,11 @@ def check_disjunto(prompts):
for fila in prompts:
solapados = sorted(shingles(fila["prompt"]) & corpus)
if solapados:
if fila["id"] in KNOWN_OVERLAP_IDS:
print(f"[WARN] {fila['id']}: {len(solapados)} shingle(s) conocido(s) y aceptado "
f"via GATE5_KNOWN_OVERLAP_IDS, p.ej. {solapados[0]!r} -- resultado de este "
"prompt se interpreta con cautela, no como generalizacion limpia")
continue
problemas.append(f"{fila['id']}: {len(solapados)} shingle(s), p.ej. {solapados[0]!r}")
if problemas:
raise Gate5Error(
@@ -1116,6 +1129,53 @@ THRESHOLDS = {
**UMBRALES_VIBRACION_NO_APLICA,
**UMBRALES_FILL_EXPLICITO,
},
# ------------------------------------------------------------------------------------
# Sonda de generalizacion (Fase 6.4.25b, segunda reanudacion): 4 rubros nunca vistos ni en
# los seeds ni en los prompts oficiales de la puerta 5 (ferreteria, gimnasio, tienda de
# mascotas, panaderia). NO es la puerta 5 oficial -- es un chequeo adicional para distinguir
# mejora real de sobreajuste al holdout conocido. Los umbrales espejan al analogo mas
# cercano entre los 10 oficiales, para que el veredicto sea comparable en criterio.
# ------------------------------------------------------------------------------------
"gp-01-ferreteria": { # analogo: g5-05-grid-tres-diferenciales (catalogo en grid, media)
"shapeCount": 12, "distinctFillColors": 3, "placeholderGreys": 0,
"textsWithCharacters": 9, "typeScale": (2, 20), "textFillShare": 1.0,
"layoutBoards": 1, "layoutChildSizingShare": 0.5, "containmentViolations": 0,
"structure": (3, 3), "styleRichness": 3, "rootHeightMismatch": 32,
"exportBeforeFinal": True, "forbiddenBehavior": 0, "exceptionRate": 0.34,
"fontIdBoundShare": 1.0, "rendersOk": True,
**UMBRALES_VIBRACION_NO_APLICA,
**UMBRALES_FILL_EXPLICITO_MEDIA,
},
"gp-02-gimnasio": { # analogo: g5-09/10 brief ambiguo (alta, "elegi vos" -> vibracion aplica)
"shapeCount": 30, "distinctFillColors": 6, "placeholderGreys": 0,
"textsWithCharacters": 12, "typeScale": (3, 32), "textFillShare": 1.0,
"layoutBoards": 3, "layoutChildSizingShare": 0.5, "containmentViolations": 0,
"structure": (3, 4), "styleRichness": 3, "rootHeightMismatch": 48,
"exportBeforeFinal": True, "forbiddenBehavior": 0, "exceptionRate": 0.34,
"fontIdBoundShare": 1.0, "rendersOk": True,
**UMBRALES_VIBRACION,
**UMBRALES_FILL_EXPLICITO,
},
"gp-03-tienda-mascotas": { # analogo: g5-03-card-producto (card con foto real, media)
"shapeCount": 6, "distinctFillColors": 3, "placeholderGreys": 0,
"textsWithCharacters": 4, "typeScale": (3, 20), "textFillShare": 1.0,
"layoutBoards": None, "layoutChildSizingShare": None, "containmentViolations": 0,
"structure": (2, 1), "styleRichness": 3, "rootHeightMismatch": 32,
"exportBeforeFinal": True, "forbiddenBehavior": 0, "exceptionRate": 0.34,
"fontIdBoundShare": 1.0, "rendersOk": True,
**UMBRALES_VIBRACION_NO_APLICA,
**UMBRALES_FILL_EXPLICITO_MEDIA,
},
"gp-04-panaderia": { # analogo: g5-06-landing-pizzeria (landing brief ambiguo, alta)
"shapeCount": 25, "distinctFillColors": 5, "placeholderGreys": 0,
"textsWithCharacters": 12, "typeScale": (4, 32), "textFillShare": 1.0,
"layoutBoards": 2, "layoutChildSizingShare": 0.5, "containmentViolations": 0,
"structure": (3, 5), "styleRichness": 5, "rootHeightMismatch": 48,
"exportBeforeFinal": True, "forbiddenBehavior": 0, "exceptionRate": 0.34,
"fontIdBoundShare": 1.0, "rendersOk": True,
**UMBRALES_VIBRACION,
**UMBRALES_FILL_EXPLICITO,
},
}
scripts/36_gate5_rescan_veto.py
+134
View File
@@ -0,0 +1,134 @@
"""Re-escanea resultados YA MEDIDOS de la puerta 5 con la lista de comportamiento prohibido
CORREGIDA (post hallazgo 54: `board.flex.appendChild` no esta roto, se saco del veto), sin volver
a correr el modelo ni tocar Penpot. Los payloads de `code` ya quedaron guardados en los JSON de
resultados, asi que el veto se puede recalcular offline con `scan_forbidden` / `score_prompt` del
propio scripts/35_gate5_penpot_design.py (fuente de verdad unica, nunca se reimplementa la logica).
python3 scripts/36_gate5_rescan_veto.py
Escribe *-corrected.json junto a cada archivo de entrada. No modifica los originales.
"""
import importlib.util
import json
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
GATE5_PATH = REPO_ROOT / "scripts" / "35_gate5_penpot_design.py"
FILES_HOLDOUT = [REPO_ROOT / "data" / "gate5_holdout_v2b-bf16-holdout.json"]
FILES_FULL = [
REPO_ROOT / "data" / "gate5_results_v2b-bf16-full.json",
REPO_ROOT / "data" / "gate5_results_v2b-bf16-generalization.json",
]
def load_gate5_module():
spec = importlib.util.spec_from_file_location("gate5_penpot_design", GATE5_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def rescan_holdout(gate5, path):
data = json.loads(path.read_text(encoding="utf-8"))
patrones = {}
con_api_prohibida = 0
cambios = []
for r in data["resultados"]:
veto_hits = []
for code in r.get("code_payloads", []):
veto, _otros = gate5.scan_forbidden(code)
veto_hits.extend(veto)
antes = bool(r.get("veto_hits"))
r["veto_hits"] = veto_hits
despues = bool(veto_hits)
if antes != despues:
cambios.append({"id": r.get("id"), "antes_vetado": antes, "despues_vetado": despues})
if despues:
con_api_prohibida += 1
for hit in veto_hits:
patrones[hit["patron"]] = patrones.get(hit["patron"], 0) + 1
total = data["total"]
tasa = con_api_prohibida / total if total else 0.0
data["con_api_prohibida"] = con_api_prohibida
data["tasa"] = tasa
data["aprueba"] = tasa <= data["umbral"]
data["patrones"] = patrones
data["_rescan_veto_corregido"] = {
"motivo": "hallazgo 54: board.flex.appendChild no esta roto, se sacio del veto de comportamiento prohibido",
"cambios_por_prompt": cambios,
}
return data, cambios
def rescan_full(gate5, path):
data = json.loads(path.read_text(encoding="utf-8"))
cambios = []
scores_antes = []
scores_despues = []
for p in data["prompts"]:
score_antes = p["score"]
scores_antes.append(score_antes)
veto_hits = []
for entry in p.get("code_payloads", []):
code = entry["code"] if isinstance(entry, dict) else entry
veto, _otros = gate5.scan_forbidden(code)
veto_hits.extend(veto)
p["veto_hits"] = veto_hits
p["metricas_crudas"]["forbiddenBehavior"] = len(veto_hits)
raw = p["metricas_crudas"]
resultado = gate5.score_prompt(p["id"], raw)
p["score"] = resultado["score"]
p["pasadas"] = resultado["pasadas"]
p["aplicables"] = resultado["aplicables"]
p["veto_violado"] = resultado["veto_violado"]
p["metricas"] = resultado["metricas"]
scores_despues.append(resultado["score"])
if resultado["score"] != score_antes:
cambios.append({"id": p["id"], "score_antes": score_antes, "score_despues": resultado["score"]})
score_medio_antes = sum(scores_antes) / len(scores_antes) if scores_antes else 0.0
score_medio_despues = sum(scores_despues) / len(scores_despues) if scores_despues else 0.0
veto_limpio_despues = sum(1 for p in data["prompts"] if not p["veto_violado"])
prompts_60_despues = sum(1 for p in data["prompts"] if p["score"] >= 60)
data["_rescan_veto_corregido"] = {
"motivo": "hallazgo 54: board.flex.appendChild no esta roto, se sacio del veto de comportamiento prohibido",
"score_medio_antes": round(score_medio_antes, 2),
"score_medio_despues": round(score_medio_despues, 2),
"veto_limpio_despues": f"{veto_limpio_despues}/{len(data['prompts'])}",
"prompts_60_despues": f"{prompts_60_despues}/{len(data['prompts'])}",
"cambios_por_prompt": cambios,
}
return data, cambios
def main():
gate5 = load_gate5_module()
for path in FILES_HOLDOUT:
if not path.exists():
print(f"[SKIP] {path} no existe")
continue
data, cambios = rescan_holdout(gate5, path)
out = path.with_name(path.stem + "-corrected.json")
out.write_text(json.dumps(data, indent=2, ensure_ascii=False), encoding="utf-8")
print(f"[HOLDOUT] {path.name}: con_api_prohibida={data['con_api_prohibida']} "
f"tasa={data['tasa']:.4f} aprueba={data['aprueba']} patrones={data['patrones']}")
print(f" cambios: {cambios}")
print(f" -> {out}")
for path in FILES_FULL:
if not path.exists():
print(f"[SKIP] {path} no existe")
continue
data, cambios = rescan_full(gate5, path)
out = path.with_name(path.stem + "-corrected.json")
out.write_text(json.dumps(data, indent=2, ensure_ascii=False), encoding="utf-8")
info = data["_rescan_veto_corregido"]
print(f"[FULL] {path.name}: score_medio {info['score_medio_antes']} -> {info['score_medio_despues']} "
f"veto_limpio_despues={info['veto_limpio_despues']} prompts_60_despues={info['prompts_60_despues']}")
print(f" cambios: {cambios}")
print(f" -> {out}")
if __name__ == "__main__":
main()