aleleba
5f0ddd962c
Ran the full agent loop against vllm-qwen36 on port 8000 (read-only HTTP)
with the Penpot plugin live, before asking for any downtime. Without this
file "it improved" would be a claim rather than a measurement.
Result over the 8 graded prompts: mean score 14.9, zero prompts at or above
60, veto violated on 2 of 8, 66% of execute_code calls raised, and 7 of 8
prompts burned all 14 turns without producing a final message.
The plan predicted production would score near zero on distinct colours and
style richness while producing a high shape count - grey boxes. The shape
count is also zero. On a fresh page it creates nothing at all, so the
failure sits upstream of the grey boxes: the model invents a Figma-shaped
API wholesale and every call throws. From the captured turns:
penpot.currentPage() is a property, not a function
penpot.createRectangle(page, 200, 56) takes no arguments
penpot.createText(page, ...) takes one, the text
penpot.getPageById(...) lives on penpotUtils
fills = [{type:'solid', color:{r,g,b,a}}] is {fillColor, fillOpacity}
shadows = [{type:'drop', x, y, blur, ...}] is {style, offsetX, offsetY}
It then spends the remaining turns querying penpot_api_info without
recovering. So the reported symptom understated it.
Two robustness fixes the run itself forced, both after losing a completed
run to them:
- A ConnectionError does not just drop the request, it can drop the MCP
session, so retrying the same tools/call against a dead session fails
identically every time - which is exactly what the first attempt showed,
four retries and four identical ConnectionErrors. The client now redoes
the handshake before retrying, and that recovered two drops in this run.
- Results are written after every prompt. The first attempt died on prompt
4 and lost the three already measured, which is the expensive data
precisely because it requires production to be up.
78 lines
4.7 KiB
Plaintext
78 lines
4.7 KiB
Plaintext
[OK] los 8 prompts son disjuntos de train/eval/seeds (7472 shingles de referencia)
|
|
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
|
|
[MCP] sonda de conectividad del plugin: {
|
|
"result": {
|
|
"gate5Probe": true
|
|
},
|
|
"log": ""
|
|
}
|
|
|
|
--- g5-01-boton-primario (baja) -> pagina gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z
|
|
[MCP] fallo transitorio en 'tools/call' (intento 1/5): error de transporte contra el MCP de Penpot: ConnectionError. Reintento en 4s
|
|
[MCP] rehaciendo el handshake (la sesion pudo haber muerto)
|
|
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
|
|
[WARN] g5-01-boton-primario: se agotaron los 14 turnos sin mensaje final
|
|
[WARN] g5-01-boton-primario: export_shape no devolvio binario
|
|
score=0.0 (2/14) grises=0 prohibido=7 shapes=0 colores=0 textos=0
|
|
|
|
--- g5-02-navbar-flex (baja) -> pagina gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z
|
|
[WARN] g5-02-navbar-flex: se agotaron los 14 turnos sin mensaje final
|
|
[WARN] g5-02-navbar-flex: export_shape no devolvio binario
|
|
score=17.6 (3/17) grises=0 prohibido=0 shapes=0 colores=0 textos=0
|
|
|
|
--- g5-03-card-producto (media) -> pagina gate5/prod-baseline/g5-03-card-producto/20260730T175753Z
|
|
[WARN] g5-03-card-producto: se agotaron los 14 turnos sin mensaje final
|
|
[WARN] g5-03-card-producto: export_shape no devolvio binario
|
|
score=20.0 (3/15) grises=0 prohibido=0 shapes=0 colores=0 textos=0
|
|
|
|
--- g5-04-tokens-biblioteca (media) -> pagina gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z
|
|
[WARN] g5-04-tokens-biblioteca: se agotaron los 14 turnos sin mensaje final
|
|
[WARN] g5-04-tokens-biblioteca: export_shape no devolvio binario
|
|
score=20.0 (3/15) grises=0 prohibido=0 shapes=0 colores=0 textos=0
|
|
|
|
--- g5-05-grid-tres-diferenciales (media) -> pagina gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z
|
|
[WARN] g5-05-grid-tres-diferenciales: se agotaron los 14 turnos sin mensaje final
|
|
[WARN] g5-05-grid-tres-diferenciales: export_shape no devolvio binario
|
|
score=17.6 (3/17) grises=0 prohibido=0 shapes=0 colores=0 textos=0
|
|
|
|
--- g5-06-landing-pizzeria (alta) -> pagina gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z
|
|
[MCP] fallo transitorio en 'tools/call' (intento 1/5): error de transporte contra el MCP de Penpot: ConnectionError. Reintento en 4s
|
|
[MCP] rehaciendo el handshake (la sesion pudo haber muerto)
|
|
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
|
|
[WARN] g5-06-landing-pizzeria: export_shape no devolvio binario
|
|
score=0.0 (3/17) grises=0 prohibido=5 shapes=0 colores=0 textos=0
|
|
|
|
--- g5-07-onboarding-mobile (alta) -> pagina gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z
|
|
[WARN] g5-07-onboarding-mobile: se agotaron los 14 turnos sin mensaje final
|
|
[WARN] g5-07-onboarding-mobile: export_shape no devolvio binario
|
|
score=17.6 (3/17) grises=0 prohibido=0 shapes=0 colores=0 textos=0
|
|
|
|
--- g5-08-reparacion-grises (alta) -> pagina gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z
|
|
[WARN] g5-08-reparacion-grises: se agotaron los 14 turnos sin mensaje final
|
|
[WARN] g5-08-reparacion-grises: export_shape no devolvio binario
|
|
score=26.7 (4/15) grises=0 prohibido=0 shapes=0 colores=0 textos=0
|
|
|
|
=== Puerta 5 -- calidad de diseno en Penpot (tag=prod-baseline) ===
|
|
|
|
prompt dif score pasa grises prohib shapes colores textos
|
|
g5-01-boton-primario baja 0.0 2/14 0 7 0 0 0
|
|
g5-02-navbar-flex baja 17.6 3/17 0 0 0 0 0
|
|
g5-03-card-producto media 20.0 3/15 0 0 0 0 0
|
|
g5-04-tokens-biblioteca media 20.0 3/15 0 0 0 0 0
|
|
g5-05-grid-tres-diferenciales media 17.6 3/17 0 0 0 0 0
|
|
g5-06-landing-pizzeria alta 0.0 3/17 0 5 0 0 0
|
|
g5-07-onboarding-mobile alta 17.6 3/17 0 0 0 0 0
|
|
g5-08-reparacion-grises alta 26.7 4/15 0 0 0 0 0
|
|
|
|
condiciones de aprobacion:
|
|
[NO ] score medio >= 65 -> 14.9
|
|
[NO ] veto limpio en 8/8 -> 6/8
|
|
[NO ] >= 6/8 prompts >= 60 -> 0/8
|
|
[NO ] landing de pizzeria >= 60 -> 0.0
|
|
|
|
VEREDICTO GLOBAL: NO APRUEBA
|
|
|
|
tiempo total: 15.1 min
|
|
resultados en data/gate5_results_prod-baseline.json
|
|
PNGs en data/gate5_png_prod-baseline/ (la puerta no borra nada: las paginas quedan para inspeccion humana)
|