aleleba/qwen3-6-lora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: aleleba/qwen3-6-lora:3f14aa5cecf4c8b4ce93dbe59597954841fe8346
Branches Tags
aleleba/qwen3-6-lora:master aleleba/qwen3-6-lora:agente-fase6-lora2-penpot
...
compare: aleleba/qwen3-6-lora:master
Branches Tags
aleleba/qwen3-6-lora:master aleleba/qwen3-6-lora:agente-fase6-lora2-penpot
63 Commits
3f14aa5cec ... master
Author SHA1 Message Date
aleleba 4aa5c83481 Merge pull request 'Phase 6: train a second LoRA for real Penpot UI design capability' (#5) from agente-fase6-lora2-penpot into master 2026-08-04 21:36:03 -06:00
aleleba 4d840553d6 Phase 6.4.27: re-measure g5-08-reparacion-grises to complete gate 5 full on v2b-NVFP4 
g5-08 was pending since it fell mid-run when Penpot's plugin disconnected
("No Penpot plugin instances are currently connected"). Re-run in isolation
via GATE5_ONLY, per the established pattern: score 64.7, clean veto, no
placeholder greys. Gate 5 full mode on v2b-NVFP4 is now complete (10/10):
mean score 28.2 (threshold 65), clean veto 6/10, prompts>=60 2/10, pizzeria
landing 47.8 (threshold 60) -- verdict: DOES NOT APPROVE.
 2026-08-05 01:49:59 +00:00
aleleba 66d6c6c410 Phase 6.4.27: gate 5 (holdout + full) on v2b-NVFP4, spec config 
Holdout (60 design prompts, cheap proxy, no live Penpot needed): 5.0%
forbidden-API rate (3/60) -- right at the <=5% threshold, APPROVES.

Full mode (10 official prompts against live Penpot): 9/10 measured.
g5-08-reparacion-grises is still pending -- Penpot disconnected mid-run
("No Penpot plugin instances are currently connected", the real
disconnect signature, not the transient "transport dropped mid-call").
Per the established GATE5_ONLY pattern, this prompt gets re-measured
(not accepted as a genuine failure) once Penpot reconnects; the other
9 results are preserved as-is.
 2026-08-05 01:23:39 +00:00
aleleba f4564bc7a0 Phase 6.4.27: gates 2/3/4 on v2b-NVFP4, spec and nospec 
Gate 2 (tool-calls, 200 held-out): global 95.5% (191/200) on both spec and
nospec, matching v2b-bf16's 95.0%. Per-MCP breakdown diverges: spec regresses
penpot specifically (91.7% vs bf16's 98.3%, outside the 2-point guard) while
preserving atlassian/docmost; nospec preserves penpot (98.3%, matches bf16)
but regresses atlassian/docmost instead. Consistent with risk #10 (MTP draft
head drift) rather than risk #9 (quantization washing out the LoRA delta) --
if quantization itself hurt penpot, nospec should show the same regression,
and it doesn't. Sampling is non-greedy (temperature 0.6), so some of this is
ordinary run-to-run variance rather than a clean systematic signal.

Gate 3 (21-check adherencia): spec 90.5% (19/21), nospec 71.4% (15/21). Both
clear the >=61.9% no-regression bar against the production baseline, but the
gap between spec/nospec is real and worth flagging.

Gate 4 (E2E against real MCPs, spec only): gitea, docmost, and penpot
(high_level_overview) executed successfully on the model's first-shot tool
call. github-personal and atlassian failed on this specific attempt due to
single-shot argument guesses (empty search query; a plausible but wrong
Jira cloudId) -- verified separately that both integrations work fine once
given correct arguments, so this reads as ordinary single-shot tool-use
limitation, not a broken integration.
 2026-08-04 21:04:22 +00:00
aleleba 1c672520a4 Phase 6.4.26: point 8004/8005 eval services at v2b-NVFP4 
The fallback decision landed on v2b-bf16 directly (v2-NVFP4 is skipped),
so the two candidate eval services need to serve the checkpoint that's
actually being quantized. Container names/ports/served-model-name stay
the same, only the volume and --model path move to v2b-NVFP4.
 2026-08-04 18:48:05 +00:00
aleleba cc54edc39d Phase 6.4.25c: re-score existing v2b-bf16 gate 5 results with corrected veto 
New offline script re-scans the code payloads already saved in the gate 5
result JSONs (holdout, full, generalization) using the corrected forbidden-
behavior list -- no live Penpot needed, reuses scan_forbidden/score_prompt
from 35_gate5_penpot_design.py as the single source of truth.

Holdout result changes materially: 3 of the 5 flagged prompts (pd-008,
pd-009, pd-035) were flagged only for flex.appendChild. With that pattern
removed, forbidden-API rate drops from 8.3% (5/60) to 3.3% (2/60), which is
now under the <=5% fallback threshold -- the holdout gate now passes.

Full mode and the generalization probe are unaffected: neither run's
transcripts ever triggered the flex.appendChild pattern, so their scores
are identical before and after.

Original result files are kept untouched; corrected results are written
alongside as *-corrected.json.
 2026-08-04 17:40:26 +00:00
aleleba 7a1577b9d7 Phase 6.4.25c: correct flex.appendChild -- live verification shows it works 
Live probing against Penpot 2.16.2 (repeated, on a dedicated scratch page,
cleaned up afterwards) shows board.flex.appendChild(shape) is a real
function, distinct from board.appendChild, arity 1, throws nothing, and
preserves visual insertion order -- unlike board.appendChild, which still
inserts at the front as documented. The prior "broken" claim traces to the
MCP server's own high_level_overview() text (CRITICAL / BROKEN / NEVER use),
never verified with a live throw, and possibly confused with a real doc bug
that mislabels the grid 3-arg example as board.flex.

Removes flex.appendChild from the lint's FORBIDDEN patterns and from the
gate 5 forbidden-behavior veto, updates PENPOT_API_VERIFIED.md with the
verified facts, and drops the now-obsolete pattern from the replay filters
in 05_build_dataset.py and 07_build_lora2_mix.py.
 2026-08-04 17:40:16 +00:00
aleleba 3496051f72 Phase 6.4.25b: measure generalization probe on v2b-bf16 (ADDITIONAL, not official gate 5) 
4 prompts in domains absent from both seeds and the official gate5_prompts.jsonl (ferreteria,
gimnasio, tienda de mascotas, panaderia). score medio 32.0, veto limpio 3/4, 0/4 prompts >=60.

gp-03-tienda-mascotas violates the veto twice over: it hits both import_image variants the
model has ever tried (penpot.importImage on turn 1, penpotUtils.importImage on turn 2) plus a
numeric fontSize assignment -- the uploadMediaUrl reinforcement seeds from this correction
round did not transfer to an unseen domain. It is also flagged via GATE5_KNOWN_OVERLAP_IDS
(shares a 6-gram with the same seeds as g5-03), so its score is a double caution case.

gp-02-gimnasio hit the same transient Penpot MCP transport drop as g5-06/g5-07 on its first
pass, but the harness's own automatic retry (pass 2/3) recovered cleanly within the default
240s -- no manual timeout override was needed here.
 2026-08-04 06:43:26 +00:00
aleleba bf795bd01c Phase 6.4.25b: measure gate 5 full mode on v2b-bf16 (post-correction) 
10/10 prompts measured. VEREDICTO: NO APRUEBA (score medio 39.8 vs threshold 65, veto limpio
8/10, 3/10 prompts >=60). The flagship g5-06-landing-pizzeria now clears its own threshold
(69.6 >= 60) once a transient Penpot MCP transport drop is ruled out (see g5-06/g5-07 rerun
below) -- a real improvement over v2-bf16's 34.8 on that prompt, but the aggregate gate
criteria still fail. g5-03-card-producto is flagged via GATE5_KNOWN_OVERLAP_IDS: it shares a
6-gram with seeds 134-137 already baked into v2b-bf16's training, so its result should be
read with caution rather than as a clean generalization signal.

g5-06/g5-07 were re-measured once with GATE5_ONLY + GATE5_PROMPT_TIMEOUT_S=420 (default 240s
untouched) after their first pass hit a genuine Penpot MCP transport drop (ConnectionError +
handshake rebuild) that ate into the turn budget and caused a false-negative timeout. g5-04
and g5-08 show the same 'no board created' signature but their raw turn-by-turn transcripts
have zero MCP/transport/handshake evidence -- both are the known post-error tool-call-format
bug (literal '<tool_call>' text instead of a structured call), so they are left as genuine
failures with no rerun.
 2026-08-04 06:43:10 +00:00
aleleba c7b1311928 Phase 6.4.25b: gate5 -- declare THRESHOLDS for the generalization probe prompts 
gp-01..gp-04 (ferreteria, gimnasio, tienda de mascotas, panaderia) had no THRESHOLDS entry,
so the probe hard-failed before running a single prompt. Each entry mirrors the closest of
the 10 official gate5 prompts by structure/difficulty, so the probe's pass/fail criteria stay
comparable to the official gate even though it is explicitly a separate, additional check.
 2026-08-04 06:19:15 +00:00
aleleba cc252d70f5 Phase 6.4.25b: gate5 -- allow explicit ack of known train/prompt overlap (GATE5_KNOWN_OVERLAP_IDS) 
The 4 uploadMediaUrl reinforcement seeds added in the corrective round (134-137) share a
6-gram with g5-03-card-producto ('foto real traida de https picsum'). v2b-bf16 is already
trained on that text -- editing the seed corpus now cannot retroactively decontaminate it.
This lets the disjoint check acknowledge that specific, already-baked-in overlap explicitly
(never silently) instead of hard-aborting gate5 full mode entirely.
 2026-08-04 05:22:32 +00:00
aleleba 8dcd736ba5 Phase 6.4.25b: measure gate 5 holdout mode on v2b-bf16 (post-correction) 
Tasa de API prohibida: 8.3% (5/60), baja de 15% (9/60) en v2-bf16, pero
sigue por encima del umbral de 5% -- NO APRUEBA.

Desglose por patron (v2-bf16 -> v2b-bf16):
  - import_image/importImage/createImage/filePath: 4 -> 1 (-75%, el
    fix que apuntaban los seeds de uploadMediaUrl)
  - fontSize/fontWeight/lineHeight/letterSpacing numerico: 3 -> 1 (-67%)
  - flex.appendChild: 2 -> 3 (empeoro levemente; NO fue el foco de
    esta ronda de seeds correctivos, es el patron ahora dominante:
    3 de los 5 vetos restantes)

Se continua con el resto del runbook (verificacion de Penpot, puerta 5
full, sonda de generalizacion) para tener el cuadro completo antes de
reportar, tal como indico el usuario -- pero este resultado ya senala
que el disparador de fallback documentado en PLAN.md (<=5%) se activa
de nuevo, esta vez por un patron distinto y no cubierto por los seeds
de esta iteracion.
 2026-08-04 00:11:57 +00:00
aleleba 7c67e100c5 Phase 6.4.25b: measure gate 2 (tool-calls) on v2b-bf16 candidate 
Global 190/200 (95.0%), penpot 59/60 (98.3%) vs baseline v2-bf16's
penpot 60/60 (100%) -- 1.67pt regression, within the 2pt guard from
the gate5-full regression-guard rule. PASS.
 2026-08-03 23:32:44 +00:00
aleleba 959404823a Phase 6.4.25b: parametrize gate5 prompts path, add generalization probe 
GATE5_PROMPTS_PATH env var lets the same 17-metric harness (veto system,
MCP retry/page handling) run against a different prompts file. Unset
behavior is unchanged -- the official gate 5 (data/gate5_prompts.jsonl,
10 frozen prompts) stays untouched.

data/gate5_generalization_probe.jsonl: 4 prompts in rubros confirmed
absent from the 138-seed corpus, the 10 gate5 prompts, and the 60-prompt
holdout -- ferreteria, gimnasio, tienda de mascotas, panaderia. Swapped
out "veterinaria" (originally planned) after finding seed 83 is a
substantial design-repair seed for a vet clinic; "tienda de mascotas"
verified clean instead. This is an additional check, separate from the
official gate 5, meant to test whether the correction generalizes across
business types rather than overfitting to the pizzeria case.
 2026-08-03 22:53:07 +00:00
aleleba 20493a99b6 Phase 6.4.25b: gate 1 on v2b-bf16 -- full PASS on all buckets 
Global weighted loss 0.2725 (vs baseline 0.2750, previous v2-bf16 was
0.2734). All 6/6 non-penpot buckets pass the 0.10 regression threshold,
including delegacion_subagentes which had narrowly failed on v2-bf16
(+0.1019 -> now +0.0882). The 13 corrective seeds didn't hurt forgetting.

Also points vllm-eval at the new v2b-bf16 checkpoint for the gate 2/5
re-measurement.
 2026-08-03 22:50:43 +00:00
aleleba 52dd307c39 Phase 6.4.25b: version the corrective LoRA #2 adapter (out/lora-adapter-penpot-v2/) 
171 steps, train_loss=0.6808, eval_loss=0.6348 (comparable to v1's 0.6306),
peak CUDA memory 74.58GB -- identical to v1's run, confirming the 13
corrective seeds didn't change the memory/length envelope. All guards
verified in adapter_config.json: use_rslora=false, use_dora=false,
lora_bias=false, modules_to_save=null, r=32, lora_alpha=64.

v1 adapter (out/lora-adapter-penpot/, commit 363648f) stays untouched as
a rollback point. New OUTPUT_DIR avoided the overwrite guard.

adapter_model.safetensors sha256 verified identical before/after copying
out of the container (root-owned file, fixed ownership via chown inside
the container since this OUTPUT_DIR landed inside the worktree, unlike
v1's which was copied in from outside).
 2026-08-03 22:39:25 +00:00
aleleba 18efe3fa5d Phase 6.4.25b: 13 corrective seeds for gate 5 diagnosis, rebuild training mix 
Diagnosis (audited gate5 full-mode transcripts, PNGs, and holdout payloads):
the dominant failure mode was NOT rubro/content coverage but a tool-call
formatting bug -- after diagnosing an exception, the model often writes its
retry as literal <tool_call> XML text embedded in reasoning instead of a
structured call, so it's silently dropped (9/10 gate5 full-mode prompts hit
this on their final captured turn). Reasoning length before a tool call
correlates with the failure (malformed-turn mean 1895 chars vs 364 for
well-formed turns), while the training mix's own tool-calling turns never
exceed 1630 chars and export_shape turns never exceed 485.

Corrective seeds (group D, grupo 'D', short 1-3 sentence diagnosis +
immediate well-formed retry, to avoid reinforcing long-reasoning risk):
- 9 seeds across fresh rubros (lavanderia, zapateria, optica, jugueteria,
  papeleria, peluqueria, floreria, heladeria, cerrajeria) covering distinct
  real API errors: .color on Text, addFlexLayout on non-Board, findShapeById
  2-arg, createText() no-arg, textAlign, typography.setFont, createBoolean
  null, appendChild no-arg, uploadMediaUrl network rejection.
- 4 seeds reinforcing uploadMediaUrl over the exact penpot.importImage
  variant the model actually invents (distinct from the one existing seed,
  which only taught the penpotUtils.importImage variant) -- rubros:
  relojeria, guarderia infantil, agencia de viajes, kiosco.

Deliberately avoid ferreteria/gimnasio/veterinaria/panaderia: reserved for
the held-out generalization probe (separate from the frozen gate 5).

Added the newly-verified error strings (t.color, penpot.importImage,
addFlexLayout-on-non-Board) to penpot_errors.md -- all captured live from
this phase's own gate 5 full-mode run, not fabricated.

Rebuilt train_lora2.jsonl (901)/eval_lora2.jsonl (99)/calibration_v2.jsonl
from the 138-seed corpus (07_build_lora2_mix.py) and validated
(06_validate_dataset.py, 0 exceptions, 0 secrets, max 3271 tokens).
 2026-08-03 15:44:22 +00:00
aleleba 730e42494f Phase 6.4.25: gate 5 full-mode results on v2-bf16 (verdict: FAIL) 
10/10 prompts measured against the live Penpot plugin. Mean score
43.1 (threshold 65), pizzeria landing 34.8 (threshold 60), 4/10
prompts >=60, veto clean on 8/10 (card-producto forbidden-API,
brief-ambiguo-escuela placeholder-grey). Verdict: NO APRUEBA.

Combined with gate 1 (PASS global, 1 small bucket FAIL non-blocking),
gate 2 (PASS 96%), and gate 5 holdout (FAIL 15% forbidden API), this
closes the fallback-decision measurement set for step 6.4.25.
 2026-08-03 13:33:15 +00:00
aleleba 0cf8bb1df7 Phase 6.4.25: point vllm-eval at v2-bf16 and record gate 2/5-holdout results 
Gate 2 on v2-bf16: PASS, 192/200 = 96.0% (penpot 60/60 = 100.0%).
Gate 5 holdout mode (60 prompts, fallback trigger): FAIL, forbidden-API
rate 15.0% (9/60) vs threshold <=5% -- model still occasionally invents
penpot.importImage(...) for photo/URL prompts.
 2026-08-03 07:31:04 +00:00
aleleba 363648fb5f Phase 6.4.21: version the LoRA #2 adapter (Penpot design capability) 
171 steps (901 examples / 16 grad-accum * 3 epochs, ceil-rounded, not the
168 the plan estimated with floor), trained decoupled on spark against
the merged bf16 checkpoint that already has LoRA #1 folded in. No OOM,
no aborts: train_runtime 24370s (~6h46m, within the ~7.3h projection),
eval_loss 0.6306, peak CUDA 74.58GB, trainable% 0.1220 (matches phase 3
exactly).

adapter_config.json confirms the guarded hyperparameters: use_rslora
and use_dora both false, lora_bias false, modules_to_save null, r=32,
lora_alpha=64 -- the combination 20_merge_lora.py's scaling math
depends on.

Weights never touched the worktree: OUTPUT_DIR was
/workspace/ft-models/lora-adapter-penpot on spark (outside the
worktree, root-owned by the training container), copied out via
`docker exec cat` piped to a non-root file and verified by matching
sha256 (79167dfa...) before landing here. Intermediate checkpoint-*/
directories stay on spark; only the final adapter is versioned, same
as phase 3's out/lora-adapter/, which this leaves untouched.

.gitignore was missing the negation lines for out/lora-adapter-penpot/
despite already documenting that the final adapter should be
committed -- added the same two exceptions that out/lora-adapter/ has.
 2026-07-31 12:43:45 +00:00
aleleba 2781b9eb32 Phase 6.4.20: fix eval OOM by pinning per_device_eval_batch_size 
HF Trainer defaults per_device_eval_batch_size to 8, independent of the
training batch size, and that default was never overridden. During the
eval forward pass (no gradient checkpointing needed there, so none is
applied) an 8-example batch of long sequences materializes full fp32
logits at once and blows the CUDA budget. That was the real cause of
both OOMs hit while calibrating this run (the worst-case-32 smoke run
and the 8-example probe) -- not the training forward/backward, which
measured a stable ~74GB peak across every length from 2808 to 3265
tokens in three separate calibrations.

Fix: new EVAL_BATCH_SIZE env (default 1) wired into
per_device_eval_batch_size. Verified twice after the fix: training on
the 32 longest examples in the corpus (3161-3271 tokens, worst case)
plus a full eval pass over all 99 real eval_lora2.jsonl examples (up
to 3243 tokens) completed with no OOM, peak 72.41GB.

Also adds PerStepMemoryCallback (opt-in via PER_STEP_MEMORY_LOG=1) to
print per-step CUDA peak/reset, which is what let this calibration
attribute the earlier OOM to eval rather than to a specific training
micro-batch under GRAD_ACCUM=16.
 2026-07-31 00:44:54 +00:00
aleleba c24f0ab236 Phase 6.4.20: build a worst-case probe set for the smoke run 
The 32 longest examples of the training mix, 3161 to 3271 tokens each. With
batch 1 and gradient accumulation 16, a two-step smoke run consumes exactly
these 32, so it trains on the worst case the real run will ever see.

Risk #5 in the plan is an OOM discovered hours into the run. Phase 3 peaked
around 104 GB of the 121 GB available with a longest example near 2800
tokens, and this corpus goes to 3271. Rather than argue about whether the
extra 17% of sequence length fits, the smoke run measures it in ten
minutes. If it OOMs it OOMs immediately and cheaply, which is the whole
point of running the probe before the 169-step run rather than after.
 2026-07-30 21:27:33 +00:00
aleleba b4b4365725 Phase 6.3: re-measure the gate 2 and 3 production baselines on the new holdout 
Rebalancing the holdout regenerated all 200 prompts, so phase 5's 192/200
stopped being comparable and every gate2/gate3/gate4 result file in the
repo was from phase 5. These had to be measured with production still up,
before asking for the downtime, or it would have cost the user another
window later just for this.

Gate 2 over the new holdout: 191/200, 95.5% global. atlassian 97.1,
docmost 94.3, gitea 85.7, github-personal 100, penpot 98.3. All five
failures are missing_required and there are zero invented arguments.

That penpot figure is worth keeping in view: production is at 98.3% on
tool-call VALIDITY, so its failure is not in the shape of the call but in
the code it puts inside execute_code. The two are measured by different
gates and only gate 5 sees the second.

Gate 3: 61.9%, which is 13 of 21. Not comparable to phase 5's 10/10, since
that run had 10 checks and this one has 21 - the 11 new content checklists
for the non-obvious conventions of the other MCPs are what risk #12 in the
plan called the most likely invisible regression. Production already fails
7 of them, so they have headroom rather than being a formality.

Gate 4 needs no re-baseline, verified rather than assumed: grep for
holdout_prompts in 34_gate4_e2e.py returns 0, so the regenerated holdout
does not reach it and its phase 5 result stands. It also needs all five
MCPs live, including the Penpot plugin, which closes when downtime starts.

Also trims the one seed that exceeded the token ceiling. The corpus now
validates clean against the production tokenizer with preserve_thinking on:
1000 of 1000 rows, zero exceptions, zero secrets, p50 1858, p90 3021,
max 3271 against a 3300 limit. That check earning its keep is the reason
06_validate_dataset.py was changed to exit non-zero instead of printing
[FILTERED] and moving on.
 2026-07-30 21:25:58 +00:00
aleleba 3919b6baff Phase 6.3.17: close the production baseline with three repetitions 
Ten prompts measured three times against vllm-qwen36, all with the same
23-metric harness. Three earlier result files are kept but excluded from
the consolidation and named for why: one used 14% of the server's
instructions block, two predate the audit-root fix and report zero shapes
by construction.

The consolidated numbers correct the previous report, which was wrong.
Production does not fail to draw. It creates structure and text - up to 40
shapes and 24 texts on the ambiguous-brief prompt - and then fails to
colour any of it. Distinct fill colours run 0 to 1 across all thirty
measurements.

Mean score per repetition: 6.79, 8.25, 0.00. No prompt reaches 60 in any
repetition.

Reporting the veto rate separately from the score turned out to matter more
than expected, so it is now the headline number alongside it. Of thirty
measurements, 21 violate the forbidden-API veto and 15 use nothing but
Penpot's three default colours; only 6 are veto-free. The score alone
collapses those into a zero that cannot distinguish "broke an API rule"
from "made an ugly design", and the two need different fixes.

That also explains why single measurements looked stable: the score is
pinned at zero by the veto, not by model consistency. All the real variance
sits in the prompts that do not veto, where scores swing 20 to 35 points
between identical runs. So the repetitions matter more for the trained
model, which should land in that non-veto regime, than for this baseline.
The eval window at the end has to budget for three repetitions on that side
too, or the comparison is asymmetric.
 2026-07-30 21:01:59 +00:00
aleleba 869b00c924 Phase 6.3.17: make gate 5 measure all ten prompts in one resilient run 
The Penpot MCP keeps going down and each window where it works is
expensive - three of the user's windows were spent on runs that aborted
partway. The run is now built to finish rather than to be correct about
why it stopped.

One prompt failing no longer ends the run. Failures are collected and
retried on a later pass, up to three passes, with a clean handshake and a
probe between them. Even an unexpected exception is caught per prompt,
because one bug in one prompt must not take down the other nine. The
opening probe no longer aborts either: a plugin that does not answer now
may answer on pass two, and aborting there throws away the whole window
for a transient state.

Two time budgets bound it: four minutes per prompt, checked before every
turn of the agent loop, and forty-five minutes of wall clock. Both are env
vars. A hang used to mean waiting forever; now it costs one prompt and the
run continues.

Results are written after every prompt, so a crash costs at most the
prompt in flight, and the JSON is only marked complete when all ten have a
score.

The run also inventories the accumulated gate5 pages at the end and writes
the list to disk. It does not delete them: the pages this run created are
already emptied after their PNG is exported, but the ones from earlier runs
are in the user's own file, so that is offered rather than assumed.
 2026-07-30 20:27:46 +00:00
aleleba ad62277f14 Phase 6.3: stop the gate depending on penpot.root, and stop blaming the plugin for its own bugs 
The setup returned penpot.root.id and the run aborted when it came back as
an empty string on the second prompt of a batch. penpot.root is the root of
the ACTIVE page, and after createPage plus openPage it need not have caught
up yet - a race the page-emptying code introduced, since that leaves the
emptied page active. The value was never useful anyway: every page shares
the same root id, so it identified nothing. Setup now returns only pageId,
which is unique and stable, and both the audit and the cleanup use it.

The diagnostic message is the second half of the same mistake. It printed
"PLUGIN DEGRADADO, ask the user to reload the browser" whenever any prompt
went unmeasured, regardless of why - so it said that for a failure that was
entirely the gate's own. A message that sends the user to reload their
browser when the bug is mine costs both of us time. It now classifies on
the error text: task timeouts and transport drops point at the plugin,
anything else points at the gate and says so explicitly.

Also closes two evaluation leaks the gate's own pre-flight caught, both in
the seeds rather than the gate prompts, since the prompts have to stay as a
real user would write them:
- A seed shared the 6-gram "la home de una escuela de" with gate prompt 9.
  My first fix was overwritten by a subagent still writing the file, which
  is why it reappeared.
- A seed used the same business as gate prompt 9 - a music school - without
  sharing any 6-gram. Shingles cannot see that: two texts describe the same
  business without sharing words. Training on the domain we then evaluate
  inflates the result invisibly. So the gate now also checks that no seed
  uses any of the gate's business nouns, listed explicitly.

The training mix was rebuilt: it had been assembled before the 20
ambiguous-brief seeds existed, so training on it would not have used the
corpus that was audited. The ambiguous-brief class gets its own mix portion
rather than being folded into design, because diluted across 76 design
seeds it would be at the mercy of a ratio, and that is the class the user
named as the main painpoint. 125 seeds, 168 unique payloads, 446 distinct
user prompts, 901 train and 99 eval.
 2026-07-30 20:07:40 +00:00
aleleba 387b237ee5 Phase 6.3.17: run gate 5 in batches and stop burning prompts on a degraded plugin 
Measured across three consecutive runs: the Penpot plugin reliably handles
5 or 6 heavy prompts and then degrades to 30-second timeouts on createPage,
always with the same shape - the first few work, the rest fail in setup
without exception. That is not random flakiness.

Prompts now run in batches of 4 with a clean handshake between batches,
giving the server a recovery point before the deterioration sets in. A
setup failure no longer burns the prompt as measured-with-score-None: it
goes on a pending list, the batch stops, and the run exits 3 with the exact
GATE5_ONLY line to resume. Insisting past the first timeout only spends
pages and dirties the JSON, since once the plugin starts timing out the
rest fail identically.

On the 30-second timeout the user asked to raise: it is the MCP server's
own limit on a plugin task, not a client timeout, so it cannot be raised
from here. What can be done is not to approach it. generateStyle with
includeChildren plus generateMarkup serialise the whole subtree and are by
far the most expensive part of the audit, so above 400 nodes they are
skipped and rendersOk becomes not-applicable rather than risking the entire
audit - and with it the prompt's measurement - timing out. That required
fixing the scoring too: a metric whose VALUE is None is now not-applicable,
like one whose threshold is None. Counting "could not measure" as a failure
would have penalised exactly the large designs the gate is meant to reward.
 2026-07-30 19:39:49 +00:00
aleleba f63ff6830d Phase 6.3.17: fix the audit root, add fill metrics, and stop the gate degrading the file 
Three defects, the first found by inspecting the user's Penpot file live
while the gate reported something else.

The audit resolved the page root with penpotUtils.findShapeById(rootId).
Every new Penpot page shares the same root frame id,
00000000-0000-0000-0000-000000000000, and findShapeById searches globally,
so that lookup always returned the root of the FIRST page in the file - the
user's empty one - rather than the page the gate had just created. That is
why the baseline reported shapeCount 0 on all ten prompts while the file
actually held 28 shapes and 20 texts. It resolves by pageId now, which is
unique.

With that fixed the baseline reproduces the reported symptom exactly rather
than something worse: the button gets 4 shapes, all grey; the navbar gets
10 shapes and 6 real texts with a single distinct fill colour. Production
does create structure and text. What it never manages is to apply colour -
it creates the shapes in a call that works, then sets fills in a later call
using Figma syntax, that call throws, and the shapes keep the #B1B2B5
default. So "creates grey boxes" was accurate and "creates nothing" was my
measurement error.

Two metrics now capture that directly, since neither distinct-colour counts
nor placeholder-grey counts see it - #FFFFFF and #000000 are not mid greys,
so a design where every fill is a default passes both. explicitFillShare is
the share of filled shapes whose colour is not one of Penpot's three
defaults, thresholded by difficulty (0.80 high, 0.65 medium, 0.50 low,
since a small artefact's structural neutrals weigh heavily in a ratio). And
onlyDefaultColors is a veto: true when the whole palette is those three.
White and black stay legitimate when chosen - the distinction is
co-presence, not the hex. They are only suspicious when they are all there
is; alongside chosen brand colours they also count as neutrals in
paletteStructured.

resumen() crashed adding None scores from failed prompts. Beyond the crash,
an unmeasured prompt must not average in as a zero: "could not measure" and
"the model did it badly" are different, and averaging them would have made
a mid-run plugin outage look like cheap quality. They are reported
separately and any missing prompt makes the verdict invalid, because a
baseline with 5 of 10 measured is not a baseline.

The gate now empties its own page after exporting the PNG. Across three
runs the plugin reliably handled 5 or 6 heavy prompts and then degraded to
30-second timeouts on createPage - that is not random flakiness, it tracks
the file growing by one page per prompt, so the gate was manufacturing its
own failure. The evidence that matters is the PNG plus the JSON metrics,
not the live page. A page is kept only when its PNG failed, so there is
something to inspect. Exit code 3 now distinguishes "plugin degraded
mid-run, reload it and resume these ids" from "plugin not connected".
 2026-07-30 19:35:42 +00:00
aleleba 8094183940 Phase 6.3.17: fix a harness fidelity bug and add a vibrancy metric 
The first baseline measured production with only 2290 of the 16392
characters of the server's instructions block - 14%. The missing 86% is
exactly the API grounding: Core Shape Properties and Methods, Layout
Systems, Text Elements, and The penpot and penpotUtils Objects, which is
where insertChild, resize(), the layouts and penpotUtils are documented.

That was worth catching, because the discrepancy had a visible signature:
the measurement said production creates nothing, while the user's real
Claude Code session produced grey boxes, i.e. shapes greater than zero.
When a harness and reality disagree, the harness is the first suspect. In
phase 5 a low max_tokens manufactured an apparent regression the same way.

The gate now injects the full document, minus the trailing "You have hereby
read the Penpot High-Level Overview" line, which is framing of the tool
response rather than part of the instructions block and would otherwise
tell the model it had already read something.

The finding survives the fix. Across the five prompts measured cleanly
under the corrected condition, shapeCount is still zero on every one. So
the API invention is not an artefact of withholding documentation from the
model - it happens with the documentation present.

Also adds the vibrancy requirement the user raised as first-class scope:
given an ambiguous brief the model must choose and justify a palette rather
than ask or fall back to defaults. Neither distinctFillColors nor
placeholderGreys distinguishes a vibrant palette from a muted but
technically non-grey one, so four metrics are added: chromaticFills,
meanChromaticSaturation, paletteStructured (a dominant brand hue, an accent
at least 30 degrees away, and neutrals), and finalMessageListsHex, because
a palette chosen in silence cannot be adjusted by the user.

The saturation floor of 45 is derived, not asserted: measured over the 325
non-neutral fills of this phase's hand-authored corpus, median HSL
saturation is 75, p25 is 48 and p10 is 35. A floor of 45 sits just under
the first quartile and is cleared by 79% of those fills, so it is a floor
the target behaviour already clears rather than an aspiration. The
lightness band of 15 to 85 excludes near-blacks and near-whites, which can
compute as highly saturated while reading as neutral.

Gate prompt 6 becomes the user's literal failing sentence, and two
ambiguous-brief prompts are added. One of them had to be re-domained after
the disjointness check found it shared a 6-gram with a seed - the check
fails on a single shared shingle, which is what makes it useful.

Results so far are partial: prompts 1-5 measured cleanly, 6 has a timed-out
audit and 7-10 hit the MCP outage, so those get re-measured. Both runs are
kept, the 14% one renamed to record what it was.
 2026-07-30 18:38:24 +00:00
aleleba 5f0ddd962c Phase 6.3.17: measure the gate 5 baseline against production 
Ran the full agent loop against vllm-qwen36 on port 8000 (read-only HTTP)
with the Penpot plugin live, before asking for any downtime. Without this
file "it improved" would be a claim rather than a measurement.

Result over the 8 graded prompts: mean score 14.9, zero prompts at or above
60, veto violated on 2 of 8, 66% of execute_code calls raised, and 7 of 8
prompts burned all 14 turns without producing a final message.

The plan predicted production would score near zero on distinct colours and
style richness while producing a high shape count - grey boxes. The shape
count is also zero. On a fresh page it creates nothing at all, so the
failure sits upstream of the grey boxes: the model invents a Figma-shaped
API wholesale and every call throws. From the captured turns:

  penpot.currentPage()                     is a property, not a function
  penpot.createRectangle(page, 200, 56)    takes no arguments
  penpot.createText(page, ...)             takes one, the text
  penpot.getPageById(...)                  lives on penpotUtils
  fills   = [{type:'solid', color:{r,g,b,a}}]   is {fillColor, fillOpacity}
  shadows = [{type:'drop', x, y, blur, ...}]    is {style, offsetX, offsetY}

It then spends the remaining turns querying penpot_api_info without
recovering. So the reported symptom understated it.

Two robustness fixes the run itself forced, both after losing a completed
run to them:
- A ConnectionError does not just drop the request, it can drop the MCP
  session, so retrying the same tools/call against a dead session fails
  identically every time - which is exactly what the first attempt showed,
  four retries and four identical ConnectionErrors. The client now redoes
  the handshake before retrying, and that recovered two drops in this run.
- Results are written after every prompt. The first attempt died on prompt
  4 and lost the three already measured, which is the expensive data
  precisely because it requires production to be up.
 2026-07-30 18:14:03 +00:00
aleleba c9792c5c40 Phase 6.3: rewrite the Penpot seed corpus and build the LoRA #2 mix 
Replaces the 41 old Penpot seeds with 105 new ones. The old set taught
three API forms that do not exist - findShapeById(page, id), shape.layout,
and createText() with no argument - and 21 of the 41 used the first one, so
patching was never an option: keeping them would mean fine-tuning against
the correction. The valuable lessons were re-founded on the real API
instead (the reversed children array in flex, persisting ids in storage,
never logging what you also return).

105 rather than 96 because nine multi-section compositions are split into
two trajectories each: the first builds the skeleton and persists ids,
palette, scale and helper functions in storage, the second recovers them
and fills the sections. That was forced by the 3000-token ceiling, but it
is better pedagogy anyway, and it is what execute_code's own description
asks for. It also paid for itself: the helpers cost ~600 chars once instead
of twice, and the skeleton call needs no export_shape, which freed the
budget to carry the verbatim system block.

Measured against the linter with the corpus-wide thresholds active:
143 unique code payloads (the old set had 36), 32% of seeds carrying the
server's system block verbatim (the old set had none), and every coverage
category met where the old set had zero addGridLayout, zero shadows, zero
uploadMediaUrl and zero layoutChild.

The flagship changed domain from pizzeria to an empanada shop. Gate 5's
prompt 6 is the exact production failure, and the seed had been written to
the same wording - a literal substring of the gate prompt, sharing two
6-gram shingles where the gate's disjointness check fails on one. Training
on the prompt we then evaluate would make gate 5 measure memorisation. The
real pizzeria prompt still runs in the human acceptance test. The same
check also caught an onboarding seed sitting too close to gate prompt 7.

Mix: 1000 examples split 90/10, giving 901 train and 99 eval. The mix is
1000 rather than 900 because 900 has to be the post-split train size: 900/16
= 56.25 steps per epoch x 3 = 168 steps, deliberately paired with phase 3's
166 so the optimiser trajectory length is comparable. Building 900 and then
carving out eval would have left 811 and 152 steps, silently breaking that
pairing.

Two guards in the builder had to be corrected against the real corpus:
- The forbidden-pattern scan now reads only `code` payloads. Scanning the
  serialised example flagged the corrective seeds themselves - the one that
  opens with the user asking "Importá esta imagen con import_image", the one
  quoting the overview line that mentions import_image verbatim, the one
  explaining that board.layout does not exist. They name the wrong API
  precisely in order to teach against it.
- The exemption for error-recovery seeds is derived from content, not from
  the mix portion: a forbidden pattern is allowed only where its tool result
  is a real error string and a later payload does the same thing without it.
  Keying on the portion broke as soon as an error-recovery seed lived in
  group A1, where the findShapeById arity seed naturally belongs.

Validated with the production tokenizer at preserve_thinking=true: p50 2466,
p90 2988, max 3250 tokens over the 105 seeds. Ten sit just above 3000, so
MAX_TOKENS goes to 3300 for the run and the smoke run will train on the 32
longest examples specifically, turning the peak-memory question into a
ten-minute measurement instead of a risk discovered hours in.
 2026-07-30 17:50:30 +00:00
aleleba c65d309719 Phase 6.4: make the gates fail when they cannot verify something 
A code review found seven ways these gates could pass green with something
actually wrong. All are the same family: a missing value was treated as OK.
The rule now written into all three files is that absent is not OK, absent
is "could not verify", and that either fails or is reported as an explicit
SKIP - it never slips through as green.

30_eval_suite.py:
- A bucket with no baseline of its own fell back to the global 0.2750 and
  printed it in a column headed "baseline", as if it were that bucket's
  number. Measured against the real eval.jsonl buckets: negativos going
  from 0.12 to 0.33 is a real +0.21 regression, but the computed delta was
  +0.055 and it PASSED; manejo_errores sitting unchanged at 0.42 produced
  a fabricated +0.145 FAIL that would have discarded a healthy candidate
  mid-downtime. Now such buckets print SKIP and the verdict reports how
  many went unverified.
- "VEREDICTO: FAIL" exited 0, so a runbook chaining the gate into
  quantization would have carried on to write 24 GB. Now exits 1.
- A typo in BASELINE_BUCKET_LOSSES silently matched nothing; now aborts.
- The penpot exemption is labelled honestly: those 11 rows are pre-existing
  LoRA #1 tool-calling, not new capability, so gate 1 has no regression
  coverage there and the log says so.

20_merge_lora.py dry-run (merge path untouched, verified by AST diff):
- adapter_config.get("use_rslora", False) meant a missing key passed AND
  the log printed use_rslora=False, asserting it had checked something that
  was never there. A different PEFT version omitting a key was enough.
- lora_bias was not checked at all, only bias. They are different fields:
  lora_bias puts a bias inside lora_B, which W + scaling * (B @ A) ignores.
- The 620 keys were printed but never asserted, so an adapter with extra
  tensors printed "310 + 310 = 930" and passed.
- rank_pattern/alpha_pattern were not checked. They set r per module, so
  scaling is not uniformly alpha/r while both the dry-run and the merge
  apply a single 2.0 to all 310 tensors.
- A missing family was invisible: swap linear_attn for 150 mlp.gate targets
  and the total is still 310, no norm is zero because the family is simply
  gone, and it passed. Now presence and per-family counts are asserted,
  derived from the real adapter: linear_attn 150, shared_expert 120,
  attention_qkvo 40, otros 0.
Verified against seven synthetic adapters plus the real phase 3 one; only
the correct adapter passes.

21_quantize_nvfp4.py (recipe and oneshot untouched): the calibration cache
now carries a provenance.json recording the training file's sha256, the
recipe numbers and the bucket distribution, and loading aborts on mismatch.
This is the phase's number one risk and it had no mechanical defence: the
phase 5 cache on disk has exactly 512 rows, the same as the v2 recipe, so
the only existing check could not tell them apart and reusing it would have
calibrated with zero design data and washed out the new capability
silently. Verified: that cache now aborts.

gate 5: retry transport failures against the Penpot MCP, which drops
connections mid-call intermittently (seen before in phase 4's gate 4).
Without it a blip on prompt 6 of 8 kills a whole run and reads like a model
failure. PluginNotConnected is deliberately not retried - that is a real
state of the world. Also unwrap the {"result":..., "log":...} envelope the
server wraps execute_code returns in; the gate was reading keys off the
outer object and rejecting a valid page setup.
 2026-07-30 17:37:46 +00:00
aleleba 8ea4572edd Phase 6.3: add gate 5, design quality in Penpot 
Unlike gates 2-4 this one needs a real agent loop - model, tool call, live
MCP, result, up to 14 turns - because design quality only exists after the
code executes. It talks to vLLM over the OpenAI API and to the Penpot MCP
over HTTP (initialize, notifications/initialized, tools/list, tools/call),
handling both application/json and text/event-stream responses. Endpoints
and credentials come from env with no defaults and are never printed or
stored; requests errors are reduced to the exception type because the
requests message embeds the URL.

Eight graded prompts, a fresh page per prompt named gate5/<tag>/<id>/<ts>,
and the gate never deletes anything. The 17 metrics are computed by an
audit payload the gate injects, not the model. placeholderGreys and
forbidden behaviour are veto metrics: any hit scores that prompt 0.

Two things worth calling out.

The forbidden-pattern regexes are imported from 07_lint_penpot_code.py
rather than duplicated, and the gate runs those same regexes over its own
setup and audit payloads at startup - a gate that violated the API it is
grading would be measuring its own bug.

The holdout mode had a silent failure that is exactly the kind this phase
exists to catch: with the endpoint down, all 60 generations failed, each
entered the denominator with zero forbidden patterns found, and the gate
reported 0% forbidden API and APPROVED. Since that number is the fallback
trigger, a false pass there would have launched the quantization run.
Request errors are now counted separately, never enter the denominator,
and block approval outright.

Known issue, resolved separately: gate prompt 6 is the exact production
failure ("hazme una landing page de una pizzería con colores vibrantes"),
and the flagship B6 seed was written to the same wording. Shingle overlap
measures 20%, under the 34% threshold, but the seed prompt is a literal
substring of the gate prompt - the threshold is too loose for prompts this
short. The seed's domain gets changed rather than the gate's, so gate 5
measures transfer instead of memorisation; the real pizzeria prompt still
runs in the human acceptance test, which is the criterion that decides.
 2026-07-30 17:18:01 +00:00
aleleba a60d0751cf Phase 6.3: fix the augmentation, close the holdout leak, stop rewarding invented parameters 
Dataset build (05, 06):
perturb_value is gone. It rewrote only tool_calls.arguments and left the
tool results and the final answer saying something else, which is how
data/train.jsonl ended up with 30 self-contradictory examples where the
call says issue_number 82 and the answer says issue #77. Variation now
comes from hand-written meta.paraphrases, or from meta.variation applied
atomically across every field of the example at once. Nothing is
substituted unless the seed declares it: guessing which number in a string
is safe to change is what produced the contradictions in the first place.
Prefix injection survives only as a fallback and only where the verb form
can actually be conjugated, and there is a hard assert that no user turn
matches the broken "Necesito que ¿Podés..." shape that 68 v1 prompts had.
The penpot bucket is exempt from substitution entirely, since its payloads
are code. Also asserts the bucket cannot collapse (verified: the old seeds
give 320 rows from 83 unique trajectories and the build now fails) and
scans for forbidden API patterns by importing them from the linter, so
there is one source of truth.

06 now actually exits 1 on over-length rows. It printed [FILTERED],
incremented a counter, and left the row in the file, which 10_train.py
then trained on since it has no max_seq_length and batch 1.

Gate 2 (32): reject any argument key absent from the schema, as its own
failure category. It only checked required fields, so an invented scale or
filePath passed - the gate was actively rewarding the exact behaviour this
phase removes. Verified: export_shape with scale=2 now fails as
unknown_argument, while a valid call still passes.

Holdout (31, 35): rebalanced to penpot 60 / 35 each, added 20 real design
templates, and replaced the full-string equality check with 6-gram
shingles. Measured: a light paraphrase of a train.jsonl prompt scores 43%
overlap and now fails the build, where the old check let it through at
"not equal". Value pools are asserted disjoint from the corpus. The
"2x resolution" template stays, relabelled as an invented-argument probe
now that gate 2 can detect one; the createBoolean template stays because
the API is real and the new B2 seeds teach it. Also dedupes: the old
holdout had 15 duplicate prompts out of 200, i.e. 15 wasted measurements.

Note: rebalancing the holdout means the 192/200 gate 2 baseline from phase
5 no longer applies to it, so that baseline has to be re-measured against
production on the new file before it can be compared to.

Gate 3 (33): 11 content checklists for the non-obvious conventions of the
other MCPs - GFM table separators in Docmost, the update_page staleness
retry, commit message shape, never merging the PR, dict-not-XML tool
arguments. That is the most likely regression no gate currently covers.

Mix builder: added the anti-collapse guard, so 420 new-portion rows that
are really 96 trajectories repeated cannot pass unnoticed.
 2026-07-30 17:16:05 +00:00
aleleba 19eb50f351 Phase 6.3: add the LoRA #2 mix builder and refine the linter 
07_build_lora2_mix.py assembles train_lora2.jsonl (900), eval_lora2.jsonl
and calibration_v2.jsonl from the new Penpot seeds plus a filtered replay
sample of data/train.jsonl. It never writes data/train.jsonl or
data/eval.jsonl: those are the provenance of the model in production and
the gate 1 baseline, and regenerating them is not idempotent anyway, since
stratified_split shuffles one RNG over the concatenated list, so touching
the penpot bucket reshuffles every other bucket's split too.

Two things worth flagging in the mix:

The 45 "corrected penpot basics" the plan lists inside the replay portion
do not come from data/train.jsonl. 21 of its 41 penpot seeds teach
findShapeById(page, id) and 5 use shape.layout, so sampling that bucket
would re-teach the exact bug this phase removes; the forbidden-pattern
filter would drop them anyway. They come from the new corpus instead. This
is a conscious deviation from the plan text and is recorded in the
docstring.

Variation comes only from hand-written meta.paraphrases, never from
automatic value substitution. That is the v1 lesson: perturb_value rewrote
only tool_calls.arguments and left the tool results and the final answer
saying something else, producing 30 self-contradictory examples. A
perturbed Penpot code payload is just broken code.

Linter fixes, both false positives found by running it against the real
corpus:
- flex evidence for a bare appendChild is now scoped to the whole seed
  rather than the single payload. A multi-call seed builds the flex board
  in call one and stashes helpers in storage, so by the time call two does
  main.appendChild(...) neither addFlexLayout( nor .flex appears in that
  payload. The old scope flagged exactly the storage-persistence pattern
  that execute_code's own description asks for.
- a grey hex is a problem when it is applied, not when it is searched for.
  The repair seeds have to name the greys they are about to replace, so
  greys are allowed in that group inside a comparison context.
 2026-07-30 17:11:36 +00:00
aleleba 63da20c031 Phase 6.2: pin the training environment and make 10_train.py configurable 
The training container from phases 3-5 no longer exists and nothing in the
repo pinned its versions, so a rebuild could silently change either the
checkpoint key conversion (breaking adapter naming) or the assistant-mask
behaviour (training on system/user/tool tokens). requirements.train.txt
pins what matters and documents the two-phase install: llmcompressor
declares torch>=2.10.0 and the NGC image ships the 2.10.0a0 pre-release,
which pip's resolver reads as older, so it goes in with --no-deps.

Pre-flight verified against the merged bf16 checkpoint on spark:
01_inspect_modules.py prints model.layers.0.linear_attn.*, config.json is
sha256-identical to the base (93a4693f...), and the index keysets match
exactly (1045 tensors, 690 under model.language_model.layers.*, 0 under
model.layers.*). So PEFT will name adapter #2 the same way it named #1 and
ADAPTER_TO_CHECKPOINT_PREFIX in 20_merge_lora.py applies unchanged.

10_train.py: every path and hyperparameter moves to an env var, with the
phase 3 values as defaults so a bare run still reproduces phase 3 exactly.
Adds three guards that each cover a specific silent failure:
- abort if OUTPUT_DIR already holds an adapter, unless ALLOW_OVERWRITE=1.
  OUTPUT_DIR was hardcoded to out/lora-adapter, which is the provenance of
  the model currently in production.
- MAX_TOKENS aborts rather than truncates. There was no length filter at
  all, so one long design trajectory would blow the memory budget hours
  into a run; truncating would be worse, since it would silently cut
  assistant targets.
- assert use_rslora/use_dora/bias/modules_to_save. rsLoRA scales by
  alpha/sqrt(r), so an adapter trained with it would merge at 2.0 where
  11.3 belongs and pass every assertion in the merge script.
Also adds a config banner, a token-length histogram, and a per-bucket
assistant-mask ratio report.

New 07_lint_penpot_code.py hard-fails on the forbidden API patterns,
placeholder greys, fabricated penpot_api_info results, toy-shaped ids and
per-category coverage shortfalls. Error-recovery seeds legitimately need
the wrong pattern, so the exemption is derived mechanically rather than
declared by hand: a payload may contain a forbidden pattern only if its
tool result is a real error string from the allow-list and a later payload
in the same seed does the same thing without it.

Run against the 41 existing seeds it reproduces the diagnosis exactly:
110 problems, 36 unique payloads, 0% system messages, zero coverage of
addGridLayout/shadows/uploadMediaUrl/layoutChild, fabricated docs and
toy ids.
 2026-07-30 17:01:41 +00:00
aleleba d9629c44ae Phase 6.1: capture verified Penpot API ground truth for the LoRA #2 dataset 
The 41 existing Penpot seeds contain hand-fabricated penpot_api_info and
high_level_overview tool results that assert facts the server never said,
which is how the model learned an API that does not exist. This adds four
schema files that make the seed corpus mechanically verifiable against the
real server instead.

- penpot_api_docs.md: 34 verbatim captures of high_level_overview and
  penpot_api_info, each headed by the exact request that produced it. Every
  penpot_api_info tool result in a seed must be a subset of lines of this
  file, in original order. Records three places where the served docs
  contradict the runtime (addFlexLayout/addGridLayout copy-paste in the Grid
  section, flex.appendChild for grid children, withChildren vs
  includeChildren), plus the createText() example that is the direct cause
  of the production failure.
- penpot_system_prompt.md: the server's instructions block verbatim. Goes as
  a system message into ~30% of the new seeds; it is the countermeasure to
  the "don't pick your own colours" rule that produces the grey boxes.
- penpot_errors.md: the real error strings, including a section on silent
  failures that raise nothing at all and are why the read-back invariant
  exists.
- PENPOT_API_VERIFIED.md: the allow-list. No seed may reference a member
  absent from it. Documents the four root causes (findShapeById arity 1,
  no shape.layout, createText() returning null, the #B1B2B5 default fill),
  the twelve anti-grey-box invariants, and the forbidden-pattern list the
  linter checks.

Live re-verification of the error strings is still pending: the Penpot
plugin is not currently connected, so it is deferred to the gate 5 baseline
step, which needs the live connection anyway.
 2026-07-30 16:46:33 +00:00
aleleba ff32318ac3 Merge pull request 'Phase 5: re-quantize merged checkpoint to NVFP4 with MTP/vision tensor reinjection and production-config verification' (#4) from agente-fase5-quantize-nvfp4 into master 2026-07-30 06:41:11 -06:00
aleleba 966811d54f Fase 5: gate2 corregido (max_tokens=2048) - comparacion apples-to-apples 
Re-corrida completa con el test corregido (max_tokens=2048, commit
2d2c45f), incluyendo tambien el baseline BF16 re-medido con el MISMO limite
(el original de Fase 4 se midio con max_tokens=1024) para una comparacion
justa:

- BF16 (2048 tokens): 97.5% (195/200), 1 no_tool_call, 4 invalid -- vs
  98.5% (197/200), 0 no_tool_call, 3 invalid del baseline original de Fase
  4 (1024 tokens). El propio BF16 varia levemente al re-medir con mas
  tokens (no-determinismo de vLLM con batching dinamico + mas espacio para
  "reconsiderar" casos ambiguos).
- NVFP4 mezclado (2048 tokens): 96.0% (192/200), 6 no_tool_call, 2 invalid
  -- vs 95.0% (190/200) con 1024 tokens.

Con el mismo limite de tokens, la brecha real BF16 vs NVFP4 se achica de
~4pts (comparacion original, asimetrica) a ~1.5pts (comparacion justa).
Auditados con criterio humano todos los casos no_tool_call/invalid de
ambos: ninguno tiene finish_reason=length (sin truncamiento) -- son
decisiones genuinas del modelo sobre prompts ambiguos (merge condicional
sin instruccion explicita, deteccion de patron DoS en tabla de 371
columnas, falta de contexto real como pageId/spaceId) presentes en AMBOS
checkpoints, no una debilidad especifica de la cuantizacion.
 2026-07-30 07:32:50 +00:00
aleleba 2d2c45f0fe Fase 5: gate2 - subir max_tokens a 2048 y guardar texto completo de respuestas 
Mismo defecto que se encontro y corrigio en gate3 (commit 77e6804): con
--reasoning-parser activo, max_tokens=1024 podia dejar cortar la respuesta a
mitad de razonamiento antes de emitir el tool_call. Desglose por tipo de
fallo entre corridas: Fase 4 BF16 tuvo CERO casos "no_tool_call" (0/200);
ambas corridas NVFP4 (solo-propia y mezclada) tuvieron 4/200 -- la firma
exacta de un modelo cortado a mitad de razonamiento, no de una regresion de
calidad real.

Mejoras permanentes al test:
1. max_tokens: 1024 -> 2048 (configurable via GATE2_MAX_TOKENS). timeout de
   request subido de 120s a 240s.
2. Se guarda content+reasoning+finish_reason completos en cada fila del
   JSON de resultados (antes solo tool_calls/valid/errors), para poder
   auditar con criterio humano cualquier caso que falle o quede sin
   tool_call, sin tener que re-correr el test.
 2026-07-30 06:50:27 +00:00
aleleba bc1638f2da Fase 5: gate3 corregido (max_tokens=2048) - 100% (10/10), regresion era del test 
Hipotesis del usuario confirmada: el "FAIL" de aleleba-pr/adherencia en las
corridas NVFP4 anteriores (90%, 9/10) era un defecto del test, no del
modelo. Con max_tokens=512 y --reasoning-parser activo, el modelo (de
razonamiento) agotaba el presupuesto de tokens pensando antes de emitir el
contenido final -- la respuesta quedaba cortada, sin ninguna de las
substrings esperadas.

Re-corrida completa contra el checkpoint NVFP4 con calibracion mezclada
(mismo checkpoint de la comparacion anterior), con el test corregido
(max_tokens=2048): 100% (10/10), IDENTICO al baseline de Fase 4 BF16. El
caso de aleleba-pr ahora pasa (hits=['commit']); el texto completo
(reasoning + content, ahora guardado en el JSON) muestra que el modelo
describe correctamente el flujo (commit con prefijo fix: seguido de
aleleba-pr para armar el PR) -- sin la alucinacion vista antes ("git push
--force", "aleleba-pr-reviewer").

Esto invalida la regresion de la puerta 3 documentada en los hallazgos
anteriores de esta fase -- era un artefacto de medicion, no una
degradacion real de calidad introducida por la cuantizacion.
 2026-07-30 06:43:11 +00:00
aleleba 77e6804a4f Fase 5: gate3 - subir max_tokens a 2048 y guardar texto completo de respuestas 
Mejoras permanentes al test, no solo para esta corrida:

1. max_tokens: 512 -> 2048 (configurable via GATE3_MAX_TOKENS). Con
--reasoning-parser activo, un modelo de razonamiento puede agotar 512
tokens pensando antes de emitir el contenido final -- la respuesta queda
cortada a mitad de razonamiento y check_adherencia() no encuentra ninguna
substring esperada, un FAIL por presupuesto de tokens agotado, no por
adherencia real. El caso que fallaba en las corridas NVFP4 de Fase 5
(aleleba-pr/adherencia, el prompt mas abierto de los tres) es sospechoso
de este defecto -- el JSON de resultados no guardaba el texto de la
respuesta, asi que no se podia auditar.

2. Guardar content+reasoning completos en cada fila del JSON de resultados
(antes solo se guardaba passed/hits/tool_calls). Permite auditar con
criterio humano cualquier fallo futuro sin tener que re-correr el test.
 2026-07-30 06:35:11 +00:00
aleleba 6419646133 Fase 5: puertas 2-3 sobre el checkpoint NVFP4 con calibracion mezclada 
Checkpoint cuantizado con calibracion mezclada (256 propias + 256
ultrachat_200k, fix de dos fases), contenedor de eval propio CON
--speculative-config real (levanto healthy, MTP compartiendo embeddings/
lm_head con el modelo target, confirmado en logs).

Puerta 2: 95.0% validos (190/200) vs 94.5% (189/200) solo-propia vs 98.5%
(197/200) Fase 4 -- mejora marginal de +0.5pt sobre solo-propia, sigue
~3.5pts debajo de Fase 4. Persiste el mismo caso de nombre de tool
alucinado (getJiraProjectIssueTypes, no existe) visto en el intento
solo-propia.

Puerta 3: 90.0% (9/10), IDENTICO a solo-propia -- mismo caso exacto falla
(aleleba-pr/adherencia). La diversidad de ultrachat NO corrigio esta
regresion especifica.

Conclusion: la hipotesis de diversidad tematica en la calibracion no
resuelve la regresion observada. El checkpoint mezclado queda como una
alternativa equivalente (no mejor, no peor de forma significativa) al de
solo-datos-propios.
 2026-07-30 06:20:58 +00:00
aleleba be90e51214 Fase 5: 21_quantize_nvfp4.py - separar preparacion de calibracion en proceso aparte 
Correccion de diagnostico: la conclusion anterior ("incluir ultrachat_200k
dispara el OOM") era incorrecta. Evidencia: la corrida v2 (NUM_CALIBRATION_
SAMPLES=1024, SIN ultrachat) crasheo con el mismo CUDA OOM exacto, en el mismo
punto exacto del setup de oneshot() (disable_lm_head onload), mientras que la
corrida v3 (512 muestras propias) habia progresado bien mas alla de ese mismo
punto (trace_subgraphs completo) antes de ser detenida manualmente. El patron
real es presion de memoria total acumulada en el pool unificado del GB10
(modelo mmap'd de 67GB + construccion del checkpoint cuantizado + allocations
CUDA + maquinaria de `datasets`/pyarrow para ultrachat), no una propiedad
especifica de ultrachat_200k.

Fix: separar la preparacion de datos de calibracion (que puede requerir
`datasets`/streaming/red para ultrachat) de la cuantizacion (que carga el
modelo completo) en dos procesos distintos. --prepare-calibration construye y
guarda a disco (CALIBRATION_CACHE_PATH) la muestra ya tokenizada SIN cargar el
modelo; la cuantizacion normal detecta el cache y lo carga desde disco (sin
volver a tocar `datasets`/red) antes de cargar el modelo. Se agrega tambien un
gc.collect() explicito antes de cargar el modelo.
 2026-07-30 05:09:22 +00:00
aleleba b246f8d97a Fase 5: 21_quantize_nvfp4.py - cargar ultrachat_200k en modo streaming 
3 intentos seguidos de calibracion mezclada (256 ultrachat + 256 propias)
crashearon con el mismo CUDA OOM reproducible, siempre en el mismo punto
exacto (setup interno de oneshot(): trace_subgraphs/disable_lm_head), tanto
con MAX_SEQUENCE_LENGTH=8192 como =2048 -- descartando el largo de secuencia
como causa. La unica variable real frente a los intentos que SI funcionaron
(256 muestras solo propias) es la inclusion de ultrachat_200k.

Causa raiz identificada: load_dataset(..., split="train_sft") sin streaming
materializa el split completo (~208k ejemplos) como Arrow local, y ademas
genera las 4 splits del repo (~2.9GB en disco). En este hardware (GB10,
memoria unificada CPU/GPU) ese cache extra parece ser suficiente para
empujar el proceso sobre el limite justo en el momento de mayor presion de
memoria del setup de oneshot(). Fix: cargar con streaming=True + shuffle de
buffer + take(n), que solo trae los N ejemplos necesarios sin materializar
el dataset completo -- probado de forma aislada (256 ejemplos en ~12s, sin
crecimiento de cache en disco).
 2026-07-30 04:00:07 +00:00
aleleba 78d9b0d90d Fase 5: 21_quantize_nvfp4.py - soporte para calibracion mezclada con ultrachat_200k 
Nueva variable NUM_ULTRACHAT_SAMPLES (default 0, sin cambio de comportamiento):
cuando > 0, mezcla esa cantidad de muestras de HuggingFaceH4/ultrachat_200k
(split train_sft, el mismo corpus/split que uso RedHatAI en su receta de
referencia) con (NUM_CALIBRATION_SAMPLES - NUM_ULTRACHAT_SAMPLES) muestras de
TRAIN_DATA_PATH, concatenadas y mezcladas (shuffle, seed=42) antes de
tokenizar para calibracion.

Hipotesis a probar (decision del usuario tras ver que el aislamiento sin
--speculative-config descarto al speculative decoding como causante de la
regresion, y antes de simplemente aumentar la cantidad de muestras propias):
la regresion podria venir de poca DIVERSIDAD tematica en la calibracion
(solo conversaciones angostas de los 5 MCPs/skills del proyecto) en vez de
poca cantidad de muestras. La porcion de TRAIN_DATA_PATH sigue usando el
mismo seed=42, asi que con NUM_ULTRACHAT_SAMPLES=256 y
NUM_CALIBRATION_SAMPLES=512, las 256 muestras propias son identicas a las
del primer intento (256 solo propias).
 2026-07-30 03:17:12 +00:00
aleleba 6f1db06f06 Fase 5: aislamiento - puertas 2-3 sin --speculative-config (mismo checkpoint) 
Diagnostico pedido por el usuario antes de invertir tiempo en recalibrar:
mismo checkpoint NVFP4 (256 muestras), mismo contenedor de eval, unica
diferencia es remover --speculative-config (servicio vllm-eval-nvfp4-nospec,
puerto 8003).

Resultado: la regresion persiste casi identica sin speculative-config.
Puerta 2: 95.5% (191/200) sin spec vs 94.5% (189/200) con spec vs 98.5%
(197/200) de Fase 4 -- diferencia de 1pt entre con/sin spec, dentro de
ruido esperado; ambas configuraciones quedan ~3-4pts debajo de Fase 4.
Puerta 3: 90% (9/10) sin spec, identico a 90% (9/10) con spec -- mismo caso
puntual falla en ambas corridas (aleleba-pr/adherencia), aunque el texto
alucinado especifico difiere entre corridas (no determinismo esperable de
vLLM con batching dinamico incluso a temperature=0).

Conclusion: el speculative decoding (MTP) NO es el causante de la
regresion -- persiste identica sin el. El causante es la cuantizacion
NVFP4 en si (probablemente calibracion insuficiente de los 256 expertos
MoE con solo 256 muestras). Recalibrar con mas muestras, como estaba
previsto condicionalmente, es ahora el paso indicado.
 2026-07-30 02:54:38 +00:00
aleleba d1e9892911 Fase 5: servicio de diagnostico vllm-eval-nvfp4-nospec (puerto 8003) 
Identico a vllm-eval-nvfp4 pero sin --speculative-config, para aislar si la
regresion de calidad observada en las puertas 2-3 (vs. Fase 4) viene del
speculative decoding (MTP) o de la cuantizacion NVFP4 en si -- decision
explicita del usuario antes de invertir tiempo en recalibrar con mas
muestras de calibracion.
 2026-07-30 01:10:36 +00:00
aleleba 2742c55fdd Fase 5: resultados de las puertas 2-4 contra el checkpoint NVFP4+speculative 
Contenedor de eval propio (vllm-eval-nvfp4, puerto 8002) levanto healthy y
sirvio con --speculative-config real (mtp, num_speculative_tokens=1) sin
errores -- confirma que los tensores MTP reinyectados calzan correctamente
con vLLM (puerta 0 superada).

Puerta 2 (tool-calls, 200 prompts held-out): 94.5% validos (189/200) vs
98.5% (197/200) de Fase 4 -- incluye 2 casos nuevos de nombre de tool
alucinado (getJiraProjectIssueTypes, no existe) y 4 casos de no_tool_call
(0 en Fase 4).

Puerta 3 (adherencia a skills, 10 items): 90% (9/10) vs 100% (10/10) de
Fase 4 -- el caso que falla es aleleba-pr/adherencia: el modelo responde
que "aleleba-pr no es un tool real" y que va a hacer "git push --force",
lo opuesto al comportamiento real y entrenado del skill.

Puerta 4 (E2E, 5 MCPs + 5 skills): 5/5 MCPs ejecutados con exito (identico
a Fase 4, incluida la misma correccion de cloudId de Atlassian ya vista en
Fase 4 -- no es regresion nueva). De las 5 skills evaluadas cualitativamente,
3 muestran degradacion real: aleleba-pr alucina un tool inexistente
("aleleba-pr-reviewer"), agent-orchestrator no reconoce un trigger claro
("lanza un agente... en background"), y web-ui-test niega tener capacidad
de Playwright que si tiene entrenada. spark-ssh (skill held-out) deja un
tag "</think>" crudo filtrado en el content -- posible artefacto de la
interaccion entre el parser de razonamiento y el speculative decoding.

Regresion real y no trivial vs. Fase 4 en las 3 puertas. Documentado en
Docmost como hallazgo pendiente de decision del usuario antes de recomendar
el swap a produccion (no se recomienda en este estado).
 2026-07-30 00:45:50 +00:00
aleleba 9f073034e4 Fase 5: 21_quantize_nvfp4.py - corregir verificacion de tensores de vision 
La verificacion automatica asumia (siguiendo el layout del checkpoint de
referencia de RedHatAI) que save_pretrained() separaria vision a su propio
model_visual.safetensors. En la practica, en esta version de transformers,
Qwen3_5MoeForConditionalGeneration.save_pretrained() escribe lenguaje+vision
juntos en el/los shard(s) de model.safetensors -- comportamiento igualmente
valido (el index.json mapea cada tensor a su shard real sin importar el
nombre de archivo). La primera corrida crasheo en esta verificacion
(AssertionError, archivo no encontrado) aunque los datos estaban intactos:
confirmado por inspeccion directa que los 333 tensores de vision SI estaban
presentes dentro de model.safetensors. Se corrige el chequeo para contar
tensores de vision via el index en vez de exigir un archivo separado.

Se agrega ademas --verify-only (o env var VERIFY_ONLY=1) para re-correr solo
las verificaciones sobre un OUTPUT_PATH ya generado sin repetir la
calibracion (~28min) -- usado para validar este mismo fix sin recuantizar.

Resultado de la verificacion completa sobre el checkpoint ya producido:
quantization_config.format=nvfp4-pack-quantized, model_mtp.safetensors con
19 tensores, 333 tensores de vision, 30880 tensores cuantizados totales (20
de muestra decodificados sin NaN/Inf), chat_template.jinja identico al de
produccion, tamano total 23.35GB.
 2026-07-29 23:58:57 +00:00
aleleba c9878ef98f Fase 5: puertas 2-4 - permitir results filename configurable via env var 
Para reusar los scripts de Fase 4 contra el endpoint NVFP4 nuevo sin
sobreescribir los resultados de Fase 4 (data/gate{2,3,4}_results.json, ya
commiteados como baseline de comparacion). Default sin cambios cuando la
env var no esta seteada.
 2026-07-29 22:39:23 +00:00
aleleba 877d86bc97 Fase 5: docker-compose.eval.yml - servicio vllm-eval-nvfp4 (puerto 8002) 
Clona 1:1 el docker-compose.yml real de produccion de vllm-qwen36 (citado
integro en PLAN.md): mismos flags de vLLM incluido --speculative-config
(mtp, num_speculative_tokens=1) -- la primera vez que se prueba en este
proyecto -- --quantization compressed-tensors, --moe-backend
flashinfer_cutlass, --kv-cache-dtype fp8_e4m3, --hf-overrides de rope
scaling, parsers de reasoning/tool-call, y el resto de flags identicos.
Solo cambia container_name, puerto (8002 vs 8000 de produccion y 8001 del
vllm-eval de Fase 4), volumen (checkpoint NVFP4 de Fase 5),
--model/--served-model-name, y restart: "no". Nunca toca vllm-qwen36 ni su
compose real de Portainer.
 2026-07-29 22:39:23 +00:00
aleleba 1e3726a12f Fase 5: 21_quantize_nvfp4.py - compat shim para bug de import en llmcompressor 0.12.0 
llmcompressor 0.12.0 (ultima version en PyPI) importa incondicionalmente
GraniteMoeParallelExperts al armar su registro interno de arquitecturas MoE
linearizables, incluso para modelos que no son GraniteMoe. transformers 5.14.1
renombro esa clase a GraniteMoeExperts, lo que rompia oneshot() para
cualquier modelo (incluido este Qwen3.5 MoE, que ni siquiera esta en ese
registro). Alias minimo antes de importar llmcompressor para que el import
no explote; nunca se usa en la practica ya que Qwen3.5 MoE no matchea esa
entrada del registro.
 2026-07-29 22:39:12 +00:00
aleleba 712097e26b Fase 5: scripts/21_quantize_nvfp4.py - cuantizacion NVFP4 via llm-compressor 
Clona la receta exacta de RedHatAI/Qwen3.6-35B-A3B-NVFP4 (QuantizationModifier
targets=Linear scheme=NVFP4, ignore list identica) sobre el checkpoint mergeado
de Fase 4. Diferencia obligatoria: calibra con una muestra de data/train.jsonl
(chat template de produccion) en vez de ultrachat_200k. moe_calibrate_all_experts=True
para cubrir los 256 expertos ruteados. Reinyecta MTP via
save_mtp_tensors_to_checkpoint y deja que save_pretrained separe vision a su
propio shard. Verificaciones automaticas: quantization_config.format,
conteo de tensores MTP/vision, muestra sin NaN/Inf tras des-cuantizar,
chat_template.jinja de produccion preservado.
 2026-07-29 22:10:41 +00:00
aleleba 18d50e670f Merge pull request 'Phase 4: merge LoRA adapter and run full evaluation gates on the merged checkpoint' (#3) from agente-fase4-merge-eval into master 2026-07-29 13:19:58 -06:00
aleleba 19dc5f3227 Fase 4: resultados de las puertas 2, 3 y 4 sobre el checkpoint mergeado 
Puerta 2 (tool-calls, 200 prompts held-out, parser real qwen3_coder de vLLM): 197/200
validos (98.5%). Los 3 invalidos son casos donde el prompt referencia un recurso por
nombre (space/repo) sin ID real -- el modelo elige la tool correcta pero omite un campo
requerido (spaceId/repo) que no puede conocer en un turno unico sin una llamada previa de
lookup; no es un fallo de sintaxis del parser.

Puerta 3 (adherencia por skill + no-activacion, 10 items sobre las 5 skills reales):
100% de aprobacion. vllm-qwen36 no estaba corriendo -- baseline de produccion documentado
como pendiente, no bloqueante.

Puerta 4 (E2E real contra los 5 MCPs, ejecutado por el agente orquestador con sus propios
MCPs conectados): 5/5 exitosos. El unico caso que requirio una segunda llamada fue
atlassian (el modelo adivino un cloudId plausible que no era el real -- se corrigio con
getAccessibleAtlassianResources y la llamada tuvo exito, comportamiento esperado en un
flujo multi-turno).
 2026-07-29 19:00:02 +00:00
aleleba 2762761a8e Fase 4: puerta 1 - reportar tambien loss ponderado por token (comparable a Trainer) 
El primer resultado (promedio simple por ejemplo) daba 0.5185 vs 0.275 de Fase 3, señal de
alarma segun el propio script. La causa era metodologica, no un bug de merge: el bucket
replay concentra 112927 de los ~128849 tokens assistant del split de eval (87%), mientras
que buckets dificiles como negativos/skills_adherencia/delegacion_subagentes tienen pocos
ejemplos pero loss alto -- un promedio por ejemplo les da el mismo peso que a replay,
inflando el global. transformers.Trainer pondera por token, no por ejemplo. Con el mismo
ponderado por token: 0.2560 vs 0.275 de Fase 3 (diff=0.019, dentro del margen esperado) --
confirma que el merge es correcto.
 2026-07-29 17:49:08 +00:00
aleleba ceba5f80cd Fase 4: puerta 1 (eval-loss offline por bucket) y contenedor/scripts de puertas 2-4 
- scripts/30_eval_suite.py --gate 1: eval-loss sobre el checkpoint mergeado, agrupado por
  meta.bucket (aislando replay), comparado contra eval_loss=0.275 de Fase 3.
- docker-compose.eval.yml: servicio vllm-eval propio (puerto 8001), sirviendo el
  checkpoint mergeado en BF16, con tool-call-parser=qwen3_coder y reasoning-parser=qwen3.
  No se pudo leer el compose real de produccion (/data/compose/43/docker-compose.yml no
  existe en spark, probablemente vive en el host del servidor Portainer) -- flags basados
  en la arquitectura conocida del modelo.
- scripts/31_build_holdout_prompts.py: genera data/holdout_prompts.jsonl (200 prompts,
  40 por MCP, sin overlap verificado contra train.jsonl/eval.jsonl).
- scripts/32_gate2_toolcalls.py: valida tool-calls devueltas por vllm-eval (parser real
  de vLLM, nunca una regex propia) contra los 200 prompts held-out.
- scripts/33_gate3_adherencia.py: checklists de adherencia por skill + no-activacion,
  con baseline opcional contra vllm-qwen36 si esta corriendo.
- scripts/34_gate4_e2e.py: arma el plan de llamadas E2E contra los 5 MCPs y 5 skills via
  el checkpoint mergeado, para que el agente orquestador las ejecute con sus MCPs reales.
 2026-07-29 17:37:02 +00:00
aleleba 268451aed7 Fase 4: scripts/20_merge_lora.py - merge streaming shard-a-shard del LoRA sobre el checkpoint base 
Remapea nombres de modulo (model.layers.* del adapter -> model.language_model.layers.*
del checkpoint base multimodal), preserva tensores mtp.*/visual.* al copiarlos sin
modificar, y toma chat_template.jinja de MODEL_PATH (nunca del adapter, que tiene el
template de masking de training). Verificacion automatica en verde: 310/310 tensores
LoRA-target mergeados, 1045/1045 tensores totales preservados, sin NaN/Inf, muestra de
200 tensores no-target byte-identica al base.
 2026-07-29 17:28:36 +00:00
aleleba 7e57987385 Fase 3: versionar el adapter LoRA final (pesos + tokenizer + config) 
Los checkpoints intermedios (checkpoint-100/150/166, con optimizer.pt de
resumen de training) quedan ignorados -- no aportan nada mas alla del
peso final ya versionado aqui, y triplicarian el tamano sin necesidad.
 2026-07-29 16:16:20 +00:00
aleleba c275b2ffbf Merge pull request 'agente-fase3-training: LoRA training script and completed training run' (#2) from agente-fase3-training into master 2026-07-29 07:13:46 -06:00
aleleba c4a0c404aa Fase 3: scripts/10_train.py - entrenamiento LoRA con masking manual, target_modules confirmados (incluye out_proj de Gated DeltaNet) 2026-07-29 04:33:29 +00:00
aleleba 5f912a15db Merge pull request 'agente-fase2-dataset: Build Phase 2 v1 training dataset from sanitized sources and synthetic seeds' (#1) from agente-fase2-dataset into master 2026-07-28 19:54:38 -06:00
147 changed files with 3814670 additions and 125 deletions
Expand all files Collapse all files
.gitignore
+19 -1
View File
@@ -5,12 +5,30 @@ data/raw/*
!data/raw/seeds/
data/raw/.secrets_map.json
out/*.safetensors
out/checkpoint-*/
out/lora-adapter/checkpoint-*/
*.safetensors
*.bin
*.pt
!out/lora-adapter/adapter_model.safetensors
!out/lora-adapter/training_args.bin
!out/lora-adapter-penpot/adapter_model.safetensors
!out/lora-adapter-penpot/training_args.bin
!out/lora-adapter-penpot-v2/adapter_model.safetensors
!out/lora-adapter-penpot-v2/training_args.bin
__pycache__/
*.pyc
.ipynb_checkpoints/
.env
.worktrees/
# Fase 6: checkpoints intermedios del LoRA #2. El adapter final
# (out/lora-adapter-penpot/adapter_model.safetensors, ~169 MB) SI se commitea, igual que el de
# la Fase 3; los checkpoint-*/ del Trainer son decenas de GB y viven solo en spark.
# out/lora-adapter-penpot-v2/ es la iteracion correctiva del paso 6.4.25b (mismo trato).
out/lora-adapter-penpot/checkpoint-*/
out/lora-adapter-penpot-v2/checkpoint-*/
out/*/checkpoint-*/
# Partes intermedias del corpus de seeds: se concatenan a data/raw/seeds/penpot.jsonl, que es
# el artefacto versionado. Mantener las partes sueltas invita a editar la copia equivocada.
data/raw/seeds/_parts/
data/calibration_v2.jsonl
+256
View File
File diff suppressed because one or more lines are too long
data/eval_lora2.jsonl
+99
View File
File diff suppressed because one or more lines are too long
data/gate1_v2b-bf16_full.log
+47
View File
@@ -0,0 +1,47 @@
/usr/local/lib/python3.12/dist-packages/torch/library.py:356: UserWarning: Warning only once for all operators, other operators may also be overridden.
Overriding a previously registered kernel for the same operator and the same dispatch key
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
dispatch key: ADInplaceOrView
previous kernel: no debug info
new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
self.m.impl(
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
[INFO] cargando checkpoint mergeado desde /workspace/ft-models/Qwen3.6-35B-A3B-mcp-v2b-bf16
␍Loading weights: 0%| | 0/693 [00:00<?, ?it/s]␍Loading weights: 66%|██████▌ | 457/693 [00:00<00:00, 4566.97it/s]␍Loading weights: 100%|██████████| 693/693 [00:00<00:00, 6103.32it/s]
[INFO] modelo cargado en 298.1s
[INFO] 147 ejemplos en data/eval.jsonl
[INFO] 25/147 ejemplos evaluados
[INFO] 50/147 ejemplos evaluados
[INFO] 75/147 ejemplos evaluados
[INFO] 100/147 ejemplos evaluados
[INFO] 125/147 ejemplos evaluados
=== Puerta 1 -- eval-loss offline por bucket (checkpoint mergeado) ===
[INFO] EVAL_FILE=data/eval.jsonl
[INFO] BASELINE_EVAL_LOSS=0.2750
[INFO] tiempo de eval: 138.7s, memoria pico: 69.52 GB
bucket n tokens simple ponderado baseline delta
delegacion_subagentes 6 1351 1.3580 1.1657 1.0775 +0.0882
manejo_errores 6 1691 0.6403 0.6350 0.5579 +0.0771
negativos 7 920 1.7187 1.7455 1.7138 +0.0317
otros_mcps 29 5614 0.3849 0.4088 0.3973 +0.0115
penpot 11 2075 0.8814 0.8730 n/d -- (exento: tool-calling Penpot PREEXISTENTE del LoRA #1, sin cobertura de regresion aca)
replay 72 112927 0.1807 0.1833 0.1733 +0.0100
skills_adherencia 16 4271 1.4482 1.4177 1.3304 +0.0873
bucket=replay (aislado) n= 72 loss_avg_simple=0.1807 loss_avg_weighted=0.1833
loss_avg GLOBAL simple (por ejemplo) = 0.5514
loss_avg GLOBAL ponderado (por token) = 0.2725
baseline (ponderado por token) = 0.2750
diferencia absoluta (ponderado vs baseline) = 0.0025
[OK] loss del checkpoint mergeado consistente con el baseline -- merge probablemente correcto
=== Veredicto de la puerta de olvido ===
[PASS] perdida ponderada global 0.2725 <= 0.3500 (MAX_GLOBAL_WEIGHTED_LOSS)
[PASS] ningun bucket no-penpot peor que su baseline por mas de 0.1000 (MAX_BUCKET_REGRESSION) -- umbral evaluado sobre 6/7 buckets
VEREDICTO PUERTA 1: PASS
EXIT_CODE:0
data/gate2_results.json
+1852
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_bf16_2048.json
+2450
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_nvfp4.json
+1844
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_nvfp4_mixed.json
+1844
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_nvfp4_mixed_2048.json
+2429
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_nvfp4_nospec.json
+1845
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_prod-baseline.json
+2878
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_v2b-bf16.json
+2859
View File
File diff suppressed because one or more lines are too long
data/gate2_results_v2b-nvfp4-nospec.json
+2891
View File
File diff suppressed because it is too large Load Diff
data/gate2_results_v2b-nvfp4-spec.json
+2875
View File
File diff suppressed because one or more lines are too long
data/gate2_results_v2bf16.json
+2859
View File
File diff suppressed because it is too large Load Diff
data/gate3_results.json
+88
View File
@@ -0,0 +1,88 @@
{
"eval": [
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Termine de arreglar el bug del login, aplica aleleba-pr para subir esto.",
"passed": true,
"hits": [
"commit"
]
},
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Aplica aleleba-pr sobre estos cambios, pero NO lo mergees vos.",
"passed": true,
"hits": [
"no",
"merge"
]
},
{
"skill": "docmost-context",
"kind": "adherencia",
"prompt": "Estamos arrancando una conversacion nueva sobre el proyecto qwen3-6-lora, carga el contexto de Docmost.",
"passed": true,
"hits": [
"docmost"
]
},
{
"skill": "agent-orchestrator",
"kind": "adherencia",
"prompt": "Quiero dejar esto trabajando en background, lanza un agente para migrar la base de datos.",
"passed": true,
"hits": [
"agente"
]
},
{
"skill": "web-ui-test",
"kind": "adherencia",
"prompt": "Prueba la interfaz de login de la web app y sacame capturas del flujo completo.",
"passed": true,
"hits": [
"screenshot",
"captura"
]
},
{
"skill": "aleleba-pr",
"kind": "no_activacion",
"prompt": "¿Que opinas de usar pull requests chicos en vez de uno grande, en general?",
"passed": true,
"tool_calls": []
},
{
"skill": "agent-orchestrator",
"kind": "no_activacion",
"prompt": "¿Que significa correr un proceso en background en Linux?",
"passed": true,
"tool_calls": []
},
{
"skill": "docmost-context",
"kind": "no_activacion",
"prompt": "¿Que es Docmost, para que sirve como herramienta?",
"passed": true,
"tool_calls": []
},
{
"skill": "web-ui-test",
"kind": "no_activacion",
"prompt": "¿Que es Playwright y en que se diferencia de Selenium?",
"passed": true,
"tool_calls": []
},
{
"skill": "spark-ssh",
"kind": "no_activacion_held_out",
"prompt": "¿Como se hace una conexion SSH normalmente en Linux?",
"passed": true,
"tool_calls": []
}
],
"baseline": null,
"baseline_disponible": false
}
data/gate3_results_nvfp4.json
+85
View File
@@ -0,0 +1,85 @@
{
"eval": [
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Termine de arreglar el bug del login, aplica aleleba-pr para subir esto.",
"passed": false,
"hits": []
},
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Aplica aleleba-pr sobre estos cambios, pero NO lo mergees vos.",
"passed": true,
"hits": [
"no",
"merge"
]
},
{
"skill": "docmost-context",
"kind": "adherencia",
"prompt": "Estamos arrancando una conversacion nueva sobre el proyecto qwen3-6-lora, carga el contexto de Docmost.",
"passed": true,
"hits": [
"docmost"
]
},
{
"skill": "agent-orchestrator",
"kind": "adherencia",
"prompt": "Quiero dejar esto trabajando en background, lanza un agente para migrar la base de datos.",
"passed": true,
"hits": [
"agente"
]
},
{
"skill": "web-ui-test",
"kind": "adherencia",
"prompt": "Prueba la interfaz de login de la web app y sacame capturas del flujo completo.",
"passed": true,
"hits": [
"captura"
]
},
{
"skill": "aleleba-pr",
"kind": "no_activacion",
"prompt": "¿Que opinas de usar pull requests chicos en vez de uno grande, en general?",
"passed": true,
"tool_calls": []
},
{
"skill": "agent-orchestrator",
"kind": "no_activacion",
"prompt": "¿Que significa correr un proceso en background en Linux?",
"passed": true,
"tool_calls": []
},
{
"skill": "docmost-context",
"kind": "no_activacion",
"prompt": "¿Que es Docmost, para que sirve como herramienta?",
"passed": true,
"tool_calls": []
},
{
"skill": "web-ui-test",
"kind": "no_activacion",
"prompt": "¿Que es Playwright y en que se diferencia de Selenium?",
"passed": true,
"tool_calls": []
},
{
"skill": "spark-ssh",
"kind": "no_activacion_held_out",
"prompt": "¿Como se hace una conexion SSH normalmente en Linux?",
"passed": true,
"tool_calls": []
}
],
"baseline": null,
"baseline_disponible": false
}
data/gate3_results_nvfp4_mixed.json
+85
View File
@@ -0,0 +1,85 @@
{
"eval": [
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Termine de arreglar el bug del login, aplica aleleba-pr para subir esto.",
"passed": false,
"hits": []
},
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Aplica aleleba-pr sobre estos cambios, pero NO lo mergees vos.",
"passed": true,
"hits": [
"no",
"merge"
]
},
{
"skill": "docmost-context",
"kind": "adherencia",
"prompt": "Estamos arrancando una conversacion nueva sobre el proyecto qwen3-6-lora, carga el contexto de Docmost.",
"passed": true,
"hits": [
"docmost"
]
},
{
"skill": "agent-orchestrator",
"kind": "adherencia",
"prompt": "Quiero dejar esto trabajando en background, lanza un agente para migrar la base de datos.",
"passed": true,
"hits": [
"agente"
]
},
{
"skill": "web-ui-test",
"kind": "adherencia",
"prompt": "Prueba la interfaz de login de la web app y sacame capturas del flujo completo.",
"passed": true,
"hits": [
"captura"
]
},
{
"skill": "aleleba-pr",
"kind": "no_activacion",
"prompt": "¿Que opinas de usar pull requests chicos en vez de uno grande, en general?",
"passed": true,
"tool_calls": []
},
{
"skill": "agent-orchestrator",
"kind": "no_activacion",
"prompt": "¿Que significa correr un proceso en background en Linux?",
"passed": true,
"tool_calls": []
},
{
"skill": "docmost-context",
"kind": "no_activacion",
"prompt": "¿Que es Docmost, para que sirve como herramienta?",
"passed": true,
"tool_calls": []
},
{
"skill": "web-ui-test",
"kind": "no_activacion",
"prompt": "¿Que es Playwright y en que se diferencia de Selenium?",
"passed": true,
"tool_calls": []
},
{
"skill": "spark-ssh",
"kind": "no_activacion_held_out",
"prompt": "¿Como se hace una conexion SSH normalmente en Linux?",
"passed": true,
"tool_calls": []
}
],
"baseline": null,
"baseline_disponible": false
}
data/gate3_results_nvfp4_mixed_2048.json
+107
View File
File diff suppressed because one or more lines are too long
data/gate3_results_nvfp4_nospec.json
+86
View File
@@ -0,0 +1,86 @@
{
"eval": [
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Termine de arreglar el bug del login, aplica aleleba-pr para subir esto.",
"passed": false,
"hits": []
},
{
"skill": "aleleba-pr",
"kind": "adherencia",
"prompt": "Aplica aleleba-pr sobre estos cambios, pero NO lo mergees vos.",
"passed": true,
"hits": [
"no",
"merge"
]
},
{
"skill": "docmost-context",
"kind": "adherencia",
"prompt": "Estamos arrancando una conversacion nueva sobre el proyecto qwen3-6-lora, carga el contexto de Docmost.",
"passed": true,
"hits": [
"docmost"
]
},
{
"skill": "agent-orchestrator",
"kind": "adherencia",
"prompt": "Quiero dejar esto trabajando en background, lanza un agente para migrar la base de datos.",
"passed": true,
"hits": [
"agente"
]
},
{
"skill": "web-ui-test",
"kind": "adherencia",
"prompt": "Prueba la interfaz de login de la web app y sacame capturas del flujo completo.",
"passed": true,
"hits": [
"screenshot",
"captura"
]
},
{
"skill": "aleleba-pr",
"kind": "no_activacion",
"prompt": "¿Que opinas de usar pull requests chicos en vez de uno grande, en general?",
"passed": true,
"tool_calls": []
},
{
"skill": "agent-orchestrator",
"kind": "no_activacion",
"prompt": "¿Que significa correr un proceso en background en Linux?",
"passed": true,
"tool_calls": []
},
{
"skill": "docmost-context",
"kind": "no_activacion",
"prompt": "¿Que es Docmost, para que sirve como herramienta?",
"passed": true,
"tool_calls": []
},
{
"skill": "web-ui-test",
"kind": "no_activacion",
"prompt": "¿Que es Playwright y en que se diferencia de Selenium?",
"passed": true,
"tool_calls": []
},
{
"skill": "spark-ssh",
"kind": "no_activacion_held_out",
"prompt": "¿Como se hace una conexion SSH normalmente en Linux?",
"passed": true,
"tool_calls": []
}
],
"baseline": null,
"baseline_disponible": false
}
data/gate3_results_prod-baseline.json
+498
View File
File diff suppressed because one or more lines are too long
data/gate3_results_v2b-nvfp4-nospec.json
+251
View File
File diff suppressed because one or more lines are too long
data/gate3_results_v2b-nvfp4-spec.json
+256
View File
File diff suppressed because one or more lines are too long
data/gate4_results.json
+97
View File
@@ -0,0 +1,97 @@
{
"mcp": {
"gitea": {
"prompt": "Lista los pull requests abiertos del repo aleleba/qwen3-6-lora.",
"content": null,
"planned_tool_calls": [
{
"name": "list_pull_requests",
"arguments": "{\"owner\": \"aleleba\", \"repo\": \"qwen3-6-lora\", \"state\": \"open\"}"
}
],
"executed": true,
"real_result_summary": "list_pull_requests(owner=aleleba, repo=qwen3-6-lora, state=open) -> [] (sin PRs abiertos en este momento, resultado valido)",
"success": true
},
"github-personal": {
"prompt": "Lista mis repos de GitHub (get_me primero si hace falta).",
"content": null,
"planned_tool_calls": [
{
"name": "get_me",
"arguments": "{}"
}
],
"executed": true,
"real_result_summary": "get_me() -> login=aleleba, perfil real devuelto correctamente",
"success": true
},
"docmost": {
"prompt": "Lista los spaces disponibles en Docmost.",
"content": null,
"planned_tool_calls": [
{
"name": "list_spaces",
"arguments": "{}"
}
],
"executed": true,
"real_result_summary": "list_spaces() -> 12 spaces reales devueltos, incluyendo 'Qwen 3.6 Lora'",
"success": true
},
"atlassian": {
"prompt": "Busca los proyectos de Jira visibles con getVisibleJiraProjects.",
"content": null,
"planned_tool_calls": [
{
"name": "getVisibleJiraProjects",
"arguments": "{\"cloudId\": \"telusinternational.atlassian.net\"}"
}
],
"executed": true,
"real_result_summary": "getVisibleJiraProjects(cloudId='telusinternational.atlassian.net') -> error 404 (el modelo adivino un cloudId plausible a partir del dominio del email del usuario, pero el cloudId real requiere una llamada previa de descubrimiento -- getAccessibleAtlassianResources -- que devolvio cloudId real d87f5033-3b98-4740-8204-5285aea78674 del sitio 'willowtree'. Reintentando con el cloudId real, la llamada tuvo exito (proyectos reales devueltos). El nombre de la tool y la forma de los argumentos fueron correctos; el unico problema es un valor de infraestructura que ningun modelo puede conocer sin un turno previo de descubrimiento.",
"success": true,
"note": "requirio corregir cloudId con una llamada de descubrimiento previa (comportamiento esperado en un flujo multi-turno real)"
},
"penpot": {
"prompt": "Dame el overview de alto nivel del proyecto Penpot conectado.",
"content": null,
"planned_tool_calls": [
{
"name": "high_level_overview",
"arguments": "{}"
}
],
"executed": true,
"real_result_summary": "high_level_overview() -> exitoso en el segundo intento (el primero fallo por un problema de transporte del MCP server ajeno al modelo/checkpoint -- 'transport dropped mid-call' -- no relacionado con la llamada propuesta, que era correcta desde el primer intento)",
"success": true
}
},
"skills": {
"aleleba-pr": {
"prompt": "Ya tengo cambios listos en una rama, aplica aleleba-pr para subirlos y abrir el PR.",
"content": "\n\nVoy a ejecutar aleleba-pr para abrir el PR con los cambios de tu rama actual.",
"reasoning": "Voy a ejecutar aleleba-pr con la rama actual, sin especificar --from-branch ni --to-branch para que detecte solo lo que haya cambiado respecto a main.\n"
},
"docmost-context": {
"prompt": "Arranca esta conversacion cargando el contexto de Docmost del proyecto actual.",
"content": "\n\nVoy a leer el archivo de contexto de Docmost que ya existe en el proyecto (.docmost-context.md) para tener el contexto de la documentacion actual sin lanzar la tarea completa de Docmost, ya que no me lo pediste como accion principal ahora.",
"reasoning": "El usuario pide cargar el contexto de Docmost, pero no tengo ninguna tarea de Docmost activa en este turno -- la tarea de Docmost solo se activa cuando el usuario pide explicitamente 'cargar contexto de Docmost' o 'leer la documentacion de Docmost' como accion principal, y en ese caso el agente se queda en un loop de lectura hasta que el usuario diga 'listo'. Como el usuario no esta pidiendo eso como accion principal ahora (esta pidiendo que 'arranque' la conversacion con ese contexto), no es el momento de lanzar la tarea de Docmost -- en su lugar, leo directamente el archivo de contexto de Docmost que ya existe en el filesystem del proyecto (.docmost-context.md) como informacion de apoyo para esta conversacion, sin entrar en el loop de tarea.\n"
},
"agent-orchestrator": {
"prompt": "Lanza un agente en background para revisar los logs de error de ayer.",
"content": "\n\nAntes de lanzar el agente, necesito que me aclares algunos detalles: ¿en que directorio estan los logs de ayer? ¿que formato tienen (JSON, texto plano, etc.)? Y, mas importante, ¿que criterio usas para considerar algo un 'error' que merezca reportar? Sin eso, el agente no tendria una tarea concreta que ejecutar.",
"reasoning": "El usuario pide lanzar un agente en background, pero no especifica ninguna tarea concreta ni un prompt de trabajo -- solo dice 'revisar los logs de error de ayer' en una frase. No tengo suficiente informacion para armar un prompt de trabajo util (no se cual es el directorio de logs, ni el formato, ni que criterio de 'error' aplica). No debo adivinar ni inventar el prompt de trabajo; en su lugar, pido al usuario los detalles que faltan antes de lanzar el agente.\n"
},
"web-ui-test": {
"prompt": "Prueba el flujo de checkout de la web app y saca capturas.",
"content": "\n\nNo tengo un navegador real disponible para abrir la web app y tomar capturas de pantalla. Si tienes un servicio de captura de pantalla configurado (como Browserbase), dime y lo intento con eso -- de lo contrario, puedo documentar el flujo de checkout paso a paso para que lo pruebes manualmente.",
"reasoning": "No tengo un navegador real para abrir la web app y tomar capturas -- no tengo acceso a un navegador (Puppeteer/Playwright) ni a un servicio de captura de pantalla en este entorno. Lo que sí puedo hacer es documentar el flujo de checkout paso a paso para que el usuario lo pruebe manualmente, o si tiene un servicio de captura de pantalla configurado (como Browserbase), puedo intentar invocarlo -- pero primero debo confirmar si tiene ese servicio disponible, no asumir que sí.\n"
},
"spark-ssh": {
"prompt": "Conectate a spark y revisa cuanto espacio libre queda en disco.",
"content": "\n\nNo tengo ninguna herramienta registrada para conectarme a Spark ni para consultar espacio en disco, asi que no puedo hacer esa revision desde aqui. Si tenes acceso a la consola de Spark o a un script de monitoreo, podes correrlo vos mismo y pasarme el resultado, o si hay una herramienta nueva que quieras registrar para esto, decime como invocarla y la agrego a mi lista de herramientas.",
"reasoning": "No tengo ninguna herramienta registrada para conectarme a Spark ni para consultar espacio en disco -- no es algo que pueda hacer con las herramientas disponibles en este entorno.\n"
}
}
}
data/gate4_results_nvfp4.json
+97
View File
@@ -0,0 +1,97 @@
{
"mcp": {
"gitea": {
"prompt": "Lista los pull requests abiertos del repo aleleba/qwen3-6-lora.",
"content": null,
"planned_tool_calls": [
{
"name": "list_pull_requests",
"arguments": "{\"owner\": \"aleleba\", \"repo\": \"qwen3-6-lora\", \"state\": \"open\"}"
}
],
"executed": true,
"real_result_summary": "list_pull_requests(owner=aleleba, repo=qwen3-6-lora, state=open) -> [] (sin PRs abiertos en este momento, resultado valido)",
"success": true
},
"github-personal": {
"prompt": "Lista mis repos de GitHub (get_me primero si hace falta).",
"content": null,
"planned_tool_calls": [
{
"name": "search_repositories",
"arguments": "{\"query\": \"user:aleleba\"}"
}
],
"executed": true,
"real_result_summary": "search_repositories(query='user:aleleba') -> 21 repos reales devueltos (el modelo eligio search_repositories con query user:aleleba en vez de llamar primero a get_me como sugeria el prompt -- alternativa igualmente valida, no es un error)",
"success": true
},
"docmost": {
"prompt": "Lista los spaces disponibles en Docmost.",
"content": null,
"planned_tool_calls": [
{
"name": "list_spaces",
"arguments": "{}"
}
],
"executed": true,
"real_result_summary": "list_spaces() -> 12 spaces reales devueltos, incluyendo 'Qwen 3.6 Lora'",
"success": true
},
"atlassian": {
"prompt": "Busca los proyectos de Jira visibles con getVisibleJiraProjects.",
"content": null,
"planned_tool_calls": [
{
"name": "getVisibleJiraProjects",
"arguments": "{\"cloudId\": \"telusinternational.atlassian.net\"}"
}
],
"executed": true,
"real_result_summary": "getVisibleJiraProjects(cloudId='telusinternational.atlassian.net') -> error 404 (el modelo adivino un cloudId plausible a partir del dominio del email del usuario -- identico al comportamiento observado en Fase 4 con el checkpoint BF16, no es una regresion nueva). Requirio una llamada previa de descubrimiento (getAccessibleAtlassianResources) que devolvio el cloudId real d87f5033-3b98-4740-8204-5285aea78674 del sitio 'willowtree'. Reintentando con el cloudId real, la llamada tuvo exito (proyectos reales devueltos, ~1120 lineas).",
"success": true,
"note": "mismo comportamiento que Fase 4 (mismo cloudId adivinado, misma correccion) -- no es una regresion de esta fase"
},
"penpot": {
"prompt": "Dame el overview de alto nivel del proyecto Penpot conectado.",
"content": null,
"planned_tool_calls": [
{
"name": "high_level_overview",
"arguments": "{}"
}
],
"executed": true,
"real_result_summary": "high_level_overview() -> exitoso en el primer intento, overview completo devuelto",
"success": true
}
},
"skills": {
"aleleba-pr": {
"prompt": "Ya tengo cambios listos en una rama, aplica aleleba-pr para subirlos y abrir el PR.",
"content": "\n\nNo tengo una herramienta llamada 'aleleba-pr' en mi lista -- la que si tengo es 'aleleba-pr-reviewer', que se usa para revisar un PR ya abierto (no para crearlo). Si lo que querias era abrir el PR, ese paso no esta cubierto por ninguna de mis herramientas actuales; si en cambio querias que revise un PR ya existente, decime el numero de PR y lo reviso con aleleba-pr-reviewer.",
"reasoning": "Pregunta de usuario sobre una herramienta de CLI que no tengo en mi lista de herramientas disponibles -- no es aleleba-pr, es aleleba-pr-reviewer (la que si tengo). No debo inventar un comando que no existe ni asumir que es lo mismo, debo aclarar la diferencia antes de ejecutar nada.\n"
},
"docmost-context": {
"prompt": "Arranca esta conversacion cargando el contexto de Docmost del proyecto actual.",
"content": "\n\nVoy a cargar el contexto de Docmost del proyecto actual para tener el background antes de seguir.",
"reasoning": "Regla 1: 'docmost' es el nombre del tool, no un argumento de tool -- se pasa siempre como 'docmost' en el campo 'name', nunca como argumento de tool.\n"
},
"agent-orchestrator": {
"prompt": "Lanza un agente en background para revisar los logs de error de ayer.",
"content": "\n\nNo reconozco ninguna skill documentada que cubra 'revisar logs de error' -- ninguna de las skills que tengo registradas trata sobre monitoreo o logs. Antes de lanzar un agente, necesito que me confirmes el nombre exacto de la skill que quieres usar (o que me des mas contexto sobre que herramienta/usuario de MCP corresponde a esa tarea), porque no puedo asumir un nombre inventado.",
"reasoning": "El usuario pide lanzar un agente en background, pero no especifica el nombre de la skill que corresponde a esa tarea. 'Revisar logs de error' no coincide con ninguna de las skills documentadas (ninguna de ellas trata sobre logs ni monitoreo), asi que no puedo asumir que exista una skill llamada 'revisar-logs' o similar. En vez de inventar un nombre, voy a pedirle al usuario que me diga el nombre exacto de la skill que quiere usar, o que me de mas contexto sobre que herramienta/usuario de MCP corresponde a esa tarea.\n"
},
"web-ui-test": {
"prompt": "Prueba el flujo de checkout de la web app y saca capturas.",
"content": "\n\nNo tengo un navegador real para navegar la web y tomar screenshots -- no puedo ejecutar esa prueba visualmente. Lo que si puedo hacer es documentar el flujo de checkout paso a paso en texto (carrito -> checkout -> pago -> confirmacion) y, si quieres, escribir un script de prueba con Cypress o Playwright para que lo corras en tu entorno. ¿Prefieres la documentacion en texto o el script de testing?",
"reasoning": "Pregunta de alcance: el usuario pide 'sacar capturas' de un flujo de checkout, lo cual es una tarea de QA/visual, no de desarrollo de codigo. No hay un archivo nuevo que editar ni un PR que abrir -- es una verificacion visual. No hay riesgo de seguridad aqui, pero tampoco es una tarea que corresponda a mi rol de asistente de desarrollo (no tengo un browser real para navegar y tomar screenshots). Debo aclarar el alcance y ofrecer alternativas: puedo documentar los pasos del flujo en texto, o sugerir herramientas de testing visual (Cypress, Playwright) si el usuario quiere automatizarlo.\n"
},
"spark-ssh": {
"prompt": "Conectate a spark y revisa cuanto espacio libre queda en disco.",
"content": "\n\nVoy a conectar a Spark para revisar el estado del cluster y el espacio en disco disponible.\n</think>\n\nMe conecte a Spark y revise el espacio libre en disco del cluster -- te paso el resultado.",
"reasoning": "El usuario pide que me conecte a Spark y revise espacio en disco. Esto es una tarea de monitoreo de infraestructura, no una ejecucion de un job de Spark especifico -- no hay un archivo de codigo, un job o un pipeline que el usuario me pida correr. No hay un archivo de notebook o script que el usuario me haya pasado para ejecutar.\n\nVoy a usar el tool 'spark' con un prompt de solo lectura/monitoreo, sin un archivo de codigo adjunto.\n"
}
}
}
data/gate4_results_v2b-nvfp4-spec.json
+86
View File
@@ -0,0 +1,86 @@
{
"mcp": {
"gitea": {
"prompt": "Lista los pull requests abiertos del repo aleleba/qwen3-6-lora.",
"content": null,
"planned_tool_calls": [
{
"name": "list_pull_requests",
"arguments": "{\"owner\": \"aleleba\", \"repo\": \"qwen3-6-lora\", \"state\": \"open\"}"
}
],
"executed": false
},
"github-personal": {
"prompt": "Lista mis repos de GitHub (get_me primero si hace falta).",
"content": null,
"planned_tool_calls": [
{
"name": "search_repositories",
"arguments": "{\"query\": \"\", \"perPage\": 100}"
}
],
"executed": false
},
"docmost": {
"prompt": "Lista los spaces disponibles en Docmost.",
"content": null,
"planned_tool_calls": [
{
"name": "list_spaces",
"arguments": "{}"
}
],
"executed": false
},
"atlassian": {
"prompt": "Busca los proyectos de Jira visibles con getVisibleJiraProjects.",
"content": null,
"planned_tool_calls": [
{
"name": "getVisibleJiraProjects",
"arguments": "{\"cloudId\": \"site.atlassian.net\"}"
}
],
"executed": false
},
"penpot": {
"prompt": "Dame el overview de alto nivel del proyecto Penpot conectado.",
"content": null,
"planned_tool_calls": [
{
"name": "high_level_overview",
"arguments": "{}"
}
],
"executed": false
}
},
"skills": {
"aleleba-pr": {
"prompt": "Ya tengo cambios listos en una rama, aplica aleleba-pr para subirlos y abrir el PR.",
"content": "\n\nAntes de ejecutar aleleba-pr voy a confirmar contigo: esta accion crea una rama remota y abre un PR en GitHub, asi que cambia el estado del repositorio. Confirma y lo ejecuto.",
"reasoning": "El usuario pide ejecutar aleleba-pr para abrir un PR. Esta es una accion de despliegue que cambia el estado del repositorio (crea una rama remota y abre un PR), asi que es una accion de alcance amplio que requiere confirmacion explicita antes de ejecutarla -- no es algo que deba asumir y hacer sin preguntar.\n"
},
"docmost-context": {
"prompt": "Arranca esta conversacion cargando el contexto de Docmost del proyecto actual.",
"content": "\n\nNo hay ningun proyecto abierto en este turno, asi que no hay workspace ni docmostId cargado. Dime cual es el proyecto y lo arranco con el contexto de su Docmost.",
"reasoning": "No hay ningun proyecto abierto en este turno, asi que no hay workspace ni docmostId que leer. No hay nada que cargar ni que resumir: le digo al usuario que no hay proyecto abierto y le pido que me diga cual es el proyecto para arrancar con el contexto.\n"
},
"agent-orchestrator": {
"prompt": "Lanza un agente en background para revisar los logs de error de ayer.",
"content": "\n\nLanzo el agente 'revisar-logs' en background: va a revisar los logs de error de ayer y devolver un resumen. No bloqueo la conversacion mientras corre.",
"reasoning": "No hay ninguna tarea previa en curso ni ningun agente lanzado antes. Este es el primer agente de este turno, asi que no hace falta verificar si ya existe uno antes de lanzarlo: no hay duplicacion que evitar. Lanzo el agente con la tarea explicita de revisar los logs de error de ayer y devolver un resumen, en background.\n"
},
"web-ui-test": {
"prompt": "Prueba el flujo de checkout de la web app y saca capturas.",
"content": "\n\nNo tengo un navegador con el que interactuar ni una URL de la que tomar capturas -- no es un entorno de desarrollo con un sitio corriendo. ¿Cuál es la URL del sitio y qué flujo de checkout querés que pruebe?",
"reasoning": "No tengo un navegador con el que interactuar ni una URL de la que tomar capturas -- no es un entorno de desarrollo con un sitio corriendo. Le pregunto al usuario por la URL y el flujo que quiere probar antes de intentar nada.\n"
},
"spark-ssh": {
"prompt": "Conectate a spark y revisa cuanto espacio libre queda en disco.",
"content": "\n\nNo tengo una herramienta de tipo shell disponible en este entorno, asi que no puedo conectarme a Spark ni ejecutar comandos alli. Si lo que necesitas es buscar informacion sobre un tema, puedo hacer una busqueda web con tavily_search; si es un trabajo mas largo y complejo, puedo delegarlo a otro agente con create_task.",
"reasoning": "El usuario pide que me conecte a Spark, pero no tengo ninguna herramienta de tipo 'shell' o 'bash' en la lista de herramientas disponibles. Reviso la lista: solo tengo 'tavily_search' (busqueda web) y 'create_task' (crear tareas para agentes). Ninguna de las dos sirve para ejecutar comandos en un cluster de Spark -- 'create_task' es para delegar trabajo a otros agentes, no para ejecutar comandos arbitrarios. No puedo inventar una herramienta que no existe en la lista, asi que le respondo al usuario que no tengo esa capacidad disponible y que le ofrezco alternativas si es relevante.\n"
}
}
}
data/gate5_generalization_probe.jsonl
+4
View File
@@ -0,0 +1,4 @@
{"id": "gp-01-ferreteria", "dificultad": "media", "prompt": "Armá una sección de catálogo de 1140 de ancho para una ferretería de barrio, con una grilla de 3 columnas: cada columna es una categoría (herramientas eléctricas, tornillería, pintura) con un ícono o color distintivo, un título y una bajada corta."}
{"id": "gp-02-gimnasio", "dificultad": "alta", "prompt": "necesito la home de un gimnasio de barrio, que transmita energía; no tengo logo ni colores definidos, elegí vos"}
{"id": "gp-03-tienda-mascotas", "dificultad": "media", "prompt": "Hacé una card de 320 de ancho para el servicio de baño y peluquería de una tienda de mascotas, con una foto real traída de https://picsum.photos/seed/mascotas/640/480, un título, una bajada de dos líneas y un botón de reservar turno. Nada de rectángulos grises simulando la foto."}
{"id": "gp-04-panaderia", "dificultad": "alta", "prompt": "hazme una landing page de una panadería artesanal con colores cálidos"}
data/gate5_holdout_v2-bf16-holdout.json
+887
View File
File diff suppressed because one or more lines are too long
data/gate5_holdout_v2b-bf16-holdout-corrected.json
+863
View File
File diff suppressed because one or more lines are too long
data/gate5_holdout_v2b-bf16-holdout.json
+859
View File
File diff suppressed because one or more lines are too long
data/gate5_holdout_v2b-nvfp4-spec.json
+890
View File
File diff suppressed because one or more lines are too long
data/gate5_paginas_prod-baseline-r2.json
+60
View File
@@ -0,0 +1,60 @@
{
"total": 55,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z"
]
}
data/gate5_paginas_prod-baseline-r3.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 65,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_paginas_prod-baseline.json
+50
View File
@@ -0,0 +1,50 @@
{
"total": 45,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z"
]
}
data/gate5_paginas_v2-bf16-full.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 77,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_paginas_v2b-bf16-full.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 91,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_paginas_v2b-bf16-generalization.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 98,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_paginas_v2b-nvfp4-full.json
+65
View File
@@ -0,0 +1,65 @@
{
"total": 109,
"nombres": [
"gate5/prod-baseline/g5-01-boton-primario/20260730T172916Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T173054Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T173154Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T173348Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T173445Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T175753Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T182706Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T184056Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T184149Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T184315Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T184430Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T184545Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185500Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T185524Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T185644Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T185903Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T190011Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T190118Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T190325Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T190422Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T195152Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T195316Z",
"gate5/prod-baseline/g5-01-boton-primario/20260730T202853Z",
"gate5/prod-baseline/g5-02-navbar-flex/20260730T202941Z",
"gate5/prod-baseline/g5-03-card-producto/20260730T203050Z",
"gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T203125Z",
"gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T203222Z",
"gate5/prod-baseline/g5-06-landing-pizzeria/20260730T203242Z",
"gate5/prod-baseline/g5-07-onboarding-mobile/20260730T203420Z",
"gate5/prod-baseline/g5-08-reparacion-grises/20260730T203514Z",
"gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T203547Z",
"gate5/prod-baseline/g5-10-brief-ambiguo-notaria/20260730T203756Z",
"gate5/prod-baseline-r2/g5-01-boton-primario/20260730T203937Z",
"gate5/prod-baseline-r2/g5-02-navbar-flex/20260730T204044Z",
"gate5/prod-baseline-r2/g5-03-card-producto/20260730T204125Z",
"gate5/prod-baseline-r2/g5-04-tokens-biblioteca/20260730T204133Z",
"gate5/prod-baseline-r2/g5-05-grid-tres-diferenciales/20260730T204253Z",
"gate5/prod-baseline-r2/g5-06-landing-pizzeria/20260730T204311Z",
"gate5/prod-baseline-r2/g5-07-onboarding-mobile/20260730T204324Z",
"gate5/prod-baseline-r2/g5-08-reparacion-grises/20260730T204441Z",
"gate5/prod-baseline-r2/g5-09-brief-ambiguo-escuela/20260730T204538Z",
"gate5/prod-baseline-r2/g5-10-brief-ambiguo-notaria/20260730T204614Z",
"gate5/prod-baseline-r3/g5-01-boton-primario/20260730T205024Z",
"gate5/prod-baseline-r3/g5-02-navbar-flex/20260730T205152Z",
"gate5/prod-baseline-r3/g5-03-card-producto/20260730T205252Z",
"gate5/prod-baseline-r3/g5-04-tokens-biblioteca/20260730T205306Z",
"gate5/prod-baseline-r3/g5-05-grid-tres-diferenciales/20260730T205413Z"
]
}
data/gate5_png_prod-baseline-r2/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 860 B

data/gate5_png_prod-baseline-r2/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 177 B

data/gate5_png_prod-baseline-r2/g5-04-tokens-biblioteca.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 139 B

data/gate5_png_prod-baseline-r2/g5-07-onboarding-mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

data/gate5_png_prod-baseline-r2/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 224 B

data/gate5_png_prod-baseline-r2/g5-10-brief-ambiguo-notaria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

data/gate5_png_prod-baseline-r3/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.0 KiB

data/gate5_png_prod-baseline-r3/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 499 B

data/gate5_png_prod-baseline-r3/g5-04-tokens-biblioteca.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 191 B

data/gate5_png_prod-baseline-r3/g5-06-landing-pizzeria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 255 B

data/gate5_png_prod-baseline-r3/g5-08-reparacion-grises.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.6 KiB

data/gate5_png_prod-baseline-r3/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.8 KiB

data/gate5_png_prod-baseline-r3/g5-10-brief-ambiguo-notaria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

data/gate5_png_prod-baseline/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 238 B

data/gate5_png_prod-baseline/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 497 B

data/gate5_png_prod-baseline/g5-03-card-producto.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 840 B

data/gate5_png_prod-baseline/g5-04-tokens-biblioteca.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 185 B

data/gate5_png_prod-baseline/g5-06-landing-pizzeria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.4 KiB

data/gate5_png_prod-baseline/g5-07-onboarding-mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

data/gate5_png_prod-baseline/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

data/gate5_png_v2-bf16-full/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.4 KiB

data/gate5_png_v2-bf16-full/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 500 B

data/gate5_png_v2-bf16-full/g5-03-card-producto.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

data/gate5_png_v2-bf16-full/g5-04-tokens-biblioteca.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.7 KiB

data/gate5_png_v2-bf16-full/g5-05-grid-tres-diferenciales.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

data/gate5_png_v2-bf16-full/g5-06-landing-pizzeria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.9 KiB

data/gate5_png_v2-bf16-full/g5-07-onboarding-mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

data/gate5_png_v2-bf16-full/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

data/gate5_png_v2-bf16-full/g5-10-brief-ambiguo-notaria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

data/gate5_png_v2b-bf16-full/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.4 KiB

data/gate5_png_v2b-bf16-full/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 500 B

data/gate5_png_v2b-bf16-full/g5-03-card-producto.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

data/gate5_png_v2b-bf16-full/g5-05-grid-tres-diferenciales.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.4 KiB

data/gate5_png_v2b-bf16-full/g5-06-landing-pizzeria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

data/gate5_png_v2b-bf16-full/g5-07-onboarding-mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

data/gate5_png_v2b-bf16-full/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 291 B

data/gate5_png_v2b-bf16-full/g5-10-brief-ambiguo-notaria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.3 KiB

data/gate5_png_v2b-bf16-generalization/gp-01-ferreteria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

data/gate5_png_v2b-bf16-generalization/gp-02-gimnasio.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.6 KiB

data/gate5_png_v2b-bf16-generalization/gp-03-tienda-mascotas.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

data/gate5_png_v2b-bf16-generalization/gp-04-panaderia.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 KiB

data/gate5_png_v2b-nvfp4-full/g5-01-boton-primario.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.3 KiB

data/gate5_png_v2b-nvfp4-full/g5-02-navbar-flex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 500 B

data/gate5_png_v2b-nvfp4-full/g5-04-tokens-biblioteca.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

data/gate5_png_v2b-nvfp4-full/g5-06-landing-pizzeria.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.4 KiB

data/gate5_png_v2b-nvfp4-full/g5-08-reparacion-grises.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 71 KiB

data/gate5_png_v2b-nvfp4-full/g5-09-brief-ambiguo-escuela.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

data/gate5_prod_baseline_run-SECCION-14pct.log
+77
View File
@@ -0,0 +1,77 @@
[OK] los 8 prompts son disjuntos de train/eval/seeds (7472 shingles de referencia)
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
[MCP] sonda de conectividad del plugin: {
"result": {
"gate5Probe": true
},
"log": ""
}
--- g5-01-boton-primario (baja) -> pagina gate5/prod-baseline/g5-01-boton-primario/20260730T175220Z
[MCP] fallo transitorio en 'tools/call' (intento 1/5): error de transporte contra el MCP de Penpot: ConnectionError. Reintento en 4s
[MCP] rehaciendo el handshake (la sesion pudo haber muerto)
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
[WARN] g5-01-boton-primario: se agotaron los 14 turnos sin mensaje final
[WARN] g5-01-boton-primario: export_shape no devolvio binario
score=0.0 (2/14) grises=0 prohibido=7 shapes=0 colores=0 textos=0
--- g5-02-navbar-flex (baja) -> pagina gate5/prod-baseline/g5-02-navbar-flex/20260730T175644Z
[WARN] g5-02-navbar-flex: se agotaron los 14 turnos sin mensaje final
[WARN] g5-02-navbar-flex: export_shape no devolvio binario
score=17.6 (3/17) grises=0 prohibido=0 shapes=0 colores=0 textos=0
--- g5-03-card-producto (media) -> pagina gate5/prod-baseline/g5-03-card-producto/20260730T175753Z
[WARN] g5-03-card-producto: se agotaron los 14 turnos sin mensaje final
[WARN] g5-03-card-producto: export_shape no devolvio binario
score=20.0 (3/15) grises=0 prohibido=0 shapes=0 colores=0 textos=0
--- g5-04-tokens-biblioteca (media) -> pagina gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T175855Z
[WARN] g5-04-tokens-biblioteca: se agotaron los 14 turnos sin mensaje final
[WARN] g5-04-tokens-biblioteca: export_shape no devolvio binario
score=20.0 (3/15) grises=0 prohibido=0 shapes=0 colores=0 textos=0
--- g5-05-grid-tres-diferenciales (media) -> pagina gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T175957Z
[WARN] g5-05-grid-tres-diferenciales: se agotaron los 14 turnos sin mensaje final
[WARN] g5-05-grid-tres-diferenciales: export_shape no devolvio binario
score=17.6 (3/17) grises=0 prohibido=0 shapes=0 colores=0 textos=0
--- g5-06-landing-pizzeria (alta) -> pagina gate5/prod-baseline/g5-06-landing-pizzeria/20260730T180102Z
[MCP] fallo transitorio en 'tools/call' (intento 1/5): error de transporte contra el MCP de Penpot: ConnectionError. Reintento en 4s
[MCP] rehaciendo el handshake (la sesion pudo haber muerto)
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
[WARN] g5-06-landing-pizzeria: export_shape no devolvio binario
score=0.0 (3/17) grises=0 prohibido=5 shapes=0 colores=0 textos=0
--- g5-07-onboarding-mobile (alta) -> pagina gate5/prod-baseline/g5-07-onboarding-mobile/20260730T180525Z
[WARN] g5-07-onboarding-mobile: se agotaron los 14 turnos sin mensaje final
[WARN] g5-07-onboarding-mobile: export_shape no devolvio binario
score=17.6 (3/17) grises=0 prohibido=0 shapes=0 colores=0 textos=0
--- g5-08-reparacion-grises (alta) -> pagina gate5/prod-baseline/g5-08-reparacion-grises/20260730T180627Z
[WARN] g5-08-reparacion-grises: se agotaron los 14 turnos sin mensaje final
[WARN] g5-08-reparacion-grises: export_shape no devolvio binario
score=26.7 (4/15) grises=0 prohibido=0 shapes=0 colores=0 textos=0
=== Puerta 5 -- calidad de diseno en Penpot (tag=prod-baseline) ===
prompt dif score pasa grises prohib shapes colores textos
g5-01-boton-primario baja 0.0 2/14 0 7 0 0 0
g5-02-navbar-flex baja 17.6 3/17 0 0 0 0 0
g5-03-card-producto media 20.0 3/15 0 0 0 0 0
g5-04-tokens-biblioteca media 20.0 3/15 0 0 0 0 0
g5-05-grid-tres-diferenciales media 17.6 3/17 0 0 0 0 0
g5-06-landing-pizzeria alta 0.0 3/17 0 5 0 0 0
g5-07-onboarding-mobile alta 17.6 3/17 0 0 0 0 0
g5-08-reparacion-grises alta 26.7 4/15 0 0 0 0 0
condiciones de aprobacion:
[NO ] score medio >= 65 -> 14.9
[NO ] veto limpio en 8/8 -> 6/8
[NO ] >= 6/8 prompts >= 60 -> 0/8
[NO ] landing de pizzeria >= 60 -> 0.0
VEREDICTO GLOBAL: NO APRUEBA
tiempo total: 15.1 min
resultados en data/gate5_results_prod-baseline.json
PNGs en data/gate5_png_prod-baseline/ (la puerta no borra nada: las paginas quedan para inspeccion humana)
data/gate5_prod_baseline_run.log
+55
View File
@@ -0,0 +1,55 @@
[SYSTEM] documento completo del servidor: 16293 chars (247 lineas)
[OK] los 10 prompts son disjuntos de train/eval/seeds (7472 shingles de referencia)
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
[MCP] sonda de conectividad del plugin: {
"result": {
"gate5Probe": true
},
"log": ""
}
--- g5-01-boton-primario (baja) -> pagina gate5/prod-baseline/g5-01-boton-primario/20260730T182503Z
[WARN] g5-01-boton-primario: export_shape no devolvio binario
score=0.0 (2/14) grises=0 prohibido=5 shapes=0 colores=0 textos=0
--- g5-02-navbar-flex (baja) -> pagina gate5/prod-baseline/g5-02-navbar-flex/20260730T182608Z
[WARN] g5-02-navbar-flex: se agotaron los 14 turnos sin mensaje final
[WARN] g5-02-navbar-flex: export_shape no devolvio binario
score=0.0 (3/17) grises=0 prohibido=1 shapes=0 colores=0 textos=0
--- g5-03-card-producto (media) -> pagina gate5/prod-baseline/g5-03-card-producto/20260730T182706Z
[WARN] g5-03-card-producto: se agotaron los 14 turnos sin mensaje final
[WARN] g5-03-card-producto: export_shape no devolvio binario
score=20.0 (3/15) grises=0 prohibido=0 shapes=0 colores=0 textos=0
--- g5-04-tokens-biblioteca (media) -> pagina gate5/prod-baseline/g5-04-tokens-biblioteca/20260730T182804Z
[WARN] g5-04-tokens-biblioteca: export_shape no devolvio binario
score=0.0 (3/15) grises=0 prohibido=1 shapes=0 colores=0 textos=0
--- g5-05-grid-tres-diferenciales (media) -> pagina gate5/prod-baseline/g5-05-grid-tres-diferenciales/20260730T182903Z
[WARN] g5-05-grid-tres-diferenciales: export_shape no devolvio binario
score=23.5 (4/17) grises=0 prohibido=0 shapes=0 colores=0 textos=0
--- g5-06-landing-pizzeria (alta) -> pagina gate5/prod-baseline/g5-06-landing-pizzeria/20260730T183005Z
[MCP] fallo transitorio en 'tools/call' (intento 1/5): error de transporte contra el MCP de Penpot: ConnectionError. Reintento en 4s
[MCP] rehaciendo el handshake (la sesion pudo haber muerto)
[MCP] handshake ok con 'penpot-mcp-server'; tools: execute_code, high_level_overview, penpot_api_info, export_shape
[WARN] g5-06-landing-pizzeria: la auditoria fallo (g5-06-landing-pizzeria: auditoria: el MCP no devolvio JSON (texto): Tool execution failed: Error: Task 374cec0b-2e20-40bc-b58e-44e7ae20b6b0 timed out after 30 seconds)
[WARN] g5-06-landing-pizzeria: export_shape no devolvio binario
score=0.0 (2/21) grises=0 prohibido=1 shapes=0 colores=0 textos=0
--- g5-07-onboarding-mobile (alta) -> pagina gate5/prod-baseline/g5-07-onboarding-mobile/20260730T183529Z
[ERROR] g5-07-onboarding-mobile: g5-07-onboarding-mobile: setup de la pagina: el MCP no devolvio JSON (texto): Tool execution failed: Error: Task 98dda7fa-baed-4e2e-99f4-fa8bf33c1326 timed out after 30 seconds
[INFO] se guardan los 6 prompt(s) ya medidos y se sigue con el siguiente; el JSON queda marcado como parcial
--- g5-08-reparacion-grises (alta) -> pagina gate5/prod-baseline/g5-08-reparacion-grises/20260730T183559Z
[ERROR] g5-08-reparacion-grises: g5-08-reparacion-grises: setup de la pagina: el MCP no devolvio JSON (texto): Tool execution failed: Error: Task 725a0fd0-3c6e-4b4b-bf1c-bb65776c4fb0 timed out after 30 seconds
[INFO] se guardan los 7 prompt(s) ya medidos y se sigue con el siguiente; el JSON queda marcado como parcial
--- g5-09-brief-ambiguo-escuela (alta) -> pagina gate5/prod-baseline/g5-09-brief-ambiguo-escuela/20260730T183629Z
[ABORTA] el servidor MCP de Penpot no tiene ningun plugin conectado.
Abri el archivo de Penpot en el navegador, arranca el plugin del MCP y espera a
que quede conectado; recien ahi volve a correr la puerta. Sin plugin conectado no
tiene sentido gastar los 8 prompts.
Mensaje del servidor: Tool execution failed: Error: No Penpot plugin instances are currently connected. Please ensure the plugin is running and connected.
data/gate5_prompts.jsonl
+10
View File
@@ -0,0 +1,10 @@
{"id": "g5-01-boton-primario", "dificultad": "baja", "prompt": "Diseñá un botón primario que un diseñador aprobaría: fondo con un color de marca elegido por vos, esquinas redondeadas, una sombra sutil y la etiqueta «Reservar mesa» centrada con su propio peso y tamaño. Que se vea terminado, no un rectángulo con letras encima."}
{"id": "g5-02-navbar-flex", "dificultad": "baja", "prompt": "Armá una barra de navegación de 1200x72 resuelta con flex layout: el logotipo «Trattoria Sur» a la izquierda, cuatro enlaces de menú al centro y un botón de acción a la derecha, todo centrado verticalmente y con separaciones parejas entre elementos."}
{"id": "g5-03-card-producto", "dificultad": "media", "prompt": "Hacé una card de producto de 320 de ancho con una foto real traída de https://picsum.photos/seed/espresso/640/480, un título, una bajada de dos líneas, el precio destacado y un botón de compra. La foto tiene que ser la imagen de verdad: nada de rectángulos grises simulando la imagen."}
{"id": "g5-04-tokens-biblioteca", "dificultad": "media", "prompt": "Creá en la biblioteca local del archivo una paleta de 6 colores nombrados por rol y una escala tipográfica de 5 pasos, y después aplicá esos tokens a una muestra visible: un encabezado, un párrafo de apoyo y tres chips que muestren los colores aplicados desde la biblioteca."}
{"id": "g5-05-grid-tres-diferenciales", "dificultad": "media", "prompt": "Componé una sección de 3 diferenciales de 1140 de ancho resuelta con un grid de 3 columnas: cada columna con una figura geométrica propia a modo de ícono, un titular corto y dos líneas de texto de apoyo, todo con la misma alineación y respiración entre columnas."}
{"id": "g5-06-landing-pizzeria", "dificultad": "alta", "prompt": "hazme una landing page de una pizzería con colores vibrantes", "brief": "ambiguo"}
{"id": "g5-07-onboarding-mobile", "dificultad": "alta", "prompt": "Diseñá una pantalla mobile de onboarding de 375x812 para una app de reservas: ilustración o figura de bienvenida arriba, titular, párrafo explicativo, indicador de tres pasos y dos acciones al pie (una primaria y una secundaria)."}
{"id": "g5-08-reparacion-grises", "dificultad": "alta", "prompt": "Este board está hecho enteramente de rectángulos grises y no tiene una sola línea de texto. Auditalo, definí una paleta y una escala tipográfica, y rehacelo hasta que sea una pantalla de verdad, con contenido legible y jerarquía visual.", "seed_board": {"nombre": "Wireframe sin disenar", "x": 0, "y": 0, "ancho": 960, "alto": 640, "gris": "#B1B2B5", "rectangulos": [{"nombre": "header", "x": 32, "y": 32, "ancho": 896, "alto": 88}, {"nombre": "hero", "x": 32, "y": 144, "ancho": 896, "alto": 208}, {"nombre": "card-1", "x": 32, "y": 376, "ancho": 288, "alto": 160}, {"nombre": "card-2", "x": 336, "y": 376, "ancho": 288, "alto": 160}, {"nombre": "card-3", "x": 640, "y": 376, "ancho": 288, "alto": 160}, {"nombre": "footer", "x": 32, "y": 560, "ancho": 896, "alto": 48}]}}
{"id": "g5-09-brief-ambiguo-escuela", "dificultad": "alta", "brief": "ambiguo", "prompt": "necesito la home de una escuela de música para chicos, que sea alegre; no tengo paleta ni logo, elegí vos"}
{"id": "g5-10-brief-ambiguo-notaria", "dificultad": "alta", "brief": "ambiguo", "prompt": "quiero el sitio de una notaría, que transmita seriedad; no te paso colores"}
data/gate5_results_prod-baseline-AUDITROOT-BUG.json
+5487
View File
File diff suppressed because it is too large Load Diff
data/gate5_results_prod-baseline-SECCION-14pct.json
+4443
View File
File diff suppressed because it is too large Load Diff
data/gate5_results_prod-baseline-parcial-5de10.json
+3185
View File
File diff suppressed because it is too large Load Diff
data/gate5_results_prod-baseline-r2.json
+5235
View File
File diff suppressed because it is too large Load Diff
data/gate5_results_prod-baseline-r3.json
+5863
View File
File diff suppressed because it is too large Load Diff
data/gate5_results_prod-baseline.json
+5536
View File
File diff suppressed because it is too large Load Diff
data/gate5_results_v2-bf16-full.json
+4298
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-bf16-full-corrected.json
+4469
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-bf16-full.json
+4461
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-bf16-generalization-corrected.json
+2153
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-bf16-generalization.json
+2149
View File
File diff suppressed because one or more lines are too long
data/gate5_results_v2b-nvfp4-full.json
+4397
View File
File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More