c24f0ab23672dedd1481fac43287077006650e52
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 aleleba
|
ad62277f14
|
Phase 6.3: stop the gate depending on penpot.root, and stop blaming the plugin for its own bugs
The setup returned penpot.root.id and the run aborted when it came back as an empty string on the second prompt of a batch. penpot.root is the root of the ACTIVE page, and after createPage plus openPage it need not have caught up yet - a race the page-emptying code introduced, since that leaves the emptied page active. The value was never useful anyway: every page shares the same root id, so it identified nothing. Setup now returns only pageId, which is unique and stable, and both the audit and the cleanup use it. The diagnostic message is the second half of the same mistake. It printed "PLUGIN DEGRADADO, ask the user to reload the browser" whenever any prompt went unmeasured, regardless of why - so it said that for a failure that was entirely the gate's own. A message that sends the user to reload their browser when the bug is mine costs both of us time. It now classifies on the error text: task timeouts and transport drops point at the plugin, anything else points at the gate and says so explicitly. Also closes two evaluation leaks the gate's own pre-flight caught, both in the seeds rather than the gate prompts, since the prompts have to stay as a real user would write them: - A seed shared the 6-gram "la home de una escuela de" with gate prompt 9. My first fix was overwritten by a subagent still writing the file, which is why it reappeared. - A seed used the same business as gate prompt 9 - a music school - without sharing any 6-gram. Shingles cannot see that: two texts describe the same business without sharing words. Training on the domain we then evaluate inflates the result invisibly. So the gate now also checks that no seed uses any of the gate's business nouns, listed explicitly. The training mix was rebuilt: it had been assembled before the 20 ambiguous-brief seeds existed, so training on it would not have used the corpus that was audited. The ambiguous-brief class gets its own mix portion rather than being folded into design, because diluted across 76 design seeds it would be at the mercy of a ratio, and that is the class the user named as the main painpoint. 125 seeds, 168 unique payloads, 446 distinct user prompts, 901 train and 99 eval. |
||
|
aleleba
|
c9792c5c40
|
Phase 6.3: rewrite the Penpot seed corpus and build the LoRA #2 mix
Replaces the 41 old Penpot seeds with 105 new ones. The old set taught three API forms that do not exist - findShapeById(page, id), shape.layout, and createText() with no argument - and 21 of the 41 used the first one, so patching was never an option: keeping them would mean fine-tuning against the correction. The valuable lessons were re-founded on the real API instead (the reversed children array in flex, persisting ids in storage, never logging what you also return). 105 rather than 96 because nine multi-section compositions are split into two trajectories each: the first builds the skeleton and persists ids, palette, scale and helper functions in storage, the second recovers them and fills the sections. That was forced by the 3000-token ceiling, but it is better pedagogy anyway, and it is what execute_code's own description asks for. It also paid for itself: the helpers cost ~600 chars once instead of twice, and the skeleton call needs no export_shape, which freed the budget to carry the verbatim system block. Measured against the linter with the corpus-wide thresholds active: 143 unique code payloads (the old set had 36), 32% of seeds carrying the server's system block verbatim (the old set had none), and every coverage category met where the old set had zero addGridLayout, zero shadows, zero uploadMediaUrl and zero layoutChild. The flagship changed domain from pizzeria to an empanada shop. Gate 5's prompt 6 is the exact production failure, and the seed had been written to the same wording - a literal substring of the gate prompt, sharing two 6-gram shingles where the gate's disjointness check fails on one. Training on the prompt we then evaluate would make gate 5 measure memorisation. The real pizzeria prompt still runs in the human acceptance test. The same check also caught an onboarding seed sitting too close to gate prompt 7. Mix: 1000 examples split 90/10, giving 901 train and 99 eval. The mix is 1000 rather than 900 because 900 has to be the post-split train size: 900/16 = 56.25 steps per epoch x 3 = 168 steps, deliberately paired with phase 3's 166 so the optimiser trajectory length is comparable. Building 900 and then carving out eval would have left 811 and 152 steps, silently breaking that pairing. Two guards in the builder had to be corrected against the real corpus: - The forbidden-pattern scan now reads only `code` payloads. Scanning the serialised example flagged the corrective seeds themselves - the one that opens with the user asking "Importá esta imagen con import_image", the one quoting the overview line that mentions import_image verbatim, the one explaining that board.layout does not exist. They name the wrong API precisely in order to teach against it. - The exemption for error-recovery seeds is derived from content, not from the mix portion: a forbidden pattern is allowed only where its tool result is a real error string and a later payload does the same thing without it. Keying on the portion broke as soon as an error-recovery seed lived in group A1, where the findShapeById arity seed naturally belongs. Validated with the production tokenizer at preserve_thinking=true: p50 2466, p90 2988, max 3250 tokens over the 105 seeds. Ten sit just above 3000, so MAX_TOKENS goes to 3300 for the run and the smoke run will train on the 32 longest examples specifically, turning the peak-memory question into a ten-minute measurement instead of a risk discovered hours in. |
||
|
aleleba
|
a60d0751cf
|
Phase 6.3: fix the augmentation, close the holdout leak, stop rewarding invented parameters
Dataset build (05, 06): perturb_value is gone. It rewrote only tool_calls.arguments and left the tool results and the final answer saying something else, which is how data/train.jsonl ended up with 30 self-contradictory examples where the call says issue_number 82 and the answer says issue #77. Variation now comes from hand-written meta.paraphrases, or from meta.variation applied atomically across every field of the example at once. Nothing is substituted unless the seed declares it: guessing which number in a string is safe to change is what produced the contradictions in the first place. Prefix injection survives only as a fallback and only where the verb form can actually be conjugated, and there is a hard assert that no user turn matches the broken "Necesito que ¿Podés..." shape that 68 v1 prompts had. The penpot bucket is exempt from substitution entirely, since its payloads are code. Also asserts the bucket cannot collapse (verified: the old seeds give 320 rows from 83 unique trajectories and the build now fails) and scans for forbidden API patterns by importing them from the linter, so there is one source of truth. 06 now actually exits 1 on over-length rows. It printed [FILTERED], incremented a counter, and left the row in the file, which 10_train.py then trained on since it has no max_seq_length and batch 1. Gate 2 (32): reject any argument key absent from the schema, as its own failure category. It only checked required fields, so an invented scale or filePath passed - the gate was actively rewarding the exact behaviour this phase removes. Verified: export_shape with scale=2 now fails as unknown_argument, while a valid call still passes. Holdout (31, 35): rebalanced to penpot 60 / 35 each, added 20 real design templates, and replaced the full-string equality check with 6-gram shingles. Measured: a light paraphrase of a train.jsonl prompt scores 43% overlap and now fails the build, where the old check let it through at "not equal". Value pools are asserted disjoint from the corpus. The "2x resolution" template stays, relabelled as an invented-argument probe now that gate 2 can detect one; the createBoolean template stays because the API is real and the new B2 seeds teach it. Also dedupes: the old holdout had 15 duplicate prompts out of 200, i.e. 15 wasted measurements. Note: rebalancing the holdout means the 192/200 gate 2 baseline from phase 5 no longer applies to it, so that baseline has to be re-measured against production on the new file before it can be compared to. Gate 3 (33): 11 content checklists for the non-obvious conventions of the other MCPs - GFM table separators in Docmost, the update_page staleness retry, commit message shape, never merging the PR, dict-not-XML tool arguments. That is the most likely regression no gate currently covers. Mix builder: added the anti-collapse guard, so 420 new-portion rows that are really 96 trajectories repeated cannot pass unnoticed. |
||
|
aleleba
|
19eb50f351
|
Phase 6.3: add the LoRA #2 mix builder and refine the linter
07_build_lora2_mix.py assembles train_lora2.jsonl (900), eval_lora2.jsonl and calibration_v2.jsonl from the new Penpot seeds plus a filtered replay sample of data/train.jsonl. It never writes data/train.jsonl or data/eval.jsonl: those are the provenance of the model in production and the gate 1 baseline, and regenerating them is not idempotent anyway, since stratified_split shuffles one RNG over the concatenated list, so touching the penpot bucket reshuffles every other bucket's split too. Two things worth flagging in the mix: The 45 "corrected penpot basics" the plan lists inside the replay portion do not come from data/train.jsonl. 21 of its 41 penpot seeds teach findShapeById(page, id) and 5 use shape.layout, so sampling that bucket would re-teach the exact bug this phase removes; the forbidden-pattern filter would drop them anyway. They come from the new corpus instead. This is a conscious deviation from the plan text and is recorded in the docstring. Variation comes only from hand-written meta.paraphrases, never from automatic value substitution. That is the v1 lesson: perturb_value rewrote only tool_calls.arguments and left the tool results and the final answer saying something else, producing 30 self-contradictory examples. A perturbed Penpot code payload is just broken code. Linter fixes, both false positives found by running it against the real corpus: - flex evidence for a bare appendChild is now scoped to the whole seed rather than the single payload. A multi-call seed builds the flex board in call one and stashes helpers in storage, so by the time call two does main.appendChild(...) neither addFlexLayout( nor .flex appears in that payload. The old scope flagged exactly the storage-persistence pattern that execute_code's own description asks for. - a grey hex is a problem when it is applied, not when it is searched for. The repair seeds have to name the greys they are about to replace, so greys are allowed in that group inside a comparison context. |