aleleba/qwen3-6-lora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 6.3: fix the augmentation, close the holdout leak, stop rewarding invented parameters

Browse Source 
Dataset build (05, 06):
perturb_value is gone. It rewrote only tool_calls.arguments and left the
tool results and the final answer saying something else, which is how
data/train.jsonl ended up with 30 self-contradictory examples where the
call says issue_number 82 and the answer says issue #77. Variation now
comes from hand-written meta.paraphrases, or from meta.variation applied
atomically across every field of the example at once. Nothing is
substituted unless the seed declares it: guessing which number in a string
is safe to change is what produced the contradictions in the first place.
Prefix injection survives only as a fallback and only where the verb form
can actually be conjugated, and there is a hard assert that no user turn
matches the broken "Necesito que ¿Podés..." shape that 68 v1 prompts had.
The penpot bucket is exempt from substitution entirely, since its payloads
are code. Also asserts the bucket cannot collapse (verified: the old seeds
give 320 rows from 83 unique trajectories and the build now fails) and
scans for forbidden API patterns by importing them from the linter, so
there is one source of truth.

06 now actually exits 1 on over-length rows. It printed [FILTERED],
incremented a counter, and left the row in the file, which 10_train.py
then trained on since it has no max_seq_length and batch 1.

Gate 2 (32): reject any argument key absent from the schema, as its own
failure category. It only checked required fields, so an invented scale or
filePath passed - the gate was actively rewarding the exact behaviour this
phase removes. Verified: export_shape with scale=2 now fails as
unknown_argument, while a valid call still passes.

Holdout (31, 35): rebalanced to penpot 60 / 35 each, added 20 real design
templates, and replaced the full-string equality check with 6-gram
shingles. Measured: a light paraphrase of a train.jsonl prompt scores 43%
overlap and now fails the build, where the old check let it through at
"not equal". Value pools are asserted disjoint from the corpus. The
"2x resolution" template stays, relabelled as an invented-argument probe
now that gate 2 can detect one; the createBoolean template stays because
the API is real and the new B2 seeds teach it. Also dedupes: the old
holdout had 15 duplicate prompts out of 200, i.e. 15 wasted measurements.

Note: rebalancing the holdout means the 192/200 gate 2 baseline from phase
5 no longer applies to it, so that baseline has to be re-measured against
production on the new file before it can be compared to.

Gate 3 (33): 11 content checklists for the non-obvious conventions of the
other MCPs - GFM table separators in Docmost, the update_page staleness
retry, commit message shape, never merging the PR, dict-not-XML tool
arguments. That is the most likely regression no gate currently covers.

Mix builder: added the anti-collapse guard, so 420 new-portion rows that
are really 96 trajectories repeated cannot pass unnoticed.
This commit is contained in:
aleleba
2026-07-30 17:16:05 +00:00
parent 19eb50f351
commit a60d0751cf
9 changed files with 2046 additions and 402 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/32_gate2_toolcalls.py
+79 -9
View File
@@ -12,13 +12,27 @@ configurado en docker-compose.eval.yml) lo hace vLLM en el servidor -- este scri
valida la RESPUESTA ya parseada (nunca re-implementa el parser con una regex propia):
- Si el modelo decide llamar una tool: valida que el nombre exista en el schema del MCP,
que los argumentos parseen como JSON valido, y que las propiedades "required" del
schema esten presentes.
que los argumentos parseen como JSON valido, que las propiedades "required" del
schema esten presentes, y que NINGUNA clave de argumento este ausente de
`parameters.properties` del schema.
- Si el modelo NO llama ninguna tool: se cuenta aparte (no es un error per se, algunos
prompts pueden resolverse sin tool-call, pero se reporta la tasa).
Reporta: % de prompts con tool_call sintacticamente valido (parseado sin excepcion por
vLLM, arguments=JSON valido, nombre y campos requeridos correctos) por MCP y global.
vLLM, arguments=JSON valido, nombre, campos requeridos y claves de argumento correctos)
por MCP y global.
FASE 6 -- la puerta premiaba la invencion de parametros
-------------------------------------------------------
Hasta la Fase 5 `validate_tool_call` solo chequeaba los `required` del schema: un argumento
INVENTADO que el schema no declara (`scale`, `filePath` en `export_shape`) **pasaba la
puerta**. Es decir, la puerta 2 premiaba activamente el comportamiento que la Fase 6 quiere
eliminar. Ahora toda clave ausente de `parameters.properties` es un fallo, contabilizado en
su propia categoria `unknown_argument` -- separada de `missing_required`, porque son errores
distintos y queremos poder medir la invencion por si sola.
El formato del JSON de resultados se mantiene compatible con los `gate2_results*.json` de las
Fases 4-5: las claves viejas siguen ahi con el mismo significado, las nuevas son aditivas.
"""
import argparse
import json
@@ -75,25 +89,47 @@ def to_openai_tools(tools):
return openai_tools
# Categorias de fallo, contabilizadas por separado. `unknown_argument` es la que mide
# invencion de parametros y por eso no se mezcla con `missing_required`.
FAILURE_KINDS = ("invalid_json", "unknown_tool", "missing_required", "unknown_argument")
def validate_tool_call(tool_call, tools):
"""Devuelve (ok, mensaje_de_error, categoria_de_fallo).
La categoria es None cuando la llamada es valida.
"""
name = tool_call["function"]["name"]
raw_args = tool_call["function"]["arguments"]
try:
args = json.loads(raw_args)
except json.JSONDecodeError as e:
return False, f"arguments no es JSON valido: {e}"
return False, f"arguments no es JSON valido: {e}", "invalid_json"
tool_def = tool_by_name(tools, name)
if tool_def is None:
return False, f"tool_call a nombre inexistente en el schema del MCP: {name}"
return False, f"tool_call a nombre inexistente en el schema del MCP: {name}", "unknown_tool"
schema = tool_def.get("inputSchema") or tool_def.get("parameters") or {}
required = schema.get("required", [])
missing = [r for r in required if r not in args]
if missing:
return False, f"faltan campos requeridos {missing} en la llamada a {name}"
return False, f"faltan campos requeridos {missing} en la llamada a {name}", "missing_required"
return True, None
# Invencion de parametros: cualquier clave que el schema no declare. Solo se puede
# juzgar si el schema declara `properties`; si no las declara (schema abierto), no hay
# con que comparar y no se penaliza.
properties = schema.get("properties")
if isinstance(properties, dict) and properties and isinstance(args, dict):
unknown = sorted(k for k in args if k not in properties)
if unknown:
return (
False,
f"argumentos inventados {unknown} ausentes de properties del schema de {name}",
"unknown_argument",
)
return True, None, None
def call_vllm(prompt, tools, timeout=240):
@@ -121,7 +157,15 @@ def main():
print(f"[INFO] {len(examples)} prompts held-out, endpoint={BASE_URL}")
results = []
stats = defaultdict(lambda: {"total": 0, "valid_tool_call": 0, "no_tool_call": 0, "invalid": 0})
# Las cuatro claves originales se mantienen tal cual para poder comparar contra los
# gate2_results*.json de las Fases 4-5; las de FAILURE_KINDS son aditivas.
def new_stats():
base = {"total": 0, "valid_tool_call": 0, "no_tool_call": 0, "invalid": 0}
base.update({kind: 0 for kind in FAILURE_KINDS})
base["request_error"] = 0
return base
stats = defaultdict(new_stats)
t0 = time.time()
for i, ex in enumerate(examples):
@@ -134,6 +178,8 @@ def main():
results.append({"mcp": mcp, "prompt": ex["prompt"], "error": str(e)})
stats[mcp]["invalid"] += 1
stats["__global__"]["invalid"] += 1
stats[mcp]["request_error"] += 1
stats["__global__"]["request_error"] += 1
continue
message = response["choices"][0]["message"]
@@ -149,6 +195,7 @@ def main():
results.append({
"mcp": mcp,
"prompt": ex["prompt"],
"expect": ex.get("expect"),
"tool_calls": None,
"valid": None,
"content": content,
@@ -159,11 +206,14 @@ def main():
all_valid = True
errors = []
error_kinds = []
for tc in tool_calls:
ok, err = validate_tool_call(tc, ex["tools"])
ok, err, kind = validate_tool_call(tc, ex["tools"])
if not ok:
all_valid = False
errors.append(err)
if kind not in error_kinds:
error_kinds.append(kind)
if all_valid:
stats[mcp]["valid_tool_call"] += 1
@@ -171,13 +221,20 @@ def main():
else:
stats[mcp]["invalid"] += 1
stats["__global__"]["invalid"] += 1
# Un prompt puede acumular mas de una categoria si emitio varias tool_calls;
# se cuenta una vez por categoria distinta, nunca dos veces la misma.
for kind in error_kinds:
stats[mcp][kind] += 1
stats["__global__"][kind] += 1
results.append({
"mcp": mcp,
"prompt": ex["prompt"],
"expect": ex.get("expect"),
"tool_calls": [tc["function"]["name"] for tc in tool_calls],
"valid": all_valid,
"errors": errors,
"error_kinds": error_kinds,
"content": content,
"reasoning": reasoning,
"finish_reason": response["choices"][0].get("finish_reason"),
@@ -198,6 +255,19 @@ def main():
f"pct_valid={pct_valid:.1f}%"
)
print("\n--- desglose de fallos por categoria ---")
for mcp in sorted(stats):
s = stats[mcp]
detalle = " ".join(f"{kind}={s.get(kind, 0)}" for kind in FAILURE_KINDS)
print(f" {mcp:20s} {detalle} request_error={s.get('request_error', 0)}")
# La invencion de parametros se reporta aparte porque es la metrica que la Fase 6
# quiere llevar a cero: hasta la Fase 5 estos casos contaban como validos.
inventados = [r for r in results if "unknown_argument" in (r.get("error_kinds") or [])]
print(f"\n[INFO] prompts con argumentos inventados: {len(inventados)}")
for r in inventados[:10]:
print(f" - [{r['mcp']}] {r['prompt'][:90]} -> {r['errors']}")
with open(RESULTS_PATH, "w", encoding="utf-8") as f:
json.dump({"stats": stats, "results": results}, f, ensure_ascii=False, indent=2)
print(f"\n[INFO] resultados detallados en {RESULTS_PATH}")