Runs every scenario N times (default 1, unchanged). Every run is still
appended to results.jsonl; with N > 1 one "kind": "summary" row per key
follows, whose metrics are the medians of the successful runs (user_s and
sys_s included, byte/instruction counts rounded) and whose summary holds
median, p90 (nearest rank, = max for N < 10), min and max. A failed run marks
the summary failed so the gate refuses it.
--compare uses the latest row per key; the table labels each side "median
of N" or "1 run" and warns when they differ, and --fail-over refuses such
keys. --repeat with --with is rejected. docs/testing.md documents the format.
With --compare, exits 1 when a matched run got more than PCT % worse in CPU time, peak memory, max RSS
or instructions retired (wall time follows the load and is never gated).
Checks: xtask fail_over test (1 passed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TEytG1Qum4TJrMTLKjgwDa
(cherry picked from commit c34be4d12f3a61f4e386256ae54d75c6bc7f875d)
(cherry picked from commit 6e3eab96493726c551ec1e66174ac08b5591e4c2)
(cherry picked from commit 2e066f4073b300d8bace04419a7d4af3f3242b2e)
(cherry picked from commit 351a4b96c0)
(cherry picked from commit 40b232d02069925e9c72677f52ee6f356495a736)
(cherry picked from commit a2d89bda11b7217c83c98ab681b8f9b4a334786d)
(cherry picked from commit 63951595222207b6bfd0ecde72f991628c6f12d9)
(cherry picked from commit 20c2bb0dcb)
Matej Kozlovský