- 9 of 30 tasks gave different results across 3 identical baseline runs.
- So every reported number is a 3-run mean with its standard deviation.
- Each result file records the exact code, model, settings and task set that produced it.
The Problem: Single Runs Lie
fix_failing_unittest failed in a smoke test and passed in the next run,
with no code change. Across 3 baseline runs, 9 of the 30 tasks were flaky:
| Tier | Flaky tasks (baseline, 3 runs) |
|---|---|
| 1 · file ops | word_frequency |
| 2 · git / packages | find_introducing_commit, installable_package, squash_commits |
| 3 · debugging | circular_import, crash_on_edge_case_input, failing_makefile_target, log_root_cause_analysis, off_by_one_pipeline |
With that much variation, a one-task difference between two single runs means nothing. That's 3.3 points on a 30-task suite.
What I Do About It
- 3 runs per configuration (
--repeats 3), reported as mean ± sample standard deviation, overall and per tier. - Per-task pass counts in
summary.json, with flaky tasks listed by name. - A code fingerprint: a sha256 of
agent/*.pystored ascode_version. Two results are comparable only if the hashes match. - Full config recorded: model, small model, routing, retry policy, max retries, max steps, repeats.
- Fixed task sets: the 30 local tasks are in the repo. The Terminal-Bench subset is a fixed list (
tb_subset_runnable.json), filtered by the oracle on my hardware (section 05). - Defaults reproduce v1: with every experiment flag off, the agent behaves exactly like the original.
ta/bin/python evals/run_eval.py --repeats 3 --label NAME
ta/bin/python evals/compare.py results/<run_A> results/<run_B> # B − A, per taskOne run can't separate signal from noise. Ten runs would cost over three times as much per idea. Three runs give a mean and a spread at about $1.40 per configuration. That's enough to reject a change like retries (+2.3 points, inside the noise) without overspending.
Three runs give a rough spread, not a confidence interval I'd defend. A small effect, like +2 points, would need many more runs to prove or rule out.