- A from-scratch terminal agent scores 84.4% ± 1.9% on a 30-task suite I built and validated (oracle 30/30, negative controls 0/30).
- Retries improved recovery but not the pass rate. Routing cut cost by 64% and dropped the pass rate to 63.3%.
- The safety validator went from 60% to 98% catch rate with 0 false positives on its test set. That's a dev-set number, and I say so.
Method
- Same 30 tasks, same code version (
6e8df3a9), 3 runs per configuration. - Change one thing at a time against the baseline.
- Report the mean and sample standard deviation, overall and per tier.
- Judge each change on tier 3 (debugging), where the agent loses most of its points, without letting tier 1 drop.
Baseline
ta/bin/python evals/run_eval.py --repeats 3 --label baseline
Pass rate: 84.4% ± 1.9% (25, 25, 26 of 30)
Tier 1: 96.7% file ops
Tier 2: 86.7% git / packages
Tier 3: 70.0% debugging
Recovery: 48.7% failed commands fixed within 3 steps
Cost/task: $0.016An earlier single run scored 21/30 (70%). I treat it as a smoke test only: the code changed between the two runs, and one run can't tell signal from noise (section 17).
All Configurations
| Baseline | Retries on | Routing on | |
|---|---|---|---|
| Pass rate | 84.4% ± 1.9% | 86.7% ± 3.3% | 63.3% ± 5.8% |
| Tier 1 · file ops | 96.7% | 96.7% | 80.0% |
| Tier 2 · git / packages | 86.7% | 90.0% | 70.0% |
| Tier 3 · debugging | 70.0% | 73.3% | 40.0% |
| Recovery rate | 48.7% | 57.1% | 51.9% |
| Cost per task | $0.0156 | $0.0153 | $0.0056 |
Entry 1: Retries
Change: after a failed command, add a hint to the tool result; allow 2 retries. Result: +2.3 points, about one standard deviation. Recovery up, failed commands down (117 → 98). Verdict: no proven gain. Details in section 08.
Entry 2: Model Routing
Change: gpt-4o for step 0 and after failures, gpt-4o-mini otherwise (70% of calls). Result: −64% cost, −21 points, −30 on debugging. Verdict: too costly in quality as designed. Details in section 12.
Entry 3: Safety Validator v1 → v2
| v1 | v2 | |
|---|---|---|
| Dangerous commands blocked | 30/50 (60%) | 49/50 (98%) |
| Safe lookalikes blocked | 5/50 (10%) | 0/50 (0%) |
| Real solution lines blocked | 0/206 | 0/206 |
Details, and why I trust the v1 number more, in section 07.
What I Take From This
- A plausible feature isn't an improvement until it beats the noise.
- Cost savings need a quality number next to them.
- Per-tier results showed where each change hurt. The overall average hid that.
The agent numbers on this page come from my own 30-task suite. A suite I wrote can be easier to do well on than one I didn't, so I don't compare them with public Terminal-Bench scores.