- The biggest agent failure: it says "done" when the task isn't done.
- The biggest safety failure: false positives. In live runs, 3 of 5 blocks were wrong.
- 9 of 30 tasks gave different results across 3 identical runs, so I never trust a single run.
1. The Agent Says "Done" Too Early
fix_broken_bash_script failed in all 9 runs: 3 baseline, 3 with retries,
3 with routing. In each baseline run, the agent stopped on its own and reported it had finished.
The check said otherwise.
Retries can't help here. Retries react to a command that fails, but this agent runs commands that succeed and still leave the script wrong.
The agent doesn't run the script one last time to verify it before it stops. A score can't show this. Only the trace can.
2. Safety False Positives
In live runs the validator blocked 5 commands. 3 were false positives: it read the empty
backup suffix in sed -i '' '/pat/d' as an absolute path.
This matters more than it looks. A false positive doesn't just cost a step. It teaches the agent that a correct command is forbidden, so it may go around it. A safety layer that blocks too much is also a reliability problem.
3. Single Runs Lie
fix_failing_unittest failed in the smoke test and passed in the baseline,
with no code change. Across 3 baseline runs, 9 of 30 tasks were flaky:
| Tier | Flaky tasks (baseline, 3 runs) |
|---|---|
| 1 · file ops | word_frequency |
| 2 · git / packages | find_introducing_commit, installable_package, squash_commits |
| 3 · debugging | circular_import, crash_on_edge_case_input, failing_makefile_target, log_root_cause_analysis, off_by_one_pipeline |
Do 3 runs before quoting a number. Report the mean and the spread.