08 — Error Recovery & Retries

Retries made the agent better at fixing failed commands. They didn't make it pass more tasks.

In short

Recovery from failed commands rose from 48.7% to 57.1%, and failed commands fell from 117 to 98.
The pass rate moved from 84.4% to 86.7%. That's inside the noise, so I don't claim a gain.
Retries can't fix the failures that actually lose tasks: the ones where every command succeeds.

Defining Recovery Before Measuring It

"The agent recovers from errors" is vague. Before the experiment, I wrote down what counts:

Failed command: non-zero exit code, timeout or error. Exception: exit 1 from grep, diff, test and similar tools, where exit 1 means "no match" or "different", not "broken". Commands blocked by the validator don't count.
Recovered: one of the next 3 commands in the same run succeeds and uses the same program.
Recovery rate = recovered ÷ failed commands, computed from the trace.

Why the grep exception matters

Without it, every grep that finds nothing would count as an error. The baseline would look far worse than it is, and retries would get credit for "fixing" searches that were never broken.

The Retry Policy

A command fails.
The tool result gets a RETRY GUIDANCE block: the error class, a hint, and "attempt k of N".
After more than N failures in a row (N = 2 by default), the result says RETRY BUDGET EXHAUSTED and the counter resets.
Any success resets the counter.

Decision: guide, don't loop

The harness never retries a command by itself. It tells the model what went wrong and lets the model choose. Blindly re-running a broken command just repeats the failure.

Result

Retries on · 3 runs · 30 tasks

ta/bin/python evals/run_eval.py --repeats 3 --retry-policy on --label retries-on

                  Baseline        Retries on
Pass rate:        84.4% ± 1.9%    86.7% ± 3.3%
Recovery rate:    48.7%           57.1%
Failed commands:  117             98
Cost per task:    $0.0156         $0.0153

My read

Retries help the agent fix commands that fail loudly. They don't help with the failures that lose tasks, where every command succeeds and the result is still wrong (section 13). That would explain why recovery improved and the score didn't. Retries stay off by default.