← Back to AI Terminal Agent
18 · AI TERMINAL AGENT

Lessons, Future Work & Conclusion

What building and measuring this agent taught me, and where the evidence points.

LESSONS RESEARCH OPEN QUESTIONS BUILD LOG 2026-10-19
In short
  • Measure the harness before the agent. Measure the noise before the change.
  • The failures that lose tasks are quiet ones: every command succeeds and the result is wrong.
  • Next directions come straight from those findings: verify before "done", route by step type, and test safety on held-out commands.

Lessons

  1. Validate the instrument first. The oracle and negative controls (section 05) meant that every agent failure afterwards was a real failure, not a broken test.
  2. Know your noise. 9 of 30 tasks were flaky (section 17). Without 3 runs, I'd have reported retries as a win.
  3. Plausible isn't proven. Retries sounded obviously useful. They raised recovery and left the pass rate inside the noise (section 08).
  4. Never report cost alone. Routing's −64% became −52% per solved task, and it gave up the hardest tasks (section 12).
  5. Averages hide where things break. The per-tier split is what showed routing hurting debugging almost twice as much as file tasks.
  6. Label your own dev-set numbers. A validator tuned on its own test set will look better than it is (section 07).
  7. Read traces, not just scores. A pass/fail score shows that a task failed. The trace showed why: the agent said "done" too early (section 13).

How I Choose What to Work On

I work in a loop, not down the list of sections:

  1. Run a small set of tasks, 3 times.
  2. Read the traces of every failure.
  3. Add each new failure to the failure-mode list (section 13).
  4. Fix one failure mode.
  5. Run the same tasks again and compare against the baseline.
  6. Record the result in the experiment log (section 14), even when it's "no change".

Future Directions

1. Verify before "done"

Evidence: one task failed 9 times out of 9. In every baseline run, the agent stopped and reported it was done. Retries didn't help, because no command ever failed. Direction: require the agent to re-run the task's own test, or an equivalent check, before it may stop. The open question is whether that lifts debugging tasks more than it costs in extra steps.

2. Smarter routing

Evidence: error-triggered routing saved 64% but cut debugging from 70% to 40%. Direction: pick the model by the kind of step (planning or diagnosing vs. running a known command). Success means most of the saving at no more than a few points of pass rate.

3. Honest safety numbers

Evidence: v2's 98% catch rate is a dev-set score. Direction: build a held-out set, written without looking at the validator, and report v2 on it. That number, not 98%, is the one worth quoting.

4. Local suite vs benchmark

Question: does my 30-task suite predict benchmark performance? If a change helps locally but not on Terminal-Bench, the local suite measures the wrong thing. If they agree, the cheap suite can screen ideas before an expensive benchmark run.

Conclusion

Building a terminal agent turned out to be the smaller half of the work. The larger half was building the measurement around it: a validated suite, repeated runs, traces, cost tracking and honest labels on every number. That measurement is what made it possible to say "retries didn't help" and "routing hurts debugging" with evidence, instead of guessing.