04 — Agent Architecture | AI Terminal Agent

One agent loop, two harnesses: my local suite and Terminal-Bench through Harbor.

In short

A from-scratch agent, with no framework: one loop, one tool, a set of switchable features.
The same loop runs locally and inside Harbor's task containers, so results stay comparable.
A fake model client tests all the plumbing at zero API cost.

The Loop

instruction
    ↓
loop.py ──► model (gpt-4o, or gpt-4o-mini if routing)
    ↓  tool call: run_command
validator.py ── blocked? → reason sent back to the model
    ↓  allowed
tools.py / environment.exec ──► shell
    ↓  output, exit code
retry.py (optional hint) ──► back to the model
    ↓
trace.py ──► JSONL ──► langfuse_export.py (after the run)
budget.py ── checked before every model call

Each part does one job, and every optional part has a flag that is off by default:

Loop: OpenAI chat completions with one run_command tool (section 03).
Validator: checks each command before it runs (section 07).
Retries: experiment, off by default (section 08).
Routing: experiment, off by default (section 12).
Trace: one JSONL file per run (section 15).
Budget cap: a hard dollar limit (section 16).

Decision: defaults reproduce the original agent

With every flag off, the agent behaves exactly like v1. Each experiment then changes one thing, and the code version hash in each result shows which code produced it.

Why One Tool

A shell already covers reading files, editing them, running tests and using git. Separate tools like read_file or write_file would be more surface for the model to misuse and more code for me to debug, for no new capability.

Decision

Start with one general tool. Add a specialised tool only if traces show a repeated failure that it fixes.

ta/bin/python -m agent "your instruction" --workdir DIR [--model M] [--max-steps N]

The default model is gpt-4o. The default step limit is 25.

Harbor Adapter

agent/harbor_agent.py:TerminalAgent (name ta-terminal-agent) runs the same loop, with the same validator, routing and retries. Only the execution changes: commands go through environment.exec inside the task container, from the image's WORKDIR, with a 60 s timeout and output cut at 8000 characters.

harbor run -a agent.harbor_agent:TerminalAgent -m gpt-4o \
  --ak routing=on --ak small_model=gpt-4o-mini \
  --ak retry_policy=on --ak max_retries=2 --ak max_steps=30 \
  -p ../terminalAgent/benchmarks/terminal-bench-2-1 -o jobs -n 2 -y \
  -i fix-git -i git-leak-recovery

Decision: Harbor External Agent, one shared loop

A second copy of the loop for Harbor would drift from the local one, and then the local results would stop predicting benchmark behaviour. One loop with two execution backends keeps them honest.

Decision: the API key never enters the container

The key is loaded in the host process only. The agent's commands run in the container, so nothing the agent runs can read the key.

Testing for Free

Set TA_FAKE_SCRIPT to a JSON list of shell commands. The loop then uses a fake client that issues those commands and stops. That tests containers, the validator, tracing and scoring with no model and no bill.