The Terminal Agent project is a hands-on investigation into how AI agents can operate inside a real terminal environment to solve software engineering tasks.
Rather than treating an LLM as a simple chatbot, this project explores what happens when an AI system can inspect an environment, execute commands, modify files, run programs, use tools, and iteratively verify its own work.
How capable are AI agents at independently completing real terminal-based software engineering tasks?
Why This Project?
Modern AI coding systems are moving beyond code completion. They can increasingly operate as autonomous agents that interact with development environments and execute multi-step workflows.
I want to understand the engineering behind these systems: how agents are constructed, how they interact with tools, how their performance is measured, and where they fail.
What I'm Exploring
- Terminal-based AI agents
- Agent tool use and command execution
- Harbor agent infrastructure
- Terminal-Bench evaluation
- Task environments and Docker isolation
- Agent timeouts and resource constraints
- Verifiers and automated evaluation
- Rewards and benchmark scoring
- Agent reliability and failure modes
- Building and evaluating a custom terminal agent
Initial Experiment
The first experiment uses Harbor with the built-in Oracle agent against Terminal-Bench 2.1.
Dataset: terminal-bench/terminal-bench-2-1
Agent: oracle
Trials: 2
Concurrency: 2
Exceptions: 0
Mean Reward: 1.000
Runtime: 5m 08sInitial Result
Both selected trials received a reward of 1.0, resulting in a mean reward of 1.000.
This establishes an initial baseline for future experiments. The two-trial sample is intentionally small and should not be interpreted as a general Terminal-Bench performance score.
What Comes Next?
The next stages of the project are to understand the individual components of the evaluation system, reproduce experiments, inspect task and verifier behavior, and eventually compare a custom agent against established baselines.