- Baseline: about $0.016 and 10 s per task with gpt-4o. A full 3-run experiment costs about $1.40.
- Over 90% of tokens are input. Most of the cost is context the agent re-reads, not answers.
- A hard budget cap stops the agent before a call, not after the bill.
How Cost Is Measured
Every model call is priced from its real token counts, using a price table per model (per 1M tokens):
| Model | Input | Output |
|---|---|---|
gpt-4o | $2.50 | $10.00 |
gpt-4o-mini | $0.15 | $0.60 |
The baseline used about 440k input tokens and 30k output tokens over 90 task runs. Input is over 90% of the tokens, because the agent re-sends its whole history at every step. Shorter context would save more than shorter answers.
Cost, Time and Quality Together
| Configuration | Pass rate | Cost / task | Cost / solved task | Wall time / task |
|---|---|---|---|---|
| Baseline | 84.4% | $0.0156 | $0.0185 | 10.4 s |
| Retries on | 86.7% | $0.0153 | $0.0176 | 6.5 s |
| Routing on | 63.3% | $0.0056 | $0.0088 | 8.4 s |
"−64% cost" sounds like a clean win until you see the pass rate next to it. Per solved task, routing saves 52%, not 64%, and the tasks it gives up are the hard ones (section 12).
Wall time is total experiment time divided by 90 task runs, from one experiment per configuration. API response times change during the day, so I don't read the retries speed-up as a real effect yet. Cost comes from token counts, so it doesn't have that problem.
The Budget Cap
Benchmark tasks are long and run in parallel. A loop that gets stuck can keep calling the model. So the limit lives in code:
- Set
TA_BUDGET_USDto a dollar limit. - After each model call, its cost is added to a shared JSON ledger, under a file lock.
- Before each call, the agent reads the ledger.
- At or over the limit, the agent stops with the reason
budget.
If I forget to add a model to the price table, the cap fails safe. It stops too early, never too late.
The check runs before each call, so parallel trials can go over the limit by about one call each. A lock around the whole call would prevent this, but it would also make the trials run one at a time.