In short
- Cost per task fell from $0.0156 to $0.0056 (−64%), with 70% of calls on gpt-4o-mini.
- Pass rate fell from 84.4% to 63.3%. Debugging tasks fell the most: 70% → 40%.
- My hypothesis: the rule switches to the big model only after an error, and hard reasoning rarely produces one.
The Rule
Use the frontier model (gpt-4o) when the agent is most likely to need it:
- step 0, to read the task and plan;
- right after a failed command;
- right after the retry budget runs out, or a command is blocked.
Use the small model (gpt-4o-mini) for every other step.
ta/bin/python evals/run_eval.py --repeats 3 --routing on --label routing-onResult
| Baseline | Routing on | Change | |
|---|---|---|---|
| Pass rate | 84.4% ± 1.9% | 63.3% ± 5.8% | −21 pts |
| Tier 1 · file ops | 96.7% | 80.0% | −17 pts |
| Tier 2 · git / packages | 86.7% | 70.0% | −17 pts |
| Tier 3 · debugging | 70.0% | 40.0% | −30 pts |
| Calls on small model | 0% | 70% | |
| Cost per task | $0.0156 | $0.0056 | −64% |
| Cost per solved task | $0.0185 | $0.0088 | −52% |
Hypothesis
The rule escalates to the big model only after a command fails. In debugging, the hard part is reasoning about the bug, and that often happens with no failed command to trigger the switch. So the small model ends up doing the hardest thinking. A better rule would route on the kind of step (planning or diagnosing vs. running a known command), not only on errors.
What I Take From This
- "−64% cost" alone would have been misleading. Per solved task, the saving is 52%, and the tasks it gives up are the hard ones.
- The per-tier split showed where routing hurt. The overall average hid it.
- Routing stays off by default until a rule keeps quality within a few points.