05 — Harbor & Terminal-Bench

In short

Two harnesses: a 30-task local suite I built, and a 31-task Terminal-Bench 2.1 subset run through Harbor.
Local suite: the oracle passes 30/30, and every check fails on the untouched state (0/30).
Terminal-Bench subset: the oracle passes 28/31 on my hardware. I dropped the 3 that fail.

Two Harnesses, Two Jobs

Local suite (30 tasks, 3 tiers: file ops, git/packages, debugging). Fast and cheap, so I can run every experiment 3 times. Each task is instruction.md, setup.sh, check.sh and solution.sh, and runs in a fresh temp folder.
Terminal-Bench 2.1 via Harbor (31-task subset). Tasks I didn't write, in Docker containers. This is the outside reference.

ta/bin/python evals/run_eval.py [--model M] [--tier T] [--task ID] [--repeats N] [--label NAME]

Why the Oracle Comes First

An agent score is only as good as the harness under it. If a check is broken, a "fail" tells you nothing about the agent, and a "pass" is worse: it's a number you'll quote by mistake.

The oracle runs the known solution with no model. It answers one question: can this task be solved, and does the check notice?

Two Tests, Not One

A passing oracle proves a check can say "yes". It doesn't prove the check can say "no". A check that passes on everything would also pass the oracle.

So I added a negative control: run each check.sh on the initial state, before any change. Every check must fail.

Local suite · 30 tasks · 3 tiers

ta/bin/python evals/run_eval.py --oracle

Oracle (solution.sh):  30/30 pass
Negative control:       0/30 pass   ← correct: every check rejects the unsolved state

Terminal-Bench: Drop What the Oracle Can't Pass

Harbor oracle · 31 tasks

harbor run -a oracle -n 2

28/31 pass

Three tasks fail on my hardware, with no model involved:

crack-7z-hash: the oracle timed out at 1800 s.
qemu-alpine-ssh: reward 0. QEMU runs under emulation.
qemu-startup: reward 0. QEMU runs under emulation.

Decision

Exclude any task the oracle can't pass on my hardware. The runnable subset is 28 tasks (tb_subset_runnable.json).

Why

If I kept them, every agent run would start with 3 guaranteed failures that come from my machine, not the agent. That would quietly lower every score by about 10 points.