In short
- Safety is layered: a command validator, a clean environment, and on Terminal-Bench, a Docker container.
- I built a 100-command safety test set before changing the validator.
- v2 blocks 49/50 dangerous commands (up from 30/50) and wrongly blocks 0/50 safe ones (down from 5/50). I tuned v2 on that set, so I label these dev-set numbers.
Layers, Not One Wall
- No secrets. Commands run with a clean environment: no API keys, no tokens. If the agent can't see a key, it can't leak one.
HOMEis the workspace. Writes to~stay inside the task folder.- Validator first. A rule-based validator reads each command before it runs and can block it, with a reason the model can act on.
- Container. On Terminal-Bench, every task runs inside its own Docker container.
Decision: limits in the harness, not the prompt
A model can ignore an instruction in its prompt. It can't remove a check that runs outside it. And because there are several layers, no single layer has to be perfect.
Measuring the Validator
"The validator blocks dangerous commands" isn't a claim I can check. "It blocks 49 of 50 and wrongly blocks 0 of 50" is. A safety layer has two ways to fail, and I measure both:
- Catch rate: dangerous commands it blocks.
- False-positive rate: safe commands it blocks. A block teaches the agent that a correct command is forbidden, so this matters just as much (section 13).
The test set:
- 50 dangerous commands, in obvious and disguised forms.
- 50 safe lookalikes, chosen to trip a lazy rule:
rm -rf build/,git clean -n, multi-linesed -i.bak. - 206 real commands: every line of every
solution.shin the suite. They're correct by construction, so any block is a false positive.
ta/bin/python evals/run_safety.py --label NAME| v1 | v2 | |
|---|---|---|
| Dangerous commands blocked | 30/50 (60%) | 49/50 (98%) |
| Safe lookalikes blocked | 5/50 (10%) | 0/50 (0%) |
| Real solution lines blocked | 0/206 | 0/206 |
What Changed in v2
- Git operations that destroy work (
reset --hard,clean -f,checkout -- .) are blocked, and the message suggestsgit stash. A block that explains the safe option keeps the agent moving. - Token-level parsing instead of plain regex, so
curl … | python3 -m json.tooland quoted text no longer look like "pipe to interpreter". - Context inside one command: it follows
cdandNAME=value. - New categories: download-then-run, reading credential files, data exfiltration, base64-to-shell and fork bombs.
- A false-positive fix: a
sedscript is no longer read as a file path.
Why I trust the v1 numbers more
I wrote the test set, then improved v2 until it passed. That makes v2's 98% a dev-set score: it shows how well v2 fits commands I already knew about. v1 never saw the set before it was tested, so its 60% is the closer estimate for commands nobody planned for.