- Every run writes a local JSONL trace: each model call and each command.
- An optional export sends it to Langfuse after the run, so tracing can never break the agent.
- Secrets are masked before anything leaves the machine.
Layer 1: JSONL Traces
A pass/fail score tells me that something broke. Only a trace tells me why. The "agent says done too early" finding in section 13 came straight from reading the stop reason in these files.
- Ad hoc runs:
results/adhoc/<timestamp>.jsonl - Eval runs:
results/<timestamp>_<label>/traces/ - Harbor runs:
<trial>/agent/ta_trace.jsonl
Each llm_call event records which model answered. That's how I measured that routing sent 70% of calls to the small model.
Layer 2: Langfuse
ta/bin/python evals/run_eval.py --label my-run --langfuseOne run becomes one Langfuse trace:
- One generation per model call: model, tokens, estimated cost, latency.
- One span per command: command, output preview, exit code, validator decision.
- Level WARNING when a command is blocked or exits non-zero, so problems stand out.
- Scores on eval runs:
passedandrecovery_rate. - Tags for label, tier, routing and retries, so I can filter one experiment against another.
Design Decisions
The agent loop doesn't know Langfuse exists. If the export fails, the run still succeeds and prints one warning line. An observability tool must never change what it observes.
The SDK can't set the start time of an observation. For a replayed run, every span would show the export time instead of when it really happened. The REST API keeps the real timings.
Commands and outputs are masked for API keys, tokens and long hex strings before anything leaves the machine.