Agent Eval Platform

The question and the setup

The benchmark is Sierra's τ-Knowledge banking domain in tau2-bench v1.0.1, pinned and checked on every run. A simulated customer asks for help. The agent has to find the right procedure in the bank's knowledge base, unlock the tools it names, verify the customer and act within policy. Grading is official: the final database state.

I held the model fixed (gpt-5-mini) and changed only what sits around it, so any difference is the harness. Experiments ran on the development split. The 67 held-out tasks were never run, so they remain a clean test.

What I built

Six layers, each there to make one number trustworthy:

1. ModelsAgent, simulated customer and embeddings, all called through LiteLLM.gpt-5-mini (agent)gpt-5.2 (customer)text-embedding-3-large
2. Spend admission controlBefore every call, reserve its worst-case cost; refuse the call if the cap can't cover it; afterwards, settle at the billed cost. A model with no recorded price can't be called at all.
3. Benchmark, pinnedtau2-bench's orchestrator, bank tools, database and grader, verified against a fixed commit and task-file hash. The agent's shell tool runs in a sandbox I tightened after canary tests found gaps in the default setup.
4. Runtime harnessComponents behind separate flags, each with a negative control: replaying all reference solutions kept the same reward in 30 of 30.tool adapterchecks before actionscapability searchverification evidencedisclosure checknudges
5. Evaluation layerFrozen plans with decision rules written before any spend, paired and balanced batches with a resume journal, cheap replay probes, and an answer-independence check that replays everything the agent saw against an answer-erased copy of the task.→ This found the benchmark leak.
6. ReviewBlind exports, two independent model readers plus an adjudicator, and verdict functions frozen before the run.

What happened

The components moved the behaviour they targeted. Transfers per conversation fell (0.65 → 0.05). Capability search made the account-lookup document reachable in far more of the conversations that needed it (16 of 90 → 81 of 90), though reachable is not the same as used. One rewrite of the feedback after a held action cut false "transfer under way" replies (26 of 27 → 2 of 27) in a replay test.

Completed tasks did not follow. The best comparison came out 5 vs 8 of 22 completed pairs (standard vs harness). That missed the pre-registered gate, and its corrected safety audit found 11 policy violations in 7 conversations vs 3 in 2. The final comparison was 3 vs 3 of 30.

What the evaluation layer caught

• A leak in the benchmark. One listing tool exposed reference-derived data in 17 of 30 dev tasks. I reported it as tau2-bench #574, with a reproducer that needs no model calls and a local fix that leaves every grade unchanged.
• Bugs in my own pipeline. Of the 13 defects, one was the benchmark's; the rest were mine. Two examples: a per-run cost cap enforced on the upper bound silently cut off only the longer arm, and a verification rule accepted values the customer was only echoing back.
• A provenance bug you can run. An intercepted draft was counted as "shown to the customer", so a valid verification was refused. make demo-provenance reproduces it and the fix, offline, for $0.

Why I stopped, and what I'd do differently

• Reach. The last component could directly affect only 2 of 30 tasks' scores. I chose it because it was tractable, not because it mattered. Next time I'd rank ideas by reach first.
• Acting, not finding. In 16 agreed policy-step failures, the rule and the evidence were both already in the agent's context. Retrieval wasn't the only bottleneck.
• Power. One conversation per task can't detect small effects. I'd estimate power before building, and use more trials per task.

Limits: one model, one domain, development tasks only. None of these results is a leaderboard number; the official protocol is every task, four times over. Review labels come from blind model readers, not human annotators.