τ-bench

Yao, Shinn, Razavi and Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 2024. T1.

Where swe-bench hands an agent a task and walks away, τ-bench puts a simulated user on the other side of the conversation and a written domain policy over the agent. Two domains — retail and airline — each with domain-specific APIs and rules the agent is obliged to follow. Grading compares the database state at the end of the conversation against an annotated goal state, so the question is what the agent actually did, not what it said.

pass^k, and why it is the important part

The paper introduces pass^k: the probability that an agent succeeds on the same task in all k independent trials. That is a reliability measure rather than an ability measure, and it is brutal. Even state-of-the-art function-calling agents “succeed on <50% of the tasks, and are quite inconsistent” — GPT-4o’s pass^8 in retail is under 25%.

This is the metric this spoke has been missing while comparing harnesses on single-run scores. An agent that solves a task 90% of the time is not 90% deployable: run it eight times and the probability it never fails is 43%. For anything customer-facing, and for anything running unattended, that gap is the product decision.

Where it sits

• swe-bench measures whether a hard task can be done at all. τ-bench measures whether an easy task is done every time, under a policy.
• It gives agent-evals a concrete reliability instrument, and it sharpens shipping-without-human-verification: the case for removing the human requires exactly the quantity pass^k reports.
• The user is simulated, which is the standing caveat — an LLM playing a customer is more patient and more legible than a real one.