WebArena
Zhou, Xu, Zhu, Zhou, Lo, Sridhar, Cheng, Ou, Bisk, Fried, Alon and Neubig (CMU), WebArena: A Realistic Web Environment for Building Autonomous Agents, 2023 (revised 2024). T1.
A self-hosted, reproducible environment of real web applications rather than a mock: e-commerce, a social forum, collaborative software development (a GitLab instance), and content management, plus supporting tools like maps and a knowledge base. The motivation is stated as a complaint about the field — agents "are primarily created and tested in simplified synthetic environments, leading to a disconnect with real-world scenarios."
Tasks are graded on functional correctness: did the intended state change happen, not did the trajectory look right.
The headline gap: a GPT-4-based agent reached 14.41% end-to-end success where humans reached 78.24%. Published in 2023, so the absolute numbers are stale — the shape is not, and it is the reason this spoke's browser-agent claims deserve a benchmark citation rather than a demo.
What it adds to the spoke's evaluation set
Three benchmarks, three different failure surfaces:
| task | environment | what fails | |
|---|---|---|---|
| swe-bench | fix a real GitHub issue | repository + tests | reasoning across files |
| tau-bench | serve a user under policy | APIs + simulated user | consistency (pass^k) |
| WebArena | operate real web apps | live self-hosted sites | long horizons, UI state, recovery |
WebArena is the one that measures acting in an environment nobody designed for an agent — no clean
tool schema, just pages. That is also why it is the natural benchmark for the browser-agent claims
tracked in ../web-browsers-wiki, where vendors report their own numbers on their own harnesses.
Not held here: BrowserGym, which coverage edge 7 also named, and any post-2024 re-run of WebArena.