Stax (Google Labs)
The open-sourced code behind Stax, google-labs' experimental LLM-evaluation platform (formerly Evaluable AI): build a dataset of prompts, score outputs with code checks, LLM autoraters or human raters, and compare versions of an app against your own criteria. Read from the README and the developers.google.com best-practices page; the product site itself returns 403 to this hub. Arrived by Telegram 2026-09-17.
What the repository is
Not a library and not an agent tool: a deployable platform. Three layers, each named in the README — Terraform for the GCP footprint (Cloud Run, Cloud SQL/MySQL 8, Pub/Sub), a Spring Boot Java backend that runs evaluation workflows over Pub/Sub, and a Next.js dashboard (Mantine, Tailwind). Local development runs the Pub/Sub emulator, Maven, and the dev server on port 3000, with AES and token-signing secrets in the environment. Apache-2.0; OpenSSF Best Practices badge; 98 stars, 16 forks, 17 commits at read time — a code drop, not yet a community.
What the product argues
The best-practices page is the readable statement of intent: evaluation criteria from ideal behaviour, business goals and known failure modes; regression sets plus challenge sets; and the line worth quoting — "An eval set with just ten high-quality prompts is infinitely more valuable than relying on vibe testing alone." Three evaluator kinds are named: heuristic/code, LLM-as-judge (autoraters), and human raters, with the last two meant to feed each other. No model list, no price, no benchmark of the autoraters against the humans.
What it adds here
agent-evals describes evals as a practice and a pipeline gate; this is the first page here of a
platform for them from a frontier lab, and the notable thing is what Google chose to open: the
plumbing. The dataset store, the rater workflow and the dashboard are in the repo; the autoraters
are whatever model you point it at. Read against harness-disclosure and metr-scaffold-comparison,
the missing piece is the same one — nothing in the repo or the docs measures how well an autorater
agrees with a human rater, which is the number that decides whether "move from vibes to proof" is
proof. Set beside ../machine-learning-wiki's standing gap on evaluation as a discipline: this is
tooling for it, still without the discipline.
Runner-up spoke: machine-learning (evaluation as an engineering discipline). Routed here because the object is a tool for people building LLM apps and agents.
Tier T3 — first-party README and docs; the repo is small and the product page unread.
Related
agent-evals · google-labs · harness-disclosure · metr-scaffold-comparison