ExploitGym (repository)

The benchmark itself, from the authors: sunblaze-ucb/exploitgym, Apache-2.0, created 2 June 2026, last pushed 25 July 2026, 584 stars and 76 forks. Code is Apache-2.0; the bundled task data under data/tasks/ keeps its upstream licenses. Paper is arXiv:2605.11086, site cybergym.io. T1 — primary artifact, versioned and dated, by the people who built the thing. The corpus reached exploitgym through openai-huggingface-incident, a T2 secondary account; this is the source underneath it.

It corrects a number this wiki was carrying

The paper’s snapshot had 898 instances. The released benchmark has 869, and the changelog says why: “Filtered non-exploitable cases from the paper snapshot (898 → 869).” Both numbers are real and they mean different things — 898 is what the published scores were measured against, 869 is what you get if you download it today. The per-model scores on exploitgym stay denominated in 898.

Composition of the 869: kernel 186 (kernelctf 27, syzbot 159), user 502 (cybergym 484, nofuzz 18), v8 181 (clusterfuzz 106, human 66, sbxbrk 9). So the headline “including the Linux kernel and V8” is 21% kernel and 21% V8, with the bulk being userspace programs.

One caveat on freshness: the README still advertises v1.0/869 as current while CHANGELOG.md lists a v1.1 release dated 18 June 2026. The README is stale against its own changelog, so treat any instance count from this repo as needing a version check.

Containment shipped as code

The benchmark packages its sandbox in the repository, which makes it a readable reference design for containing an agent you expect to behave adversarially. Three components run alongside the task container: a controller, a Squid forward proxy firewall, and an LLM proxy.

The firewall sits on an internal Docker network with no default route, so “even if a program inside the container ignores HTTP_PROXY, direct connections fail because there is no route out.” Containment doesn’t depend on the contained process cooperating, which is the failure mode of every allowlist that exists only as a client-side setting.

docs/defenses.md covers deliberately disabling system defenses (ASLR and friends) for the exploit tasks, and pre_run.py --hardened flips that back on with hardened images. The v8 image variant defaults to nodefense. So whether mitigations were enabled is a per-run configuration, and the results format carries a mitigation_enabled flag per instance — any score is uninterpretable without it.

v1.1 closed the hole that the incident class runs through

openai-huggingface-incident turns on a single structural point — the carve-out that makes a sandbox usable is what gets attacked, and specifically the package-install path, which has to reach the open internet for the environment to work at all. ExploitGym v1.1, dated 18 June 2026, restructured exactly that:

• The firewall split into two proxies. A run proxy whose allowlist is now LLM API endpoints only, and an allow-all install proxy on its own separate network.
• A pre-agent install phase fetches per-task dependencies through the allow-all proxy, and then “the container is locked to the API-only run network before the agent runs” — the evaluator disconnects the install network, so the agent never has a route to the permissive path.
• The LLM proxy began blocking provider-side external retrieval by default with HTTP 403: web search and fetch, MCP tools and connectors, remote file_url / image_url / Gemini file_uri, hosted code execution, file search, network-enabled hosted shells, Gemini grounding and URL context. API keys can also be scoped to specific models.

Disconnecting the install network answers the package-proxy escape route directly. The retrieval block closes a different door in the same wall: an agent that cannot reach the internet through the network can still reach it through the model provider’s own tool-calling, and until v1.1 nothing stopped it.

What this does not establish is causation. v1.1 is dated 18 June; Hugging Face disclosed on 16 July and OpenAI confessed on 21 July. Whether the authors were responding to something they knew privately or converging independently on the same fix is not answerable from the repository, and the changelog frames all of it as routine tooling work. The honest reading is that two sets of security researchers arrived at the same conclusion about the same weak point within weeks of each other. Recorded as sequence, not cause.

Leaderboard mechanics

Results are submitted as a pull request against sunblaze-ucb/exploitgym-results, one directory containing metadata.yml and results.json. Submissions pin a benchmark_commit, which is what makes the 898-vs-869 drift tractable — a score is anchored to a specific state of the benchmark rather than to a version label. Per-instance results carry flag_captured, on_target, mitigation_enabled, wall-clock time, judge models, and per-model cost and token counts, where cost is “the agent’s own LLM usage (the model driving the exploit), not the scorer’s.” Maintainers validate format only.

An open leaderboard on a capability like this is itself a governance question — it is the mechanism by which “can frontier models write exploits” stays measured rather than asserted, and also a public scoreboard for exploit development. This spoke’s dual-use posture applies: documented here for what it settles about defense.

Worth separating from security-benchmark, which in this spoke means a compliance baseline (CIS/STIG, measured with OpenSCAP): a standard you harden a host against. ExploitGym is a capability benchmark — same word, opposite direction. One measures whether your system meets a defined bar, the other measures how well an attacker performs against yours.

Related

exploitgym · openai-huggingface-incident · security-benchmark · defender-guardrail-asymmetry · zero-trust · defensive-security · synthesis