ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

Geng, Huang, Li et al. — 45 authors across 25+ institutions, with AItonomy, Qwen and PhAI-Labs as the corresponding groups. arXiv 2609.19134 (cs.CL, cs.CY), v1 16 September 2026; code at github.com/aitofound/ScienceIDE. Read from the arXiv HTML; arrived by Telegram 2026-09-17 via the Hugging Face papers page.

The problem it names

Scientific repositories hold decades of executable method, but "fragmented toolchains, implicit domain conventions, and specialized correctness criteria" stop that from becoming training experience for an agent — the scientific experience bottleneck. The claim is that the bottleneck is infrastructural, so the paper builds infrastructure.

What was built

27 scientific codebases → 64 environments → 2,812 tasks: 2,515 repair, 295 implementation, 2 acceleration; domains are astrophysical flows, plasma physics, ocean/atmosphere modelling and particle simulation (PLUTO, LAPS, MITgcm appear by name).
1,076 executable checks on numerical outputs and physical invariants, with tolerances calibrated by running nominal and deliberately-varied solutions.
Pipeline: domain experts draw module boundaries and write scientific equivalence contracts; per-environment factories generate candidate tasks by deterministic mutation and excision; a candidate survives only if the known-good solution passes and the broken baseline fails ("Propose broadly; establish validity by execution").
ScienceIDE-Hard, 85 tasks, as a benchmark for existing agents.

What was measured

On ScienceIDE-Hard, strict scientific success: Fable 5.1 67.1%, Opus 5 64.6%, Astra 63.1%.
PhAI-IDE 4B / 9B / 72B (Qwen backbones): SFT on 4,567 demonstration segments from 564 tasks, trajectories collected with GPT-5.6 Sol, then RL against the verifiers. Held-out repair: PLUTO-Particles-Dust 0 → 33.33% (4B), LAPS 31.25 → 50.0% (9B). Public benchmarks: HumanEvalFix-JS +10.98 (4B), BBH word sorting +33.60 (9B).
RL, 30 steps, two environments: LAPS reward 0.357 → 0.857 (2.4×), MITgcm-biogeo 0.286 → 0.571 (2.0×); budget truncations on LAPS fell from ~39.5% to 6.6%.

What the authors say it does not show

The limitations section is unusually plain and worth carrying: tasks are reference-verifiable repair and implementation, not discovery; domains are mostly computational physics and geoscience; a one-hour budget per task; the checks encode modelling choices, not universal correctness; SFT gains are on selected benchmarks; RL gains are inside the training environments, with hints, and do not demonstrate transfer to unseen codebases; legacy images may carry contamination from public source.

Where it sits

This is environment-evolution's programme with the source of difficulty changed: not synthesised environments that get easier as models improve, but real repositories whose difficulty is whatever the science is, with correctness defined by the code's own invariants. It is also the second paper in a fortnight to put the loop's effort into the environment rather than the agent (self-improving-agents now lists four sites). Two cautions from the corpus apply. The ScienceIDE-Hard leaderboard is a single run per model with no spread — harness-disclosure's complaint — and the transfer result is the honest one: the RL number that rose 2.4× was measured where it was trained.

Tier T1 — arXiv preprint with code, 45 named authors, an explicit limitations section; results are the authors' own.

environment-evolution · agent-environment-design · self-improving-agents · harness-disclosure · dream-rsi · qwen