Agentic discovery loop

The search procedure code-as-world-report uses to recover an executable-world-representation from an observation. An LLM proposes a world, a physics engine runs it, a renderer draws it, a verifier compares the drawing to the evidence and says what is wrong, and the LLM revises. Five stages, repeated up to a budget:

propose → instantiate → execute → render → verify

The framing is deliberately abductive — inference to the best explanation, constrained by parsimony — and the paper reaches for the scientific-discovery precedent to say so.

The algorithm, plainly

An LLM A, an iteration budget K (set to 5 in the experiments), and modality-specific evidence.

1. Evidence first, outside the loop. Text input goes through an interpreter that extracts entities, spatial relations, physical events and intended outcomes. Video input goes through depth estimation, instance segmentation and object tracking, plus a generated mesh per segmented object — so the loop never sees raw pixels alone.
2. Propose / update. A writes or edits the EWR from the evidence, the current hypothesis, and last round's structured feedback.
3. Instantiate. Compile the EWR into simulator-ready parameters (MuJoCo).
4. Execute. Run it. The trajectory records object states, contacts, collisions and outcomes — the hypothesis's consequences, made inspectable.
5. Render & project. Draw the trajectory; for video input also project it back into depth maps, masks and image-plane tracks so it can be compared in the same terms as the evidence.
6. Verify. Compare at selected key frames. Text input is checked on semantic and physical constraints; video input jointly on RGB, depth, masks and trajectories. Frame-level discrepancies aggregate into structured feedback that names which component to revise.
7. Accept, or exhaust the budget and reject.

The rejection branch is worth noticing. The loop is allowed to fail and say so, rather than always returning its best guess — which is what makes the surviving worlds usable as labels.

Why iteration beats sampling

The obvious objection is that five rounds of refinement might be no better than five independent tries. The report runs that control: Best-of-5 sampling at a matched five-evaluation budget, with the loop's verifier choosing candidates and the scoring done on metrics held out of the verification signal. Iterative refinement wins on visual alignment, object IoU and trajectory error. The reported conclusion is about compute, not accuracy — the same budget spent on diagnosis and revision buys more than the same budget spent on rerolls.

What the loop is for

Not inference. The loop runs offline, to manufacture training data: each accepted world exposes exact geometry and a state trajectory, from which quantitative question–answer pairs are generated with labels read off the simulator. code-as-world-vl is then trained on those pairs.

So the agent here is a data factory, not a product. That distinction is the spoke's boundary with ../agentic-tooling-wiki: the harness machinery is that spoke's subject, and what the harness produces — supervision, and a model fitted to it — is this one's.

The limitation the loop cannot see past

Verification is consistency with rendered evidence. code-as-world-report §4.4 states the consequence: outside the simulator's modelling scope the loop "may converge to a locally plausible EWR without recovering a mechanistically accurate explanation." A loop whose only test is "does the replay look like the video" cannot distinguish a right mechanism from a wrong one that renders the same.

And by §5.5 the trained model "does not internalize the agentic discovery process itself." Hypothesis, simulation, diagnosis and revision stay outside it. See llms-cant-jump (../research-wiki).

Related

• executable-world-representation — what the loop searches for
• code-as-world-report · code-as-world · code-as-world-vl