QuantiPhy
A benchmark asking vision-language models to do physics with numbers: estimate an object's real size, velocity or acceleration from a monocular video, given one known physical quantity as a reference. From stanford-university and UST — Li, Xiang, Mao, Wei, Chen, Masood, Fei-Fei Li and Adeli — released at CVPR 2026. It bills itself as "the first benchmark that asks vision–language models to do physics with numerical accuracy."
3,300+ video–text instances over 569 unique videos, drawn from simulated, laboratory and internet sources.
The task, and why it is hard
A video gives measurements in pixels and says nothing about scale. The benchmark supplies one reference quantity with a known world value, so the model must estimate the pixel-to-world ratio and apply it — measure the target in pixels, calibrate, answer in metres or m/s. Getting the object right and the scale wrong still fails.
Four subsets crossing motion dimensionality with the kind of prior given:
| Size-based prior | Motion-based prior | |
|---|---|---|
| Planar motion | 2S | 2D |
| Depth-varying motion | 3S | 3D |
Metric: MRA (Mean Relative Accuracy). Scores relative rather than absolute error, giving partial credit across confidence thresholds from 0.1 to 0.95. A benchmark of continuous quantities cannot be scored right/wrong, and the choice of a graded metric is what makes the table readable at all.
Why this wiki holds it
Two reasons, and the second is the interesting one.
It is the yardstick for code-as-world-vl, whose whole case rests on QuantiPhy scores — so the benchmark's coverage bounds the claim. code-as-world-report §5.5 says so itself: QuantiPhy covers "only a small part of real-world physics," excluding camera motion, rotation, deformation, occlusion, fluids and long-horizon multi-object dynamics. "State-of-the-art on QuantiPhy" is a narrower statement than "physical reasoning."
And it names, from the benchmark side, the problem code-as-world exists to solve: real videos rarely carry ground-truth physical quantities. Someone has to know the ball's true speed before anyone can ask a model for it. QuantiPhy answers that with 569 curated videos; Code-as-World answers it by simulating worlds where the number is exact by construction. Same shortage, opposite strategies — see training-data-quality.
Caveat on the numbers this wiki repeats
Every QuantiPhy score on these pages comes from code-as-world-report's own runs of the official protocol, not from the benchmark's maintainers. The corpus has no independent leaderboard for it.
Related
- code-as-world-vl · code-as-world-report
- training-data-quality — where labels come from, and what they cost