Tiny Theory of Mind
Tiny Theory of Mind ("Tiny ToM") is a synthetic benchmark from axiomic-labs, released 2026-09-23, that tests theory-of-mind reasoning — tracking what another person believes, knows, intends or feels — in small, non-instruction-tuned base models. Read from the dataset card and its JSON schema via Telegram. It is Apache-2.0, 2,000 multiple-choice items, and the card states this is the org's "first attempt" at the task, so it should be read as a first benchmark version rather than a settled standard.
Why it targets small base models specifically
Every other benchmark this wiki tracks (llm-benchmarks, arc-agi) scores frontier or mid-tier chat models via instruction-following or generated answers. Tiny ToM is built the other way: each item is a short story that cuts off mid-sentence with four candidate endings, scored by length-normalized log-likelihood — the format HellaSwag popularized for base-model evaluation. No chain-of-thought, no instruction-following, no generated explanation is required; a model only has to assign higher likelihood to the correct continuation. Random-choice accuracy is 25%. That design is what makes it usable on 1M–135M-parameter checkpoints that were never chat-tuned, which is a different axis from anything else catalogued here — cost/speed/reasoning benchmarks assume a model that can follow an instruction in the first place.
Construction
- 2,000 examples, 4 endings each, drawn from 40 ToM constructs (50 examples each): first- and second-order false belief, appearance vs. reality, deception, bluffing, sarcasm/irony, hidden emotions, goal inference, reputation management, and 32 others spanning belief, perception, emotion, and intention.
- Each item also carries a difficulty (easy/medium, split 50/50) and a grade band (Pre-K–K through Grades 4–6) pitched at the age a human would need to solve it — the dataset's own caveat is that human developmental difficulty doesn't necessarily track model difficulty.
- Answer position is exactly balanced (500 each of options 0–3), closing off a positional shortcut.
- Everyday scenarios (classrooms, kitchens, playgrounds) with short character names;
source_idvalues show at least two generation batches (2_0,3_0), consistent with iterative synthetic generation rather than a single pass.
Baseline results
The card publishes 36 baseline scores, all sub-150M-parameter base models, topped by
HuggingFaceTB/SmolLM-135M at 47.2% and Axiomic Labs' own GPT-X2-125M/GPT-X2.5-135M in 4th
and 5th place (44.2–44.55%). The lowest scorer, harley-ml/dillion-1.2M, is at 28.3% — barely
above the 25% floor. No frontier or mid-size instruction-tuned model (Claude, GPT, Gemini, Llama,
Qwen) appears on the table; the benchmark hasn't yet been run against anything this wiki otherwise
tracks, which is the main limit on connecting it to the rest of the corpus.
Provenance and limits
Single source (the dataset card itself), T1 as primary data about the org's own release — but note what T1 does not cover here: the generation method beyond "synthetic," any held-out test set, and whether the 36 baseline scorers were cherry-picked. No downstream paper, leaderboard, or independent re-run has referenced it yet.