Zero-shot evaluation of single-cell foundation models (Genome Biology, 2025)

Kedzierska, Crawford, Amini and Lu (University of Oxford; Microsoft Research); Genome Biology 26:101, published 18 April 2025, open access (PMC12007350). Found during the 2026-09-22 research pass while reading geneformer, and held because it is the thing that paper cannot be: a test of the model by people who did not build it. T1, and independent of both models it grades.

What was tested

Geneformer and scGPT, used zero-shot (the pretrained cell embedding taken as it is, with no fine-tuning) against three simple baselines: highly variable gene (HVG) selection, Harmony and scVI. Five datasets, including Pancreas (16k cells), PBMC (12k) and Tabula Sapiens; two jobs, keeping cell types apart (AvgBIO, ASW) and removing batch effects.

The authors' reason for the setting: in discovery work the labels are unknown, so there is nothing to fine-tune on, and the zero-shot embedding is all a user has.

What it found

• "HVG outperforms Geneformer and scGPT across all metrics." Selecting variable genes, with no model at all, beat both foundation models.
• scGPT did better than the baselines on PBMC (12k) and was comparable to scVI on three datasets; Geneformer trailed throughout.
• Both models struggled to remove batch effects.
• Checking the pretraining task itself, Geneformer's reconstructed gene rankings correlated with the true rankings at a median of 0.56 (best 0.95) across datasets. So part of the problem may be that the pretraining objective was not learned well, not only that masked-gene pretraining makes poor cell embeddings. The paper offers both explanations and does not settle between them.

Why it matters here

geneformer reports strong results after fine-tuning; this paper shows the zero-shot embedding is weaker than a filter anyone can compute. Both can be true, and the difference is the spoke's first measured answer to what a biological foundation model's pretraining buys by itself. It is also an instance the hub's standing question keeps looking for: a third party re-running a builder's model on data the builder did not choose.

Related

geneformer · biological-foundation-model · single-cell-multiomics · synthesis