Geneformer — transfer learning in network biology (Nature, 2023)

Theodoris, Xiao, Chopra and eight others, senior authors X. Shirley Liu and Patrick Ellinor; Nature 618:616–624, published 31 May 2023. Read from the author manuscript on PubMed Central (PMC10949956) during the 2026-09-22 research pass, for growth edge 1: a biological foundation model, described — its data, its training, one confirmed prediction. This is the spoke's first held instance of the thing biological-foundation-model had only as an aim.

T1: a peer-reviewed primary paper, graded on its load-bearing claims: what the model is, what it was trained on, and a wet-lab test of one of its predictions. The accuracy figures are the authors' own, on tasks they chose; an outside test of the same model is held separately as single-cell-foundation-model-zero-shot, and it does not flatter them.

Data

Genecorpus-30M: 29.9 million human single-cell transcriptomes from 561 public datasets (GEO, SRA, Human Cell Atlas, EBI Single Cell Expression Atlas, Broad Single Cell Portal and others), droplet platforms only so that expression units compare. Cells with high mutational burden (malignant cells, immortalised lines) were excluded, because their networks are rewired in ways the corpus could not explain without genome sequencing.

Representation and training

Each cell becomes a rank value encoding: its genes sorted by expression in that cell, normalised by each gene's expression across the whole corpus. The authors state the cost: ranking discards the exact counts. The gain is a non-parametric input in which housekeeping genes sink and genes that separate cell states rise. The model is a six-layer transformer encoder: input length 2,048 genes (enough to hold 93% of the corpus's encodings in full), 256-dimensional embeddings, four attention heads per layer. It was trained on masked-gene prediction, self-supervised, with no labels.

What it predicted and what was tested

Fine-tuned with little task data (one chromatin task used 56 labelled loci in about 15,000 embryonic stem cells), it beat the baselines the authors ran on their chosen panel. The disease application is the part the edge asked for. Fine-tuned to tell cardiomyocytes from non-failing hearts apart from hypertrophic and dilated cardiomyopathy (out-of-sample accuracy 90%), the model was used for in silico deletion: which genes, removed from a cell's encoding, move its embedding toward or away from disease. Two kinds of prediction then went to the bench, in iPSC-derived cardiac microtissues edited with CRISPR:

• A dosage-sensitive gene. Knocking out TEAD4, a Hippo-pathway transcription factor the model flagged as a new candidate, reduced the tissue's contractile stress (control n=12, knockout n=11, p<0.05).
• Therapeutic targets. In a TTN+/− model of cardiomyopathy, knocking out the predicted targets PLN, GSN, ESRRG and HMGB1 improved contraction (TTN+/− control n=22; p<0.05, FDR-corrected).

That is the "one confirmed prediction" the edge named, in a cell model rather than a patient.

What it does not show

Every accuracy number here comes from fine-tuning on tasks the authors picked. Whether the pretrained embedding is useful before fine-tuning is a separate question, and when Microsoft Research and Oxford asked it two years later the answer was no: single-cell-foundation-model-zero-shot found that plain highly-variable-gene selection beat Geneformer's zero-shot embeddings on every metric they ran.

Related

biological-foundation-model · single-cell-foundation-model-zero-shot · single-cell-multiomics · hippocampus-aging-epigenome · synthesis