How to Evaluate Entity Resolution Systems: An Entity-Centric Framework
olivier-binette, Youngsoo Baek, Siddharth Engineer, Christina Jones, Abel Dasylva and Jerome P.
Reiter — arXiv, submitted 8 April 2024, 33 pages. Software released as
OlivierBinette/er-evaluation. Ingested by the 2026-08-05 research pass against growth edge #1, the
corpus’s longest-standing evidence hole.
The problem it names, which is not the problem the wiki thought it had
This spoke’s entity-resolution page has said for months that nobody measures merge quality. This paper says something worse and more useful: the obvious way to measure it is biased upward, and the bias is large enough to reverse rankings.
the naive computation of pairwise precision on benchmark data sets has been shown to provide over-optimistic results. Precision computed on benchmark data sets is often close to 1, even when the true precision for the entire data set may be much lower.
And the consequence the authors care about:
When combining biased precision estimates with recall estimates into an F1 score, this can lead to performance rank reversals: an algorithm may be assessed to perform better than another with high confidence, despite the opposite being true.
The mechanism is a sampling artifact. Finding matching pairs is “looking for a needle in a
haystack” — in n records there are O(n²) non-matches and only O(n) matches — so evaluation has
historically used clever application-specific sampling to find the matches. Benchmarks built that way
are enriched for easy positives, so precision measured on them does not transfer to the full data set.
What they propose instead
Sample fully-resolved entities, not pairs. Given a known cluster, every pair inside it is a match and every pair that crosses its boundary is a non-match, so one labeled cluster supplies both classes without a sampling scheme. From cluster-wise error metrics they then build weighted estimates of the global figures — pairwise, cluster and b-cubed precision and recall — that describe the whole data set rather than the benchmark. The framework has three parts: monitoring statistics, global performance estimates, and root-cause error analysis.
The by-product matters for this corpus: benchmarks built this way are reusable, and usable for training as well as scoring, which none of the systems here have.
The numbers
Validated on PatentsView inventor-name disambiguation (US patent data, maintained by the American Institutes for Research) and in simulation. Taking the 30 December 2021 disambiguation as ground truth, the 28 May 2018 disambiguation scores 91% pairwise precision and 94% pairwise recall — deliberately chosen because “this is a high accuracy bar, which makes the performance estimation problem challenging.”
Empirical bias stays under 0.4%, and under 0.2% at sample sizes of 400+. On RMSE, b-cubed estimators are most accurate, then cluster, then pairwise. The calibration line worth keeping:
a data-free (and not recommended) estimator of precision equal to 100% achieves RMSE of 9%, since the true pairwise precision is 91%.
Against that 9% baseline, their pairwise-precision estimator reaches 4.7% RMSE at sample size 200, 3.5% at 400, and 2.4% at 800. So a few hundred labeled clusters buys roughly a 4× improvement over assuming the system is perfect — and “assume it is perfect” is, in effect, what every system in this corpus currently reports.
What it does and does not close
It closes the methodological half of growth edge #1: there is now a T1 source giving measured precision and recall for a real resolution system, with an estimator whose error is quantified. It does not close the half about this corpus’s own systems — nobody has run this against gbrain‘s dream cycle, sift-kg‘s gated merges, or graphrag‘s community absorption, and the paper predates all three as LLM systems.
One genuine limit on transfer. This is classical record linkage over patent records: surface-form strings with a long fuzzy-matching literature behind them. entity-resolution already notes that proposition-shaped nodes “almost never match on surface form”, which is why LLM builders reach for embeddings. The entity-centric sampling idea carries over intact — a resolved cluster is a resolved cluster whatever the node is — but the assumption that annotators can cheaply label a cluster as fully resolved is doing real work, and it is less obviously true when the node is a claim rather than a person.