zk-Bench — SNARK benchmarking by people not selling a prover

Ernstberger and Steinhorst (TU Munich), Chaliasos and Livshits (Imperial College London), Jovanovic and Gervais (UCL), Kadianakis (Ethereum Foundation) and Orrù (CNRS). ePrint 2023/1503, published at SCN 2024. The first source in this spoke whose numbers were produced by somebody with no system in the comparison.

The paper's motivation is this spoke's standing tension, stated by cryptographers rather than inferred by a wiki: "claimed results are generated in isolation, and omissions in details render them irreproducible."

What it actually measures

Two backends, benchmarked separately, which is the design decision that gives the results their bite.

Arithmetic backend — 13 elliptic curves across 9 libraries in Rust, Go and JavaScript/Wasm: curve25519-dalek 4.1.1, pasta_curves 0.5.1, gnark-crypto 0.11.1, arkworks-rs 0.4.2, ffjavascript 0.3.59, halo2_curves 0.3.1, pairing_ce 0.28.5, zkcrypto 0.23, blstrs 0.7.0.

Circuit backend — 5 development tools with their proof systems: Bellman (Groth16, BLS12-381), PSE/Halo2 (Plonk with KZG, BN254), Gnark (Groth16 and Plonk, both curves), Circom via snarkjs and rapidsnark (Groth16, BN254), Starky (STARK over Goldilocks).

Hardware is five named AWS instances rather than "a laptop": m5.large (2 vCPU / 8 GB), m5.2xlarge (8 / 32), m6i.8xlarge x86, m6g.8xlarge ARM, and r6i.8xlarge memory-optimized.

The results that matter to this spoke

The stack decides the number, not the scheme. An MSM of size 2²⁰ runs 9.44× faster in G1 and 13.06× faster in G2 in gnark-crypto than in ffjavascript, and that difference propagates straight up: gnark proves SHA-256 over a 16 kB pre-image 14.02× faster than snarkjs. Same proof system, same curve, same circuit — a factor of fourteen that belongs entirely to the arithmetic library underneath.

A tool's ranking flips with the circuit. halo2 is 8.64× slower than gnark on the authors' first test vector and 1.09× faster on the second, because of circuit-level optimizations rather than anything about the proof system. The authors' phrasing is worth keeping: custom circuit implementations "can falsify the perceived performance of a library or DSL."

Two counterintuitive arithmetic findings. Field addition and multiplication in BN254's scalar field is not consistently faster than in BLS12-381's, despite the smaller characteristic. And gnark-crypto beats blstrs on BLS12-381 — 0.35 s against 0.64 s ± 0.03 s for a 2²⁰ MSM, 1.83× — which inverts the usual assumption about that pairing.

Hardware is a variable, not a constant. Some tools gain up to 50% from compute-optimized instances, others up to 40% from memory-optimized ones, depending on the tool. A single prover timing without an instance type attached is therefore under-specified by up to half.

Proof size, as expected but quantified. Starky's STARK proofs run 299.68× to 678.98× larger than the biggest Groth16 and Plonk proofs at roughly 2²⁴ constraints or rows.

What this does to the corpus

concrete-efficiency argues that every gain here is a trade and that a headline number with no stated cost is an incomplete claim. zk-Bench supplies the missing axis: a headline number with no stated arithmetic backend and instance type is also incomplete, and by a factor large enough to swamp the differences papers are published over. binius reports 5.90 s against 13.8 s — a 2.3× claim. Fourteen times that spread is available from changing the library underneath.

That does not impeach binius or flock. Both benchmark against a rival implementation of a comparable maturity and say what hardware they used. It does mean the corpus's cross-source comparisons — reading Binius's seconds beside Flock's multiples-of-native — were never sound, and now there is a source that explains why rather than a wiki's caution about it.

Limits, and the residue

The paper is 2023 work published in 2024. Its five tools are the generation before this spoke's founding sources: no binius, no flock, no binary-field system at all, and Plonk/Groth16/STARK rather than the sumcheck-and-small-fields line the spoke follows. Its estimator is instantiated for Plonk alone, at 6–32% accuracy on circuits up to millions of gates, which is useful for planning and not for adjudicating a 9% claim.

So the independence problem is answered in principle and not for the systems held here. What is still missing is a third party running binius against flock — or either against Plonky3 — on stated hardware. Growth edge 1 is re-specified rather than struck.

Half-answered 2026-09-03 by contract-validated-zk-bench. Hulek, Peresini and Homoliak (Brno) benchmark eight systems including Plonky3 on stated hardware, with nobody's own system in the comparison, and add the piece this paper's own finding made necessary: a semantic contract that declares the public claim each benchmark proves, so two systems count as running the same run even when their circuits differ. It still holds neither Binius nor Flock. Three years on, independent benchmarking is still one generation behind the systems tracked here.

Related

concrete-efficiency · succinct-argument · binius · flock · binary-fields · synthesis