CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings (2020)

Shinji Watanabe, Michael Mandel, Jon Barker and Emmanuel Vincent with 17 co-authors — Johns Hopkins, CUNY, Sheffield, Inria, Paderborn, Hitachi, Microsoft and the Linguistic Data Consortium. CHiME 2020 workshop; read from arXiv 2004.09249v2. T1 — the challenge organisers describing their own task and baselines, which is what a benchmark's definition is.

The conversational benchmark this spoke has been missing. librispeech, common-voice and fleurs are all read speech, so every WER on this wiki was earned on someone reading aloud.

The recording is the point

Twenty real dinner parties in twenty different homes, each with two or three recording areas, four participants each, "talkers speaking in a relaxed and unscripted fashion," with real domestic noise — kitchen appliances, air conditioning, movement. Captured simultaneously on multiple commercially available 4-channel microphone arrays plus binaural mics worn by each speaker. The material is CHiME-5's, re-released with accurate array synchronisation.

Two tracks: Track 1 gives the system the utterance boundaries; Track 2 does not, so a system has to diarize and recognise together — "the first challenge activity in the community to tackle an unsegmented multispeaker speech recognition scenario with a complete set of reproducible open source baselines."

The metric is not WER, and the difference matters

Scoring uses cpWER — concatenated minimum-permutation word error rate. Concatenate all utterances per speaker in both reference and hypothesis, compute WER under all 24 speaker permutations, and take the lowest. So cpWER is directly affected by the diarization: a system that transcribes every word correctly and attributes them to the wrong speakers is scored as wrong. Any comparison between a number here and a single-speaker WER elsewhere on this wiki is comparing two different measurements.

The numbers

What this does to the rest of the spoke

An order of magnitude, not a margin. The open STT leaderboard this spoke tracks reports 5.04% to 5.63% WER for current open models. The best system in this family, four years of challenge effort in, sat at 45.6%. Different era, different metric, different task — and that is exactly the point: the numbers circulating as "speech recognition accuracy" describe read speech captured close to the mouth, and the task products are actually sold for is somewhere near this page instead.

Enhancement is worth more than the model here. Swapping BeamformIt for GSS moves Track 1 from 61.2% to 51.3% on identical acoustic and language models — a 10-point absolute gain from the front-end alone. The spoke grades models; on far-field audio, the array processing in front of the model is a comparable lever, and nothing else here describes that layer.

And diarization is the tax. Losing oracle segmentation costs 61.2% → 77.9% with the same enhancement. That is the size of the pipeline gap growth edge 5 names: whisper or vosk alone does not produce a transcript of a meeting, and what fills the difference has no page here.

Caveats

• 2020, and Kaldi-era. The baselines are hybrid HMM systems. Retracted 2026-08-31: this page said nobody had published the task's numbers for the current generation of end-to-end models, and that CHiME-7 and CHiME-8 DASR were unsourced here. chime-8-dasr is now in the wiki, and its end-to-end baselines score 56.5% macro tcpWER over four scenarios — so the gap survives the model generation, on a stricter metric and a harder scenario set.
• Four speakers, English, dinner parties. Wide, natural and specific — not a general conversational benchmark.
• The organisers built the task, the baseline and the scoring; there is no independent party in this benchmark. That is normal for a shared task, and it still bounds what the numbers mean.

Related

chime-8-dasr · librispeech · common-voice · fleurs · speech-to-text · whisper · open-asr-models-2026-comparison · speech-audio-ai