The CHiME-8 DASR Challenge for Generalizable and Array-Agnostic Distant ASR and Diarization (2024)
Samuele Cornell, Taejin Park, Steve Huang, Christoph Boeddeker, Xuankai Chang, Matthew Maciejewski,
Matthew Wiesner, Paola Garcia and Shinji Watanabe — Carnegie Mellon, NVIDIA, Paderborn and Johns
Hopkins. Interspeech/CHiME 2024; read from the arXiv PDF with pypdf. T1 — the organisers
defining their own task and publishing both baselines' numbers.
This is the source growth edge 9 asked for: chime-6's numbers are Kaldi-era hybrids, and the question was what the same task costs a current end-to-end system. The answer is that the task got harder faster than the models got better.
Four scenarios, one system
| Scenario | setting | speakers | recording setup | avg. duration |
|---|---|---|---|---|
| CHiME-6 | dinner party | 4 | 6 linear arrays | ~2h–2h30 |
| DiPCo | dinner party | 4 | 5 circular arrays | ~20–30 min |
| Mixer 6 | 1-to-1 interview | 2 | 10 heterogeneous devices | ~15 min |
| NOTSOFAR-1 | office meeting | 4–8 | 1 circular array | ~6 min |
The point of the set is that one system must handle all four without knowing the array topology. NOTSOFAR-1 is the new scenario and the one that breaks things: many speakers, short meetings, one device.
The ranking metric moved to tcpWER — time-constrained concatenated minimum-permutation WER, from Meeteval — because the previous metric (DA-WER) turned out to be "basically equivalent" to cpWER with any reasonable diarizer, and neither penalised bad segmentation.
The numbers
Evaluation-set tcpWER (%), both official baselines:
| Scenario | ESPnet baseline | NeMo baseline |
|---|---|---|
| CHiME-6 | 99.1 | 73.8 |
| DiPCo | 56.6 | 57.1 |
| Mixer 6 | 43.8 | 23.1 |
| NOTSOFAR-1 | 50.7 | 72.0 |
| Macro | 62.6 | 56.5 |
The NeMo baseline is a Conformer transducer fine-tuned on CHiME-6 and Mixer 6 after GSS pre-processing, with a KenLM n-gram in the beam search — an end-to-end model, and the challenge allowed Whisper, WavLM, wav2vec 2.0, HuBERT and OWSM as starting points. So this is the modern stack, and it gets 56.5% macro tcpWER. The read-speech numbers on this spoke's own boards live near 5%. Far-field conversation is still an order of magnitude harder, and the shift to end-to-end models did not close it — the paper notes that C7DASR was the year every participant abandoned hybrid ASR, and its best main-track system bought about 30% relative cpWER over CHiME-6.
The ESPnet baseline's 99.1% on CHiME-6 eval is the sharpest single figure here: a system can get essentially every word wrong on a task it was tuned for the year before, because retuning the diarizer's clustering threshold for a new scenario broke speaker counting on an old one.
What the challenge found out, which is not about acoustics
The organisers' own conclusion is that the dominant error source is counting how many people are in the room, and everything downstream inherits it. Two things support that:
- On CHiME-6 eval the two baselines have almost the same diarization error rate — 60.0% vs 56.7% DER — and wildly different tcpWER (99.1 vs 73.8). The difference is speaker confusion and missed speakers, which DER averages away. DER does not predict WER, so a spoke that quotes diarization quality as a proxy for transcription quality is quoting the wrong number.
- The NeMo baseline estimates speaker count by clustering TitaNet embeddings with no overlap exclusion, which is fine on 4-speaker dinner parties (73.8 on CHiME-6) and collapses on 4–8 speaker 6-minute meetings (72.0 on NOTSOFAR-1, against the ESPnet baseline's 50.7). The same pipeline, the same audio quality, a different number of people.
Links
- chime-6 — the predecessor and one of the four scenarios here; its 51.3% cpWER baseline is the figure this one should be read against, keeping in mind that tcpWER is a stricter metric and the macro average now includes three other scenarios.
- whisper — permitted as a pre-trained component for the first time in this edition, on the organisers' judgement that CHiME evaluation data was probably not in its training set, "since its performance is competitive with a WavLM-based ASR model" rather than better.