Spokes.wiki Search About

Synthesis — Speech & Audio AI

The evolving thesis. Spun out as tts-wiki from the hub _inbox speech-audio-models cluster on 2026-06-05 — seeded by the parked misotts and grown with three TTS field sources the router curated at the user's request — then broadened to all speech & audio AI and renamed speech-audio-wiki the same day, and immediately grown with four more router-curated sources covering STT/ASR and audio/music generation. The wiki now spans the three production branches of speech-audio-ai — synthesis, recognition, generation — plus an emerging fourth, audio comprehension (audio-flamingo-3); TTS is still the deepest corner.

Current thesis

The 2026 speech & audio AI market — across text-to-speech (synthesis), speech-to-text (recognition), and audio-music-generation — rhymes with the text-LLM market (../llm-providers-wiki): a closed frontier leading on the headline metric, an open-weight field competing on cost, control, and licensing. The same four dynamics recur in all three branches (see speech-audio-ai):

  1. A closed frontier on the headline metric — and on TTS the lead is not thin. Proprietary leaders top each branch's ranking: TTS Elo (tts-arena-leaderboard, 2026-08-03: Speechify Simba 3.2 at 1229, Alibaba's Qwen-Audio-3.0-TTS-Plus 1227, gemini 3.1 Flash TTS 1212; by 2026-10-07 elevenlabs' Eleven v4 Turbo leads at 1334 and Simba is eighth), STT WER (ElevenLabs Scribe ~3.3% EN, Deepgram 5.26% — stt-apis-comparison), music Elo (music-arena-leaderboard, 2026-08-03: suno V5.5 at 1189 instrumental / 1171 vocals — the earlier "~1293" came from a T4 survey on a different scale). Corrected 2026-08-03: this bullet used to say the open field was "close behind and closing." On STT that holds — canary-qwen at 5.63% nearly matches the commercial APIs, and four Apache-2.0 models now sit below it. On TTS it does not: the open leader fish-audio-s2-pro ranks 20th overall and kokoro 48th, and over the summer the gap to the leader widened from 99 to 106 Elo. The wiki had been reporting Elo deltas without ranks, which made twentieth place read like a near miss. The branches diverge, and the earlier sentence generalized the recognition branch's good news across all three.
  2. License/rights — not just the score — decides, and the axis has three positions (revised 2026-08-03). The constraint sharpens as you move across branches: TTS has a research-license ceiling (fish-audio-s2-pro is the top open model but paid-commercial); STT is mostly permissive (whisper MIT, Canary CC-BY); music escalates to outright copyright war (ai-music-copyright: suno in Sony litigation vs the clean udio / stable-audio). The best-sounding option is often the least legally safe — but mureka, the quality runner-up, fits neither pole: it asserts full commercial rights and discloses no training-data provenance, no rights deals and no operating company. Contested and licensed were the two positions this wiki knew; undisclosed is a third, and a buyer comparing on legal safety cannot place it at all.
  3. "Best" is multi-axis and weekly-volatile. Each branch sorts by its own binding constraint — quality (Elo/MOS), accuracy (WER/CER), latency (TTFA / streaming RTFx), languages, capabilities, cost (tts-benchmarks). "No single model wins." Every figure here is a dated snapshot.
  4. Convergence on the LLM stack. TTS rides Llama backbones + RVQ neural-audio-codec; STT's accuracy leaders are SALM models bolting an LLM decoder onto a speech encoder (canary-qwen = FastConformer + Qwen3; Granite-Speech; Qwen3-ASR). "Speech as language modeling" in both directions — the structural bridge to ../llm-providers-wiki (gemini straddles). A fourth branch — audio comprehension — now completes the pattern: audio-flamingo-3 (NVIDIA, a fully-open Large Audio-Language Model) doesn't transcribe or synthesize, it reasons over speech, sound, and music (unified AF-Whisper encoder, on-demand chain-of-thought, ~10-min audio), SOTA on 20+ understanding benchmarks. Where codec-token TTS and SALM STT were the production directions of "the LLM eats audio from both ends," AF3 is the pure comprehension vertex — and it re-confirms dynamic 2 (license, not score, decides): open weights and data, but a non-commercial research license, the same shippability ceiling as fish-audio-s2-pro and musicgen.

Unifying tension: as in text, capability concentrates at a few closed providers while cost and access are democratized from below by open weights — but audio adds two twists text rarely faces: latency (TTFA / RTFx) as a make-or-break product axis, and rights (research-license ceilings, voice-cloning consent, music copyright) as a constraint that can outweigh quality outright.

Recurring reads

What the word-error rates were measured on

The three corpora every comparison here quotes now have pages, and reading them together changes how the spoke should treat its own numbers.

All three are read speech. librispeech is audiobooks, common-voice is prompts read into a laptop microphone, fleurs is translated sentences read aloud in 102 languages. Nothing in the spoke's evidence base measures spontaneous conversation, overlapping talkers, or a microphone across a room — which is what the products these numbers sell are used for. The WERs are real and they are all from one register.

"Clean" is a circular label. LibriSpeech's clean/other split ranks speakers by the error rate of an acoustic model trained on WSJ si-84 and cuts at the median: "the lower-WER speakers designated as 'clean' and the higher-WER speakers designated as 'other'." So test-clean is the half of the speakers that a 2015 system already handled. Improvements reported there are improvements on material pre-selected for tractability, which is worth remembering next to the observation in open-asr-models-2026-comparison that the top ten models sit inside one WER point.

The licence that makes Common Voice useful also makes it suspect as a test set. CC0 means it is in nearly every open model's reach, and the spoke has no source measuring how much of it ended up in training data. A Common Voice score is uninterpretable without that.

This is the same shape as the spoke's standing complaint about vendor latency benchmarks — the number is precise, the conditions it was taken under are the part nobody states.

Open questions

Batch accuracy does not predict streaming accuracy (added 2026-08-22)

The research pass went looking for a neutral latency number and came back with something that undermines the spoke's accuracy evidence instead.

on-device-streaming-asr-study runs six model families in both modes on the same hardware, and the ordering inverts. Qwen3-ASR-1.7B is the most accurate model in batch at 5.90% and, streaming on a pinned 32-core CPU, manages 10.45% at 0.49× real time — it cannot keep up with the audio. Parakeet TDT goes from 6.32% to 12.83%, doubling its error rate. Nemotron-0.6B goes from third in batch to first in streaming, 7.28% at 0.56 s of algorithmic latency.

Nearly every accuracy figure this wiki holds is batch: open-asr-models-2026-comparison's table, the Open ASR Leaderboard numbers relayed through it, the WERs on whisper and canary-qwen. Those figures describe a mode most speech products do not run in, and one source now shows they do not carry over. A model is not slower and equally accurate when it streams; it is a different model.

This is the second axis on which the accuracy column has failed in one day, and the two are independent. The section below on chime-6 says the leaderboard's audio is wrong — read speech scored at 5%, conversational far-field audio at 51%. This says the leaderboard's mode is wrong — batch scored, streaming shipped, and the ordering does not survive the change. Neither finding rescues the other: a streaming board on read speech would still measure the wrong audio, and a conversational board scored in batch would still measure the wrong mode. What the corpus holds is one number that is wrong twice.

RTFx and latency are separate axes, and the spoke has been reading one as the other. Canary-1B-v2 has the best throughput in the streaming table at 2.93× RTFx and the worst latency at 4.8 seconds. Nemotron has lower throughput and eight times better latency. RTFx answers how many audio-hours per wall-clock hour, which is a cost question. It does not answer when does the first word appear, which is the make-or-break axis this spine keeps naming. Figures like the 3333× carried from open-asr-models-2026-comparison price a transcription pipeline and say nothing about a conversation.

One thing the source settles cheaply: quantization is close to free on this workload. The same model at int4 k-quant is 0.67 GB against 2.47 GB for 0.17 points of WER, and int8 measures marginally better than FP32. What costs accuracy is the PyTorch-to-ONNX export (7.28% → 8.03%), not the precision — which inverts the usual worry.

And the edge it was hunting is still half open. The measurement is independent and the hardware is named to the core count, which is more than any previous candidate offered. It never varies concurrency, and it measures self-hosted open models rather than elevenlabs-class hosted APIs. The vendor claims this spoke distrusts remain unchecked by anyone without a product in the race; what changed is that the open-model half of the field now has one honest table.

Growth edges

Ranked; each names the kind of source that would close it (see ../QUALITY.md → Growth edges).

  1. A stress-accuracy number for the Russian front end (new 2026-08-30, promoted to the top of this list because it is the cheapest open edge the spoke has). ruaccent ships ten homograph checkpoints from 10.5 MB to 713 MB and publishes no accuracy figure for any of them, so the ordering implied by their names is the only guidance a user gets. The first ingest recorded that absence and assumed it was structural — "no obvious alternative to compare against." Re-reading the source on 2026-08-30 found that assumption is false: Silero publishes silero-stress (MIT, 146★, pushed 2026-02-12) for the same task in the same language, and Alpha Cephei (the vosk authors) maintain awesome-russian-speech, the catalog where such a comparison would live. So this edge needs no new instrument and no listening panel — unlike every other edge on this list, which waits on a benchmark somebody else must build. Two libraries, one shipped dictionary of ambiguous forms, and a labelled classification task on a closed vocabulary. — needs: any published homograph-accuracy comparison, or the two libraries run over a common set. HALF-CLOSED the same day (research pass) by silero-stress (Silero Team, MIT, T1): one side now has figures — ~2K homographs at F1 0.85, 91% per-word, 93% total dataset accuracy, plus 60–70% on unknown or invented words and a near-tautological "~4M known forms at 100%", which is a lookup table being right about its own keys. Trained on ~120M annotated homograph sentences. Residue, and it is narrow: the 91% is measured on Silero's ~2K set with no published protocol or scoring script, ruaccent still reports nothing, and its shipped dictionary of ambiguous forms is a different set — so the two remain non-comparable. — needs now: both libraries over one common homograph list, with the script published. Still the cheapest edge on this list: two MIT packages and an afternoon.

  2. A neutral latency benchmark. Every latency figure in this spoke is a vendor number measured under unstated conditions. HALF-CLOSED 2026-08-22 (research pass) by on-device-streaming-asr-study (Microsoft CoreAI, arXiv:2604.14493, T2) — 50+ configurations of six model families the authors did not build, on an AMD EPYC 7V12 with 32 cores pinned, latency and RTFx published per model including the losers. Independent: yes. Hardware stated: yes. Concurrency: never varied, and the subjects are self-hosted open models rather than the hosted APIs the edge was aimed at. — still needs: the same discipline applied to the APIs, with simultaneous streams varied. — new edge 1a: batch WER does not predict streaming WER (see above), and every accuracy board this spoke holds is batch — needs: a streaming-mode leaderboard, or any second measurement of the batch-to-streaming gap. Edge 1b hunted 2026-10-01 — nothing ≥ bar. Needs one benchmark that streams, states hardware and varies concurrency. Artificial Analysis's own methodology page, read today, confirms the gap from the inside: streaming audio is paced in real time in 20 ms chunks and network delay is included, but it states no concurrency, no test region and no client hardware (open-asr-leaderboard). The only concurrency curves found are vendors measuring their own models (NVIDIA's Nemotron streaming cards and NIM tables) or a paper measuring its own adapter on a shared GPU slice (arXiv 2609.04225, Pine AI). The nearest thing to the admission branch is T3: Deepgram's own Deepgram vs Google vs AssemblyAI page opens its latency section with "Independent latency data is thin" and tells readers to "test spike behavior" themselves. A vendor saying so is a finding, but a marketing page is below the bar, so nothing was paged. Not eligible again before 2026-10-15. Original text: Every latency figure in this spoke — TTFA, RTFx, "ultra-low latency", Cerebras-class claims — is a vendor number measured under unstated conditions. It is the axis the spoke calls make-or-break and the only major one with no independent board, where TTS Elo and STT WER both have one. — needs: a T1/T2 independent latency measurement across providers with stated hardware and concurrency · hunted 2026-08-08 — nothing ≥ bar. The whole result set is the vendor-comparison genre: Deepgram, Gladia and Inworld each publish a "best STT APIs 2026" page on which they rank well. The closest thing to a third party is Coval, an eval-tooling company whose own headline for the sibling piece is "why vendor benchmarks lie" — and which still states no hardware and no concurrency, the two conditions this edge exists to demand. The figures circulating (ElevenLabs Scribe v2 Realtime "under 150 ms", Inworld "~92 ms time-to-first-token") are clean-studio marketing numbers, which is the same defect as supertonic-2's table, not its cure. Not re-hunted before 2026-08-22.

  3. One neutral board per branch, held permanently. The music corner ran two months on a T4 marketing survey's cast list. TTS and STT have boards; audio understanding now has none, and it is the branch growing fastest. — needs: a T2 leaderboard or independent evaluation covering LALMs and fixed-head understanding models on the same tasks.

  4. Anything measuring an assembled loop rather than a component. The corpus grades models; the artifacts that matter are S2ST pipelines, conversational agents and on-device stacks. No metric here applies to them. — needs: a T1/T2 end-to-end evaluation of a composite speech system.

  5. The distilled/derivative tier, measured. luxtts claims parity with "models 10x larger" and publishes nothing; the benchmark pages cover labs, not derivatives, exactly as derivatives become the interesting artifacts. — needs: any independent measurement of a distilled open TTS model.

Coverage edges (added 2026-08-08, at the curator's request for a wider backlog). These widen what the spoke covers instead of answering an open question above; one ordinary solid source closes any.

  1. The pipeline around the model. Diarization, voice activity detection and forced alignment are what a real transcript needs beside whisper or vosk, and the spoke holds models only. — needs: the pyannote paper or an equivalent, T1.

  2. The corpora. CLOSED 2026-08-09 (research pass) — librispeech, common-voice and fleurs written from their own papers, all T1. Successor, and it is a real gap: all three are read speech, so every WER in this wiki was earned on someone reading aloud, while the products those numbers sell are used on meetings, calls and far-field capture. — needs: a conversational or far-field benchmark with published WERs — CHiME, AMI, Switchboard or an equivalent — plus, separately, any measurement of Common Voice contamination in open models' training sets. The conversational half CLOSED 2026-08-22 (research pass) by chime-6 (Watanabe et al., CHiME 2020, T1): twenty real dinner parties, distant 4-channel arrays, 51.3% cpWER for the official baseline and 77.9% once the system must segment for itself. The Common Voice contamination half is untouched. Successors, edges 9 and 10: nobody has published this task's numbers for a current end-to-end model (the baselines are Kaldi-era hybrids; CHiME-7/8 DASR are unsourced here), and the 10-point gap between BeamformIt and GSS says the enhancement front-end is a lever this spoke does not cover at all.

  3. Wake words and keyword spotting. The always-on tier below full transcription, running on a microcontroller budget, is absent. — needs: a paper or an on-device toolkit's docs. Cross-spoke: the hardware side is ../embedded-iot-wiki.

  4. The real-time plumbing. Growth edge 1 wants a neutral latency benchmark for a loop that is mostly transport — WebRTC, echo cancellation, noise suppression, jitter buffers — and no page describes any of it. — needs: the WebRTC specs or a measured deployment.

  5. This task, on a current model. (new 2026-08-22, successor to edge 6) CLOSED 2026-08-31 (research pass) by chime-8-dasr (Cornell et al., CMU/NVIDIA/Paderborn/JHU, 2024, T1). The modern stack is measured: a NeMo Conformer transducer over GSS, in a challenge that permitted whisper, WavLM, wav2vec 2.0, HuBERT and OWSM, scores 56.5% macro tcpWER across four scenarios, against roughly 5% for the read-speech numbers on this spoke's boards. End-to-end did not close the far-field gap; the paper notes the switch away from hybrid ASR bought about 30% relative over CHiME-6, and the harder scenario added since ate it. Successor, edge 9a: these are the organisers' baselines, not the field's best. The paper publishes no submitted-system results, so the corpus still has no figure for what the strongest system can do on far-field conversation — only what a reasonable default does. — needs: the CHiME-8 DASR workshop results, or a winning team's system description with its tcpWER.

  6. The enhancement front-end. (new 2026-08-22) Swapping BeamformIt for GSS moved CHiME-6 Track 1 by 10 points absolute with the acoustic and language models held fixed — larger than most model-to-model gaps this spoke reports. Beamforming, guided source separation and dereverberation have no page here. — needs: the GSS paper or an equivalent array-processing source, T1.

  7. A language-coverage claim, checked by anyone. (new 2026-09-05, from omnivoice) The broadest claim in the spoke is 646 languages, and the broadest multilingual board anyone runs is FLEURS-102. So there is no instrument for six-sevenths of the claim, and this is not specific to one vendor: any model claiming the long tail is unfalsifiable there by construction. Worse, the only scoring method in use — synthesise, then run ASR — fails hardest in exactly the languages where the training data is thinnest, and OmniVoice's own Cantonese figures (17.71% under Whisper, 2.27% under SenseVoice) show a 15-point swing from the choice of recogniser alone. — needs: any evaluation of a massively multilingual TTS model on languages outside FLEURS-102, ideally with native listeners rather than an ASR proxy; failing that, a published analysis of how much of a multilingual TTS score is attributable to the recogniser. T1/T2. A per-language intelligibility study by anyone other than a model's authors would close it.

  8. The front-end, measured. (new 2026-08-27) ruaccent gave this spoke its first tts-front-end page and, with it, the same hole in a new place: ten homograph checkpoints spanning 10.5 MB to 713 MB, and no accuracy figure for any of them — on a closed-vocabulary classification task with a labelled dictionary shipped in the repo, which is far cheaper to measure than the listening tests the acoustic side argues about. — needs: any published stress or G2P accuracy, or an end-to-end TTS score with the front-end varied and the acoustic model held fixed. Compare edge 10, the ASR enhancement front-end: the same question on the other branch.

  9. Russian TTS, measured this decade. (new 2026-09-06, from russian-tts) The only independent side-by-side measurement of Russian synthesis this spoke can locate is russian-tts-evaluation, 2024-07-12 — thirteen systems, five automatic metrics, an audiobook test set, and published scoring code. Every system a 2026 buyer would shortlist post-dates it: Silero v5 shipped October 2025 claiming 3–4× v3's speed and homograph handling, elevenlabs is absent from the table entirely, and the SpeechKit figures cover two voices of eighteen. Like edge 0 and unlike the rest of this list, the instrument already exists and is re-runnable (alphacep/vosk-tts, extra/tts-test/ru) — automatic metrics, no listening panel. — needs: any 2025-or-later Russian TTS comparison, or the existing harness re-run over the current models. T1/T2.

The unit is the loop, not the model, and the loop is built at every scale

The three branches aren't only sold separately — they compose, and tracking that composition has turned out to matter more than tracking any single model's score. The pattern shows up three times over, at three different scales, and the sections below were written a month apart before it was clear they were one argument.

The composite task

The three branches aren't only sold separately — they compose. speech-to-speech-translation (S2ST) is the first applied/composite task in the wiki: STT + machine translation + TTS in one pipeline. Its 2026 exemplar, gemini-live-3-5-translate (Google; 70+ languages, 2000+ pairs, near-real-time, voice-preserving, SynthID-watermarked), is the purest case of "the LLM eats audio from both ends" — both ends in one streamed, end-to-end model. Two threads it sharpens: (1) latency becomes a quality trade-off, not just a number — simultaneous translation must choose between waiting for context and translating immediately; (2) provenance/watermarking (SynthID on all output) enters the wiki as an audio-safety axis (alongside voice-cloning consent and music copyright). It also extends the closed-frontier read: gemini now leads (by its own framing) at TTS, plays at STT, and stakes the S2ST frontier — a single proprietary provider spanning all of it.

Second composite (2026-06-13): the real-time conversational agent. elevenlabs-expressive-mode (ElevenLabs' Conversational AI) is the dialogue sibling of S2ST's translation loop: Scribe v2 Realtime STT → reasoning → Eleven V3 Conversational TTS in one low-latency turn-by-turn loop. It adds a genuinely new axis to the spoke — emotion/affect: the agent infers emotion from prosody (pitch, pacing, exclamations) and applies "tone cue cards," making paralinguistic understanding the headline differentiator rather than fidelity or WER. Two thesis hooks: (1) it confirms the branches compose into products — leaders increasingly sell the integrated loop, not components; (2) another "ultra-low latency" claim with unstated conditions (T3 vendor page) — more weight on the still-open call for a neutral latency benchmark. Caveat: a marketing landing page; CSAT/latency figures unverified.

Third entrant (2026-06-14): a single-model frontier voice agent. gpt-realtime-2 (OpenAI's Realtime-API voice model, "GPT-5-class reasoning," speech-to-speech over WebRTC) is the conversational loop collapsed into one end-to-end model — the gemini-live-3-5-translate shape aimed at general dialogue rather than translation, and the contrast to ElevenLabs' assembled STT→LLM→TTS pipeline. It pushes the "the LLM eats audio from both ends" read to its limit: a speech-to-speech network carrying the reasoning of the text frontier, not just its tokens. Its document-context feature also adds a grounding axis to spoken dialogue (answer by voice about pasted text) — retrieval entering the voice loop. So the real-time-voice frontier now has three closed entrants (Google, ElevenLabs, OpenAI). Caveat: secondhand practitioner source (T4), no latency/quality numbers.

The same loop, disaggregated onto the user's own hardware

Every source above assembles the loop in the cloud. The consumption layer builds it from the other direction — locally, out of parts anyone can download — and the binding constraints flip with it.

Every source so far looked at speech/audio AI as models and their metrics — WER, Elo, latency, license. vocalinux (open-source Linux voice dictation) opens a different layer: the end-user application that assembles those engines into something you use, where the binding constraints flip from accuracy to privacy, offline operation, and OS integration. It's a wrapper, not a recognizer — it runs a chosen local backend (whisper-cpp, whisper, or vosk) and spends its real engineering on injecting text under both X11 and Wayland. Three threads fold in. (1) It gives the recurring open wedge a concrete consumer end-point: the whole value proposition is no cloud, no subscription — private dictation built entirely from off-the-shelf open engines, the thing the open-weight field makes possible that the closed frontier structurally can't. (2) It surfaces a small/local pole the WER pages omit — vosk (Kaldi, ~50 MB, 20+ languages, runs on a Pi) is chosen for constraint, not accuracy, and whisper-cpp (quantization applied to ASR) is what makes Whisper responsive on a laptop; "best" here means runs on my hardware, offline, a fifth sorting axis under the speech-to-text leaderboard. VOSK also sits at the clean end of the license axis — Apache-2.0, so unlike the TTS research-license ceiling, the small-STT corner's constrained option is also the freely-shippable one; here the recurring "best-sounding is least legally safe" tension doesn't bite. (3) It's the inverse of the frontier voice agents — where gpt-realtime-2 and gemini-live-3-5-translate collapse the loop into one cloud model, Vocalinux disaggregates it onto the local desktop for privacy. Caveat: single T3 product write-up; the app inherits whatever accuracy its backend has, so there's no new benchmark here, only a new use.

The pole extends to the microcontroller (added 2026-07-23). moonshine-pico-voice-toolkit pushes the small/local pole one order of magnitude smaller than vocalinux: Moonshine AI's micro toolkit runs a whole offline voice loop — VAD → SpellingCNN STT → neural TTS — on a Raspberry Pi Pico 2 W in ~3.6 MiB flash / 468 KiB SRAM. It sharpens the "best = fits my hardware" axis into a design principle: at MCU scale you don't ship open-vocabulary ASR, you ship a retrainable 50-token command recognizer that fits, and you accept a bounded vocabulary as the price of no cloud, no OS, no network. Two firsts for the spoke fold in — TTS on an MCU and on-device VAD — and moonshine (previously only the "edge, from 27M params" mention in open-source-stt-models) becomes a paged subject anchoring the sub-laptop end of the size axis, below vosk/whisper-cpp. It also flips the vocalinux "assemble off-the-shelf engines" pattern: micro is a purpose-built tiny stack, three models co-designed to fit, not a wrapper choosing among general backends. Same T3 caveat, harder here — the impressive claim is the fit (concrete), while recognition quality on a 50-token SpellingCNN at this size is entirely unmeasured in the source.

The loop collapsed into one pass

The frontier collapses the loop into one model; the edge shrinks it until it fits. There is a third move: collapse it into one pass, and fix at training time which questions it will answer.

sensevoice (Alibaba/FunAudioLLM) returns ASR, spoken-language ID, a seven-category emotion label and audio-event tags from a single non-autoregressive pass over the audio. It does the job this wiki opened with audio-flamingo-3 — treat sound as something to interpret, not only to transcribe — but with fixed task heads instead of a language model, and it is fast for exactly that reason: no token-by-token decode. The README claims more than 5x Whisper-Small and 15x Whisper-Large at comparable size.

That splits the audio-understanding branch in two, and the split is architectural rather than a matter of scale. A LALM can answer a question nobody anticipated; SenseVoice answers four that were fixed at training time, cheaply enough to run over everything. Neither replaces the other, and the spoke should stop treating "audio understanding" as one shelf.

It also sharpens how the STT corner is graded here. Every model on that shelf competes on word error rate and on how small it gets. SenseVoice competes on how many questions one pass answers, which no benchmark in this corpus measures. The emotion output is the weakest part of it: seven discrete categories, no per-category accuracy, no speaker demographics, no cross-corpus evaluation — the exact conditions under which speech-emotion results are known not to transfer. The audio-event labels are the defensible half.

Two openness caveats join the standing pattern. MIT code with the weights under the FunASR licence is another instance of the split-licence shape the spoke keeps recording, and only SenseVoiceSmall is released while the benchmark tables include a Large nobody outside can run.

Read together, the three scales say the same thing about how this spoke should grade work. A model card reports WER, Elo or latency for one component. None of the artifacts above is one component, and the corpus has no measurement that applies to an assembled loop — which is why the open questions below keep asking for a neutral latency benchmark and never get one.

Openness arrives three ways, and the safety machinery rides on none of them

The spoke started by treating openness as something a lab grants: a licence attached to weights at release, with the standing question being whether the permissive tier can reach the quality tier (open-weight-tts, fish-audio-s2-pro's research-licence ceiling). Two later sources show that this is only the first of three routes, and that what fails to travel along any of them is the same thing.

Route two — the census, which counts availability rather than quality

audio-deepfake has carried the safety axis since June, and every response on it sits downstream of the models: detection (ASVspoof), watermarking (SynthID), the FCC robocall ban. free-voice-clone-list shows what the upstream looks like. It is a 59 KB catalogue of 53 open audio models, 35 of its 36 TTS entries advertising cloning from a three-to-ten-second reference sample, and it contains no occurrence of consent, ethics, watermark, misuse, deepfake, responsible, abuse or disclaimer. Zero, not few.

This wiki had already flagged the same absence on luxtts five days earlier, on a project shipping three-second cloning. One project is an oversight; a catalogue of 53 is the genre's default. So the spoke's safety axis gains a second half: the harms are documented in journalism, regulation and detection research, and the layer that puts the capability in a user's hands carries no corresponding sentence. The one licence in the catalogue with behavioural use restrictions (OpenRAIL-M) belongs to the one model that cannot clone a voice — a single data point, noted rather than built on.

Two other things the census settles, both about the shape of the open field rather than its ceiling:

The source's own weakness is the caveat on all of it: no MOS, no WER, no Elo, no methodology, one pseudonymous maintainer (0xsojalsec), and a last push of 2026-04-09 read four months later, in a field this wiki's conventions call weekly-volatile.

Route three — distillation, which needs no release decision at all

The spoke has tracked openness as a property labs grant: a licence attached to weights at release, with the standing question being whether the permissive tier can reach the quality tier (open-weight-tts, fish-audio-s2-pro's research-licence ceiling). luxtts arrives by a different route. It is not a released model; it is someone else's model, distilled to four steps and fitted with a different vocoder, published Apache-2.0 by a person who did not train the original.

Three consequences worth holding:

Capability propagates without a release decision. k2-fsa's ZipVoice licence permitted the derivative; the derivative is faster, higher-fidelity at 48 kHz, and now has more community tooling than the base. Openness here is downstream of what the base licence allowed, not of what any lab chose to promote.

The safety infrastructure does not propagate with it. audio-deepfake records SynthID watermarking and ASVspoof detection as the countermeasures to non-consensual cloning. Those attach to specific model outputs and specific vendors. A distillation with a swapped vocoder inherits the capability and none of the provenance machinery, and luxtts ships no consent language at all. The cheapest, freest cloning in this corpus is also the least traceable, and that correlation is structural rather than incidental.

Evidence quality drops as the artifact count rises. The distilled model claims parity with "models 10x larger" and publishes nothing. The corpus's benchmark pages (tts-benchmarks, tts-arena-leaderboard, mean-opinion-score) exist for exactly this and cover the labs, not the derivatives. The measurement layer is organised around the release model at the moment the interesting artifacts have stopped being releases.

The common finding is the one worth carrying: consent language, watermarking and provenance attach to a release — a specific lab's specific model — and every mechanism above weakens that attachment. The census shows capability distributed without the sentence; distillation shows it distributed without the releaser. The safety axis this spoke tracks was built for a world where openness is granted, and it is now mostly not.

The corpus's real weakness is what nobody measured, at both ends of the market

Tier discipline weighs claims. Two 2026 findings, arriving from opposite ends of the field, show the same blind spot: neither was a false claim that a tier check would have caught. One was an omission, the other a vacuum.

At the bottom — a marketing survey supplied the set of things that exist

A staleness pass refreshed the music corner and found the branch narrative built on one source: ai-music-generators-2026, a T4 marketing survey. Adding a single neutral board (music-arena-leaderboard, T2) changed three things at once.

udio is not suno's close rival. This wiki said it traded "a slight quality gap" for clean rights. It ranks 16th of 18 on instrumental, below stable-audio 2.0, and ~235 Elo behind Suno V5.5. The rights case for Udio is untouched — but "slight gap for lower legal risk" and "large gap for lower legal risk" are different decisions, and the wiki was recommending the first.

A top-two provider was missing. mureka holds ranks 2 and 3 on both boards, within 2 Elo of the leader on instrumental, and appeared nowhere in this hub. The survey never mentioned it. Its own docs, read the same day, make the omission worse rather than better: Mureka has the widest API surface in the branch — generation, vocal cloning, transcription, recognition, stem separation, and text-to-speech and podcasts — undercuts suno and udio on price, and asserts full commercial rights while disclosing no training-data provenance and not naming its operating company. The branch's quality runner-up is also its least legible participant, which is exactly what a marketing survey has no incentive to surface.

The open wedge does not exist in music. musicgen is last of eighteen.

The generalizable failure is not that a T4 source was wrong about a number — the wiki already discounts T4 claims. It is that a marketing survey silently supplied the set of things that exist. Tier discipline weighs assertions; it does nothing about a source's omissions, because an omission makes no claim to weigh. Two months of careful reasoning ran on a cast list nobody had checked.

The cheap defence, and the reason this pass was worth running: for any branch the wiki ranks, hold at least one neutral board — not to source the individual claims, but to enumerate the field. TTS had one from the start (tts-arena-leaderboard) and its corresponding error was smaller and numeric (reporting Elo without rank). Music had none, and the error was structural.

At the top — a frontier lab shipped with nothing attached

Music generation had been the branch the frontier labs stayed out of. They shipped speech (gemini-live-3-5-translate) and open instrumental research (musicgen, stable-audio); the songs-with-vocals market belonged to specialists (suno, udio, ElevenLabs Music) who were also the ones carrying the copyright exposure. lyria-3-5 ends that division: google now ships vocals, lyrics and prompt-controlled song structure as a consumer product (Flow Music), competing on exactly the capability the open wedge lacks.

What the announcement withholds counts as much as what it claims. Four adjectival bullets, one demo track, and no architecture, no baseline version, no listening test, no rights statement, and — from the lab that made SynthID a headline feature of its speech model six weeks earlier — no watermarking claim. The same evidence vacuum that luxtts showed at the open end ("on par with models 10x larger," nothing published) now appears at the closed frontier, from a lab that has the evaluation apparatus and chose not to use it here.

So the branch's two poles are converging on a shared problem for this wiki rather than on quality: the artifacts that matter most are arriving with the least measurement attached. On music specifically the spoke can still rank suno by Elo and place stable-audio by licence, but it cannot place the newest entrant at all until someone independent measures it.

So both ends fail the same way, and the failure is not dishonesty. A T4 survey and a frontier announcement are equally unfalsifiable when one omits the competitors and the other omits the numbers. The defence in both cases is the same and is cheap: hold one neutral board per branch, and treat an artifact with no independent measurement as unplaced rather than as unranked.

The accuracy numbers describe a task nobody buys the product for (added 2026-08-22)

chime-6 closes the conversational-benchmark half of coverage edge 6 and reframes the spoke's whole accuracy column.

The size of it. Open STT models lead the current board at 5.04–5.63% WER on read speech. On unscripted four-person dinner-party audio captured by distant arrays, the challenge's official baseline was 51.3% and the best system of that era 45.6%. Even allowing for six years and a metric change, the corpus has been quoting one task's difficulty while describing another's use case — meetings, calls, far-field capture.

The metric is part of the finding. cpWER concatenates each speaker's utterances and scores under the best of 24 speaker permutations, so misattributed words count as errors. A "WER" that includes diarization is not the same measurement as a "WER" that assumes one speaker, and this spoke has been putting both in the same sentences.

Two levers this spoke does not grade. Changing only the enhancement front-end — BeamformIt to GSS, same acoustic and language models — moves Track 1 from 61.2% to 51.3%. Removing oracle segmentation costs 61.2% → 77.9%. Both are worth more than most model-to-model differences on the leaderboards here, and neither the array processing nor the diarization stage has a page. Growth edge 5 asked for the pipeline around the model; this is the first measurement of what that pipeline is worth.

Six years and an end-to-end rewrite later, the gap is still there (added 2026-08-31)

chime-8-dasr runs the same task with the modern stack and settles growth edge 9. The 2024 challenge's own baselines are a Conformer transducer over guided source separation, in a rulebook that permits whisper, WavLM, wav2vec 2.0, HuBERT and OWSM — and the better of the two scores 56.5% macro tcpWER over four scenarios. The corpus's read-speech board still sits near 5%.

That is worth stating carefully, because the two numbers are not the same measurement and the scenario set changed. CHiME-6 is now one of four, sitting beside DiPCo, a 2-speaker interview corpus and NOTSOFAR-1's 4-to-8-speaker six-minute office meetings; tcpWER adds a temporal constraint to cpWER, so it is the stricter metric. The honest summary is not "no progress": the organisers report that the shift to end-to-end ASR bought roughly 30% relative over the CHiME-6 era. It is that the task got harder at least as fast, and the order-of-magnitude distance between read speech and far-field conversation has not moved.

The new finding is about diarization, and it contradicts a convenient assumption. On CHiME-6 eval the two baselines are three points apart on diarization error rate — 60.0% against 56.7% — and twenty-five points apart on tcpWER (99.1 against 73.8). DER averages missed speech, false alarms and speaker confusion into one number, and the component that drives transcription is speaker counting, which DER hides. So the pipeline lever named above has a sharper form: it is not "diarization matters", it is that the standard diarization metric does not predict the thing this spoke cares about, and quoting DER as a proxy for transcription quality would mislead. The organisers' own conclusion is the same — speaker counting is the component "responsible for most downstream recognition errors."

The 99.1% itself is worth keeping. That is the previous year's baseline, retuned for a new scenario, getting essentially every word wrong on the scenario it used to handle — a reminder that these pipelines are tuned to a room, not general.

The licence axis grew a third pole, and cloning got onto the CPU (added 2026-08-25)

audio8-tts arrives as a 0.17B multilingual cloner packaged for CPU inference, and it moves three of this spoke's standing arguments at once.

The permissive/restrictive split is now a three-way one

The spoke has read licensing as two poles since June: Apache/MIT on one side, research-only or paid-commercial on the other, with the open ceiling question (open-weight-tts) turning on fish-audio-s2-pro sitting in the restrictive corner. The Audio8 Community License v1.0 is neither. Commercial use is free below US$2M in consolidated annual revenue and forbidden above it without a written licence — the permission is granted to a class of user, not to a class of use.

That breaks the census method free-voice-clone-list supplied. Counting "27 Apache-2.0, 4 MIT, 1 Research License" sorts licences by whether they restrict; this one restricts nobody who is small and everybody who is not, so it lands in either bucket depending on who is asking. The shape is borrowed from source-available software licensing rather than from anything else in the audio field, and one instance is not a trend — but the axis the spoke was counting on is no longer one-dimensional.

And the same licence carries the sentence the whole field was missing. audio-deepfake plus the census gave the spoke its sharpest finding: 53 open audio models, 35 of 36 TTS entries advertising cloning, and zero occurrences of consent, watermark, ethics or misuse anywhere in the catalogue; the one licence with behavioural restrictions (OpenRAIL-M) sat on supertonic-2, which cannot clone. Audio8's licence forbids impersonation without consent and requires disclosure of synthetic audio, on a model whose primary use is cloning. Second data point, first one on a cloner. The generalisation still holds — this is one repo against a genre default — but it is now falsifiable rather than absolute.

The small pole reached zero-shot cloning above real time on a CPU

The efficiency ladder ran 15M (kittentts) → 66M (supertonic-2) → 82M (kokoro), and the two that were actually measured dropped cloning to get there. Audio8 sits above them at ~170M plus a ~120M codec decoder and keeps zero-shot cloning at 44.1 kHz, at ~19 ms per slow-AR token and ~8 ms per fast-AR frame against 46.5 ms of audio per frame — about 0.58 RTF, faster than real time, on CPU with no GPU (our arithmetic from the card's numbers). The enabling trick is architectural: a Falcon-H1 hybrid slow branch, attention plus Mamba, exported as a per-token recurrent graph with the state passed explicitly, so nothing has to re-read a growing KV cache. That is the spoke's first state-space component on the TTS side, and it is what makes the ONNX export cheap rather than merely possible.

So the small end of open-weight-tts is no longer defined by what it gives up. Which raises the question the next source should settle: the on-device pole has been arguing about parameters when the binding constraint was decode shape.

The benchmark it loses on is the one nobody prints

Audio8 publishes speaker similarity beside WER, and the two orderings nearly reverse: near-parity with a 4.6B model on intelligibility (1.662 vs 1.607 EN WER), 8 to 26 points behind on similarity (56.7 vs 64.6 vs IndexTTS2.5's 82.3 — and IndexTTS2.5 is the worst model in that table on WER). This is the tts-benchmarks problem in a single table, and it is worse than the batch-vs-streaming and clean-vs-far-field findings above, because there the wrong number was at least measuring the task. No board this spoke tracks reports similarity at all. A "small model matches a big one at cloning" headline is available from this data and it is false.

And it contradicts its own sibling repo three times

Same publisher, six days apart, two repositories: ~170M+~120M vs "about 100M" parameters (neither is 0.1B, the name), 8 languages vs 11 (the base card states in as many words that evaluation does not expand the language claim; the derived repo adds three unevaluated languages), and Apache-2.0 vs the revenue-capped licence on weights that are the same weights. Recorded per record-don't-overwrite. The pattern the spoke keeps meeting — kokoro's cloning checkbox, canary-qwen's leaderboard position, luxtts's unmeasured SOTA claim — is that the primary source is one fetch away and the disagreement survives because nobody fetches it. Here both primaries were fetched and they disagree with each other, which is a failure mode the earlier lesson does not cover.

The front-end is a whole stage the corpus had no page for (added 2026-08-27)

Sixty-odd pages of models, three benchmark corpora, four leaderboards — and until ruaccent arrived, nothing about what happens to text before a synthesizer reads it. The tts-front-end page exists now, and it changes two readings.

A "multilingual" model is a claim about the acoustic stage only. Every language count this spoke records — 11 for audio8-tts, 16 for xtts-paper, 102 in fleurs — counts languages the model can produce sound in, given text it can already read. Russian text does not write stress, and stress in Russian decides which word it is (з+амок castle, зам+ок lock) and how every vowel in it reduces. So a synthesizer supporting Russian either has a stress model in front of it or is guessing. ruaccent is what that stage costs when someone builds it properly: five ONNX models, 21 MB of accent dictionaries, a vendored CRF part-of-speech tagger, and three years of one person's work — none of which appears in anybody's language count. The same is true of Hebrew and Arabic diacritization, Japanese pitch accent, Mandarin polyphones. English is the language where this stage is nearly free, and English is where the benchmarks are.

And it repeats the corpus's defining absence in a place where the excuse does not hold. The running complaint here is that quality claims arrive unmeasured — luxtts asserting parity with "models 10x larger" and publishing nothing, supertonic-2's vendor table, audio8-tts's unmeasured INT8. Those at least involve MOS panels or listening tests, which are genuinely expensive. Stress placement is not: it is classification over a closed set of homographs, scored against a labelled dictionary that ships inside the repository. RUAccent publishes ten checkpoints, a 68× size range between the smallest and largest, an implied ordering in their version numbers, and not one accuracy number. When measurement is this cheap and still absent, the explanation is not cost. It is that nobody in this field is asked for it.

Both points sharpen an existing one. chime-6 showed the ASR enhancement front-end moving results by 10 points absolute with the models fixed — bigger than most model-to-model gaps this wiki reports. The TTS front-end has no equivalent measurement, and no reason to be smaller in a language where one wrong mark produces the wrong word.

Amended 2026-09-06: "in front of it" was too narrow. Ingesting silero-tts found the stage in a third shape. Silero v5 (October 2025) does stress, ё restoration and homograph resolution inside the synthesizer — put_accent, put_yo, put_stress_homo, put_yo_homo are flags on the synthesis call, not a library you run first. So the corpus now holds all three arrangements: bolt-on (ruaccent), the model vendor's own standalone library (silero-stress, MIT), and folded in (silero-tts, CC BY-NC-SA). The claim that a Russian synthesizer needs a stress model survives intact; what changed is that you cannot tell from the outside whether one is there. yandex-speechkit publishes nothing about its front end and takes the lowest character error rate of any Russian system measured — evidence that something good is inside the box, and no way to see it.

A support claim is not a measurement, and 646 is the largest one yet (added 2026-09-05)

omnivoice (k2-fsa, Apache-2.0, 0.8B) arrived by Telegram claiming 646 languages on 581k hours drawn from 50 open-source datasets. It is the broadest coverage claim in this spoke by an order of magnitude, and it is worth being exact about which part of it is measured.

The paper is honest and the packaging is not. k2-fsa publish the per-language hours, and the paper names the problem in its own words — "severe data imbalance", handled by resampling at β=0.8. The spread is Chinese 111,343 hours against Dhofari Arabic 0.02 hours, roughly 5.6 million to one; Haitian gets 0.04, Tigrinya 0.08, Votic 0.10. Then docs/languages.md lists all 646 rows identically, with an hours column and no support tier, no quality level and no caveat, and the README says "600+ languages". A reader looking up their own language finds it present, and nothing tells them that present can mean seventy-two seconds of audio.

The measured fraction is smaller than the claim by a factor of six. The multilingual evaluation is FLEURS-102 and MiniMax-Multilingual-24. At most 102 of 646 languages carry a published number, so about 544 have a support claim and no measurement of any kind — not a poor score, no score. This is the spoke's standing finding ("the corpus's real weakness is what nobody measured") arriving on a new axis: previously the unmeasured thing was latency, or the assembled loop, or the distilled model's parity claim. Here it is the headline feature itself.

Two numbers in the same paper show why the measured part is also softer than it looks.

It beats the ground truth, twice. On LibriSpeech-PC the model scores WER 1.30% / UTMOS 4.28 against real recordings at 1.87 / 4.10; on FLEURS-102 it clears CER ≤5% in 82 languages against the ground truth's 75. A synthetic voice outscoring human speech on an ASR-derived metric is the metric running out of headroom — synthesised audio is cleaner and more canonically pronounced than a person in a room. It belongs beside mean-opinion-score's caution rather than in a capability claim.

And changing the judge moves a language by 15 points. The paper reports Cantonese at WER 17.71% under Whisper and 2.27% under SenseVoice-Small, reading it as an ASR failure rather than a synthesis failure. They are almost certainly right, and the consequence generalises: every multilingual TTS number in this spoke is an ASR number wearing a TTS label, and it degrades exactly where ASR coverage is thin — which is the same tail where the training data is thin. The two errors point the same way and there is no published way to separate them.

What this does not undercut. The self-run comparisons against ElevenLabs v2 (WER 10.950% against OmniVoice's 2.850% on MiniMax-24) and the RTF 0.0319 against their own ZipVoice's 0.0557 are real numbers with a stated method, and k2-fsa publishing the per-language hours at all is more than any other model in this spoke does. The complaint is not that they overclaim. It is that the field has no instrument for the thing they are claiming, so the claim cannot be checked by anyone — written as growth edge 12.

The music corner gets an analysis model (2026-09-16)

sheetsage2 is the first page here that takes recorded music apart rather than making it: audio in, lead sheet and timed annotations out, from 57M parameters of adapters on m-a-p's MERT encoder. The design is the speech pattern this spoke keeps meeting — a large pretrained encoder, small task heads — moved to music understanding. The first-party scores are high on the rhythmic tasks (beat and downbeat over 89% F1) and lower where it matters for the product claim: vocal melody at 82.51% F1 is one note in five wrong, so "editable lead sheet" is accurate and "finished lead sheet" is not. No report, no independent run, non-commercial weights; the same caveats as the generators.

A "realtime" report with no latency number (2026-09-17)

stepaudio-3-realtime gives the speech-to-speech row a second architecture beside gpt-realtime-2, and the design is legible: a 196B/11B-active MoE text model, Qwen3-Omni's audio encoder borrowed whole, and two inference streams so the model can reason privately while it talks. The benchmark table is the usual shape — the vendor's own chat set, a rival-tied agent set, and two third-party instruments — and the margins on the third-party ones are the finding: 98.9 against 98.4 on the Artificial Analysis Full-Duplex Bench is a saturated scale, and seven points on MMSU is the only result that separates anything. What is missing is the number the title promises. Nothing in the abstract or the results gives an end-to-end response latency; the only speed figure is a 1.76–2.05× speedup of the private stream. Growth edge 1 stays exactly where it was, and this page is one more vendor entry in the column it describes.

Contradictions / tensions

Cross-spoke adjacency

The board everyone quotes, read for once (2026-09-06, research pass)

Eleven pages in this spoke rest on Open ASR Leaderboard WER and none of them had read its method. open-asr-leaderboard fixes that, and the method is better than the corpus assumed: since 2026-07-24 every short-form run goes through Hugging Face Jobs on one flavor — 1× H200 (141 GB), 23 vCPU, 256 GB — as a per-family Docker image, with the cost of a full pass printed ($2.92 for parakeet-tdt-0.6b-v3, $4.75 for whisper-large-v3-turbo, $5.58 for Qwen3-ASR-1.7B) and a private test half the maintainers run after submission. That is a third party, on fixed hardware, inviting a re-run at a stated price, which is more than this corpus can say for any vendor number it holds.

It also does not measure what the spoke keeps being told about. RTFx is throughput, not latency — audio seconds per wall second on batch short-form files — and concurrency never varies. So the "sub-300 ms at p95" style of claim that fills vendor pages remains unchecked by anything here. Together with on-device-streaming-asr-study (streaming, latency, a named CPU, no concurrency) the four conditions edge 10 asks for are covered by two instruments that measure different things, which is the honest version of a half close.

The older worry stands and is now better located: every accuracy board here is batch, while the product being sold is streaming, and the one study that measured both found the ranking changes (Qwen3-ASR best in batch at 5.90% and 0.49× real time streaming; Nemotron third to first at 7.28%).

A whole language, and one table from 2024 (2026-09-06, research pass)

A curator question — which TTS is best for Russian audiobook narration — sent a research pass into russian-tts. The answer exists and rests on a single source that is two years old, which is the finding.

What can be said with numbers. russian-tts-evaluation (nickolay-shmyrev, alpha-cephei, 2024-07-12) put thirteen Russian systems through CER, xRT, UTMOS, speaker similarity and Encodec FAD, on audiobooks — ~100 speakers, ~1k utterances — and published the scoring code. Yandex takes the lowest character error rate in the field (0.6) with UTMOS 3.41–3.48; Silero v3_1 ties for second on clarity at 0.7 and is last-but-few on intonation (FAD 97–170); Piper irina, on one hour of training data, takes the highest UTMOS of all twenty-one rows.

Three results worth carrying beyond Russian. Cloning has a measured price: every multi-speaker system scores CER 2.1+ against 0.6–0.7 for single-speaker leaders — "multi-voice systems seriously suffer from fuzziness" — which is the first number this corpus holds against a feature 35 of 36 models in free-voice-clone-list advertise for free. Data quality dominates data quantity: one good hour beat three sibling voices trained the same way on worse recordings, by two full CER points. And the evaluator prefers clarity to intonation on the listener's behalf — "you can deal with plain intonation but artifacts are really annoying" — a ranking rule no leaderboard in this spoke applies.

What cannot be said, which is more. The table predates Silero v5 by fifteen months, contains no elevenlabs row, and covers two of SpeechKit's eighteen Russian voices. So the three systems anyone would actually shortlist in 2026 are, between them, unmeasured. Recorded as growth edge 13; like edge 0 it needs no new instrument, only for someone to re-run one that is already published.

And the conflict is on the record. Vosk-TTS, which wins the intonation and studio-quality columns, is the evaluator's own system. The page keeps it at T2 rather than T3 because method, metrics and code are all published and re-runnable — the distinction this corpus draws between a measurement and a claim — while weighting those two columns differently from the rest.

The licence decides more of this than the scores do. Silero's Russian models are CC BY-NC-SA and XTTS-v2 is CPML, so the two most-recommended free options both fail on a book anyone intends to sell — while silero-stress, the same team's front-end library doing the harder linguistic work, is MIT. A vendor splitting its own stack across the licence axis at the model boundary is new here, and it points the same way open-weight-tts's third pole did: the interesting licence question is no longer open-versus-closed but which part of a pipeline is which.

A model finally gets used, not just described (2026-09-24)

yue2-studio is a Windows app built around YuE2, m-a-p's song-generation model: full songs with vocals from a prompt and lyrics, an editable score, exact replay of past renders, and covers made by transcribing an existing recording with sheetsage2 first. That last part is the find. sheetsage2 has sat in this wiki since 2026-09-16 as a thin node — a model card with no application pointed at it. This is the first source here that shows it doing its actual job, and the job carries the weakness the model-card page already flagged forward: a cover's lyrics have to land on the syllables and stresses sheetsage2 transcribed, so its measured melody-F1 (82.51%, one note in five wrong) is no longer an abstract accuracy number but a direct limit on how good a cover can be.

It's also a third instance of the consumption-layer pattern this synthesis has tracked since abogen (TTS, on kokoro) and vocalinux (STT, on Whisper/VOSK): an app that contributes no model of its own and instead supplies the packaging — native install, GPU-aware backend, no Python — around someone else's open weights. What's new is that the pattern now has a face: ilya-timonin (timoncool) has built several of these Studio apps around different open audio-gen models (only this one is read here), which makes the pattern a developer's repeated move rather than a coincidence across three unrelated authors.

No neutral board covers YuE2. music-arena-leaderboard already tracks Suno, Mureka, Udio and musicgen on Elo; YuE2 has no row there, and the only performance number in this source is yue2-studio's own render-speed claim (46 seconds for a 3:38 song, RTX 4090) — a speed figure, not a quality one, and flagged as self-reported on its page rather than treated as evidence either way.

Licence split follows the usual shape, not the Silero one. The app and its inference engine (yue2.cpp) are MIT; the model weights are CC BY-NC 4.0. That's the ordinary open-tooling-around-closed- weights split already seen in abogen/kokoro and musicgen's own code/weights divide — not the same-vendor-splits-its-own-stack case Silero showed on 2026-09-06.

Diarization gets its own model, and the gain is where the benchmark is easy (2026-10-05)

diarizationlm-gemma-4-e4b is the first source here that is about speaker labelling itself. It is a 4B gemma-4-based LLM that rewrites the speaker labels on an existing transcript. The vendor's WDER falls from 5.32% to 2.99% on Fisher and from 7.74% to 4.92% on Callhome (two-speaker phone calls), but only from 14.70% to 14.10% on ICSI and 15.68% to 14.89% on AMI (meetings). That is the same shape the CHiME pages found from the other side: chime-8-dasr showed DER does not predict downstream error and that speaker counting drives most of it, and chime-6 scored cpWER because a wrong speaker is a wrong word. A text-only corrector helps most where the acoustic stage already did its job, and least where several voices overlap. That reading is ours, not the source's.

Caveats stay attached. T3: trade-press relay of model-card numbers, not read at the card. WDER assumes correct words, so it does not combine with the WER figures elsewhere in this spoke. The meeting sets are 3 and 16 meetings. No latency or hardware figure, so growth edge 1 (stated hardware, varied concurrency) is untouched. Open question added: does a post-hoc LLM relabeller help on far-field multi-speaker audio at all, or only on clean two-party calls? Nothing here measures it.

Index — Speech & Audio AI Wiki

Catalog of every page, grouped by schema.org @type. Spine: synthesis (thesis), log.md (history), this file (catalog). Domain = speech & audio AI models across three branches — TTS (synthesis · deepest), STT/ASR (recognition), audio/music generation. Some wiki-links resolve cross-wiki to ../llm-providers-wiki (gemini, open-weight-models, quantization) — intentional bridge links. Elo / WER / latency facts are dated snapshots.

DefinedTerm (concepts)

SoftwareApplication (models)

SoftwareApplication (apps over the engines)

SoftwareSourceCode (the pipeline beside the models)

Organization (providers)

Person (curators & authors)

Collection (sources)

ScholarlyArticle (sources)

WebPage (sources)

Dataset (benchmark corpora)

Article / BlogPosting / Dataset (sources)

Synthesis

Bridge nodes (live in sibling wikis, linked cross-wiki)

gemini · open-weight-models · quantization (../llm-providers-wiki) · llms-local-list (../llm-inference-wiki — the same curator's text-LLM census, see 0xsojalsec)

Related pages

Links to

Linked from