StepAudio 3 Realtime — technical report (StepFun, 2026-09-12)

stepfun's audio-language model for live spoken dialogue: 100+ authors listed alphabetically as the StepFun-Audio Team, arXiv 2609.14005 (cs.SD, eess.AS), v1 12 September 2026, CC BY-NC-ND 4.0 on the report. Read from the arXiv HTML; arrived by Telegram 2026-09-17 via the Hugging Face papers page. Weights and code: the report does not say whether either is released.

What it is

End-to-end speech-to-speech on a Step 3.7 Flash mixture-of-experts backbone — about 196B total, 11B active per token — with the AuT audio encoder from Qwen3-Omni in front and speech generation integrated behind. The loop is described as listen–converse–think–act: "Deep Perception" (acoustic cues for intent), "Seamless Duplex" (synchronised streams for pauses, backchannels and interruptions), and an asynchronous Voice Agent that runs tools without stopping the dialogue.

Think-While-Speaking

The claim the report is built around. Two concurrent inference streams: a Formulation Brain that reasons privately and an Articulation Brain that speaks, with a Speak-First mode (answer now) and a Think-First mode (short reasoning prefix). Multi-token prediction (MTP3 / MTP5) speeds the private stream by a measured 1.76×–2.05× wall-clock. The point is to keep reasoning-model answer quality without reasoning-model silence; the report says the result is "comparable to dedicated reasoning models while speaking in real time," and gives no end-to-end response-latency figure at abstract level.

The numbers, with the competitors the authors chose

So: second on its own chat benchmark, first on MMSU by seven points, first on the duplex bench by half a point, second on the agent bench by half a point. The report's own limitations: multi-turn constraint following and retail tool-use tasks "need improvement," and adaptive thinking has "uneven effects on answer quality."

What it adds here

The spoke's speech-to-speech row had one entry, gpt-realtime-2, sourced from a write-up. This is a second design with the architecture stated — MoE text backbone, Qwen's encoder borrowed, dual inference streams — and the first page here to name the Artificial Analysis Full-Duplex Bench, which is a third-party instrument even if the score on this page is the vendor's reading of it. Two of the four benchmarks are the vendor's own (StepAudioChat) or tied to a rival (τ-Voice); the duplex score is a 0.5-point margin on a saturated scale, where 98.9 against 98.4 says the bench no longer separates the top. Growth edge 1 (a neutral latency number) is untouched: no latency figure is reported at all, which for a "realtime" report is the notable absence.

Tier T3 — vendor technical report, vendor-selected comparators, release status unstated; graded below T2 for the missing latency number and the unstated weights.

Related

stepfun · gpt-realtime-2 · speech-to-speech-translation · tts-arena-leaderboard · text-to-speech · speech-to-text