Mixture of Experts (MoE) — Wikipedia

The definitional reference for the architecture under almost every 2026 open-weight flagship — the "sparse MoE" the synthesis names as the first footprint lever. Wikipedia, so T2, but for a settled architecture definition it's a clean, source-cited explainer.

Facts it pins down

• What it is. Multiple "expert" sub-networks divide the problem space; a gating network (router) picks which experts run per input. The sparse, modern form computes softmax(topk(Wx + noise)) and takes a weighted sum of only the top-k experts, not all of them — vs classical MoE which sums every expert.
• Total vs active params. Because only a subset fires per token (conditional computation), a model can carry huge total parameters while only activating a few — Wikipedia's example: an MoE used less inference compute despite 30× more parameters than a dense alternative. This is exactly the "large total / small active" pattern this wiki records (Gemma 4 25.2B/3.8B, Llama 4 Maverick 400B/17B, DeepSeek R1 671B/37B).
• k is usually 1 or 2. Switch Transformers use k=1; Mixtral 8×7B (Mistral, Dec 2023) uses k=2 over 8 experts (46.7B total).
• Shared experts. DeepSeek-V2 adds shared experts that always activate alongside the routed ones — a refinement of the basic top-k scheme.

Where 2026 practice has gone past the explainer

Two refinements in qwen38-flash-next (2026-08-26) sit outside this article's picture. Ultra-sparse MoE: hold the number of activated experts fixed and keep growing the pool — Qwen reports training loss falling steadily under global load balancing, so "how many experts exist" and "how many fire" are being tuned as independent knobs. And capacity that is not an expert at all: 51B N-gram embedding parameters, addressed deterministically by local context, added on top of a 125B/6B-active backbone and offloadable to host memory. Top-k routing is no longer the only way a model carries parameters it does not multiply by.

Why it matters here

Backs the open-weight-models "MoE dominance" trend and the synthesis claim that sparse MoE gives "frontier capability at practical inference cost" with the actual mechanism: the router's top-k selection is why total-vs-active params diverge and why serving stays cheap. The deeper serving mechanics (routing kernels, expert parallelism) belong cross-wiki to llm-inference.

Related

open-weight-models · llm-benchmarks · deepseek · mistral-ai · llm-inference · qwen38-flash-next