Mixture of Experts (MoE) — Wikipedia

The definitional reference for the architecture under almost every 2026 open-weight flagship — the “sparse MoE” the synthesis names as the first footprint lever. Wikipedia, so T2, but for a settled architecture definition it’s a clean, source-cited explainer.

Facts it pins down

• What it is. Multiple “expert” sub-networks divide the problem space; a gating network (router) picks which experts run per input. The sparse, modern form computes softmax(topk(Wx + noise)) and takes a weighted sum of only the top-k experts, not all of them — vs classical MoE which sums every expert.
• Total vs active params. Because only a subset fires per token (conditional computation), a model can carry huge total parameters while only activating a few — Wikipedia’s example: an MoE used less inference compute despite 30× more parameters than a dense alternative. This is exactly the “large total / small active” pattern this wiki records (Gemma 4 25.2B/3.8B, Llama 4 Maverick 400B/17B, DeepSeek R1 671B/37B).
• k is usually 1 or 2. Switch Transformers use k=1; Mixtral 8×7B (Mistral, Dec 2023) uses k=2 over 8 experts (46.7B total).
• Shared experts. DeepSeek-V2 adds shared experts that always activate alongside the routed ones — a refinement of the basic top-k scheme.

Why it matters here

Backs the open-weight-models “MoE dominance” trend and the synthesis claim that sparse MoE gives “frontier capability at practical inference cost” with the actual mechanism: the router’s top-k selection is why total-vs-active params diverge and why serving stays cheap. The deeper serving mechanics (routing kernels, expert parallelism) belong cross-wiki to llm-inference.

Related

open-weight-models · llm-benchmarks · deepseek · mistral-ai · llm-inference