What do dense models and MoE trade?

The bottleneck

Dense models normally use all feed-forward parameters for each token. MoE grows capacity through multiple experts while activating only a subset per token. It changes capacity versus compute, without guaranteeing lower memory, latency or training difficulty.

The mechanism

A router dispatches tokens to experts and combines their outputs. Switch studied top-1 routing; other systems use multiple and shared experts. Total parameters affect storage; active parameters better approximate dominant per-token work, but attention, routing and device transfers remain.

y(x) = Σᵢ∈Top-k(x) gᵢ(x) Eᵢ(x)

Examples and evidence

Public examples include Gemini 1.5 Pro, DeepSeek-V3 and Qwen3-235B-A22B. Qwen3 also has dense models. A company need not use one architecture across its family, and A22B does not mean only 22B parameters need to be stored.

New constraints

Uneven expert loads create hotspots, idle devices or capacity overflow. Cross-device all-to-all depends on network topology and bandwidth. Evaluate sparse gains with batch size, balancing strategy and actual serving latency.