Mixture of experts (MoE)
MoE · sparse model
ひとことで
Mixture of experts splits parts of a network into many parallel sub-networks and uses a router to send each token through only a few of them. Total parameter count stays large while the computation per token stays small, which is why several very large models are cheaper to serve than their size implies.
In a dense model, every parameter participates in processing every token. In a mixture-of-experts model, certain layers are replaced by a set of parallel experts, and a small router network selects which two or three of them each token passes through.
The economics are the point. A model may hold hundreds of billions of total parameters while activating only tens of billions per token. Capacity scales with the total; the compute bill scales with the active portion.
The trade-off is memory. All experts must be resident and reachable, so the hardware requirement follows total parameters even though the arithmetic follows active ones. That makes MoE models attractive to operators running at scale and awkward for someone trying to fit one on a single accelerator.
Training brings its own difficulty: if the router sends most tokens to a handful of favourite experts, the rest never learn anything useful. Load-balancing objectives exist to prevent this and add a tuning problem that dense models do not have.
よくある質問
- Why do MoE models report two parameter counts?
- Total parameters describe the whole model; active parameters describe how many are used per token. Compute cost tracks the active count, while memory requirements track the total.
- Are MoE models worse than dense models?
- Not inherently. They give more capacity per unit of compute, at the cost of higher memory requirements and more complex training, since the router must learn to distribute work evenly.