Models
Mixture of experts (MoE)
A mixture-of-experts model contains many specialised sub-networks and activates only a few per token, so it has the capability of a large model at the compute cost of a smaller one.
Also written: MoE
A model might hold hundreds of billions of parameters while activating only tens of billions for any given token. Capability scales with total parameters; cost scales with active parameters.
This is a large part of why capable models have become dramatically cheaper. It also means parameter count tells you even less than it used to.
In practice
Only a fraction of the parameters activate per token, which is how a very large model can be served at a price that looks impossible for its size. For anyone buying tokens the architecture is invisible; what is visible is that headline parameter counts and price have almost entirely decoupled.
Common questions
Does MoE affect how I use a model?
No — it is invisible through an API. What it affects is your intuition about price: a very large model can be served cheaply because only a fraction of it activates per token.
Is a MoE model less capable than a dense one of the same size?
Comparing by total parameters is the wrong comparison. What matters commercially is the price and the measured quality on your task, both of which you can observe without knowing the architecture.