Inference
serving · generation
In short
Inference is the act of running a trained model on new input to produce output, as opposed to training, which produces the model in the first place. Training is a one-off cost; inference repeats for every request, and over a widely used model's life it dominates both spend and energy consumption.
Inference splits into two phases with different performance characteristics.
Prefill processes the whole input at once. It parallelises well and is compute-bound, so it scales with input length and determines how long the user waits for the first token.
Decode generates output one token at a time, each step requiring a full pass over the model's weights. It is memory-bandwidth-bound, and it sets the words-per-second rate the user perceives.
Most serving optimisations attack one of these. Continuous batching keeps accelerators busy by merging requests that arrive at different times. KV caching stores intermediate attention state so earlier tokens are not recomputed at every step. Speculative decoding has a small model draft several tokens that the large model verifies in one pass. [Quantisation](/en/glossary/quantization) shrinks the weights so more fits in fast memory. [Mixture-of-experts](/en/glossary/mixture-of-experts) routes each token through only part of the network.
The economic consequence is that inference efficiency, not model size, determines whether an AI product has a viable margin — and the same optimisations that cut cost also cut energy consumption.
Frequently asked questions
- Why is the first token slower than the rest?
- The model must process the entire input before generating anything — the prefill phase. After that, tokens are produced one at a time in the decode phase, which is limited by memory bandwidth rather than raw compute.
- Does inference cost more than training overall?
- For any model in real production use, yes. Training is finite; inference repeats indefinitely, and cumulative inference cost overtakes training well within a model's service life.