Distillation
knowledge distillation · teacher-student training
กล่าวโดยย่อ
Distillation trains a smaller student model on the outputs of a larger teacher model, so the student reproduces much of the teacher's behaviour at far lower cost. It is the main reason small models improved so quickly, and it is restricted or prohibited by many commercial API terms.
The teacher model generates outputs — answers, reasoning traces, labels — and the student is trained on those, rather than on raw web text. Because the teacher's outputs are cleaner and more consistent than scraped data, the student learns faster and from far fewer examples than pre-training would require.
This is why the gap between a frontier model and a good small model narrowed so fast. It is also the mechanism behind most of the price decline in AI serving: a distilled model that handles the routine majority of requests at a tenth of the cost is the single largest saving available to most products.
Two limits are worth stating. A student inherits the teacher's mistakes along with its skills, so errors propagate rather than being averaged out. And students generalise poorly beyond the distribution they were distilled on — strong on the tasks in the training set, noticeably weaker just outside it.
The licensing question is live. "You may not use the outputs to develop models that compete with us" appears in most commercial API terms and in several open-weight licences. Whether a given distillation is permitted is a contract question, and it has become a recurring dispute in the industry.
คำถามที่พบบ่อย
- Is distillation legal?
- It depends on the source. Most commercial API terms restrict using outputs to train competing models, and some open-weight licences do too. Distilling from a model you are licensed to use for that purpose is routine practice.
- How much capability survives distillation?
- A great deal within the domain covered by the training examples, and much less outside it. Students match teachers closely on the distribution they were distilled on and degrade beyond it.