Photo by www.quantamagazine.org.

Model distillation is a method by which knowledge is transferred from a larger and more powerful model to a smaller and more efficient one; the bigger model is typically referred to as the teacher and the smaller one as the student. Rather than training the student only on the original dataset labels, it is also trained to mimic the teacher’s outputs. As a result, the student gains access to more information than it would if it were given only a correct-or-incorrect label.

For example, suppose a model designed to identify whether or not a picture contains a cat, a dog, or a fox were to try to do so.

A normal training label might simply say:

Cat = 1
Dog = 0
Fox = 0

But a teacher model might output something closer to:

Cat = 0.82
Dog = 0.14
Fox = 0.04

The extra information is useful since it shows the student not just the right answer but also how the teacher views the relationships between the classes. When carrying out distillation, the student learns to mimic this behavior by using fewer parameters and less computing power.

Why is this procedure carried out? A very large model might perform extremely well. However, it can be expensive to run, slow, memory-intensive, and difficult to deploy on phones, edge devices, embedded systems, or high-volume applications. A distilled model can often maintain much of the original model’s performance while being smaller, faster, and cheaper to run.

It is particularly useful in cases where the cost of inference is more important than the cost of training.

Model distillation is commonly used for:

• Compressing large neural networks
• Reducing inference latency
• Lowering memory requirements
• Deploying models on mobile and edge devices
• Reducing serving costs at scale
• Transferring knowledge from an ensemble into a single model

The key thing to realize is that distillation isn’t just about reducing a model’s size; it’s about teaching a smaller model to act like a more powerful one. You can see it as compressing both the model and part of the knowledge contained in its predictions.

In modern AI systems, where the size of the models and the cost of running them can become major engineering constraints, distillation is one of the most practical methods of exchanging a small amount of performance for a large improvement in efficiency.