Back to series
· 15 min read

Model Distillation and Small Model Deployment


I. What Is Model Distillation

Model distillation (Knowledge Distillation, KD) is a model compression and knowledge transfer technique. It uses a larger, more capable teacher model (Teacher Model) to guide the training of a smaller student model (Student Model), so that the student model can retain as much of the teacher model’s capability as possible while significantly reducing parameter count, GPU memory usage, and inference cost.

Traditional supervised learning mainly relies on ground-truth labels annotated by humans, whereas model distillation can also use the teacher model’s output probability distributions, intermediate-layer features, reasoning processes, and generated results as additional supervision signals.

For example:

  • Teacher model: a 70B-parameter large language model.
  • Student model: a 7B-parameter small language model.
  • Training data: the original corpus, responses generated by the teacher model, and reasoning samples.
  • Training objective: enable the 7B model to learn the 70B model’s output behavior and part of its knowledge capabilities.

Distillation is not simply shrinking the parameters of a large model; rather, it trains an independent small model to learn the knowledge and behavior patterns captured by the large model.

1.1 Why Model Distillation Is Needed

Large models are highly capable, but their deployment costs are also high.

Comparison dimension Large model Small model
Parameter scale Billions to hundreds of billions Hundreds of millions to billions
GPU memory requirement High Low
Inference latency Relatively high Usually lower
Deployment cost High Low
Concurrency capability Limited by GPU memory and compute Usually easier to scale on the same hardware
Knowledge coverage Usually broader Depends on training data and model capability
Complex reasoning Usually stronger May suffer capability loss
On-device deployment Limited by hardware Easier to implement

Model distillation mainly addresses the following problems:

  1. Lower deployment costs: reduce GPU memory usage, compute resource consumption, and inference costs.
  2. Reduce response latency: small models usually require less computation and have shorter inference paths.
  3. Improve deployment flexibility: they can be deployed on a single GPU, edge devices, or local servers.
  4. Reduce service pressure: support more requests under the same resource conditions.
  5. Enable domain specialization: let small models focus on specific tasks such as customer service, classification, extraction, and code assistance.

Note that a reduction in model parameters does not mean inference speed will necessarily improve proportionally in all scenarios. Actual performance is also affected by model architecture, memory bandwidth, operator implementation, batching strategy, and hardware utilization.

II. Basic Principles of Model Distillation

2.1 Teacher Model and Student Model

Model distillation usually involves two core roles.

Teacher Model

  • Usually larger and more capable.
  • Provides soft labels, generated samples, or intermediate-layer knowledge.
  • Its parameters are generally kept frozen during training.
  • It can be a general-purpose large model or a domain-expert model.

Student Model

  • Usually smaller.
  • Receives supervision signals provided by the teacher model.
  • Updates its parameters through backpropagation.
  • Ultimately performs inference tasks independently.

Overall workflow:

flowchart TD
    A["Training data"] --> B["Teacher Model"]
    B --> C["Generate knowledge supervision signals"]
    A --> D["Student Model"]
    C --> E["Compute distillation loss"]
    D --> E
    E --> F["Backpropagate and update student model"]
    F --> G["Evaluate student model"]
    G --> H{"Reached target?"}
    H -->|No| F
    H -->|Yes| I["Export student model"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

2.2 Hard Labels and Soft Labels

Understanding hard labels and soft labels is key to understanding knowledge distillation.

Hard Label

A hard label is the ground-truth class or target result directly provided in the dataset.

For example, given a piece of text:

This movie is excellent, and the actors give outstanding performances.

The hard label for a classification task might be:

Positive sentiment: 1
Negative sentiment: 0

It only tells the model what the correct answer is and does not include the degree of similarity between other classes.

Soft Label

A soft label is the probability distribution output by the teacher model.

For example:

Class Teacher model probability
Positive 0.82
Neutral 0.15
Negative 0.03

Soft labels contain more information.

For example, the teacher model not only determines that the text expresses positive sentiment, but also indicates that it has some similarity to neutral sentiment.

This relationship between classes is also known as Dark Knowledge.

When the student model learns soft labels, it can obtain richer supervisory information than when learning only hard labels.

2.3 Why soft labels are more valuable

Suppose an image classification task contains three classes: cat, dog, and wolf.

A hard label only tells the student model:

Correct class: cat

But the teacher model may output:

Cat: 0.85
Dog: 0.10
Wolf: 0.05

This probability distribution also expresses:

  • Cats and dogs have some visual similarity.
  • Cats and wolves have relatively low similarity.
  • The teacher model has high confidence in the correct class.

The student model learns not only the correct answer but also the teacher model's judgments about relationships among different classes.

For large language models, soft labels can correspond to the predicted probability distribution of each Token in the vocabulary, enabling the student model to learn the teacher model's language prediction behavior.

3. Temperature Coefficient and Probability Distribution

Knowledge distillation typically uses a temperature coefficient (Temperature) to adjust the probability distribution output by the teacher model.

3.1 Softmax Temperature Formula

Standard Softmax:

p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}

After introducing the temperature coefficient:

p_i^{(T)} = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}

where:

  • z_i: the Logit output by the model.
  • T: the temperature coefficient.
  • p_i^{(T)}: the temperature-adjusted probability.

The temperature coefficient determines the smoothness of the probability distribution.

3.2 Effects of Different Temperatures

Temperature T Probability Distribution Main Effect
T = 1 Original distribution Maintains normal model predictions
T > 1 Smoother Reveals more relationships between classes
T approaches 0 Sharper Closer to the maximum-probability class
Large T Over-smoothing risk May weaken meaningful class differences

For example, the teacher model's original output is:

Cat: 0.90
Dog: 0.08
Wolf: 0.02

After increasing the temperature, the probability distribution may become:

Cat: 0.65
Dog: 0.25
Wolf: 0.10

The student model can therefore learn more information about classes other than the maximum-probability class.

Note that a higher temperature does not mean better distillation. Too high a temperature may make the probability distribution overly flat and lose important judgment information from the teacher model.

3.3 Why the Temperature Coefficient Affects the Distillation Loss

Distillation typically uses the soft probability distributions of the teacher and student models to calculate KL divergence.

L_{KD} = T^2 D_{KL} \left( P_T^{(T)} \parallel P_S^{(T)} \right)

where:

  • P_T^{(T)}: the teacher model's temperature-scaled probability distribution.
  • P_S^{(T)}: the student model's temperature-scaled probability distribution.
  • D_{KL}: KL divergence.
  • T^2: temperature compensation coefficient.

T^2 is introduced to compensate for the effect of temperature changes on the gradient scale, so that the distillation loss at different temperatures has a more reasonable gradient magnitude.

In actual training, the temperature, loss weight, and learning rate usually need to be tuned jointly.

4. Main Methods of Model Distillation

Model distillation is not limited to a single implementation. Depending on the type of knowledge transferred, it can be divided into multiple methods.

4.1 Response-Based Distillation

This is the most classic distillation approach.

The student model directly learns the teacher model's outputs, including:

  • Classification probabilities.
  • Token probability distributions.
  • Logits.
  • Text generation outputs.
  • Task prediction results.

For classification models, KL divergence is usually calculated using the probability distributions of the teacher and student models.

For large language models, the student model can be made to learn the teacher model's Token distribution at each position.

Advantages:

  • Simple to implement.
  • Does not require the internal architectures of the teacher and student models to be identical.
  • Suitable for heterogeneous model distillation.

Disadvantages:

  • Mainly transfers output behavior.
  • Cannot directly leverage the teacher model's internal features.
  • If only final text answers are available, the supervision signal may not be sufficiently rich.

4.2 Feature-Based Distillation

Feature-based distillation has the student model learn the hidden representations of the teacher model's intermediate layers.

Each Transformer layer produces hidden states:

H_l \in \mathbb{R}^{B\times S\times d}

where:

  • B: Batch Size.
  • S: Sequence length.
  • d: Hidden dimension.
  • l: Transformer layer index.

An intermediate-layer feature loss can be used to make the student model's hidden representations close to those of the teacher model.

For example:

L_{feature} = \frac{1}{N} \sum_{l} \left\| W_l H_l^S-H_{\phi(l)}^T \right\|_2^2

where:

  • H_l^S: The hidden state of the student model at layer l.
  • H_{\phi(l)}^T: The hidden state of the corresponding layer in the teacher model.
  • W_l: The feature dimension alignment mapping.
  • \phi(l): The mapping relationship between teacher layers and student layers.

If the teacher and student have different hidden dimensions, a linear projection layer can be used for dimension alignment.

Advantages:

  • Can transfer intermediate-layer representation capability.
  • Can learn how the teacher model organizes its features.
  • Suitable for models with similar architectures.

Disadvantages:

  • Requires access to the teacher model's internal hidden states.
  • Requires handling different numbers of layers and hidden dimensions.
  • Higher training memory and computational overhead.

4.3 Relation-Based Distillation

Relation-based distillation does not directly require the student model to reproduce a specific layer's features from the teacher model; instead, it learns relationships between samples or between tokens.

For example:

  • Similarity between samples.
  • Attention relationships between tokens.
  • Distances between different feature vectors.
  • Associations between different classes.

Suppose the teacher and student models each produce feature vectors h_i; the similarity between samples can be computed as:

R_{ij} = \frac{h_i^\top h_j}

The student model is then made to learn the teacher model's relation matrix.

This approach is suitable for knowledge transfer between different model architectures because layer-by-layer correspondence is not necessarily required.

4.4 Generation-Based Distillation

In large language models, generation-based distillation is very important.

The teacher model generates high-quality training data according to task instructions, and the student model then learns from this data through supervised fine-tuning.

For example:

Input:
Explain what RAG is.

Teacher model output:
RAG is a retrieval-augmented generation technique that provides contextual information to a language model through external knowledge retrieval, thereby improving the accuracy and timeliness of its answers.

Combine the teacher model's input and output into training samples:

{
  "instruction": "解释什么是 RAG",
  "output": "RAG 是一种检索增强生成技术,它通过外部知识检索为语言模型提供上下文信息,从而提高回答的准确性和知识时效性。"
}

These samples are then used to train the student model.

This method is also often called synthetic data distillation.

It is particularly suitable for:

  • Instruction following.
  • Dialogue capability transfer.
  • Domain question answering.
  • Code generation.
  • Structured information extraction.
  • Reasoning capability training.

Note: Generation-based distillation typically learns the teacher model's output behavior and is not equivalent to fully transferring the teacher model's internal knowledge distribution.

4.5 Multi-Stage Distillation

In real projects, multiple distillation methods can be combined.

For example:

  1. Use the teacher model to generate high-quality instruction data.
  2. Use supervised fine-tuning to teach the student model the output behavior.
  3. Use Logits distillation to learn the teacher model's probability distribution.
  4. Use feature distillation to further align intermediate-layer representations.
  5. Use preference data to optimize answer quality.

Different methods can be combined, but the additional computational cost needs to be evaluated to determine whether it is worthwhile.

5. Core Techniques for Large Language Model Distillation

Distilling large language models differs markedly from distilling traditional classification models.

Traditional classification models usually have an output space of only dozens or hundreds of classes, whereas the output space of a large language model is a massive token vocabulary and requires autoregressive prediction over long sequences.

5.1 Logits Distillation

Logits are the raw scores output by a model before Softmax.

Suppose the vocabulary size is V and the sequence length is S; the teacher model outputs:

Z_T\in\mathbb{R}^{B\times S\times V}

Student model output:

Z_S\in\mathbb{R}^{B\times S\times V}

Apply Softmax to the logits of both models at the same temperature to obtain probability distributions, then compute the KL divergence.

L_{KD} = T^2 \sum_{b,s} D_{KL} \left( P_T^{(T)}(b,s) \parallel P_S^{(T)}(b,s) \right)

In practice, padding tokens and invalid positions usually need to be masked so that they do not participate in the loss computation.

Advantages of logit distillation:

  • The student can learn the teacher model's token prediction distribution.
  • It provides more supervisory signal than learning only from the final generated text.
  • It suits scenarios where the model architecture and tokenizer are compatible.

Main limitations:

  • The teacher and student models must have aligned vocabularies.
  • Storing full-vocabulary logits consumes a large amount of GPU memory.
  • The teacher model's forward computation is expensive.
  • With different tokenizers, full logits cannot be compared directly position by position.

For example, assuming a vocabulary size of 128000 and a sequence length of 4096, the full FP16 logits of a single sample require approximately:

128000\times4096\times2 \approx 0.98\text{ GiB}

This does not yet account for the batch dimension or other tensors during training.

Therefore, in practice one can:

  • Compute teacher logits online.
  • Compute the distillation loss in chunks.
  • Store only the probabilities and indices of the Top-K tokens.
  • Use shorter training sequences.
  • Control memory usage through caching and data pipelines.

Top-K logit distillation can reduce the data volume, but if the remaining probability mass is not handled, what the student model learns is no longer the full teacher distribution.

5.2 Sequence-Level Distillation

Sequence-Level Distillation does not directly learn the teacher model's full probability distribution at each position; instead, it learns the complete sequences generated by the teacher model.

For example:

Input question:
What is a Transformer?

Teacher model:
A Transformer is a neural network architecture based on the attention mechanism,
capable of modeling relationships between different positions in a sequence through self-attention.

Student model:
Train using the answer generated by the teacher as the target sequence.

The student model learns the teacher-generated token sequence through a cross-entropy loss:

L_{seq} = -\sum_{t=1}^{S} \log P_S(y_t\mid y_{<t},x)

where:

  • x: the input.
  • y_t: the target token generated by the teacher.
  • y_{<t}: the preceding target tokens.

Advantages of sequence-level distillation:

  • No need to store full logits.
  • Models of different sizes can be used.
  • It is easier to combine with instruction data to build a training set.
  • It suits large-scale data generation and supervised fine-tuning.

However, the student model can only learn the target sequences provided by the teacher and cannot directly obtain the teacher model's probability information over other candidate tokens.

5.3 Chain-of-Thought Distillation

Chain-of-Thought (CoT) distillation uses reasoning processes generated by the teacher model to help the student model learn multi-step problem-solving ability.

For example:

Question:
An item has an original price of 200 yuan. It is discounted by 20% and then reduced by another 20 yuan. What is the final price?

Teacher model reasoning:
Step 1: Compute the 20%-off price, 200 × 0.8 = 160.
Step 2: Subtract another 20 yuan, 160 - 20 = 140.
Final answer: 140 yuan.

What the student model learns is not just the final answer, but also how the intermediate steps are organized.

However, reasoning text does not necessarily equal the model's actual internal computation process. What CoT distillation transfers is the reasoning expression and problem-solving patterns provided by the teacher model.

When using it, pay special attention to the following:

  • The reasoning data should be verifiable.
  • Math and code tasks should be programmatically verified wherever possible.
  • Not all reasoning text generated by the teacher model should be treated as correct.
  • Avoid a situation where the student model only learns verbose expression without improving its actual problem-solving ability.

5.4 Instruction Distillation

Instruction distillation mainly lets the student model learn the teacher model's instruction-following ability.

Training samples typically contain:

  • System Prompt.
  • User Instruction.
  • Context.
  • Assistant Response.

For example:

{
  "instruction": "Summarize the following text into three key points",
  "input": "Here is a rather long technical article……",
  "output": "1. Core concept……\n2. Key techniques……\n3. Application scenarios……"
}

During training, the dialogue template must be unified and the loss mask must be set correctly.

Usually, the loss is computed only on the assistant's target response, not on the user input or on template parts that should not be learned.

6. Designing the Distillation Loss Function

In actual training, a single loss function is usually not the only one used.

6.1 Cross-Entropy Loss

The cross-entropy loss is used to make the student model learn the target labels or the target sequences generated by the teacher.

L_{CE} = -\sum_{i=1}^{V} y_i\log p_i^S

Where:

  • y_i: target label distribution.
  • p_i^S: student model predicted probability.

If the target is a hard label, y_i is usually a One-Hot distribution.

6.2 KL Divergence Loss

KL divergence measures the difference between two probability distributions.

D_{KL}(P\parallel Q) = \sum_i P_i\log\frac{P_i}{Q_i}

In distillation, the following is generally used:

L_{KD} = T^2D_{KL}(P_T^{(T)}\parallel P_S^{(T)})

Note that KL divergence is directional:

D_{KL}(P\parallel Q) \neq D_{KL}(Q\parallel P)

Usually, the teacher model's distribution is used as the target distribution, and the student model's distribution as the distribution to be optimized.

6.3 Feature Loss

Used to bring the intermediate-layer features of the student model close to those of the teacher model.

Common methods include:

  • Mean Squared Error.
  • Cosine Similarity Loss.
  • Attention Map Loss.
  • Hidden State Alignment.

For example, mean squared error:

L_{MSE} = \frac{1}{N} \sum_{i=1}^{N} (h_i^S-h_i^T)^2

6.4 Combined Distillation Loss

When combining multiple supervision methods, a combined loss can be constructed:

L = \alpha L_{CE} + \beta L_{KD} + \gamma L_{feature} + \delta L_{relation}

Where:

  • L_{CE}: target sequence supervision loss.
  • L_{KD}: Logits distillation loss.
  • L_{feature}: intermediate-layer feature loss.
  • L_{relation}: relation distillation loss.
  • \alpha,\beta,\gamma,\delta: weights of each loss term.

More loss terms do not necessarily make things better.

In practice, projects should start with a simple approach—for example, first use cross-entropy for instruction fine-tuning, then add Logits distillation, and finally decide whether to introduce feature distillation based on the results.

VII. Construction of the Distillation Dataset

In large language model distillation, data quality is often more important than simply increasing the amount of data.

7.1 Data Generation Process

flowchart TD
    A["Raw data and task requirements"] --> B["Task classification and data sampling"]
    B --> C["Construct teacher model Prompt"]
    C --> D["Teacher model generates responses"]
    D --> E["Quality check"]
    E --> F{"Sample qualified?"}
    F -->|No| G["Discard or regenerate"]
    G --> C
    F -->|Yes| H["Deduplication and format standardization"]
    H --> I["Dataset splitting"]
    I --> J["Distillation training"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

7.2 Data Type Design

It is recommended to build multiple types of training data rather than only having the teacher model generate ordinary question-answer pairs.

Data Type Training Objective Example
Foundational Knowledge Knowledge Representation Concept Explanation
Instruction Following Understanding Task Requirements Summarization, Rewriting
Multi-turn Dialogue Context Management Consecutive Follow-up Questions
Reasoning Data Multi-step Problem Solving Mathematics, Logic
Code Data Code Generation and Understanding Programming Tasks
Structured Data Format Adherence JSON Output
Domain Data Professional Knowledge Transfer Finance, Law, Healthcare
Refusal Data Safe Behavior Handling Inappropriate Requests

7.3 Data Quality Checks

Teacher models may also generate incorrect content, so not all outputs can be directly added to the training set.

It is recommended to establish the following quality checks:

  1. Format check: Whether formats such as JSON, Markdown, and code are correct.
  2. Fact check: Whether answers contain obvious errors.
  3. Logic check: Whether the reasoning process is self-consistent.
  4. Duplication check: Filter highly similar training samples.
  5. Length check: Filter samples that are too short, too long, or abnormally truncated.
  6. Safety check: Filter non-compliant or harmful training content.
  7. Task coverage check: Avoid training data being overly concentrated on a certain type of task.

For code, mathematics, and structured output tasks, prioritize executable validators or rule-based validation rather than relying entirely on another model for judgment.

7.4 Data Contamination Issues

If there is a large amount of duplicate samples between the training set and the test set, evaluation results may be distorted.

Therefore, it is necessary to:

  • Deduplicate data before training.
  • Perform similarity detection on the training set and validation set.
  • Keep an independent test set.
  • Avoid using test questions directly for teacher data generation.
  • Build additional test sets using real business scenarios.

8. Selection of Teacher and Student Models

The effectiveness of distillation depends not only on the teacher model's capabilities but also on whether the student model is suitable for the target task.

8.1 Teacher Model Selection

For the teacher model, key considerations include:

  • Whether task capabilities are sufficiently strong.
  • Whether output quality is stable.
  • Whether it supports obtaining Logits.
  • Whether intermediate-layer features can be accessed.
  • Whether the tokenizer is compatible with the student model.
  • Whether the license agreement permits the relevant training and deployment.

If only generative data distillation is performed, the teacher model does not necessarily need to expose its internal weights, but it must have stable text generation capabilities.

If Logits distillation or feature distillation is to be performed, it is usually necessary to be able to access the corresponding outputs of the model.

8.2 Student Model Selection

For the student model, the main considerations are:

  • Parameter scale.
  • Transformer architecture.
  • Hidden layer dimension.
  • Number of Attention Heads.
  • Context length.
  • Tokenizer.
  • Support in inference frameworks.
  • Operator compatibility with target hardware.

The student model does not have to be completely identical in structure to the teacher model.

However, the greater the structural difference, the more difficult it is usually to directly align intermediate-layer features.

8.3 Comparison of Teacher and Student Models

Combination Type Characteristics Applicable Scenarios
Same architecture, different scale Features are easy to align Classical distillation
Different architectures High flexibility Heterogeneous model transfer
Same Tokenizer Logits alignment is convenient Probability distillation
Different Tokenizers Requires additional alignment Text generation distillation
Multiple teacher models Can integrate multiple capabilities Multi-task distillation

9. Complete Training Workflow for Model Distillation

A practical large language model distillation project usually includes the following stages:

  1. Define deployment goals and capability metrics.
  2. Select the teacher model and student model.
  3. Build domain data and instruction data.
  4. Use the teacher model to generate supervised samples.
  5. Clean, deduplicate, and validate the data.
  6. Perform supervised fine-tuning on the student model.
  7. Add Logits or feature distillation as appropriate.
  8. Evaluate on an independent test set.
  9. Quantize and optimize the student model for deployment.
  10. Conduct stress testing with real business traffic.

Training workflow:

flowchart TD
    A["Define task and deployment goals"] --> B["Select Teacher and Student"]
    B --> C["Build distillation dataset"]
    C --> D["Teacher model generates supervision data"]
    D --> E["Data cleaning and quality validation"]
    E --> F["Student model initialization"]
    F --> G["Supervised fine-tuning SFT"]
    G --> H["Logits or feature distillation"]
    H --> I["Validation set evaluation"]
    I --> J{"Does quality meet the standard?"}
    J -->|No| K["Analyze errors and adjust data and parameters"]
    K --> G
    J -->|Yes| L["Model export"]
    L --> M["Quantization and inference optimization"]
    M --> N["Deployment and online evaluation"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style M fill:#EAF3EC,stroke:#83A995,color:#315D49
    style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F

10. Using PyTorch to Implement Basic Knowledge Distillation

The following uses a classification task to illustrate the core implementation of distillation.

10.1 Teacher Model and Student Model

import torch
import torch.nn as nn
import torch.nn.functional as F


class TeacherModel(nn.Module):
    def __init__(self, input_dim=128, hidden_dim=512, num_classes=10):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, num_classes)
        )

    def forward(self, x):
        return self.network(x)


class StudentModel(nn.Module):
    def __init__(self, input_dim=128, hidden_dim=64, num_classes=10):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, num_classes)
        )

    def forward(self, x):
        return self.network(x)

The teacher model has larger hidden layers and more parameters, while the student model has a smaller architecture.

10.2 Distillation Loss Function

def distillation_loss(
    student_logits,
    teacher_logits,
    labels,
    temperature=4.0,
    alpha=0.7
):
    # Hard-label loss of the student model
    hard_loss = F.cross_entropy(
        student_logits,
        labels
    )

    # Soft-label distribution from the teacher model
    teacher_probs = F.softmax(
        teacher_logits / temperature,
        dim=-1
    )

    # Soft prediction distribution of the student model
    student_log_probs = F.log_softmax(
        student_logits / temperature,
        dim=-1
    )

    # KL distillation loss
    soft_loss = F.kl_div(
        student_log_probs,
        teacher_probs,
        reduction="batchmean"
    ) * (temperature ** 2)

    return alpha * soft_loss + (1 - alpha) * hard_loss

Here:

  • temperature=4.0: Increases the smoothness of the probability distribution.
  • alpha=0.7: Weight of the distillation loss.
  • 1-alpha: Weight of the ground-truth label loss.

10.3 Train the student model

teacher = TeacherModel()
student = StudentModel()

# Assume the teacher model has already been trained
teacher.eval()

for param in teacher.parameters():
    param.requires_grad = False

optimizer = torch.optim.AdamW(
    student.parameters(),
    lr=1e-3
)

for epoch in range(10):
    student.train()

    for inputs, labels in train_loader:
        with torch.no_grad():
            teacher_logits = teacher(inputs)

        student_logits = student(inputs)

        loss = distillation_loss(
            student_logits,
            teacher_logits,
            labels,
            temperature=4.0,
            alpha=0.7
        )

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    print(f"Epoch {epoch + 1}, Loss: {loss.item():.4f}")

This example shows the classic distillation process:

  • The teacher model's parameters are not updated.
  • The student model performs the forward pass as usual.
  • The teacher outputs soft labels.
  • The student learns both soft labels and ground-truth labels.
  • The student model is updated via backpropagation.

In real projects, you also need to handle data devices, mixed precision, gradient accumulation, validation sets, model saving, and other issues.

XI. Engineering Practices for Large Language Model Distillation

11.1 SFT Based on Teacher-Generated Data

This is a distillation approach that is relatively easy to implement in real projects.

Assume the teacher-generated data is already prepared:

{
  "messages": [
    {
      "role": "system",
      "content": "你是一个专业的技术助手。"
    },
    {
      "role": "user",
      "content": "解释什么是 KV Cache。"
    },
    {
      "role": "assistant",
      "content": "KV Cache 是一种用于缓存 Transformer 注意力机制中 Key 和 Value 张量的技术,可以避免自回归生成时重复计算历史 Token 的键值表示。"
    }
  ]
}

Use these samples for supervised fine-tuning of the student model.

During training, you need to handle the following correctly:

  • Chat Template.
  • Tokenizer.
  • EOS Token.
  • Padding.
  • Attention Mask.
  • Assistant Loss Mask.
  • Maximum sequence length.

If Assistant Loss Mask is not set correctly in the training data, the model may also treat user input, the System Prompt, or padding tokens as prediction targets, affecting training effectiveness.

11.2 Training with Hugging Face Transformers

Below is a simplified SFT training example; replace the placeholders with the actual paths for the model and dataset.