Back to series
· 18 min read

Model Quantization: INT8, INT4, and Low-Precision Computing


1. What Is Model Quantization?

Model quantization is the technique of mapping the high-precision numerical values in a model to low-precision values.

In deep learning models, weights, activations, and intermediate computation results are typically represented using FP32, FP16, or BF16.

For example, a single FP32 parameter occupies 4 bytes:

FP32: 0.123456789
FP16: 0.1235
INT8:  31
INT4:   7

The core goals of quantization are:

  • Reduce the storage space for model weights.
  • Reduce GPU memory usage.
  • Lower the bandwidth required to read weights.
  • Use low-precision hardware instructions to increase computational throughput.
  • Reduce model deployment costs.
  • Improve inference efficiency within an acceptable range of accuracy loss.

To be clear: quantization is not just about shrinking data types; it is a systems engineering effort involving numerical mapping, error control, compute kernels, and hardware execution.

1.1 Why Do Large Models Need Quantization?

Large models have a very large number of parameters, and model weights are typically a major source of memory overhead in inference deployment.

Suppose a model has 70B parameters:

Data Type Theoretical Storage per Parameter Theoretical Weight Footprint
FP32 4 Bytes 280 GB
FP16 2 Bytes 140 GB
BF16 2 Bytes 140 GB
FP8 1 Byte 70 GB
INT8 1 Byte 70 GB
INT4 0.5 Byte 35 GB

The figures above are theoretical weight sizes in decimal units, excluding overheads such as quantization Scale, Zero Point, alignment padding, runtime workspace, and KV Cache.

As can be seen from the table:

  • Compared with FP32, FP16 reduces weight storage by about 50%.
  • Compared with FP16, INT8 reduces theoretical weight storage by about 50%.
  • Compared with FP16, INT4 reduces theoretical weight storage by about 75%.

This means a model that originally required multiple GPUs to deploy may, after quantization, fit into fewer GPUs — or even run on a single card.

1.2 The Three Main Targets of Quantization

Model quantization typically involves three types of data.

Target Meaning Purpose of Quantization
Weight Quantization Weight quantization Reduce model parameter storage
Activation Quantization Activation quantization Reduce the storage and compute overhead of intermediate tensors
KV Cache Quantization KV Cache quantization Reduce cache memory for long-context inference

The three differ in quantization difficulty.

Weights are usually fixed before model deployment, so their distribution can be measured in advance and quantization completed ahead of time.

Activations depend on the input data, and their distribution may vary from request to request, requiring calibration data, dynamic statistics, or special quantization methods.

KV Cache, meanwhile, keeps growing throughout the generation process, and its quantization must also account for Attention's sensitivity to the precision of historical Keys and Values.


2. Numerical Precision Fundamentals: FP32, FP16, BF16, FP8, and INT

To understand quantization, you first need to understand how different numerical formats are represented.

2.1 The Basic Structure of Floating-Point Numbers

An IEEE 754 floating-point number usually consists of three parts:

  • Sign: the sign bit.
  • Exponent: the exponent bits.
  • Mantissa: the mantissa bits, also called the Fraction.

A floating-point number can be abstractly expressed as:

x=(-1)^s \times 2^{e-bias} \times (1+m)

where:

  • s: the sign bit.
  • e: the exponent encoding.
  • bias: the exponent offset.
  • m: the mantissa part.

The exponent bits determine the numerical range, and the mantissa bits determine the numerical precision.

The more exponent bits there are, the larger the dynamic range that can typically be represented.

The more mantissa bits there are, the finer the numerical variation that can typically be represented.

2.2 Common Floating-Point Formats

Format Total bits Exponent bits Mantissa bits Characteristics
FP32 32 8 23 High precision, relatively large range
FP16 16 5 10 Small storage footprint, but a relatively narrow dynamic range
BF16 16 8 7 Dynamic range close to FP32
FP8 E4M3 8 4 3 Relatively high precision, smaller range
FP8 E5M2 8 5 2 Larger range, lower precision

FP8 has two common formats:

  • E4M3: 4 exponent bits and 3 mantissa bits; generally better suited to computations with higher precision requirements.
  • E5M2: 5 exponent bits and 2 mantissa bits; generally has a larger dynamic range.

Actual FP8 formats also have special encodings, exponent biases, and finite-value conventions; the full numerical range cannot be inferred from the exponent and mantissa bits alone.

2.3 Integer Formats

INT8 and INT4 are fixed-point integer representations.

Format Bits Signed integer range
INT8 8 -128 to 127
INT4 4 -8 to 7
INT2 2 -2 to 1

Integers themselves have no floating-point exponent or mantissa.

They can only represent a finite number of discrete values, so floating-point numbers need to be mapped into the integer space using Scale and Zero Point.

2.4 Differences Between FP16 and INT8

FP16 and INT8 are both low-precision representations of 16 bits or less, but their numerical structures are completely different.

FP16:

  • Uses floating-point representation.
  • Has exponent and mantissa bits.
  • Can represent values of different orders of magnitude.
  • Can usually participate directly in floating-point matrix operations.

INT8:

  • Uses integer representation.
  • Requires Scale and Zero Point to establish numerical mapping.
  • Has a limited numerical range.
  • Usually requires integer multiply-add or quantized compute kernels.

Therefore, INT8 is not simply FP16 truncated to 8 bits.


3. Mathematical Principles of Linear Quantization

Linear quantization is the most basic method for integer weight quantization such as INT8 and INT4.

It maps floating-point numbers to a finite range of integers through a linear mapping.

3.1 Quantization: Quantization Formula

The most common affine quantization (Affine Quantization) can be expressed as:

q=\operatorname{clip} \left( \operatorname{round}\left(\frac{x}{s}\right)+z, q_{\min},q_{\max} \right)

where:

  • x: original floating-point number.
  • q: quantized integer.
  • s: Scale, the scaling factor.
  • z: Zero Point, the zero-point offset.
  • q_{\min}: minimum integer value.
  • q_{\max}: maximum integer value.
  • round: rounding operation.
  • clip: clips to the valid integer range.

Dequantization formula:

\hat{x}=s(q-z)

where \hat{x} is the approximate floating-point value recovered after quantization.

Note: Dequantization cannot recover the original values; it can only recover an approximation of the quantized values.

3.2 Calculating the Scale

For asymmetric quantization, if the floating-point data range is:

x\in[x_{\min},x_{\max}]

and the integer range is:

q\in[q_{\min},q_{\max}]

the Scale is usually calculated as:

s=\frac{x_{\max}-x_{\min}} {q_{\max}-q_{\min}}

Zero Point:

z=\operatorname{round} \left( q_{\min}-\frac{x_{\min}}{s} \right)

Practical implementations usually also clamp the Zero Point to the integer range.

3.3 Symmetric Quantization

Symmetric Quantization usually sets the Zero Point to 0.

For signed integers:

s=\frac{\max(|x_{\min}|,|x_{\max}|)} {q_{\max}}

Quantization formula:

q=\operatorname{clip} \left( \operatorname{round}\left(\frac{x}{s}\right), q_{\min},q_{\max} \right)

Dequantization:

\hat{x}=sq

Advantages of symmetric quantization:

  • No additional Zero Point is needed.
  • Integer matrix multiplication is easier to optimize.
  • The computation process is relatively simple.
  • It suits many weight tensors.

However, if the data distribution clearly deviates from zero, symmetric quantization may not fully utilize the integer representation range.

3.4 Quantization Calculation Example

Suppose the weight range is:

x\in[-1,1]

Using symmetric INT8 quantization, take:

q_{\max}=127

Then:

s=\frac{1}{127} \approx 0.007874

Original weight:

x=0.5

Quantization result:

q=\operatorname{round}(0.5\times127) =64

Dequantization:

\hat{x}=\frac{64}{127} \approx0.50394

Quantization error:

e=x-\hat{x} \approx-0.00394

This is the numerical error introduced by quantization.

3.5 Where Does Quantization Error Come From?

The main sources include:

  1. Rounding error: error produced when floating-point numbers are mapped to discrete integers.
  2. Clipping error: values outside the quantization range are clipped.
  3. Parameter estimation error: the Scale and Zero Point are not accurate enough.
  4. Distribution shift: the calibration data and the real inference data are inconsistent.
  5. Accumulated error: errors from multi-layer quantization propagate layer by layer.

Among these, clipping error is usually more severe than ordinary rounding error.

For example, if a weight is 12 but the quantization range can only represent up to 1, then that value will be clipped, and after dequantization it can still only yield a value close to 1.


4. Quantization Granularity: Per-Tensor, Per-Channel, and Group-wise

Quantization requires not only choosing INT8 or INT4 but also deciding the scope over which the Scale applies.

Quantization Granularity directly affects quantization error, metadata overhead, and computational efficiency.

4.1 Per-Tensor Quantization

Per-Tensor means the entire tensor shares one Scale and Zero Point.

For example, for a weight matrix:

W\in\mathbb{R}^{4096\times4096}

all elements share the same set of quantization parameters.

Advantages:

  • Lowest parameter overhead.
  • Simple implementation.
  • Easy to use high-performance matrix computation kernels.

Disadvantages:

  • The value distributions of different channels may differ greatly.
  • Extreme values may compress the effective quantization range of other elements.
  • Precision loss may be significant under INT4.

4.2 Per-Channel Quantization

Per-Channel computes the Scale independently for each output channel.

For a linear layer:

Y=XW^T

Suppose:

W\in\mathbb{R}^{N\times K}

where:

  • N: output dimension.
  • K: input dimension.

An independent Scale can be assigned to each output channel:

s_1,s_2,\ldots,s_N

This better adapts to the weight distributions of different output channels.

Advantages:

  • Usually more accurate than Per-Tensor.
  • Reduces the error caused by differences in value distributions across channels.
  • Suitable for many INT8 weight quantization scenarios.

Disadvantages:

  • More quantization parameters must be stored.
  • The computation kernel must support channel-level Scales.
  • Different matrix layouts affect how the Scale is applied.

4.3 Group-wise Quantization

Group-wise Quantization divides a channel or matrix into multiple small groups, and each group computes its Scale independently.

For example:

A row of the weight matrix:

Group 1: 128 weights, sharing Scale 1
Group 2: 128 weights, sharing Scale 2
Group 3: 128 weights, sharing Scale 3
Group 4: 128 weights, sharing Scale 4

Common Group Sizes:

  • 32.
  • 64.
  • 128.
  • 256.

The specific size depends on the quantization algorithm, model structure, and compute kernel.

The smaller the Group Size:

  • The Scale is closer to the local data distribution.
  • Quantization error may be lower.
  • The amount of metadata is larger.
  • Kernel reads and computation management are more complex.

The larger the Group Size:

  • Scale overhead is lower.
  • Computation implementation may be simpler.
  • Distribution differences across local regions may increase quantization error.

4.4 Comparison of the Three Granularities

Method Scale Scope Metadata Overhead Accuracy Performance
Per-Tensor Entire tensor Lowest Depends on overall distribution
Per-Channel Each channel Relatively low Usually better
Group-wise Each group Medium Flexible; can further reduce error

4.5 Schematic Diagram of Quantization Granularity

flowchart TD
    A["Original weight matrix"] --> B{"Choose quantization granularity"}
    B --> C["Per-Tensor"]
    B --> D["Per-Channel"]
    B --> E["Group-wise"]
    C --> F["Entire matrix shares Scale"]
    D --> G["Independent Scale per output channel"]
    E --> H["Independent Scale per group of weights"]
    F --> I["Low metadata overhead"]
    G --> J["Adapts to channel distribution differences"]
    H --> K["Controls local quantization error"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F

V. INT8 Quantization: 8-Bit Integer Computation

INT8 is a very common low-precision integer computation format today.

It uses 8-bit signed integers to represent values, and in theory each parameter occupies 1 Byte.

5.1 Basic Characteristics of INT8

  • The integer range is usually -128 to 127.
  • Weight storage takes about half as much as FP16.
  • Integer matrix multiplication instructions can be used.
  • A Scale is needed to restore the numerical scale.
  • It can usually achieve model compression with relatively low precision loss.

INT8 is commonly used in model inference in two ways:

  1. Weights use INT8, and activations use FP16 or BF16.
  2. Both weights and activations use INT8.

The first is usually called W8A16.

The second is usually called W8A8.

5.2 W8A16: INT8 Weights, FP16 Activations

W8A16 means:

  • Weight: INT8.
  • Activation: FP16.

Model weights are stored in INT8. During matrix computation, depending on the compute kernel, the weights are converted or fused into a form suitable for computation.

Illustration:

Y=X_{FP16}W_{INT8}^{T}

Its advantages are:

  • Weight VRAM usage is significantly reduced.
  • Activations still retain higher precision.
  • Lower quantization requirements for activation distributions.
  • It is easier to achieve good deployment results on some GPUs.

But note: W8A16 does not guarantee that all hardware can directly execute native mixed matrix multiplication of INT8 weights and FP16 activations.

In practice, it may be implemented through dequantization, fused kernels, or specific mixed-precision kernels.

5.3 W8A8: INT8 Weights, INT8 Activations

W8A8 means:

  • Weight: INT8.
  • Activation: INT8.

Matrix multiplication:

Y=A_{INT8}W_{INT8}^{T}

Practical integer matrix multiplication usually uses a wider accumulation type, such as INT32.

The computation result needs to be combined with the input and weight scales to restore the numerical scale.

For symmetric quantization:

A\approx s_a q_a
W\approx s_w q_w

Therefore:

Y\approx s_as_w(q_aq_w^T)

Here, the integer multiply-accumulate result is usually first accumulated into INT32 and then converted to the target output format through scaling.

Advantages of W8A8:

  • Both weights and activations can be compressed.
  • Reduces data transfer and storage overhead.
  • May achieve higher throughput on hardware that supports INT8 Tensor Cores.

Challenges:

  • Activation distributions change dynamically.
  • Outliers may affect the Scale.
  • Activation quantization error may affect model output.
  • More complex quantization and calibration strategies are required.

5.4 Accumulation Precision of INT8

An easily overlooked issue is:

INT8 inputs do not mean the entire computation process uses INT8.

For example:

C=\sum_{k=1}^{K}a_kw_k

If a_k and w_k are both INT8, then an individual product can be handled in the integer compute path, but the accumulation process usually requires a wider integer type.

If INT8 is used for accumulation throughout, values can easily overflow.

Therefore, a common compute path is:

INT8 × INT8
      ↓
INT32 accumulation
      ↓
Scale restoration
      ↓
FP16 / BF16 / FP32 output

This design reduces the storage cost of input data while avoiding severe overflow problems caused by low-bit-width accumulation.


6. INT4 Quantization: 4-bit Weight Compression

INT4 uses 4 bits to represent an integer.

Its core value is further reducing model weight storage and read overhead.

6.1 Numeric Range of INT4

Signed INT4 typically represents:

q\in[-8,7]

Each parameter theoretically occupies only half a byte.

Since a byte has 8 bits, two INT4 values can be packed into one byte.

For example:

High 4 bits: Weight A
Low 4 bits: Weight B

This method is called Packing.

Actual quantization formats may use different encoding methods, such as symmetric signed encoding, asymmetric encoding, or custom low-bit-width formats.

6.2 Main Advantages of INT4

  • Weight storage is only one quarter that of FP16.
  • Theoretically significantly reduces GPU memory requirements.
  • Reduces weight read bandwidth.
  • Facilitates single-GPU deployment of large models.
  • May provide significant benefits in scenarios where weight reading is the bottleneck.

6.3 Main Challenges of INT4

INT4 has only 16 discrete values.

This means the representable values are very limited.

For example:

x\in[-1,1]

If the entire tensor is mapped to the limited integer range of INT4, relatively large quantization steps are needed to represent different floating-point values.

Compared with INT8, INT4 usually has larger rounding error.

In addition, if a tensor contains a small number of extreme values, the quantization range may be dominated by these extreme values, causing most ordinary weights to use fewer effective quantization levels.

6.4 Group-wise INT4

To reduce INT4 quantization error, practical models usually adopt Group-wise Quantization.

For example:

  • Every 128 weights form a group.
  • The Scale is computed separately for each group.
  • Different groups have independent quantization ranges.

This avoids a small number of outliers in the entire matrix affecting all weights.

A typical INT4 weight representation may include:

Quantized weights: INT4 Packed Values
Scaling parameters: FP16 / BF16 Scale
Zero-point parameters: Optional
Group size: Group Size

Therefore, the actual memory footprint of an INT4 model is not equal to the number of parameters multiplied by 0.5 byte.

Scale, Zero Point, alignment, and other metadata overhead also need to be accounted for.

6.5 INT4 Dequantization Process

Suppose a group of weights uses symmetric INT4 quantization:

q=\operatorname{clip} \left( \operatorname{round}\left(\frac{x}{s}\right), -8,7 \right)

Dequantization:

\hat{x}=sq

In actual GPU inference, it is not necessarily required to first fully dequantize the entire weight matrix into FP16.

A more efficient approach is:

  1. Read Packed INT4 from GPU memory.
  2. Unpack it inside the kernel.
  3. Fetch the corresponding Group Scale.
  4. Convert in registers or along the compute path.
  5. Perform matrix multiplication.
  6. Output the accumulation result.

This approach avoids creating an additional full FP16 weight copy.


7. Comparison of INT8 and INT4

Both INT8 and INT4 are forms of integer weight quantization, but they differ significantly in precision, storage, and computation methods.

Comparison dimension INT8 INT4
Bit width 8 Bit 4 Bit
Theoretical storage 1 Byte/parameter 0.5 Byte/parameter
Integer representation levels 256 encoded values 16 encoded values
Quantization error Usually lower Usually higher
Scale overhead Relatively low Requires attention for group-wise
Weight read volume Lower Even lower
Hardware support Relatively broad Depends on hardware and Kernel
Deployment complexity Relatively low Usually higher
Applicable scenarios Balance between accuracy and performance GPU memory-constrained, large-model compression

7.1 INT4 Is Not Necessarily Faster Than INT8

This is a very important engineering misconception in model quantization.

INT4 weights have a smaller data volume, but that does not mean inference will necessarily be faster.

The actual speed depends on:

  • Whether the GPU supports efficient INT4 computation.
  • INT4 unpacking overhead.
  • Scale read overhead.
  • Whether dequantization is fused into matrix multiplication.
  • Whether the Kernel fully utilizes Tensor Cores.
  • Whether it is limited by memory bandwidth.
  • Batch Size and matrix shape.

If the unpacking and dequantization overhead of the INT4 Kernel is large, it may not fully exploit the bandwidth advantage of lower bit width.

7.2 Choosing Between INT4 and INT8

Selection can usually be based on the following goals:

Goal Recommended approach
Minimize GPU memory usage INT4
Control quantization error as much as possible INT8
Deploy a larger model on a single GPU INT4
Higher requirements for output quality INT8 or higher precision
Need a mature integer computation path INT8
GPU has efficient low-bit-width computation support Evaluate INT4

Ultimately, quality evaluation and performance testing on real business datasets should be the deciding factors.


8. FP8: Low-Precision Floating-Point Computing

FP8 and INT8 both use 8-bit representation, but their computational logic is different.

FP8 retains a floating-point exponent and mantissa, and can represent values across different orders of magnitude.

INT8, by contrast, needs to be mapped to a finite integer range through Scale and Zero Point.

8.1 Characteristics of FP8

FP8 is mainly used to reduce the data precision and storage overhead of floating-point computation.

Common formats:

Format Structure Key characteristics
E4M3 4-bit exponent, 3-bit mantissa Relatively high precision
E5M2 5-bit exponent, 2-bit mantissa Larger dynamic range

Advantages of FP8:

  • A single element occupies 1 Byte.
  • It can preserve the exponent structure of floating-point numbers.
  • It is suitable for certain matrix multiplications and deep learning computations.
  • Higher throughput can be achieved on GPUs that support FP8 Tensor Cores.

8.2 Differences Between FP8 and INT8

Dimension FP8 INT8
Numeric type Floating point Integer
Exponent structure Yes No
Scale Common implementations require scale management Usually required
Zero Point Typically does not use affine integer zero points Can be used
Numeric range Depends on the floating-point format Depends on integer encoding and Scale
Main application Low-precision floating-point computation Integer computation for weights and activations
Hardware execution FP8 Tensor Cores, etc. INT8 Tensor Cores, etc.

8.3 Scale Management for FP8

Although FP8 has a more flexible dynamic range than INT8, it is still limited.

If input values exceed the representable range, overflow or saturation will occur.

If values are too small, underflow and precision loss may also occur.

Therefore, practical computation usually requires a scaling factor:

x_{fp8}=\operatorname{cast}(x\cdot s)

After the computation:

y\approx \frac{y_{fp8}}{s}

For matrix multiplication, the inputs and outputs may have different scaling parameters.

FP8 training and inference systems may also use dynamic Scale, delayed Scale, or Scale management strategies based on historical statistics.


9. Weight Quantization, Activation Quantization, and KV Cache Quantization

Although all three types of quantization are low-precision optimizations, they act in different locations and present different engineering challenges.

9.1 Weight Quantization

Weight Quantization is the process of converting model parameters into a lower-precision format.

For example:

Original model:
W1: FP16
W2: FP16
W3: FP16

After quantization:
W1: INT4
W2: INT4
W3: INT4

Weight quantization is usually completed before model deployment.

Main benefits:

  • Reduce model file size.
  • Reduce GPU memory used for weights.
  • Reduce weight read bandwidth.

Common approaches:

  • INT8 Weight Quantization.
  • INT4 Weight Quantization.
  • GPTQ.
  • AWQ.
  • GGUF quantization format.
  • BitsAndBytes low-bit weight quantization.

9.2 Activation Quantization

Activation Quantization is the process of converting intermediate computation results of the model into a lower-precision format.

For example:

Embedding
    ↓
Attention
    ↓
Activation Quantization
    ↓
FFN
    ↓
Activation Quantization
    ↓
Next layer

Activation values depend on the input data, model layer, and runtime state.

Therefore, activation quantization is usually more complex than static weight quantization.

Main challenges:

  • Input distributions vary with requests.
  • Numerical ranges differ significantly across layers.
  • Outliers can degrade quantization accuracy.
  • Calibration or dynamic scales are required.
  • Quantization and dequantization operations may introduce additional overhead.

9.3 KV Cache Quantization

The KV Cache stores the Key and Value of historical tokens.

In long-context, high-concurrency inference, the KV Cache may consume a large amount of GPU memory.

Therefore, the KV Cache can be converted from FP16 or BF16 to FP8, INT8, or other low-precision formats.

Its main benefits are:

  • Reduce cache memory usage.
  • Increase the number of concurrent requests supported by the same GPU.
  • Reduce read bandwidth for historical Keys and Values.

However, KV Cache quantization requires particular attention to Attention errors.

Historical Keys and Values participate in the Attention computation for every subsequent token, so quantization errors may continuously affect the generated results.

9.4 Comparison of the Three Types of Quantization

Quantization Target Quantization Timing Main Benefit Main Risk
Weight Usually before deployment Reduce weight memory Model capability degradation
Activation During execution or configured before deployment Reduce intermediate tensor overhead Distribution shifts and outliers
KV Cache During inference Reduce context cache usage Accumulation of Attention errors

10. Mainstream Quantization Algorithms: GPTQ, AWQ, SmoothQuant, and bitsandbytes

The differences among quantization algorithms mainly lie in how they determine quantization parameters, how they handle outliers, and how they reduce model output error.

10.1 GPTQ

GPTQ is a post-training quantization method for Transformer weights.

The core idea of GPTQ is that when quantizing weights, it considers not only the difference between the current weights and the quantized result, but also the impact of weight error on the model output.

It uses approximate second-order information to guide the quantization process.

Core Principle

For a linear layer:

Y=XW^T

The quantized weights are:

\hat{W}=W+\Delta W

Output error:

\Delta Y=X\Delta W^T

If some directions of the input activations X are more important, then a weight error of the same magnitude may cause output changes of different degrees.

GPTQ uses calibration data to compute input-related statistics, approximately measuring the impact of weight error on the output.

A common objective can be written as:

\min_{\hat{W}} \|XW^T-X\hat{W}^T\|_F^2

where \|\cdot\|_F denotes the Frobenius norm.

GPTQ reduces this objective through approximate second-order optimization and a step-by-step quantization strategy.

Main Characteristics of GPTQ

  • It belongs to Post-Training Quantization, PTQ.
  • It usually does not require fully retraining the model.
  • It relies on representative calibration data.
  • It is commonly used for INT4 weight quantization.
  • It can be combined with Group-wise Quantization.

Limitations of GPTQ

  • The quantization process requires computing statistical matrices.
  • The quantization stage may consume considerable CPU/GPU memory.
  • The choice of calibration data affects the results.
  • Inference speed depends on whether the corresponding Kernel is efficient.

10.2 AWQ

AWQ stands for Activation-aware Weight Quantization.

The core idea of AWQ is that the importance of weights is related to input activation values.

Different weights do not have the same impact on model output.

If the activation values of some input channels are large, the quantization error on the corresponding weights may be more significant.

AWQ uses activation statistics to identify important channels and protects the quantization accuracy of key weights through scaling transformations.

Core Idea

For a linear layer:

Y=XW^T

Scale the input channels:

X'=XS^{-1}

Adjust the weights accordingly:

W'=WS

Therefore:

X'W'^T=XW^T

Under ideal mathematical conditions, the model computation results remain unchanged.

Then quantization is applied to the adjusted weights.

This transformation can change the numerical distribution of the weights, allowing the quantization process to better protect important channels.

Main Features of AWQ

  • Uses activation statistics to guide weight quantization.
  • Commonly used for low-bit weight quantization.
  • Does not require full model fine-tuning.
  • Commonly used for INT4 deployment.
  • Suitable for combination with high-performance low-bit matrix computation kernels.

Limitations of AWQ

  • Requires calibration data.
  • Requires handling weight scaling and model architecture adaptation.
  • Support varies across operators and model architectures.
  • Final inference performance depends on the kernel implementation.

10.3 SmoothQuant

SmoothQuant mainly addresses the problem of outliers in activations.

In Transformers, activations may have a small number of channels with particularly large magnitudes.

If INT8 quantization is applied directly to activations, these outliers may reduce the effective quantization precision of most other values.

The core idea of SmoothQuant is to shift part of the quantization difficulty from activations to weights.

It adjusts the numerical distributions of inputs and weights through an equivalent scaling transformation.

For:

Y=XW^T

we can construct:

X'=XS^{-1}
W'=WS

Thus:

X'W'^T=XW^T

Without changing the mathematical computation results, activations become easier to quantize.

Main Features of SmoothQuant

  • Focuses on activation outliers.
  • Commonly used for W8A8.
  • Can improve INT8 activation quantization for Transformers.
  • Reduces the dynamic quantization burden at inference through offline scaling.

10.4 bitsandbytes

bitsandbytes is a commonly used low-bit quantization library in the deep learning ecosystem.

Common capabilities include:

  • 8-bit optimizer-related features.
  • 8-bit model weight loading.
  • 4-bit weight quantization.
  • NF4.
  • FP4.
  • Integration with tools such as Transformers.

NF4 (NormalFloat4) is a 4-bit data representation designed for approximately normally distributed weights.

It is not ordinary uniform INT4.

NF4 designs quantization levels according to the target distribution, making the limited 4-bit codes better suited to represent weights of a specific distribution.

10.5 Comparison of Mainstream Algorithms

Algorithm Core Idea Common Target Requires Calibration Data
GPTQ Uses approximate second-order information to reduce weight error INT4 / INT8 weights Usually required
AWQ Protects weight channels based on activation importance INT4 weights Usually required
SmoothQuant Shifts the impact of activation outliers to weights W8A8 Usually required
bitsandbytes Provides low-bit data formats and computation support INT8 / 4-bit Depends on the specific scheme

11. Quantization Calibration: Calibration

Quantization is not simply choosing INT8 or INT4 and then directly converting the model.

It is necessary to determine parameters such as data distribution, Scale, Zero Point, and outlier handling methods.

This process is usually called Calibration.

11.1 Why Is Calibration Needed?

Suppose the actual distribution of activations in a certain layer is:

X\in[-0.5,0.5]

If calibration incorrectly assumes:

X\in[-10,10]

then the quantization range will be severely wasted.

A large number of normal values can only use a small number of integer codes, reducing effective precision.

Conversely, if the calibration range is too narrow, large values in real inference data may be clipped.

11.2 Calibration Dataset

The calibration dataset should be as close as possible to real business inputs.

For example:

  • General conversation.
  • Code generation.
  • Mathematical reasoning.
  • Long context.
  • Multilingual input.
  • Structured data.
  • Tool-call requests.

If the model is mainly used for code generation but is calibrated only with ordinary chat data, it may not sufficiently cover the real activation distribution.

11.3 Calibration Methods

Common statistical methods include:

Min-Max

Determine the quantization range based on the minimum and maximum values.

The advantage is simplicity.

The disadvantage is susceptibility to outliers.

Percentile

Use percentiles to determine the quantization range.

For example, ignore the extreme 0.1% or 1% of data.

The advantage is that it can reduce the impact of outliers.

The disadvantage is that extreme values may be truncated.

Histogram

Compute an activation histogram and select the quantization range based on the distribution.

This can more finely balance quantization error and clipping error.

MSE Optimization

Search for Scale to minimize the mean squared error between data before and after quantization.

MSE=\frac{1}{N}\sum_{i=1}^{N}(x_i-\hat{x}_i)^2

This method directly optimizes reconstruction error, but it may not fully reflect the final task quality of the model.

11.4 Calibration Process

flowchart TD
    A["Prepare pretrained model"] --> B["Prepare representative calibration data"]
    B --> C["Run model forward pass"]
    C --> D["Collect weight and activation statistics"]
    D --> E["Analyze numerical distribution and outliers"]
    E --> F["Determine quantization granularity"]
    F --> G["Compute Scale and Zero Point"]
    G --> H["Perform quantization conversion"]
    H --> I["Evaluate quantization error"]
    I --> J{"Does the error meet requirements?"}
    J -->|"No"| K["Adjust calibration data or quantization strategy"]
    K --> C
    J -->|"Yes"| L["Save quantized model"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F

12. PTQ and QAT: Post-Training Quantization and Quantization-Aware Training

Model quantization mainly falls into two approaches:

  • PTQ: Post-Training Quantization (post-training quantization).
  • QAT: Quantization-Aware Training (quantization-aware training).

12.1 PTQ: Post-Training Quantization

PTQ performs quantization after model training is complete.

Typical process:

  1. Load an already trained FP16 or BF16 model.
  2. Prepare calibration data.
  3. Collect weight or activation distributions.
  4. Compute quantization parameters.
  5. Convert weights and related computation paths.
  6. Evaluate model quality.

Advantages:

  • No need to perform full model training again.
  • Lower cost.
  • Relatively simple deployment process.
  • Suitable for quickly compressing existing models.

Disadvantages:

  • Accuracy loss may be larger at low bit widths.
  • Calibration data affects the results.
  • Higher adaptation requirements for some model architectures.

GPTQ, AWQ, and SmoothQuant are all typical post-training quantization approaches.

12.2 QAT: Quantization-Aware Training

QAT simulates quantization error during model training or fine-tuning.

During training, the model can still use high-precision parameters for gradient updates, but the forward pass simulates low-precision quantization behavior.

A typical approach is to use Fake Quantization.

Fake Quantization does not necessarily actually store parameters as INT8; instead, it performs the following in the forward pass:

x\rightarrow Q(x)\rightarrow DQ(Q(x))

Where:

  • Q: quantization.
  • DQ: dequantization.

In this way, the training process can perceive quantization error and adapt to low-precision computation through parameter updates.

12.3 Straight-Through Estimator

The quantization function includes a rounding operation:

q=\operatorname{round}(x)

Strictly speaking, the rounding function is non-differentiable almost everywhere.

Therefore, QAT usually adopts the Straight-Through Estimator, or STE for short.

STE uses an approximate gradient during backpropagation, allowing gradients to pass through the quantization operation.

A simplified form:

\frac{\partial Q(x)}{\partial x}\approx1

In practice, implementations may set gradients to zero outside the quantization range, to prevent values outside the valid range from continuously affecting training.

12.4 Comparison of PTQ and QAT

Comparison Dimension PTQ QAT
Execution timing After training is complete During training or fine-tuning
Requires training Usually not required Required
Computational cost Relatively low Relatively high
Adaptation to quantization error Relies on calibration and algorithms The model can learn to adapt
Ultra-low bit width Requires refined algorithms Has the potential for better accuracy
Deployment complexity Relatively low Training pipeline is more complex