I. Why Does Large Model Deployment Need GPUs?
Large model inference involves a large number of matrix multiplications, vector operations, and tensor computations. Although CPUs can also perform these operations, they generally struggle to match GPU throughput for large-scale parallel computing.
The core advantage of a GPU is its large number of compute units, which can process large amounts of data simultaneously, making it well suited to the matrix operations in Transformers.
1. Differences Between CPUs and GPUs
| Comparison Dimension | CPU | GPU |
|---|---|---|
| Number of cores | Fewer cores, strong single-core performance | Large number of compute cores |
| Computation style | Good at complex control and serial tasks | Good at large-scale parallel tasks |
| Memory | System memory | Dedicated VRAM or shared memory |
| Typical use | Data processing, scheduling, business logic | Matrix multiplication, tensor computation |
| Large model inference | Can run, but usually slower | Better suited to high-performance inference |
CPUs and GPUs are not substitutes for each other.
In actual deployment, the CPU handles request reception, task scheduling, data preprocessing, and network communication, while the GPU handles model computation.
2. Role of GPUs in Large Model Inference
Transformer inference mainly consists of two stages:
- Prefill: Process the input prompt and compute the attention and hidden states of the input tokens.
- Decode: Generate output tokens one by one; each generation requires model computation.
GPUs play an important role in both stages.
The Prefill stage usually has a high degree of matrix computation parallelism and is highly compute-intensive.
In the Decode stage, the number of tokens generated each time is small, but model weights and the KV cache need to be read repeatedly, making it prone to being limited by VRAM bandwidth.
Therefore, large model inference performance depends not only on GPU compute capability but also on VRAM capacity, VRAM bandwidth, and communication efficiency between GPUs.
II. Core Hardware Metrics of GPUs
When choosing a GPU to deploy a model, you should not look only at VRAM size; you also need to comprehensively consider compute capability, bandwidth, interconnect method, and software support.
1. GPU Memory Capacity (VRAM)
VRAM is the high-speed storage space used by the GPU to store model weights, activations, KV cache, and intermediate computation results.
Common GPU VRAM capacities include:
| VRAM Capacity | Typical Use |
|---|---|
| 8GB | Small models, lightweight experiments |
| 12GB | Small quantized models |
| 16GB | Partial quantized deployment of 7B-class models |
| 24GB | Quantized deployment of 7B and some 14B models |
| 40GB | Medium and large model inference |
| 48GB | Suitable for larger quantized models and higher concurrency |
| 80GB | Large model inference, training, and high-concurrency services |
| 141GB and above | Larger models and high-load inference scenarios |
These are only rough references. Whether deployment is actually possible depends on model architecture, quantization format, context length, concurrency, and the VRAM management method of the inference framework.
2. VRAM Bandwidth (Memory Bandwidth)
VRAM bandwidth indicates how much data the GPU can read from or write to VRAM per second.
Common units are GB/s or TB/s.
For example:
- 500 GB/s: approximately 500 GB of data transferred per second.
- 1 TB/s: approximately 1 TB of data transferred per second.
- 3 TB/s: approximately 3 TB of data transferred per second.
In the Decode stage of large model inference, the GPU often needs to read model weights repeatedly.
When the compute units process very quickly but VRAM cannot supply data in time, a bandwidth bottleneck occurs.
The following formula can be used to roughly estimate the theoretical data read time:
Where:
- W_{\text{read}}: the amount of data to be read.
- B_{\text{memory}}: VRAM bandwidth.
For example, suppose one inference pass needs to read 14 GB of weights and the VRAM bandwidth is 1 TB/s; then, under ideal conditions, the data read time is approximately:
Actual inference also involves computation, cache hits, data transfer, kernel scheduling, and parallel efficiency, so this result cannot be directly treated as real token latency.
3. GPU Compute Capability
GPU compute capability is usually expressed in FLOPS, i.e., floating-point operations per second.
Common metrics include:
- FP32: 32-bit floating-point compute capability.
- FP16: 16-bit floating-point compute capability.
- BF16: Brain Floating Point 16-bit compute capability.
- FP8: 8-bit floating-point compute capability.
- INT8: 8-bit integer compute capability.
Different precisions correspond to different computational throughput and numerical representation ranges.
Large model inference often uses FP16, BF16, FP8, INT8, or lower-bit-width quantization formats to reduce VRAM usage and improve computational efficiency.
Note that the peak FLOPS published by vendors do not equal actual model inference performance. Actual performance is also affected by operator implementation, VRAM bandwidth, batch size, communication overhead, and software optimization.
4. GPU Interconnect Bandwidth
When a model cannot fit into a single GPU, it needs to be partitioned across multiple GPUs for execution.
GPUs need to exchange intermediate computation results, so interconnect bandwidth affects the efficiency of multi-GPU deployment.
Common interconnect technologies:
- PCIe: A common GPU interconnect with broad deployment.
- NVLink: A GPU interconnect technology that provides higher bandwidth.
- NVSwitch: Used to build multi-GPU high-bandwidth interconnect topologies.
- InfiniBand: Commonly used for high-speed network communication across servers.
When deploying multiple GPUs, it is not enough to look only at the number of GPUs; you also need to consider the communication topology between the GPUs.
III. What Exactly Consumes GPU Memory?
When deploying large models, GPU memory is not used only to store model parameters.
Actual GPU memory usage mainly consists of the following parts:
- Model Weights
- KV Cache
- Activations
- Temporary Computation Space (Workspace)
- Framework and CUDA runtime overhead
- GPU memory fragmentation and reserved space
The overall relationship can be expressed as:
flowchart TD
A["Total GPU Memory"] --> B["Model Weights"]
A --> C["KV Cache"]
A --> D["Activations"]
A --> E["Temporary Computation Space"]
A --> F["Framework and Runtime Overhead"]
A --> G["Reserved GPU Memory and Fragmentation"]
B --> H["Determines whether the model can be loaded"]
C --> I["Affects context length and concurrency"]
D --> J["Related to batching and the computation process"]
E --> K["Related to operators and the inference framework"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
1. Model Weight Usage
Model parameters need to be stored in GPU memory. The number of parameters and the data precision together determine weight usage.
The basic formula is:
Where:
- P: number of model parameters.
- b: number of bits used per parameter.
- M_{\text{weights}}: number of bytes occupied by model weights.
Suppose a 7B model, i.e., 7 billion parameters:
| Precision | Memory per Parameter | Theoretical Weight Memory |
|---|---|---|
| FP32 | 4 Byte | 28GB |
| FP16 | 2 Byte | 14GB |
| BF16 | 2 Byte | 14GB |
| INT8 | 1 Byte | 7GB |
| INT4 | 0.5 Byte | 3.5GB |
The above uses decimal GB for estimation. Actual deployment is also affected by quantization metadata, alignment, Scale, Zero Point, and framework implementation.
Therefore, the actual GPU memory usage of an INT4 model usually will not exactly equal the theoretical value.
2. KV Cache Usage
KV Cache is used during large model inference to cache the Key and Value tensors of historical Tokens.
During autoregressive generation, each new Token requires the attention information of previous Tokens.
If the Key and Value of historical Tokens were recomputed every time, the computational cost would be very high.
KV Cache caches historical computation results to reduce redundant computation.
Its GPU memory usage can be approximated as:
Where:
- 2: represents Key and Value, respectively.
- L: number of Transformer layers.
- B: Batch Size.
- S: number of context tokens.
- H_{\text{KV}}: number of KV Heads.
- D: dimension of each Head.
- b: bit width of KV Cache data precision.
For example:
- Number of model layers: 32.
- KV Heads: 8.
- Head Dimension: 128.
- Batch Size: 1.
- Context length: 8192.
- KV Cache precision: FP16.
Calculation:
The result is approximately 1GiB.
This shows that even if the model weights have already been quantized, the KV Cache may still occupy a large amount of GPU memory.
Especially for long-context and high-concurrency services, the KV Cache often becomes a deployment bottleneck.
3. Activation Memory Usage
Activations are intermediate tensors produced during the model's forward computation.
For example:
- Intermediate results of Attention.
- Intermediate results of the FFN.
- Hidden layer outputs.
- Temporary matrix computation results.
In inference scenarios, activations usually do not need to retain backpropagation information for all layers over the long term as in training, but they still require some temporary GPU memory.
Activation memory usage is related to the following factors:
- Batch Size.
- Number of input tokens.
- Model hidden layer dimension.
- Inference framework.
- Operator fusion method.
- Implementation of Prefill computation.
4. Temporary Space and GPU Memory Reservation
Inference frameworks usually also need to allocate temporary workspace.
For example, matrix multiplication, Attention kernels, and CUDA operators may require additional temporary buffers.
Some inference frameworks preallocate a GPU memory pool to reduce the overhead of frequently requesting and releasing GPU memory at runtime.
Therefore, the GPU memory usage observed using nvidia-smi does not necessarily exactly match the theoretical calculation for model weights and KV Cache.
IV. How to Estimate How Much GPU Memory a Model Needs?
Before deploying a model, you should first estimate the model weight memory usage, and then consider the KV Cache, activations, and runtime overhead.
1. GPU Memory Estimation for Models with Different Parameter Counts
The following is a theoretical estimate of FP16/BF16 model weight memory:
| Model Size | FP16/BF16 Weight Memory | Theoretical INT8 Weight Memory | Theoretical INT4 Weight Memory |
|---|---|---|---|
| 1B | 2GB | 1GB | 0.5GB |
| 3B | 6GB | 3GB | 1.5GB |
| 7B | 14GB | 7GB | 3.5GB |
| 8B | 16GB | 8GB | 4GB |
| 14B | 28GB | 14GB | 7GB |
| 32B | 64GB | 32GB | 16GB |
| 70B | 140GB | 70GB | 35GB |
| 72B | 144GB | 72GB | 36GB |
These values represent only the theoretical footprint of the weights, not the full deployment requirements.
For example, a 70B model using INT4 quantization has a theoretical weight footprint of about 35GB, but actual deployment must also account for quantization metadata, KV Cache, context length, and inference framework overhead.
2. Complete VRAM Estimation Formula
The following formula can be used for a preliminary estimate:
In engineering planning, some VRAM headroom also needs to be reserved:
Where:
- M_{\text{GPU}}: Available GPU memory.
- M_{\text{total}}: Estimated actual VRAM usage.
- U: Target VRAM utilization.
For example, suppose an estimated 20GB of VRAM is required and the target utilization is 85%:
Therefore, choosing a GPU with 24GB of VRAM may come close to meeting the requirement, but actual stress testing is still needed to confirm whether sufficient headroom remains.
3. VRAM Estimation Example: 70B Model
Suppose you are deploying a 70B model:
- Model parameters: 70B.
- Weight precision: INT4.
- Theoretical weight footprint: 35GB.
- KV Cache: estimated at 12GB based on context and concurrency.
- Other runtime overhead: about 5GB.
Then:
If VRAM utilization is to be kept at 85%:
This means at least about 62GB of available VRAM capacity is required.
In practice, you should also consider whether GPU VRAM is occupied by other processes, the quantization implementation, and the memory allocation strategy of the inference framework.
V. Quantization: A Key Technology for Reducing VRAM Usage
Quantization is a technique that converts model parameters from high-precision representations to low-precision representations.
It can reduce model weight footprint and may improve inference speed.
1. Common Quantization Precisions
| Precision | Bit width | Characteristics |
|---|---|---|
| FP32 | 32 bit | Higher precision, large memory footprint |
| FP16 | 16 bit | Common GPU inference precision |
| BF16 | 16 bit | Larger dynamic range, suitable for some models |
| FP8 | 8 bit | Balances computational efficiency and numerical precision |
| INT8 | 8 bit | Notable weight compression, mature in practice |
| INT4 | 4 bit | Significant memory savings, but precision loss needs attention |
| INT2 | 2 bit | More aggressive compression, limited applicability |
2. Basic Principles of Quantization
The core idea of quantization is to use fewer bits to represent values that were originally high precision.
A common form of linear quantization:
Where:
- x: the original floating-point number.
- s: the scale factor (Scale).
- z: the zero point (Zero Point).
- q: the quantized integer value.
Dequantization can be expressed as:
After quantization, the model uses less storage space to represent weights, but this introduces a certain amount of numerical error.
3. Common Quantization Schemes
- GPTQ: aimed at weight quantization, typically using calibration data to reduce quantization error.
- AWQ: protects important weights through an activation-aware approach.
- GGUF: a model file format commonly used for local model deployment, supporting multiple quantization types.
- bitsandbytes: provides common 8-bit and 4-bit quantized loading capabilities.
- FP8: uses a low-precision floating-point format to reduce computation and storage costs.
Note that GGUF is a model file format and its associated quantization ecosystem, and should not be simply equated with quantization algorithms such as GPTQ and AWQ.
4. Impact of Quantization on Deployment
| Dimension of impact | Possible change |
|---|---|
| Memory footprint | Significantly reduced |
| Model loading | Usually easier |
| Inference speed | May increase, or may decrease |
| Output quality | Precision loss may occur |
| Hardware requirements | Depends on the quantization format and GPU support |
| Operator support | Varies considerably across inference frameworks |
Quantization does not necessarily mean inference speed will inevitably improve.
If the GPU lacks the corresponding low-precision compute capability, or if the quantized operators are not implemented efficiently enough, actual inference speed may drop rather than rise.
6. KV Cache and Long-Context Deployment
As context length increases, KV Cache usage grows accordingly.
Assuming all other conditions remain unchanged:
- Increasing context length from 4K to 8K roughly doubles the KV Cache.
- From 8K to 32K, the KV Cache grows roughly fourfold.
- From 32K to 128K, the KV Cache grows roughly fourfold.
This is also why long-context models often require larger memory capacity.
1. Impact of GQA and MQA on KV Cache
Different attention architectures differ noticeably in their KV Cache usage.
- MHA: each Query Head typically has a corresponding Key and Value Head.
- GQA: multiple Query Heads share a set of Key and Value Heads.
- MQA: all Query Heads share a single set of Key and Value Heads.
KV Cache usage is directly related to the number of KV Heads.
With other parameters held equal, GQA and MQA typically use less KV Cache memory than MHA.
2. KV Cache Quantization
The KV Cache itself can also be stored at low precision.
For example:
- FP16 KV Cache.
- BF16 KV Cache.
- INT8 KV Cache.
- INT4 KV Cache.
Lowering KV Cache precision can reduce memory usage, but may introduce additional quantization error.
Actual deployment requires testing based on model quality requirements and inference framework support.
3. PagedAttention
PagedAttention is a technique for efficiently managing the KV Cache.
Traditional KV Cache management may require allocating large contiguous chunks of memory, which easily leads to memory fragmentation.
PagedAttention divides the KV Cache into fixed-size Blocks and manages physical memory blocks through logical mapping.
It improves VRAM utilization and makes memory management more efficient in dynamic request and concurrent multi-request scenarios.
flowchart TD
A["User request arrives"] --> B["Allocate logical KV Block"]
B --> C["Map to physical VRAM Block"]
C --> D["Run model inference"]
D --> E{"New Token generated?"}
E -->|"Yes"| F["Request or reuse KV Block"]
F --> G["Update Block mapping"]
G --> D
E -->|"No"| H["Release Blocks held by the request"]
H --> I["Reclaim physical VRAM"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
VII. Single-GPU and Multi-GPU Deployment
When model weights, KV Cache, and runtime overhead exceed the available VRAM of a single GPU, multi-GPU deployment must be considered.
1. Single-GPU Deployment
Single-GPU deployment is the simplest approach:
- All model weights reside on a single GPU.
- KV Cache is managed on the same GPU.
- Inference computation is performed on the same GPU.
Advantages:
- Simple architecture.
- Low GPU communication overhead.
- Relatively easy to deploy and troubleshoot.
- Suitable for small and medium-sized models.
Disadvantages:
- Limited by single-card VRAM.
- Model scale and concurrency capacity are constrained.
- A single-card failure affects the entire model instance.
2. Tensor Parallelism
Tensor parallelism splits the matrix computations in the model across multiple GPUs.
For example, a large weight matrix is split into several parts, and each GPU handles one part of the computation.
After the computation is complete, results must be synchronized over the GPU interconnect.
flowchart TD
A["Input Hidden States"] --> B["Tensor parallel computation"]
B --> C["GPU 0 computes matrix shard"]
B --> D["GPU 1 computes matrix shard"]
B --> E["GPU 2 computes matrix shard"]
B --> F["GPU 3 computes matrix shard"]
C --> G["All-Reduce / All-Gather"]
D --> G
E --> G
F --> G
G --> H["Merge computation results"]
H --> I["Proceed to next layer"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Advantages:
- Reduces the VRAM pressure of weights on each card.
- Suitable for models with heavy per-layer computation.
- Can leverage the compute resources of multiple GPUs.
Disadvantages:
- Requires frequent inter-GPU communication.
- Insufficient communication bandwidth will degrade performance.
- Multi-GPU splitting does not necessarily bring linear speedup.
3. Pipeline Parallelism
Pipeline parallelism splits the model by layer.
For example, a 32-layer Transformer can be split into multiple stages:
- GPU 0: Layers 1–8.
- GPU 1: Layers 9–16.
- GPU 2: Layers 17–24.
- GPU 3: Layers 25–32.
Data is passed between GPUs in layer order.
Advantages:
- Can spread model weights across GPUs.
- Suitable for large models with many layers.
- GPU utilization can be improved through micro-batching.
Disadvantages:
- There is pipeline idle time.
- Inter-layer transfers incur communication overhead.
- Uneven load across stages reduces overall efficiency.
4. Expert Parallelism
Expert parallelism is mainly used for MoE (Mixture of Experts) models.
An MoE model contains multiple expert networks, and each token is routed to only some experts for computation.
Expert parallelism places different experts on different GPUs, using multiple GPUs to share expert parameters and computation tasks.
This approach is suitable for MoE architectures with a large number of experts and a large model parameter scale.
However, it needs to handle token routing, cross-GPU data exchange, and expert load balancing.
5. Comparison of Multi-GPU Deployment Strategies
| Parallelism Method | What Is Split | Primary Purpose | Main Challenges |
|---|---|---|---|
| Data Parallelism | Requests or data batches | Increase throughput | The model must be replicated across multiple GPUs |
| Tensor Parallelism | Matrices and tensors within a single layer | Distribute computation and weights of a single layer | Frequent communication |
| Pipeline Parallelism | Transformer layers | Distribute model layers | Pipeline bubbles |
| Expert Parallelism | MoE experts | Distribute expert computation | Token routing and load balancing |
It is important to distinguish the following:
Data parallelism generally does not reduce the weight VRAM footprint of a single model replica, because each GPU may store a complete model replica.
If the goal is to fit a single model into multiple GPUs with smaller VRAM, focus on tensor parallelism, pipeline parallelism, or other partitioning schemes suitable for the model architecture.
VIII. How to Choose Among Different GPU Configurations?
When deploying GPUs, choices should be made based on model scale, quantization method, context length, concurrency requirements, and budget.
1. Personal Learning and Local Experiments
Suitable for:
- Lightweight models.
- Quantized versions of 7B-class models.
- RAG prototypes.
- Agent development.
- Prompt debugging and tool-calling tests.
Recommended considerations:
- 12GB–24GB VRAM.
- Compatibility with common inference frameworks.
- Support for CUDA or corresponding compute platforms.
- Local model loading and quantization capabilities.
2. Development and Small-to-Medium-Scale Services
Suitable for:
- 7B–14B models.
- Quantized model deployment.
- Small-scale concurrent requests.
- Internal knowledge base Q&A.
- Enterprise internal Agents.
Recommended considerations:
- 24GB–48GB VRAM.
- Higher memory bandwidth.
- Multi-request scheduling capability.
- Model serving framework compatibility.
- Runtime monitoring and failure recovery.
3. Enterprise-Grade Model Services
Suitable for:
- 32B–70B and larger models.
- Multi-user concurrent access.
- Long-context tasks.
- Multi-Agent workflows.
- High-availability inference services.
Recommended considerations:
- 80GB-class or larger VRAM configurations.
- Multi-GPU interconnect capability.
- High-speed networking.
- Distributed inference frameworks.
- Model replicas and load balancing.
- GPU fault isolation and service degradation.