Back to series
· 12 min read

GPU, VRAM, and Model Deployment Requirements


I. Why Does Large Model Deployment Need GPUs?

Large model inference involves a large number of matrix multiplications, vector operations, and tensor computations. Although CPUs can also perform these operations, they generally struggle to match GPU throughput for large-scale parallel computing.

The core advantage of a GPU is its large number of compute units, which can process large amounts of data simultaneously, making it well suited to the matrix operations in Transformers.

1. Differences Between CPUs and GPUs

Comparison Dimension CPU GPU
Number of cores Fewer cores, strong single-core performance Large number of compute cores
Computation style Good at complex control and serial tasks Good at large-scale parallel tasks
Memory System memory Dedicated VRAM or shared memory
Typical use Data processing, scheduling, business logic Matrix multiplication, tensor computation
Large model inference Can run, but usually slower Better suited to high-performance inference

CPUs and GPUs are not substitutes for each other.

In actual deployment, the CPU handles request reception, task scheduling, data preprocessing, and network communication, while the GPU handles model computation.

2. Role of GPUs in Large Model Inference

Transformer inference mainly consists of two stages:

  • Prefill: Process the input prompt and compute the attention and hidden states of the input tokens.
  • Decode: Generate output tokens one by one; each generation requires model computation.

GPUs play an important role in both stages.

The Prefill stage usually has a high degree of matrix computation parallelism and is highly compute-intensive.

In the Decode stage, the number of tokens generated each time is small, but model weights and the KV cache need to be read repeatedly, making it prone to being limited by VRAM bandwidth.

Therefore, large model inference performance depends not only on GPU compute capability but also on VRAM capacity, VRAM bandwidth, and communication efficiency between GPUs.

II. Core Hardware Metrics of GPUs

When choosing a GPU to deploy a model, you should not look only at VRAM size; you also need to comprehensively consider compute capability, bandwidth, interconnect method, and software support.

1. GPU Memory Capacity (VRAM)

VRAM is the high-speed storage space used by the GPU to store model weights, activations, KV cache, and intermediate computation results.

Common GPU VRAM capacities include:

VRAM Capacity Typical Use
8GB Small models, lightweight experiments
12GB Small quantized models
16GB Partial quantized deployment of 7B-class models
24GB Quantized deployment of 7B and some 14B models
40GB Medium and large model inference
48GB Suitable for larger quantized models and higher concurrency
80GB Large model inference, training, and high-concurrency services
141GB and above Larger models and high-load inference scenarios

These are only rough references. Whether deployment is actually possible depends on model architecture, quantization format, context length, concurrency, and the VRAM management method of the inference framework.

2. VRAM Bandwidth (Memory Bandwidth)

VRAM bandwidth indicates how much data the GPU can read from or write to VRAM per second.

Common units are GB/s or TB/s.

For example:

  • 500 GB/s: approximately 500 GB of data transferred per second.
  • 1 TB/s: approximately 1 TB of data transferred per second.
  • 3 TB/s: approximately 3 TB of data transferred per second.

In the Decode stage of large model inference, the GPU often needs to read model weights repeatedly.

When the compute units process very quickly but VRAM cannot supply data in time, a bandwidth bottleneck occurs.

The following formula can be used to roughly estimate the theoretical data read time:

T_{\text{memory}} \approx \frac{W_{\text{read}}}{B_{\text{memory}}}

Where:

  • W_{\text{read}}: the amount of data to be read.
  • B_{\text{memory}}: VRAM bandwidth.

For example, suppose one inference pass needs to read 14 GB of weights and the VRAM bandwidth is 1 TB/s; then, under ideal conditions, the data read time is approximately:

T \approx \frac{14}{1000}=0.014s

Actual inference also involves computation, cache hits, data transfer, kernel scheduling, and parallel efficiency, so this result cannot be directly treated as real token latency.

3. GPU Compute Capability

GPU compute capability is usually expressed in FLOPS, i.e., floating-point operations per second.

Common metrics include:

  • FP32: 32-bit floating-point compute capability.
  • FP16: 16-bit floating-point compute capability.
  • BF16: Brain Floating Point 16-bit compute capability.
  • FP8: 8-bit floating-point compute capability.
  • INT8: 8-bit integer compute capability.

Different precisions correspond to different computational throughput and numerical representation ranges.

Large model inference often uses FP16, BF16, FP8, INT8, or lower-bit-width quantization formats to reduce VRAM usage and improve computational efficiency.

Note that the peak FLOPS published by vendors do not equal actual model inference performance. Actual performance is also affected by operator implementation, VRAM bandwidth, batch size, communication overhead, and software optimization.

4. GPU Interconnect Bandwidth

When a model cannot fit into a single GPU, it needs to be partitioned across multiple GPUs for execution.

GPUs need to exchange intermediate computation results, so interconnect bandwidth affects the efficiency of multi-GPU deployment.

Common interconnect technologies:

  • PCIe: A common GPU interconnect with broad deployment.
  • NVLink: A GPU interconnect technology that provides higher bandwidth.
  • NVSwitch: Used to build multi-GPU high-bandwidth interconnect topologies.
  • InfiniBand: Commonly used for high-speed network communication across servers.

When deploying multiple GPUs, it is not enough to look only at the number of GPUs; you also need to consider the communication topology between the GPUs.

III. What Exactly Consumes GPU Memory?

When deploying large models, GPU memory is not used only to store model parameters.

Actual GPU memory usage mainly consists of the following parts:

  1. Model Weights
  2. KV Cache
  3. Activations
  4. Temporary Computation Space (Workspace)
  5. Framework and CUDA runtime overhead
  6. GPU memory fragmentation and reserved space

The overall relationship can be expressed as:

flowchart TD
    A["Total GPU Memory"] --> B["Model Weights"]
    A --> C["KV Cache"]
    A --> D["Activations"]
    A --> E["Temporary Computation Space"]
    A --> F["Framework and Runtime Overhead"]
    A --> G["Reserved GPU Memory and Fragmentation"]

    B --> H["Determines whether the model can be loaded"]
    C --> I["Affects context length and concurrency"]
    D --> J["Related to batching and the computation process"]
    E --> K["Related to operators and the inference framework"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49

1. Model Weight Usage

Model parameters need to be stored in GPU memory. The number of parameters and the data precision together determine weight usage.

The basic formula is:

M_{\text{weights}} = P \times \frac{b}{8}

Where:

  • P: number of model parameters.
  • b: number of bits used per parameter.
  • M_{\text{weights}}: number of bytes occupied by model weights.

Suppose a 7B model, i.e., 7 billion parameters:

Precision Memory per Parameter Theoretical Weight Memory
FP32 4 Byte 28GB
FP16 2 Byte 14GB
BF16 2 Byte 14GB
INT8 1 Byte 7GB
INT4 0.5 Byte 3.5GB

The above uses decimal GB for estimation. Actual deployment is also affected by quantization metadata, alignment, Scale, Zero Point, and framework implementation.

Therefore, the actual GPU memory usage of an INT4 model usually will not exactly equal the theoretical value.

2. KV Cache Usage

KV Cache is used during large model inference to cache the Key and Value tensors of historical Tokens.

During autoregressive generation, each new Token requires the attention information of previous Tokens.

If the Key and Value of historical Tokens were recomputed every time, the computational cost would be very high.

KV Cache caches historical computation results to reduce redundant computation.

Its GPU memory usage can be approximated as:

M_{\text{KV}} = 2 \times L \times B \times S \times H_{\text{KV}} \times D \times \frac{b}{8}

Where:

  • 2: represents Key and Value, respectively.
  • L: number of Transformer layers.
  • B: Batch Size.
  • S: number of context tokens.
  • H_{\text{KV}}: number of KV Heads.
  • D: dimension of each Head.
  • b: bit width of KV Cache data precision.

For example:

  • Number of model layers: 32.
  • KV Heads: 8.
  • Head Dimension: 128.
  • Batch Size: 1.
  • Context length: 8192.
  • KV Cache precision: FP16.

Calculation:

\begin{aligned} M_{\text{KV}} ={}& 2 \times 32 \times 1 \times 8192\\ &\times 8 \times 128 \times 2 \end{aligned}

The result is approximately 1GiB.

This shows that even if the model weights have already been quantized, the KV Cache may still occupy a large amount of GPU memory.

Especially for long-context and high-concurrency services, the KV Cache often becomes a deployment bottleneck.

3. Activation Memory Usage

Activations are intermediate tensors produced during the model's forward computation.

For example:

  • Intermediate results of Attention.
  • Intermediate results of the FFN.
  • Hidden layer outputs.
  • Temporary matrix computation results.

In inference scenarios, activations usually do not need to retain backpropagation information for all layers over the long term as in training, but they still require some temporary GPU memory.

Activation memory usage is related to the following factors:

  • Batch Size.
  • Number of input tokens.
  • Model hidden layer dimension.
  • Inference framework.
  • Operator fusion method.
  • Implementation of Prefill computation.

4. Temporary Space and GPU Memory Reservation

Inference frameworks usually also need to allocate temporary workspace.

For example, matrix multiplication, Attention kernels, and CUDA operators may require additional temporary buffers.

Some inference frameworks preallocate a GPU memory pool to reduce the overhead of frequently requesting and releasing GPU memory at runtime.

Therefore, the GPU memory usage observed using nvidia-smi does not necessarily exactly match the theoretical calculation for model weights and KV Cache.

IV. How to Estimate How Much GPU Memory a Model Needs?

Before deploying a model, you should first estimate the model weight memory usage, and then consider the KV Cache, activations, and runtime overhead.

1. GPU Memory Estimation for Models with Different Parameter Counts

The following is a theoretical estimate of FP16/BF16 model weight memory:

Model Size FP16/BF16 Weight Memory Theoretical INT8 Weight Memory Theoretical INT4 Weight Memory
1B 2GB 1GB 0.5GB
3B 6GB 3GB 1.5GB
7B 14GB 7GB 3.5GB
8B 16GB 8GB 4GB
14B 28GB 14GB 7GB
32B 64GB 32GB 16GB
70B 140GB 70GB 35GB
72B 144GB 72GB 36GB

These values represent only the theoretical footprint of the weights, not the full deployment requirements.

For example, a 70B model using INT4 quantization has a theoretical weight footprint of about 35GB, but actual deployment must also account for quantization metadata, KV Cache, context length, and inference framework overhead.

2. Complete VRAM Estimation Formula

The following formula can be used for a preliminary estimate:

M_{\text{total}} = M_{\text{weights}} + M_{\text{KV}} + M_{\text{activations}} + M_{\text{workspace}} + M_{\text{runtime}}

In engineering planning, some VRAM headroom also needs to be reserved:

M_{\text{GPU}} \geq \frac{M_{\text{total}}}{U}

Where:

  • M_{\text{GPU}}: Available GPU memory.
  • M_{\text{total}}: Estimated actual VRAM usage.
  • U: Target VRAM utilization.

For example, suppose an estimated 20GB of VRAM is required and the target utilization is 85%:

M_{\text{GPU}} \geq \frac{20}{0.85} \approx 23.53GB

Therefore, choosing a GPU with 24GB of VRAM may come close to meeting the requirement, but actual stress testing is still needed to confirm whether sufficient headroom remains.

3. VRAM Estimation Example: 70B Model

Suppose you are deploying a 70B model:

  • Model parameters: 70B.
  • Weight precision: INT4.
  • Theoretical weight footprint: 35GB.
  • KV Cache: estimated at 12GB based on context and concurrency.
  • Other runtime overhead: about 5GB.

Then:

M_{\text{total}} = 35+12+5=52GB

If VRAM utilization is to be kept at 85%:

M_{\text{GPU}} \geq \frac{52}{0.85} \approx 61.18GB

This means at least about 62GB of available VRAM capacity is required.

In practice, you should also consider whether GPU VRAM is occupied by other processes, the quantization implementation, and the memory allocation strategy of the inference framework.

V. Quantization: A Key Technology for Reducing VRAM Usage

Quantization is a technique that converts model parameters from high-precision representations to low-precision representations.

It can reduce model weight footprint and may improve inference speed.

1. Common Quantization Precisions

Precision Bit width Characteristics
FP32 32 bit Higher precision, large memory footprint
FP16 16 bit Common GPU inference precision
BF16 16 bit Larger dynamic range, suitable for some models
FP8 8 bit Balances computational efficiency and numerical precision
INT8 8 bit Notable weight compression, mature in practice
INT4 4 bit Significant memory savings, but precision loss needs attention
INT2 2 bit More aggressive compression, limited applicability

2. Basic Principles of Quantization

The core idea of quantization is to use fewer bits to represent values that were originally high precision.

A common form of linear quantization:

q = \operatorname{round}\left(\frac{x}{s}\right)+z

Where:

  • x: the original floating-point number.
  • s: the scale factor (Scale).
  • z: the zero point (Zero Point).
  • q: the quantized integer value.

Dequantization can be expressed as:

\hat{x}=s(q-z)

After quantization, the model uses less storage space to represent weights, but this introduces a certain amount of numerical error.

3. Common Quantization Schemes

  • GPTQ: aimed at weight quantization, typically using calibration data to reduce quantization error.
  • AWQ: protects important weights through an activation-aware approach.
  • GGUF: a model file format commonly used for local model deployment, supporting multiple quantization types.
  • bitsandbytes: provides common 8-bit and 4-bit quantized loading capabilities.
  • FP8: uses a low-precision floating-point format to reduce computation and storage costs.

Note that GGUF is a model file format and its associated quantization ecosystem, and should not be simply equated with quantization algorithms such as GPTQ and AWQ.

4. Impact of Quantization on Deployment

Dimension of impact Possible change
Memory footprint Significantly reduced
Model loading Usually easier
Inference speed May increase, or may decrease
Output quality Precision loss may occur
Hardware requirements Depends on the quantization format and GPU support
Operator support Varies considerably across inference frameworks

Quantization does not necessarily mean inference speed will inevitably improve.

If the GPU lacks the corresponding low-precision compute capability, or if the quantized operators are not implemented efficiently enough, actual inference speed may drop rather than rise.

6. KV Cache and Long-Context Deployment

As context length increases, KV Cache usage grows accordingly.

Assuming all other conditions remain unchanged:

  • Increasing context length from 4K to 8K roughly doubles the KV Cache.
  • From 8K to 32K, the KV Cache grows roughly fourfold.
  • From 32K to 128K, the KV Cache grows roughly fourfold.

This is also why long-context models often require larger memory capacity.

1. Impact of GQA and MQA on KV Cache

Different attention architectures differ noticeably in their KV Cache usage.

  • MHA: each Query Head typically has a corresponding Key and Value Head.
  • GQA: multiple Query Heads share a set of Key and Value Heads.
  • MQA: all Query Heads share a single set of Key and Value Heads.

KV Cache usage is directly related to the number of KV Heads.

With other parameters held equal, GQA and MQA typically use less KV Cache memory than MHA.

2. KV Cache Quantization

The KV Cache itself can also be stored at low precision.

For example:

  • FP16 KV Cache.
  • BF16 KV Cache.
  • INT8 KV Cache.
  • INT4 KV Cache.

Lowering KV Cache precision can reduce memory usage, but may introduce additional quantization error.

Actual deployment requires testing based on model quality requirements and inference framework support.

3. PagedAttention

PagedAttention is a technique for efficiently managing the KV Cache.

Traditional KV Cache management may require allocating large contiguous chunks of memory, which easily leads to memory fragmentation.

PagedAttention divides the KV Cache into fixed-size Blocks and manages physical memory blocks through logical mapping.

It improves VRAM utilization and makes memory management more efficient in dynamic request and concurrent multi-request scenarios.

flowchart TD
    A["User request arrives"] --> B["Allocate logical KV Block"]
    B --> C["Map to physical VRAM Block"]
    C --> D["Run model inference"]
    D --> E{"New Token generated?"}
    E -->|"Yes"| F["Request or reuse KV Block"]
    F --> G["Update Block mapping"]
    G --> D
    E -->|"No"| H["Release Blocks held by the request"]
    H --> I["Reclaim physical VRAM"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

VII. Single-GPU and Multi-GPU Deployment

When model weights, KV Cache, and runtime overhead exceed the available VRAM of a single GPU, multi-GPU deployment must be considered.

1. Single-GPU Deployment

Single-GPU deployment is the simplest approach:

  • All model weights reside on a single GPU.
  • KV Cache is managed on the same GPU.
  • Inference computation is performed on the same GPU.

Advantages:

  • Simple architecture.
  • Low GPU communication overhead.
  • Relatively easy to deploy and troubleshoot.
  • Suitable for small and medium-sized models.

Disadvantages:

  • Limited by single-card VRAM.
  • Model scale and concurrency capacity are constrained.
  • A single-card failure affects the entire model instance.

2. Tensor Parallelism

Tensor parallelism splits the matrix computations in the model across multiple GPUs.

For example, a large weight matrix is split into several parts, and each GPU handles one part of the computation.

After the computation is complete, results must be synchronized over the GPU interconnect.

flowchart TD
    A["Input Hidden States"] --> B["Tensor parallel computation"]
    B --> C["GPU 0 computes matrix shard"]
    B --> D["GPU 1 computes matrix shard"]
    B --> E["GPU 2 computes matrix shard"]
    B --> F["GPU 3 computes matrix shard"]
    C --> G["All-Reduce / All-Gather"]
    D --> G
    E --> G
    F --> G
    G --> H["Merge computation results"]
    H --> I["Proceed to next layer"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Advantages:

  • Reduces the VRAM pressure of weights on each card.
  • Suitable for models with heavy per-layer computation.
  • Can leverage the compute resources of multiple GPUs.

Disadvantages:

  • Requires frequent inter-GPU communication.
  • Insufficient communication bandwidth will degrade performance.
  • Multi-GPU splitting does not necessarily bring linear speedup.

3. Pipeline Parallelism

Pipeline parallelism splits the model by layer.

For example, a 32-layer Transformer can be split into multiple stages:

  • GPU 0: Layers 1–8.
  • GPU 1: Layers 9–16.
  • GPU 2: Layers 17–24.
  • GPU 3: Layers 25–32.

Data is passed between GPUs in layer order.

Advantages:

  • Can spread model weights across GPUs.
  • Suitable for large models with many layers.
  • GPU utilization can be improved through micro-batching.

Disadvantages:

  • There is pipeline idle time.
  • Inter-layer transfers incur communication overhead.
  • Uneven load across stages reduces overall efficiency.

4. Expert Parallelism

Expert parallelism is mainly used for MoE (Mixture of Experts) models.

An MoE model contains multiple expert networks, and each token is routed to only some experts for computation.

Expert parallelism places different experts on different GPUs, using multiple GPUs to share expert parameters and computation tasks.

This approach is suitable for MoE architectures with a large number of experts and a large model parameter scale.

However, it needs to handle token routing, cross-GPU data exchange, and expert load balancing.

5. Comparison of Multi-GPU Deployment Strategies

Parallelism Method What Is Split Primary Purpose Main Challenges
Data Parallelism Requests or data batches Increase throughput The model must be replicated across multiple GPUs
Tensor Parallelism Matrices and tensors within a single layer Distribute computation and weights of a single layer Frequent communication
Pipeline Parallelism Transformer layers Distribute model layers Pipeline bubbles
Expert Parallelism MoE experts Distribute expert computation Token routing and load balancing

It is important to distinguish the following:

Data parallelism generally does not reduce the weight VRAM footprint of a single model replica, because each GPU may store a complete model replica.

If the goal is to fit a single model into multiple GPUs with smaller VRAM, focus on tensor parallelism, pipeline parallelism, or other partitioning schemes suitable for the model architecture.

VIII. How to Choose Among Different GPU Configurations?

When deploying GPUs, choices should be made based on model scale, quantization method, context length, concurrency requirements, and budget.

1. Personal Learning and Local Experiments

Suitable for:

  • Lightweight models.
  • Quantized versions of 7B-class models.
  • RAG prototypes.
  • Agent development.
  • Prompt debugging and tool-calling tests.

Recommended considerations:

  • 12GB–24GB VRAM.
  • Compatibility with common inference frameworks.
  • Support for CUDA or corresponding compute platforms.
  • Local model loading and quantization capabilities.

2. Development and Small-to-Medium-Scale Services

Suitable for:

  • 7B–14B models.
  • Quantized model deployment.
  • Small-scale concurrent requests.
  • Internal knowledge base Q&A.
  • Enterprise internal Agents.

Recommended considerations:

  • 24GB–48GB VRAM.
  • Higher memory bandwidth.
  • Multi-request scheduling capability.
  • Model serving framework compatibility.
  • Runtime monitoring and failure recovery.

3. Enterprise-Grade Model Services

Suitable for:

  • 32B–70B and larger models.
  • Multi-user concurrent access.
  • Long-context tasks.
  • Multi-Agent workflows.
  • High-availability inference services.

Recommended considerations:

  • 80GB-class or larger VRAM configurations.
  • Multi-GPU interconnect capability.
  • High-speed networking.
  • Distributed inference frameworks.
  • Model replicas and load balancing.
  • GPU fault isolation and service degradation.

4. Reference for GPU Configuration Selection