I. What Is vLLM?
vLLM is a high-performance inference and serving framework for large language models (LLMs). Its core goal is to let more requests execute concurrently under limited GPU resources, while reducing inference latency and increasing token throughput.
Traditional inference approaches typically use a relatively static batching strategy. Because requests differ in input length and output length, GPU compute and memory resources are hard to fully utilize.
For example:
- Request A needs to generate 20 tokens.
- Request B needs to generate 200 tokens.
- Request C needs to generate 500 tokens.
With traditional batching, a short request may have to wait for a long request after it finishes, or it keeps occupying a resource slot in the batch, which lowers GPU utilization.
vLLM solves these problems through more flexible request scheduling and KV cache management.
1. Core Capabilities of vLLM
| Capability | Purpose | Problem Solved |
|---|---|---|
| PagedAttention | Manages the KV cache in pages | Memory fragmentation and contiguous allocation issues |
| Continuous Batching | Dynamically adjusts execution batches | Resource waste caused by differing request lengths |
| KV Cache Management | Manages the Key and Value of historical tokens | Avoids recomputing attention |
| Prefix Caching | Reuses the KV cache of identical prefixes | Reduces repeated prompt computation |
| Chunked Prefill | Executes long prompts in chunks | Reduces blocking of decode by long inputs |
| Scheduler | Schedules requests and token budgets | Improves GPU resource utilization |
| Tensor Parallel | Multi-GPU cooperative computation | Enables deployment of larger models |
| Speculative Decoding | Accelerates generation with a draft model | Reduces autoregressive decoding latency |
2. Differences Between vLLM and Ordinary Transformers Inference
Transformers leans more toward model loading, training, and general-purpose inference interfaces, while vLLM focuses more on high-concurrency online inference serving.
| Comparison Dimension | Conventional Transformers Inference | vLLM |
|---|---|---|
| Core positioning | General-purpose model framework | High-performance inference engine |
| Batching | Commonly static batching | Continuous Batching |
| KV Cache | Managed by the model's inference logic | Unified block management mechanism |
| Memory management | Typically conventional tensor allocation | Paged management via PagedAttention |
| Request scheduling | Usually handled by an external serving layer | Built-in Scheduler |
| Concurrent serving | Requires building additional serving logic | Provides online inference serving capability |
| Use cases | Model experimentation, training, general inference | High-throughput, many-request inference |
Note that Transformers itself continues to gain more efficient caching, batching, and inference capabilities. The two are not simply a matter of one being faster than the other; they have different priorities.
II. Overall Architecture of vLLM
The core of vLLM can be understood as four cooperating parts:
- Request entry layer: receives client requests, performs parameter parsing and request management.
- Scheduling layer: decides which requests enter the current execution batch and how much compute budget to allocate.
- Execution layer: invokes the model's computation logic to run prefill and decode.
- Cache management layer: manages the allocation, reuse, and release of KV cache blocks.
1. Overall Architecture Diagram
flowchart TD
A["Client Request"] --> B["API Server"]
B --> C["Request Queue"]
C --> D["Scheduler"]
D --> E["Model Runner"]
E --> F["GPU Model Execution"]
F --> G["Prefill / Decode"]
G --> H["PagedAttention"]
H --> I["KV Cache Block Manager"]
I --> J["GPU Memory"]
G --> K["Token Sampler"]
K --> L["Output Processing"]
L --> M["Streaming Response to Client"]
D --> N["Scheduling Budget and Request State"]
N --> D
I --> D
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
2. Core Module Responsibilities
| Module | Core Responsibilities |
|---|---|
| API Server | Receives HTTP requests and handles the interface protocol |
| Engine | Manages inference tasks and the overall execution lifecycle |
| Scheduler | Selects requests to be executed and controls batch and token budgets |
| Model Runner | Assembles model inputs and executes GPU computation |
| Worker | Manages GPU execution resources |
| Attention Backend | Executes Attention operators |
| Block Manager | Manages KV Cache Blocks |
| Sampler | Samples and generates Tokens from Logits |
| Output Processor | Organizes results and returns them to the client |
The internal module names and call boundaries may differ across vLLM versions, but these responsibilities form the core abstraction for understanding its architecture.
III. The Request Execution Lifecycle in vLLM
From entering vLLM to producing the final result, a request typically goes through the following stages:
- Receive the request and validate its parameters.
- Add the request to the waiting queue.
- The Scheduler selects the request.
- Allocate KV Cache Blocks for the request.
- Execute Prefill to process the input Prompt.
- Execute Decode to generate Tokens step by step.
- Determine whether to stop based on the stopping conditions.
- Release resources such as the KV Cache.
- Return the final result.
1. Request Lifecycle Flowchart
flowchart TD
A["Receive inference request"] --> B["Parameter validation and Tokenize"]
B --> C["Add to waiting queue"]
C --> D["Scheduler selects request"]
D --> E{"Are resources sufficient?"}
E -->|"Yes"| F["Allocate KV Cache Block"]
E -->|"No"| G["Wait or adjust scheduling"]
G --> D
F --> H["Prefill stage"]
H --> I["Generate first Token"]
I --> J["Decode stage"]
J --> K{"Is stop condition met?"}
K -->|"No"| L["Update KV Cache"]
L --> J
K -->|"Yes"| M["Release cache resources"]
M --> N["Organize output results"]
N --> O["Return to client"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2. Prefill Stage
Prefill is responsible for processing the user's input Prompt.
Assume the input:
Please explain how Transformer works.
After Tokenize, the Token sequence is:
[token_1, token_2, token_3, ..., token_n]
The model performs forward computation on the input Tokens, completing:
- Embedding lookup.
- Transformer Layer computation.
- Attention computation.
- FFN computation.
- Output Logits.
- Build the KV Cache for each layer.
The Prefill stage can usually process multiple input Tokens in parallel, so it has strong matrix computation characteristics.
3. Decode Stage
After Prefill is complete, the model starts generating content Token by Token.
For example:
Input Prompt
↓
Prefill
↓
Generate Token A
↓
Generate Token B
↓
Generate Token C
↓
Until the stop condition is met
In the Decode stage, usually only the new Token needs to be fed into the model while reading the historical KV Cache, without recomputing the Key and Value of all historical Tokens.
The Decode stage usually has more obvious GPU memory bandwidth pressure and is limited by autoregressive dependencies.
IV. PagedAttention: The Core Technology of vLLM
PagedAttention is one of vLLM's most representative technologies.
It draws on the idea of paged virtual memory management in operating systems, splitting the KV Cache into fixed-size Blocks and no longer requiring each request to use one contiguous block of GPU memory.
1. Problems with Traditional KV Cache Management
In conventional contiguous allocation, each request needs to reserve contiguous GPU memory space for the KV Cache.
Suppose there are three requests:
| Request | Estimated Token Count | Actual Token Count |
|---|---|---|
| A | 1024 | 200 |
| B | 1024 | 600 |
| C | 1024 | 900 |
If preallocated according to the maximum length, each request needs to reserve cache space for 1024 tokens.
If actual usage is far less than the reserved amount, VRAM is wasted.
In addition, different requests finish at different times, so after VRAM is released, many non-contiguous free regions may appear.
This is the VRAM fragmentation problem.
2. The Paging Concept of PagedAttention
PagedAttention divides the KV Cache into fixed-size Blocks.
Suppose each Block can store the KV Cache of 16 tokens:
| Request | Actual Token Count | Required Block Count |
|---|---|---|
| A | 20 | 2 |
| B | 35 | 3 |
| C | 50 | 4 |
The number of Blocks is calculated by rounding up:
Where:
- N_{\text{tokens}}: the number of tokens that the request currently needs to cache.
- B: the number of tokens each Block can hold.
Each request uses a Block Table to record the mapping between logical Blocks and physical Blocks.
For example:
Request A:
Logical Block 0 → Physical Block 3
Logical Block 1 → Physical Block 8
Request B:
Logical Block 0 → Physical Block 1
Logical Block 1 → Physical Block 5
Logical Block 2 → Physical Block 9
Physical Blocks do not need to be arranged contiguously.
Therefore, the KV Cache can be allocated from any available location, reducing the constraints of contiguous memory allocation.
3. The Role of the Block Table
The Block Table is an important mapping structure through which PagedAttention accesses the KV Cache.
It is responsible for maintaining:
- The logical Block number of a request.
- The corresponding physical Block number.
- Block usage status.
- Block reference relationships.
- Block allocation and release information.
A logical token position needs to be mapped through the Block Table to the corresponding physical cache location.
For example:
Logical token position
↓
Calculate logical Block number
↓
Look up the Block Table
↓
Get physical Block number
↓
Calculate offset within the Block
↓
Access the KV Cache
4. The Execution Principle of PagedAttention
Traditional Attention needs to read the Key and Value of historical tokens.
PagedAttention stores the historical KV Cache across multiple physical Blocks, and the Attention Kernel locates the data according to the Block Table and performs the Attention computation.
It does not change the mathematical formula of Attention; rather, it changes the physical organization and access method of the KV Cache.
The standard Attention computation can still be expressed as:
Where:
- Q: Query.
- K: Key.
- V: Value.
- d_k: the dimension of the Key vector.
5. Advantages of PagedAttention
| Advantage | Description |
|---|---|
| Reduces memory fragmentation | Does not require a request's cache to be allocated contiguously |
| Improves memory utilization | Blocks are allocated on demand |
| Supports dynamic expansion | More Blocks can be allocated as the number of Tokens grows |
| Supports cache sharing | Requests with the same prefix can share Blocks through references |
| Improves concurrency | The same memory capacity can hold more requests |
Note that paged management itself also carries costs for Block metadata, addressing, and scheduling. Actual performance depends on the GPU architecture, the attention backend, the Block size, and the workload.
V. KV Cache Management Mechanism
The KV Cache stores the Key and Value of historical Tokens for every layer of the Transformer.
During autoregressive generation, the model can directly reuse previously computed results, avoiding recomputing the entire historical sequence each time a new Token is generated.
1. Calculating KV Cache Memory Usage
The memory footprint of the KV Cache can be approximated as:
Where:
| Parameter | Meaning |
|---|---|
| L | Number of Transformer layers |
| B | Number of requests in the batch |
| S | Number of cached Tokens per request |
| H_{kv} | Number of KV Heads |
| D_h | Dimension of each Head |
| P | Number of bytes per element |
| 2 | Key and Value are stored separately |
For example:
- A 32-layer Transformer.
- 8 KV Heads.
- Each Head has a dimension of 128.
- FP16, 2 bytes per element.
- 1000 Tokens.
KV Cache usage for a single request:
This is approximately 125 MiB.
If 100 requests of the same length are processed concurrently, the KV Cache can reach approximately 12.2 GiB.
This does not yet account for model weights, temporary workspaces, and other runtime overhead.
2. Impact of GQA and MQA on the KV Cache
Different attention architectures affect the size of the KV Cache.
| Attention structure | Number of KV Heads | Cache characteristics |
|---|---|---|
| MHA | Same as the number of Query Heads | Larger KV Cache |
| GQA | Fewer than the number of Query Heads | Smaller KV Cache |
| MQA | Usually 1 | Even smaller KV Cache |
For example:
A model has 32 Query Heads:
- MHA: 32 KV Heads.
- GQA: 8 KV Heads.
- MQA: 1 KV Head.
With all other parameters held equal, the KV Cache for GQA is about one quarter that of MHA, and for MQA about one thirty-second that of MHA.
3. Block Allocation and Release
vLLM does not need to allocate all Blocks for the entire maximum context length when a request starts.
As the decode phase generates new Tokens, cache space can be expanded on demand.
When a request finishes, its Blocks can be released and returned to the pool of available Blocks.
flowchart TD
A["Request enters"] --> B["Compute required initial Blocks"]
B --> C["Allocate Blocks from free pool"]
C --> D["Build Block Table"]
D --> E["Execute Prefill"]
E --> F["Decode generates new Token"]
F --> G{"Does the current Block have remaining space"}
G -->|"Yes"| H["Write to existing Block"]
G -->|"No"| I["Request a new Block"]
I --> J{"Is there a free Block"}
J -->|"Yes"| K["Update Block Table"]
J -->|"No"| L["Wait, preempt, or recompute"]
K --> H
H --> F
L --> F
F --> M["Request finishes"]
M --> N["Release or decrement Block reference"]
N --> O["Block returns to free pool"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
4. Handling Insufficient GPU Memory
When there are not enough available Blocks, the Scheduler needs to handle resource contention.
Common strategies include:
- Deferred scheduling: temporarily keep new requests out of the execution batch.
- Request preemption: release the KV Cache currently occupied by certain requests.
- Recomputation: recompute the released portion of historical state in a later execution.
- Batch adjustment: reduce the resource demand of the current batch.
The specific strategy adopted and the related parameters depend on the vLLM version and the scheduling configuration.
VI. Continuous Batching
Continuous Batching is an important mechanism by which vLLM improves concurrent throughput.
1. Problems with Traditional Static Batching
Suppose a Batch contains three requests:
| Request | Number of Generated Tokens |
|---|---|
| A | 10 |
| B | 100 |
| C | 200 |
Traditional static batching usually executes the entire batch as a unit.
Even after request A has finished generating, the other requests may still occupy the batch slots.
This causes:
- Completed requests occupying batch resources.
- New requests being unable to join in time.
- Long and short requests affecting each other.
- Unstable GPU resource utilization.
2. The Basic Idea of Continuous Batching
Continuous Batching no longer requires all requests to start and end at the same time.
After each execution round, the Scheduler can reorganize the execution batch:
- Completed requests exit.
- New requests join.
- Requests that are generating continue executing.
- The batch is dynamically adjusted according to resource conditions.
For example:
Round 1: A, B, C
Round 2: A, B, C
Round 3: A finishes, B and C continue, D joins
Round 4: B, C, D
Round 5: B finishes, C and D continue, E joins
This is a simplified illustration; actual scheduling is also affected by the Token budget, KV Cache capacity, request priority, and scheduling strategy.
3. The Continuous Batching Execution Flow
flowchart TD
A["Waiting Queue"] --> B["Scheduler selects requests"]
B --> C["Form current execution batch"]
C --> D["Execute model computation"]
D --> E["Update status of each request"]
E --> F{"Are there completed requests?"}
F -->|"Yes"| G["Remove completed requests"]
F -->|"No"| H["Keep running requests"]
G --> I["Release corresponding resources"]
I --> J["Check waiting queue"]
H --> J
J --> K{"Are there new requests that can be scheduled?"}
K -->|"Yes"| L["Add new requests to batch"]
K -->|"No"| M["Continue existing batch"]
L --> D
M --> D
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
4. Advantages of Continuous Batching
| Advantage | Description |
|---|---|
| Dynamically add requests | No need to wait for the entire batch to finish |
| Dynamically remove requests | Completed requests release resources promptly |
| Improve GPU utilization | Reduce idle execution slots |
| Improve request throughput | Process more requests at the same time |
| Accommodate different output lengths | Reduce interference between long and short requests |
Two concepts need to be distinguished:
- Static Batching: batch membership is relatively fixed.
- Continuous Batching: batch membership can change dynamically during execution.
Continuous Batching does not mean every request gets the same response latency. The actual experience still depends on request length, queuing time, and GPU load.
7. Scheduler: Inference Scheduler
The Scheduler is an important control module in vLLM, responsible for coordinating request execution and GPU resource allocation.
The problems it needs to solve include:
- Which requests should be executed currently?
- How many tokens should be processed for each request?
- Is the current KV Cache sufficient?
- Can new requests be added?
- Which requests need to wait or be preempted?
1. Main Responsibilities of the Scheduler
| Responsibility | Description |
|---|---|
| Request selection | Select requests to execute from the waiting queue |
| Batch construction | Organize the current execution batch |
| Token budget control | Limit the number of tokens computed per round |
| Cache resource check | Confirm whether the KV Cache is sufficient |
| Request state management | Manage states such as waiting, running, and completed |
| Preemption handling | Handle request contention when resources are insufficient |
2. Token Budget
vLLM scheduling does not only control the number of requests; it also considers how many tokens need to be processed per round.
For example:
Current scheduling budget: 4096 Tokens
Request A: Prefill 1000 Tokens
Request B: Decode 1 Token
Request C: Prefill 2000 Tokens
Request D: Decode 1 Token
If the current scheduling policy allows these requests to enter the same round of execution, the total token budget usage is 3002.
Token Budget can control the scale of computation per round, preventing a large number of long prompts from occupying too much GPU compute resources at once.
Note that token count is not the only indicator of GPU memory usage. KV Cache, model weights, activations, the Attention workspace, and other factors also affect scheduling feasibility.
3. Scheduling Differences Between Prefill and Decode
Prefill and Decode have different computational characteristics:
| Phase | Main Work | Characteristics |
|---|---|---|
| Prefill | Process input Prompt | Process multiple Tokens at once |
| Decode | Generate new Tokens | Usually generates one Token per request per round |
Prefill has a relatively large computational load and can easily occupy GPU compute resources.
Decode generates fewer Tokens per round, but it needs to frequently access model weights and the KV Cache.
If a long Prompt occupies too much compute resources in a given round, other requests that are in Decode may have to wait longer.
Therefore, the scheduler needs to strike a balance between Prefill and Decode.
8. Chunked Prefill: Chunked Prefill
Chunked Prefill is an optimization mechanism that splits a long Prompt into multiple small chunks for execution.
1. Why is Chunked Prefill needed?
Suppose a request contains 16,000 input Tokens.
If the entire Prefill is executed all at once, it may create a relatively large computational load.
At the same time, other requests may be in the Decode phase and need to continue executing to maintain generation speed.
If a long Prompt keeps occupying GPU compute resources, it will affect the Token output latency of other requests.
2. Chunked Execution Mechanism
Suppose 16,000 Tokens are split into 4 Chunks:
| Chunk | Token Range | Token Count |
|---|---|---|
| Chunk 1 | 1–4000 | 4000 |
| Chunk 2 | 4001–8000 | 4000 |
| Chunk 3 | 8001–12000 | 4000 |
| Chunk 4 | 12001–16000 | 4000 |
After each Chunk finishes executing, the scheduler can schedule other requests according to the budget.
flowchart TD
A["Receive long Prompt"] --> B["Split Prompt"]
B --> C["Execute Chunk 1"]
C --> D["Update KV Cache"]
D --> E["Check scheduling budget"]
E --> F{"Are there any Decode requests?"}
F -->|"Yes"| G["Execute Decode requests"]
F -->|"No"| H["Continue Prefill"]
G --> I["Return to Scheduler"]
H --> J["Execute next Chunk"]
I --> J
J --> K{"Are there any Prompt Tokens left?"}
K -->|"Yes"| E
K -->|"No"| L["Prefill complete"]
L --> M["Enter normal Decode"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
3. Benefits of Chunked Prefill
- Avoid a long Prompt occupying too much computational budget all at once.
- Improve scheduling fairness between Prefill and Decode.
- Reduce the impact of long-input requests on the generation latency of other requests.
- Control the GPU workload per round more flexibly.
It does not necessarily reduce the total amount of computation required for the model to complete Prefill; what it mainly changes is how computational tasks are organized and when they are executed.
9. Prefix Caching: Prefix Cache Reuse
Prefix Caching is another important inference optimization mechanism.
In many applications, different requests may share the same Prompt prefix.
For example:
Request A:
You are a professional Python assistant.
Please explain list comprehensions.
Request B:
You are a professional Python assistant.
Please explain decorators.
Both requests share the same system prompt.
If the full prefix Prefill is re-executed every time, this results in redundant computation.
Prefix Caching can reuse the prefix KV Cache that has already been computed.
1. Principles of Prefix Reuse
flowchart TD
A["Request A"] --> C["Common Prompt Prefix"]
B["Request B"] --> C
C --> D["Compute Prefix KV Cache"]
D --> E["Cache Block"]
E --> F["Reused by Request A"]
E --> G["Reused by Request B"]
F --> H["Process A's Unique Content"]
G --> I["Process B's Unique Content"]
H --> J["Continue Generation"]
I --> K["Continue Generation"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2. Key Conditions for Prefix Caching
Prefix caching does not simply reuse based on textual similarity.
Usually, the following must hold:
- The token sequence is identical.
- The model and the relevant computation configuration are identical.
- The computation state of the cached Block matches.
- The processing information for multimodal or other special inputs is identical.
- The cache still exists and is available.
The specific cache keys and matching strategy are determined by the vLLM implementation.
3. When Prefix Caching Applies
| Scenario | Reuse Value |
|---|---|
| Fixed system prompt | High |
| Shared history in multi-turn conversations | Depends on whether the conversation prefixes match |
| Fixed instruction template for RAG | High |
| Fixed role configuration for an Agent | High |
| Large amounts of shared document prefix | Depends on the tokenization result |
| Completely different user inputs | Low |
Prefix Caching mainly reduces redundant Prefill computation; it does not mean that all KV Caches of different requests can be shared.
10. Speculative Decoding
Speculative Decoding is an acceleration technique that uses a draft model to assist the target model in generating tokens.
Traditional autoregressive generation typically generates one token per round, requiring multiple executions of the target model.
Speculative Decoding, by contrast, first has a smaller Draft Model quickly propose multiple candidate tokens, which are then verified by the Target Model.
1. Problems with Traditional Decoding
Suppose we need to generate:
Token A → Token B → Token C → Token D
The traditional approach typically requires multiple rounds of target model execution:
Run target model → Token A
Run target model → Token B
Run target model → Token C
Run target model → Token D
Each round must wait for the previous token to finish being generated.
2. How Speculative Decoding Executes
Suppose the Draft Model proposes all at once:
Token A, Token B, Token C, Token D
The Target Model verifies these candidate tokens.
If the first few tokens conform to the target model's sampling distribution, they can be accepted; when a token fails the verification conditions, it is corrected according to the corresponding acceptance and rejection sampling algorithm, and generation continues.
```mermaid flowchart TD A["Existing context"] --> B["Draft Model"] B --> C["Generate multiple candidate tokens"] C --> D["Target Model verification"] D --> E{"Do candidate tokens pass verification?"} E -->|"Pass"| F["Accept candidate token"] E -->|"Fail"| G["Correct according to sampling rules"] F --> H{"Are there remaining candidate tokens?"} H -->|"Yes"| F H -->|"No"| I["Proceed to next speculative round"] G --> J["Obtain the corrected output"] J --> I I --> B style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976 style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D style C fill:#EAF3EC,stroke:#83A995,color:#315D49 style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976 style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F style G fill:#F7EAE7,stroke:#C98D7C,color:#754438 style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D style I fill:#EAF3EC,stroke:#83A995,color:#315D49 style J fill:#EAF3EC,stroke:#83A995,color:#315D49 ```3. Why does it speed things up?
The Draft Model is usually smaller in scale, so generating candidate tokens costs less.
The Target Model can process multiple candidate positions in a single verification computation, thereby reducing the number of times the target model must be executed token by token.
When the acceptance rate of candidate tokens is high, the overall generation speed can improve noticeably.
4. Limitations of speculative decoding
- The Draft Model requires additional GPU memory and compute resources.
- When the draft model differs greatly from the target model, the acceptance rate may be low.
- Verification and scheduling also incur additional overhead.
- In scenarios where some GPUs are already highly saturated, the speedup may be limited.
A correct implementation of speculative decoding must ensure that the output distribution matches the sampling distribution specified by the target model, rather than simply accepting all tokens generated by the draft model.
XI. Tensor Parallel
When a single GPU cannot hold the full model, or when a single card lacks sufficient compute capacity, execution can be coordinated across multiple GPUs.
Tensor Parallel (TP) splits the model's large matrix computations across multiple GPUs.
1. The basic idea of tensor parallelism
Suppose the model contains a large weight matrix:
The weight matrix can be split into multiple parts:
Each GPU is responsible for computing one of these parts.
Input X
│
├── GPU 0:X × W1
├── GPU 1:X × W2
├── GPU 2:X × W3
└── GPU 3:X × W4
│
↓
Merge results
│
↓
Output Y
The above is a simplified illustration of splitting a matrix by columns. In actual Transformers, there are also various implementations such as row parallelism, column parallelism, and Attention Head partitioning.
2. Advantages of Tensor Parallel
- Spreads model weights across multiple GPUs.
- Reduces the weight memory pressure on a single GPU.
- Uses multiple GPUs to execute matrix computations in parallel.
- Supports deployment of larger-scale models.
3. Costs of Tensor Parallel
Tensor parallelism usually requires communication between GPUs.
For example:
- AllReduce.
- AllGather.
- ReduceScatter.
If the communication bandwidth between GPUs is insufficient, communication overhead may offset part of the parallel gains.
Therefore, configuring Tensor Parallel requires considering the number of GPUs, memory capacity, NVLink topology, and model structure.
XII. Other performance optimization mechanisms in vLLM
In addition to PagedAttention and Continuous Batching, vLLM can also combine various techniques to improve inference performance.
1. FlashAttention
FlashAttention is an algorithm that optimizes Attention computation and memory access.
Traditional Attention may require explicitly constructing a large Attention Score matrix:
When the sequence length is long, the intermediate matrix may incur significant memory access overhead.