1. Why AI Systems Need Caching and Resource Scheduling
The main bottlenecks in traditional web systems are usually databases, network I/O, and application servers. AI systems face more complex resource constraints:
- High computational cost: Model inference, embedding generation, document parsing, and vector retrieval can all consume substantial computing resources.
- Expensive GPU resources: GPU memory is limited, and different models and tasks vary greatly in their memory requirements.
- Wide variation in requests: The compute required for short-text Q&A versus long-context reasoning can differ by dozens of times.
- Pervasive redundant computation: The same prompts, the same documents, and the same embeddings may be processed over and over again.
- Severe resource contention: Online inference, batch embedding, RAG index building, and model training may all compete for GPU, CPU, memory, and storage at the same time.
- Different latency objectives: Online interaction focuses on TTFT and TPOT, while offline tasks focus more on throughput and cost per task.
Without caching, the system will repeatedly perform expensive computations; without proper scheduling, even with a large number of GPUs, problems such as some GPUs sitting idle, some tasks queuing, insufficient GPU memory, and request timeouts can still occur.
Caching and resource scheduling address two interrelated problems:
| Capability | Primary Goal | Typical Techniques |
|---|---|---|
| Caching | Reduce redundant computation and data access | KV cache, semantic cache, embedding cache, result cache |
| Resource scheduling | Improve resource utilization and service stability | Queue scheduling, GPU allocation, concurrency control, priority scheduling |
| Joint optimization | Lower overall cost and end-to-end latency | Cache-aware scheduling, resource isolation, dynamic scaling |
The core idea is: reuse computations that can be reused whenever possible, and place computations that must be performed on suitable resources whenever possible.
2. The Caching System in AI Systems
There is no one universal cache in AI systems. Different caches differ in the data they store, their lifecycles, their hit conditions, and their consistency requirements.
2.1 Main Cache Layers
Caches can be divided into multiple layers based on how far the data is from the compute nodes.
| Cache Layer | Storage Location | Typical Content | Main Characteristics |
|---|---|---|---|
| L1 | Process memory | Hot configurations, objects, short-term results | Low latency, but limited capacity |
| L2 | Distributed cache | Query results, embeddings, session state | Shared across instances, suitable for cross-process reuse |
| L3 | Local disk or NVMe | Model weights, data blocks, index files | Larger capacity, faster access than remote storage |
| L4 | Object storage or persistent storage | Model files, datasets, historical results | Large capacity, suitable for long-term retention |
A typical data access path is as follows:
flowchart TD
A["Request arrives"] --> B["Query in-process cache"]
B --> C{"L1 hit?"}
C -->|Yes| H["Return cached result"]
C -->|No| D["Query distributed cache"]
D --> E{"L2 hit?"}
E -->|Yes| I["Backfill L1 and return"]
E -->|No| F["Access local disk or persistent storage"]
F --> G["Perform necessary computation or loading"]
G --> J["Write to cache"]
J --> H
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
Note here that different cache layers do not necessarily form a strict tier-by-tier access chain. For example, model weights may be downloaded directly from object storage to local NVMe and then loaded into GPU memory; the embedding cache, by contrast, may access Redis or a dedicated vector store directly.
2.2 Cache Hits and Cache Misses
A cache hit means the data required by a request already exists in the cache and can be reused directly.
A cache miss means the corresponding data is not in the cache, so lower-level storage must be accessed or the computation must be re-executed.
Cache hit rate:
Cache miss rate:
For example, over a period of time there are 10,000 cache lookups, of which 8,500 are hits:
But a high hit rate does not necessarily mean good system performance.
For example:
- The cache hits are all lightweight data, so the computation cost saved is limited.
- The cache lookup itself requires calling a remote service, which may add network latency.
- Even after a cache hit, a large amount of follow-up computation may still be required.
- Cached data may expire or become invalid, which can lead to incorrect results.
Therefore, beyond the hit rate, you should also look at the computation time the cache saves, request latency, network overhead, and cache maintenance cost.
2.3 How to Calculate the Cache Benefit
Suppose:
- T_c: time taken by a cache lookup.
- T_m: time taken to recompute when the cache misses.
- H: cache hit rate.
Ignoring other overheads, the average request time can be approximated as:
For example:
- A cache lookup takes 2 ms.
- The original computation takes 200 ms.
- The cache hit rate is 80%.
Then:
Compared with running the 200 ms computation every time, the average time is significantly lower.
In real systems, however, you also need to account for the extra costs of cache serialization, network transfer, data backfilling, invalidation handling, and concurrent contention.
III. Main Cache Types in AI Systems
3.1 Prompt Cache: Prompt Caching
Prompt Cache is used to cache prompts that have already been processed, or the related computation results.
In LLM inference, a request usually contains:
- System Prompt.
- User input.
- Conversation history.
- Tool definitions.
- RAG retrieval content.
If multiple requests share the same prefix — for example the same System Prompt, a long block of background knowledge, and tool definitions — the already-completed prefix computation can be reused.
Two different mechanisms need to be distinguished:
Application-level prompt cache
Caches the result corresponding to a complete request or prompt. The result can be reused directly only when the request meets the corresponding matching conditions.
Model-inference-level Prefix Cache
Caches the KV states produced when the same token prefix passes through the Transformer, so that subsequent requests can skip the repeated Prefill computation.
The difference between the two:
| Comparison dimension | Application-level prompt cache | Prefix Cache |
|---|---|---|
| What is cached | Request results or intermediate data | Transformer KV states |
| How it is reused | Return directly or reuse application data | Skip part of the Prefill computation |
| Main benefit | Reduces full computation | Reduces repeated prefix computation |
| Use cases | Repeated questions, fixed tasks | Inference requests that share a long prefix |
Prefix Cache does not mean the model needs no inference at all. Even with a prefix hit, the new user input and the generation phase still require computation.
3.2 KV Cache: Attention Key-Value Cache
KV Cache is one of the most important caching mechanisms in LLM inference.
In the self-attention computation of a Transformer, each token produces a Key and a Value:
The attention computation:
During autoregressive generation, a new token needs to attend to the tokens already generated.
If the Key and Value of historical tokens were recomputed every time a new token is generated, there would be a large amount of duplicate computation.
KV Cache stores the Key and Value of historical tokens; in later generation steps only the K and V for the new token are computed, and attention is then computed against the historical cache.
flowchart TD
A["Input Prompt"] --> B["Prefill Stage"]
B --> C["Compute K and V for all input tokens"]
C --> D["Store KV Cache"]
D --> E["Decode stage generates new tokens"]
E --> F["Compute Q, K, V of the new token"]
F --> G["Append new K, V to the cache"]
G --> H["Compute Attention using historical KV"]
H --> I["Output the next token"]
I --> J{"Finished?"}
J -->|No| E
J -->|Yes| K["Release KV Cache"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#E5F0E9,stroke:#83A995,color:#315D49
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#E5F0E9,stroke:#83A995,color:#315D49
The main benefit of KV Cache is avoiding recomputation of historical K and V, but it also consumes a large amount of GPU memory.
Assume:
- Batch Size is B.
- The number of Transformer layers is L.
- The number of KV heads is H_{kv}.
- The dimension of each head is D.
- The number of tokens per sequence is S.
- Each element occupies P bytes.
Ignoring alignment and extra management overhead, the memory requirement for KV Cache is approximately:
Here the 2 represents the two parts: Key and Value.
For example:
- Batch Size = 1.
- Number of Transformer layers = 32.
- Number of KV heads = 8.
- Head Dimension = 128.
- Context length = 8192.
- The data type is FP16, with each element occupying 2 bytes.
Then:
This is about 1GiB.
This means a single long-context request can already occupy a large amount of GPU memory. The more concurrent requests there are, the greater the memory pressure that KV Cache places on the GPU.
Real models may also use mechanisms such as GQA, MQA, quantized KV Cache, and paged management to reduce this footprint.
3.3 Prefix Cache: Prefix Caching
Prefix Cache is a further reuse of KV Cache.
For example, multiple requests all use the same system prompt:
You are a professional technical assistant.
Please follow these answer guidelines:
1. Use accurate technical terminology.
2. Provide the necessary code examples.
3. Clearly state the applicable conditions.
Subsequent requests only append different questions at the end:
Request A: Explain Redis's persistence mechanism.
Request B: Explain Kafka's message delivery semantics.
Request C: Explain Kubernetes's scheduling mechanism.
If the model service supports prefix KV caching, the KV state corresponding to the shared prefix can be reused.
A Prefix Cache hit usually depends on:
- Prefix consistency after tokenization.
- Identical model weights and model version.
- Compatible adapter configurations such as LoRA.
- Compatible positional encoding and related inference parameters.
- The cache blocks still exist and have not been reclaimed.
- Multi-tenant isolation and cache access permissions meet the requirements.
You cannot simply assume that because two pieces of text look similar, their KV Cache can definitely be reused.
In particular, note that Prefix Cache reuses based on the model's computational state, not on semantic similarity.
3.4 Embedding Cache: Vector Caching
An Embedding model converts text into vectors:
If the same text is embedded repeatedly, the corresponding Embedding result can be cached.
For example:
Input text:
Redis is a high-performance in-memory data structure store.
Cache result:
[0.125, -0.328, 0.741, ...]
The key of an Embedding cache should not depend only on the raw text; it should also include the configuration that affects the resulting vector:
cache_key = hash(
normalized_text
+ model_name
+ model_version
+ preprocessing_version
+ embedding_dimension
)
Consider the following:
- Vectors produced by different Embedding models cannot be mixed at will.
- Changes to text preprocessing rules may invalidate the cache.
- After a model upgrade, old and new vectors should be distinguished.
- Multilingual handling, chunking strategy and normalization configuration may affect the result.
- Large-scale Embedding caches require control over storage capacity.
An Embedding Cache mainly reduces repeated vectorization computation, but it is not the same as caching vector retrieval results.
3.5 Semantic Cache
A semantic cache reuses historical question-answer results based on semantic similarity.
For example:
Historical question:
What is Redis?
New question:
What is Redis used for?
The two are worded differently, but their meanings are close. If the historical answer still applies, it may be reused directly.
Compared with ordinary exact-match caching, a semantic cache must first generate a query vector and then perform a similarity search:
Only when the similarity exceeds a threshold is reuse of the result considered.
Typical flow:
flowchart TD
A["Receive user question"] --> B["Text normalization"]
B --> C["Generate Query Embedding"]
C --> D["Search semantic cache"]
D --> E{"Is there a result with high similarity?"}
E -->|Yes| F["Check permissions and context constraints"]
F --> G{"Can the result be reused?"}
G -->|Yes| H["Return cached answer"]
G -->|No| I["Call LLM"]
E -->|No| I
I --> J["Generate answer"]
J --> K["Check cache policy"]
K --> L["Write to semantic cache"]
L --> M["Return answer"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
Semantic caching carries an important risk: semantic similarity does not mean the business answer can be reused.
For example:
- "What is the USD exchange rate today?"
- "What was the USD exchange rate yesterday?"
The two are semantically very close, but their time conditions differ.
Another example:
- "What is the order status of Zhang San?"
- "What is the order status of Li Si?"
If user identity and data permissions are ignored, this can cause serious information leakage.
Therefore, a semantic cache must at minimum combine:
- A similarity threshold.
- Tenant and user permissions.
- Time sensitivity.
- Data version.
- Query context.
- Result freshness.
- Business rule constraints.
In scenarios such as finance, healthcare, orders, permissions and real-time status, semantic similarity alone must not be used to decide a cache hit.
3.6 RAG Cache
A RAG system has several stages that are suitable for caching:
| Cache Object | Cached Content | Invalidation Condition |
|---|---|---|
| Query Embedding | Query vector | Model or preprocessing version changes |
| Retrieval Cache | Retrieved document IDs | Index update, permission changes |
| Rerank Cache | Rerank results | Model or ranking strategy changes |
| Document Cache | Document content | Document updated or deleted |
| Prompt Cache | Assembled context | Retrieved content or prompt changes |
| LLM Result Cache | Final answer | Context, model, or generation parameter changes |
A complete RAG request can use multiple layers of caching to reduce redundant computation:
flowchart TD
A["User Question"] --> B["Query Cache"]
B --> C{"Query vector hit?"}
C -->|Yes| D["Reuse Query Embedding"]
C -->|No| E["Call Embedding Model"]
E --> F["Save vector cache"]
F --> G["Vector retrieval"]
D --> G
G --> H["Check retrieval result cache"]
H --> I{"Retrieval results reusable?"}
I -->|Yes| J["Reuse candidate documents"]
I -->|No| K["Execute vector retrieval"]
K --> L["Execute reranking"]
L --> M["Save retrieval and rerank results"]
J --> N["Assemble Prompt"]
M --> N
N --> O["Check LLM result cache"]
O --> P{"Final answer hit?"}
P -->|Yes| Q["Return answer"]
P -->|No| R["Call LLM"]
R --> S["Save result cache"]
S --> Q
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style N fill:#EAF3EC,stroke:#83A995,color:#315D49
style O fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style P fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style Q fill:#E5F0E9,stroke:#83A995,color:#315D49
style R fill:#EAF3EC,stroke:#83A995,color:#315D49
style S fill:#EAE8FA,stroke:#A49BD5,color:#51477D
Note: RAG retrieval results depend on document versions, index versions, and user permissions. A cache hit must not bypass permission filtering, nor allow already-deleted documents to keep appearing in answers.
IV. Caching Strategies and Consistency Management
A cache is not a permanent copy of data. Caching strategies must be chosen according to the characteristics of the data.
4.1 Cache Aside
Cache Aside is the most common caching pattern.
On read:
- Query the cache first.
- On a hit, return directly.
- On a miss, query the database or perform the computation.
- Write the result into the cache.
- Return the result.
On update:
- Update the database.
- Delete the corresponding cache.
- Subsequent reads reload the data.
flowchart TD
A["Read request"] --> B["Query cache"]
B --> C{"Cache hit?"}
C -->|Yes| D["Return cached data"]
C -->|No| E["Read database"]
E --> F["Write cache"]
F --> D
G["Update request"] --> H["Update database"]
H --> I["Delete cache"]
I --> J["Update complete"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#E5F0E9,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
This pattern is simple to implement and suits data that is queried frequently and updated relatively rarely.
But it has a concurrency race: after cache deletion, an old request may re-read stale data and write it back to the cache. Therefore, in systems with high consistency requirements, supplementary mechanisms such as version numbers, delayed double deletion, write validation, or event-driven invalidation are needed.
4.2 Read Through: Read-through
Read Through encapsulates cache reads and underlying data loading logic in the cache layer.
The application only accesses the cache:
Application
|
v
Cache Layer
|
+-- Cache Hit: Return directly
|
+-- Cache Miss: Automatically load data source
Compared with Cache Aside, the application layer does not need to handle the loading process for cache misses itself.
This approach suits scenarios where read logic is managed uniformly through a caching framework.
4.3 Write Through: Write-through
Write Through synchronously updates the underlying data source when writing to the cache.
Application
|
v
Cache Layer
|
+-- Update cache
|
+-- Update database
The advantage is that the update paths for the cache and data source are relatively unified.
The disadvantage is that write latency may increase, and rollback or consistency issues when database updates fail need to be handled.
4.4 Write Back: Write-back caching
Write Back updates the cache first, then asynchronously writes data to persistent storage.
It suits scenarios with frequent writes that can tolerate some persistence delay.
However, if a cache node fails, data that has not yet been written to the database may be lost. Therefore, it needs to be combined with:
- Persistent logs.
- Write queues.
- Failure recovery.
- Retries and compensation.
- A clear data-loss tolerance policy.
For orders, payments, and critical business states, you should not rely solely on a Write Back mechanism that lacks persistence guarantees.
4.5 TTL: Expiration time
TTL (Time To Live) indicates the valid duration of cached data.
For example:
redis.set(
"embedding:doc:1001",
vector_data,
ex=3600
)
This means the cache expires after 3600 seconds.
TTL is suitable for handling:
- Configuration caches.
- Hot query results.
- Temporary session state.
- Data with a clear expiration time.
However, TTL cannot fully guarantee data consistency. Data may change within its valid period, so active invalidation must be used together with TTL according to business requirements.
Recommendations:
- Use short TTLs for data with high real-time requirements.
- Use long TTLs for stable data that changes infrequently.
- Use randomized TTLs for hot data to reduce the risk of mass expiration.
- Use version numbers or event notifications for active invalidation of critical data.
4.6 Cache Breakdown, Penetration, and Avalanche
These are the three most common problems in caching systems.
| Problem | Symptom | Solution |
|---|---|---|
| Cache penetration | Queries for nonexistent data continually hit the underlying storage | Bloom filter, null-value caching, parameter validation |
| Cache breakdown | A hot key expires, and a large number of requests rebuild it simultaneously | Mutex lock, logical expiration, background refresh |
| Cache avalanche | A large number of caches expire simultaneously or the cache service fails | TTL randomization, multi-level caching, rate limiting, degradation |
Cache penetration
For example, an attacker continuously queries nonexistent document IDs. Because these data are not in the cache, requests continually hit the underlying database.
A Bloom Filter can be used to determine in advance whether an object may exist.
Note that a Bloom Filter has false positives and may return “may exist,” but it does not normally produce false negatives, provided that construction and updates are performed correctly.
Cache breakdown
For example, when a popular prompt cache suddenly expires, 10,000 concurrent requests hit the model service at the same time, causing a sharp spike in inference load.
You can use a mutex to control cache rebuilding:
import redis
client = redis.Redis()
def get_or_build(key, builder):
cached = client.get(key)
if cached is not None:
return cached
lock_key = f"lock:{key}"
acquired = client.set(
lock_key,
"1",
nx=True,
ex=10
)
if acquired:
try:
cached = client.get(key)
if cached is not None:
return cached
value = builder()
client.set(key, value, ex=3600)
return value
finally:
client.delete(lock_key)
# Simplified example: production should use bounded waiting or a fallback strategy
raise RuntimeError("Cache rebuild in progress")
In production, releasing a lock requires verifying the identity of the lock holder so that a lock held by another instance is not accidentally deleted; you also need to account for lock expiration, execution timeouts, and rebuild failures.
Cache Avalanche
If a large number of cache entries are all set to the same TTL, they may expire all at once at the same moment.
This can be mitigated by randomizing the expiration time:
import random
ttl = 3600 + random.randint(0, 600)
client.set("cache:key", value, ex=ttl)
This spreads out cache expiration times.
V. Resource Types and Resource Bottlenecks in AI Systems
Caching solves the problem of repeated computation, while resource scheduling solves the problem of how compute resources are allocated.
AI systems typically involve the following resources:
| Resource | Typical Use | Common Bottleneck |
|---|---|---|
| CPU | Tokenization, data preprocessing, request handling | CPU utilization too high |
| GPU | LLM inference, embedding, training | GPU compute saturation |
| GPU memory | Model weights, KV cache, activations | OOM, limited concurrency |
| System memory | Data loading, caching, batching | Insufficient memory, swap thrashing |
| NVMe | Model weights, datasets, indexes | Insufficient I/O throughput |
| Network | Distributed inference, model loading, data transfer | Bandwidth and latency |
| Object storage | Model files, datasets, result files | Request latency, limited throughput |
5.1 GPU Compute Resources
A GPU's main compute capability is determined jointly by factors such as its compute units, memory bandwidth, and memory capacity.
Different tasks have different GPU requirements:
- The prefill phase usually involves a large amount of matrix computation.
- The decode phase typically requires frequent access to model weights and the KV cache.
- Embedding tasks can improve throughput through batching.
- Model training requires storing gradients, optimizer states, and intermediate activations.
Therefore, you cannot judge whether a system is fully utilizing its resources by looking at GPU utilization alone.
For example:
- GPU utilization is very high, but little memory remains, indicating limited room to scale concurrency.
- GPU utilization is very low, yet requests are queuing heavily — the bottleneck may lie in CPU preprocessing, the network, or scheduling.
- Memory usage is high, but compute utilization is low, which may indicate excessive KV cache usage or an unreasonable distribution of request lengths.
5.2 Composition of GPU Memory
GPU memory is typically occupied by the following:
Where:
- M_{weights}: model weights.
- M_{KV}: KV cache.
- M_{activations}: intermediate activations.
- M_{workspace}: temporary operator workspace.
- M_{runtime}: overhead such as CUDA runtime and communication buffers.
Therefore, to determine whether a model can be deployed on a given GPU, you cannot simply compare the size of the model weights against the memory capacity.
For example, even if a 70B-parameter model's weights are compressed via quantization to the point where they fit in memory, you still need to reserve space for the KV cache, operator workspace, and runtime.
5.3 CPU–GPU Collaboration
LLM inference is not just GPU computation.
A complete request may include:
Request intake
-> Text normalization
-> Tokenization
-> Prompt construction
-> GPU inference
-> Detokenization
-> Streaming output
The CPU is responsible for extensive data processing and scheduling, while the GPU handles core tensor computation.
If CPU preprocessing capacity is insufficient, the GPU may not be fully utilized because it is waiting for input data.
Therefore, CPU, GPU, memory, network, and storage resources need to be monitored simultaneously, rather than monitoring only GPU utilization.
VI. Resource Scheduling Architecture for AI Systems
The goal of resource scheduling is to determine which compute node executes a task based on task type, priority, resource requirements, and system load.
A typical scheduling architecture includes:
- API Gateway: Receives requests.
- Scheduler: Task scheduler.
- Task Queue: Task queue.
- Resource Manager: Resource manager.
- Worker: Compute node that executes tasks.
- Model Server: Model inference service.
- Cache Service: Cache service.
- Metrics: Resource and task monitoring.
flowchart TD
A["User request"] --> B["API Gateway"]
B --> C["Task classification and resource estimation"]
C --> D["Scheduler"]
D --> E["Task queue"]
E --> F["Resource manager"]
F --> G{"Are resources sufficient?"}
G -->|Yes| H["Select compute node"]
G -->|No| I["Wait, rate-limit, or reject"]
H --> J["Worker executes task"]
J --> K["Access cache"]
K --> L{"Cache hit?"}
L -->|Yes| M["Reuse cached data"]
L -->|No| N["Call model or compute service"]
N --> O["Update cache"]
M --> P["Save result"]
O --> P
P --> Q["Return result or update task status"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#EAF3EC,stroke:#83A995,color:#315D49
style O fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style P fill:#E5F0E9,stroke:#83A995,color:#315D49
style Q fill:#E5F0E9,stroke:#83A995,color:#315D49
In actual production systems, the Scheduler, Resource Manager, and Worker are usually split into independent services, which collaborate through message queues, a resource registry, and a monitoring system.
VII. Common Resource Scheduling Strategies
7.1 FIFO: First In, First Out
FIFO (First In, First Out) executes tasks in the order in which they enter the queue.
Advantages:
- Simple to implement.
- Clear scheduling rules.
- Less likely to create complex priority management issues.
Disadvantages:
- Long tasks can block short tasks.
- Cannot reflect request priority.
- Not suitable for scenarios where task resource requirements vary greatly.
For example:
Task A: requires 30 minutes
Task B: requires 10 seconds
Task C: requires 5 seconds
With strict FIFO, B and C may need to wait until A finishes executing.
7.2 Priority Scheduling: Priority-Based Scheduling
Assign priorities to different tasks:
P0: Urgent online requests
P1: Normal online requests
P2: Interactive background tasks
P3: Batch offline tasks
P4: Low-priority maintenance tasks
The scheduler prioritizes high-priority tasks.
However, if high-priority requests are always handled first, low-priority tasks may not be executed for a long time, causing starvation.
Common solutions:
- Aging: The longer the waiting time, the higher the priority gradually becomes.
- Quota: Limits the maximum proportion of resources that high-priority tasks can occupy.
- Weighted Fair Queueing: Allocates processing capacity by weight.
- Deadline Scheduling: Schedules based on deadlines.
7.3 Round Robin: Round-Robin Scheduling
Multiple task queues obtain execution opportunities in a round-robin manner.
For example:
Tenant A -> Tenant B -> Tenant C -> Tenant A
The advantage is that it prevents a single tenant from occupying all resources for a long time.
The disadvantage is that the computational costs of different tasks may vary greatly, and simple round-robin cannot guarantee fair resource consumption.
7.4 Weighted Fair Scheduling: Weighted Fair Scheduling
Assign weights to different tenants or task categories.
For example:
| Tenant | Weight | Resource allocation target |
|---|---|---|
| A | 5 | 50% |
| B | 3 | 30% |
| C | 2 | 20% |
When all tenants continuously have tasks, the scheduler can allocate computing resources according to weights.
Real systems also need to consider the actual resource consumption of tasks, rather than simply allocating according to the number of tasks.
For example, one task generates 1000 tokens, while another generates 100 tokens; their computational costs are not the same.
7.5 Resource-aware Scheduling: Resource-Aware Scheduling
Select execution nodes based on task resource requirements.
For example:
| Task | Main resource requirements | Scheduling strategy |
|---|---|---|
| LLM inference | GPU, VRAM | Select GPU nodes with sufficient VRAM |
| Embedding | GPU or CPU | Batch and group by model |
| OCR | GPU, CPU | Select nodes based on model and image size |
| Document parsing | CPU, memory | Schedule to the CPU compute pool |
| RAG index construction | CPU, GPU, disk, network | Staged scheduling |
| Model training | GPU, VRAM, network | Reserve cluster resources |
The key to resource-aware scheduling is not simply choosing idle machines, but determining whether a machine can complete the task within the target time.
7.6 Bin Packing: Bin-Packing Scheduling
Bin Packing is an optimization problem of assigning tasks with different resource requirements to containers with limited resources.
For example:
- GPU node A: 80GB VRAM.
- GPU node B: 80GB VRAM.
- Task 1: estimated to require 30GB.
- Task 2: estimated to require 20GB.
- Task 3: estimated to require 50GB.
The scheduler needs to decide how to place tasks so that resources are fully utilized while avoiding exceeding VRAM capacity.
In its general form, this type of problem is NP-hard, so practical systems usually adopt heuristic algorithms:
- First Fit: Find the first node that can accommodate the task.
- Best Fit: Choose the node with the least remaining resources that can still accommodate the task.
- Worst Fit: Prefer the node with the most remaining resources.
- Spread: Distribute tasks as much as possible to avoid node overload.
- Pack: Concentrate tasks as much as possible to preserve complete resources for other nodes.
Different strategies suit different goals:
- Pack helps free up whole idle nodes.
- Spread helps reduce the risk of single-node failures and hotspots.
- Best Fit helps improve resource space utilization but may increase fragmentation.
8. Resource Scheduling in LLM Inference
LLM inference is the most typical resource scheduling scenario in AI systems.
8.1 Resource Characteristics of Prefill and Decode
LLM inference is mainly divided into two stages:
Prefill Stage
Process the input prompt, compute representations of the input tokens, and build the KV Cache.
Characteristics:
- The number of input tokens is usually large.
- There are many parallel matrix computations.
- There are high requirements for GPU computing power and memory bandwidth.
- Input length directly affects the Prefill workload.
Decode Stage
Generate output token by token.
Characteristics: