1. Why Do We Need a High-Concurrency AI Service Architecture?
As large model applications move from personal chat tools to enterprise-grade applications, AI services need to handle a large number of users, complex requests, and continuously changing compute loads at the same time.
Traditional web services usually rely on CPU, databases, and network I/O as their primary resources, whereas large model inference services also need to manage special resources such as GPUs, model weights, KV Cache, token generation, and dynamic batching.
For example, an AI chat service may simultaneously face:
- A large number of users sending requests at the same time.
- Prompts and generation tasks of varying lengths.
- Call requirements for different models and different versions.
- Long-running streaming responses.
- Burst traffic and periodic traffic peaks.
- Insufficient GPU memory, inference timeouts, and instance failures.
- Resource isolation and cost control across multiple tenants.
If all requests go directly to a single model instance, the system easily runs into queuing, memory exhaustion, slower responses, and even service unavailability.
Therefore, a high-concurrency AI service needs a complete service governance system beyond model inference.
1.1 Core Goals of a High-Concurrency AI Service
| Goal | Description | Key Metrics |
|---|---|---|
| High throughput | Process more requests and tokens per unit time | RPS, Output TPS |
| Low latency | Return the first token and the complete result as quickly as possible | TTFT, TPOT, E2E |
| High availability | A single instance or node failure does not affect the overall service | Availability, Error Rate |
| Scalability | Dynamically add or remove compute resources based on traffic | Scaling time, resource utilization |
| High resource utilization | Reduce GPU idling and GPU memory waste | GPU Utilization, KV Cache Usage |
| Controllable cost | Lower the inference cost per request and per token | Cost per Request, Cost per Token |
| Multi-tenant isolation | Prevent a single tenant from affecting other tenants | Quotas, rate limiting, resource isolation |
1.2 Special Challenges Faced by High-Concurrency AI Services
First, GPU resources are expensive and limited in capacity.
Model weights consume a large amount of GPU memory, and the inference process also requires dynamic allocation of KV Cache. GPUs are limited not only by compute capability but also by memory capacity and memory bandwidth.
Second, there are significant differences between requests.
Requests vary greatly in input length, output length, and generation time. Simply using fixed batching easily leads to wasted resources.
Third, streaming responses hold connections for a long time.
A single request may keep generating for tens of seconds or even several minutes. The server needs to maintain connection state, send heartbeats, and handle cancellation and disconnect/reconnect scenarios.
Fourth, inference load is dynamic.
Burst traffic can push a GPU cluster to its capacity limit within a short period. Relying solely on a fixed number of model instances makes it hard to adapt to all load conditions.
Fifth, the model invocation chain is more complex.
An AI request may pass through a model gateway, prompt processing, RAG retrieval, model inference, tool calls, and result post-processing, and any of these stages can become a bottleneck.
2. Overall Architecture of a High-Concurrency AI Service
Production-grade AI services usually adopt a layered architecture that separates traffic ingress, request governance, model scheduling, inference execution, and infrastructure management.
2.1 Overall Architecture
flowchart TB
A["Client and Business Applications"] --> B["DNS / CDN / WAF"]
B --> C["API Gateway"]
C --> D["Authentication and Quota Management"]
D --> E["Model Gateway"]
E --> F["Request Scheduling and Load Balancing"]
F --> G["AI Service Orchestration Layer"]
G --> H["RAG / Tool Calling / Prompt Processing"]
G --> I["Inference Service Cluster"]
I --> J["GPU Inference Instances"]
J --> K["Model Weights and KV Cache"]
E --> L["Cache Service"]
G --> M["Message Queue and Async Tasks"]
I --> N["Monitoring and Logging"]
E --> N
G --> N
O["Configuration Center and Service Discovery"] --> E
O --> F
O --> I
P["Autoscaling Controller"] --> I
P --> J
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style O fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style P fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
2.2 Core Architecture Layers
| Layer | Core Components | Main Responsibilities |
|---|---|---|
| Access Layer | CDN, WAF, API Gateway | Traffic ingress, security protection, protocol conversion |
| Identity and Governance Layer | Auth, Quota, Rate Limiter | Authentication, quota management, rate limiting |
| Model Gateway Layer | Model Gateway | Model routing, version selection, unified interface |
| Scheduling Layer | Scheduler, Load Balancer | Request queuing, instance selection, dynamic scheduling |
| Application Orchestration Layer | RAG, Agent, Workflow | Retrieval, tool calling, multi-step tasks |
| Inference Layer | Inference Engine | Prefill, Decode, batching |
| Resource Layer | GPU, KV Cache, model storage | Provides compute, VRAM, and model resources |
| Governance and Operations Layer | Metrics, Logs, Tracing | Monitoring, alerting, distributed tracing |
| Control Layer | Service Discovery, Autoscaling | Service discovery, configuration management, elastic scaling |
The key to architecture design is separation of responsibilities.
For example, the API Gateway should not be directly responsible for GPU scheduling; the model gateway should not handle complex model forward computation; and the inference engine should not be responsible for user authentication and billing management.
III. Request Ingress Layer: Handling High-Concurrency Traffic
The request ingress layer is the first gateway through which all AI requests enter the system.
It needs to handle a large number of concurrent connections and complete basic governance before requests reach the inference cluster.
3.1 API Gateway
The API Gateway is the unified API entry point.
Main responsibilities:
- Receive HTTP, HTTPS, or WebSocket requests.
- Perform authentication and permission checks.
- Validate request formats.
- Enforce rate limiting and quota control.
- Handle request timeouts.
- Inject Trace IDs and request context.
- Route to backend model services.
- Handle error responses in a unified way.
Common implementations:
- NGINX.
- Envoy.
- Kong.
- APISIX.
- Cloud provider API Gateway.
For large model services, API Gateway also needs to specifically consider streaming responses.
Ordinary HTTP requests can usually return all at once after the server finishes processing, while streaming requests need to continuously send data to the client.
Therefore, the gateway needs to correctly handle:
- SSE (Server-Sent Events).
- HTTP Chunked Transfer Encoding.
- WebSocket.
- Long connection timeouts.
- Proxy buffering.
- Client disconnections.
- Heartbeats and connection keep-alive.
If the gateway enables inappropriate response buffering, the model may have already generated tokens, but the client may still not receive the content for a long time.
3.2 Request Protocol Design
High-concurrency AI services usually need to support multiple interaction modes.
| Mode | Use Case | Characteristics |
|---|---|---|
| Standard HTTP | Classification, Embedding, short-text generation | Returns once after the request completes |
| SSE | Chat, streaming generation | The server continuously pushes events |
| WebSocket | Real-time voice, bidirectional interaction | Bidirectional communication |
| gRPC | Internal service calls | Efficient RPC communication |
| Asynchronous task API | Long-running tasks, batch inference | Query results by task ID after submitting the task |
For ordinary text chat, SSE is usually sufficient; for scenarios that require continuous transmission of audio, video, or bidirectional control, WebSocket is more suitable.
3.3 Request ID and Idempotency
In high-concurrency environments, requests may be submitted repeatedly due to network timeouts, client retries, or service failures.
If a request has side effects, such as an Agent executing tool calls, writing to a database, or creating a task, repeated execution may cause errors.
Therefore, it is necessary to design:
- Request ID: uniquely identifies a request.
- Idempotency Key: identifies the same business operation.
- Trace ID: correlates the complete call chain.
- Retry Count: records the number of retries.
- Deadline: records the request deadline.
For pure text generation, duplicate requests usually only increase inference cost; for Agent operations with side effects, duplicate execution must be controlled through idempotency mechanisms.
IV. Model Gateway: Unified Model Access and Routing
The Model Gateway is a core component in high-concurrency AI architecture.
It unifies the interfaces of different models, different inference instances, and different providers, providing upper-layer applications with a consistent invocation method.
4.1 Why Is a Model Gateway Needed?
Suppose an AI platform simultaneously deploys:
- General chat models.
- Code generation models.
- Embedding models.
- Reranker models.
- Multimodal models.
- Third-party cloud models.
If every business application directly calls different models, it will generate a large amount of duplicate code and management overhead.
A model gateway can centrally handle:
- Model name and version management.
- Request parameter conversion.
- Model routing.
- Instance selection.
- Timeouts and retries.
- Rate limiting and quotas.
- Access control.
- Usage statistics.
- Unified error handling.
4.2 Model Routing Strategies
The model gateway can select a target model based on different conditions.
| Routing Strategy | Basis | Use Case |
|---|---|---|
| Fixed routing | Specified model name | Stable business |
| Capability-based routing | Task type, model capability | Classification, code, vision |
| Cost-based routing | Token price and budget | Cost-sensitive business |
| Latency-based routing | Estimated response time | Real-time interaction |
| Load-based routing | GPU load, queue length | High-concurrency inference |
| Canary routing | User, traffic percentage | Model version upgrades |
| Failover | Health status | High-availability services |
For example:
- Simple classification requests are routed to small models.
- Complex reasoning requests are routed to large models.
- High-priority requests are routed to low-latency instances.
- When an instance is overloaded, switch to another available instance.
4.3 Model Routing Process
flowchart TD
A["Receive model request"] --> B["Parse model name and task type"]
B --> C["Check tenant permissions and quota"]
C --> D["Get model candidate list"]
D --> E["Check model health status"]
E --> F["Calculate routing score"]
F --> G{"Are there available instances?"}
G -->|"Yes"| H["Select target inference instance"]
G -->|"No"| I["Check fallback or backup models"]
I --> J{"Is fallback allowed?"}
J -->|"Yes"| K["Select backup model"]
J -->|"No"| L["Return service unavailable"]
H --> M["Forward request"]
K --> M
M --> N["Record invocation result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
4.4 Routing Score Design
A composite score can be calculated for each inference instance:
Where:
- L_i: Estimated latency.
- Q_i: Waiting queue length.
- C_i: Resource utilization.
- E_i: Error rate or health risk.
- w_1,w_2,w_3,w_4: Weight coefficients.
Select the healthy instance with the lowest score.
In actual systems, metrics need to be normalized, and model capabilities, GPU type, remaining KV Cache capacity, and request length should be considered.
Instances should not be selected based only on CPU or GPU utilization, because high GPU utilization does not necessarily mean that the instance can no longer accept new requests.
V. Load Balancing and Inference Instance Scheduling
Load balancing is responsible for reasonably distributing requests across different inference instances.
However, load balancing for AI services is significantly different from that for ordinary Web services.
Ordinary Web services can distribute requests mainly based on connection count, CPU load, and response time.
Large model inference also needs to consider:
- Whether the model has already been loaded.
- Whether GPU memory is sufficient.
- Remaining KV Cache capacity.
- Current Decode batch size.
- Waiting queue length.
- Prompt length.
- Estimated output length.
- Resource contention between Prefill and Decode.
5.1 Common Load Balancing Strategies
Round Robin
Distribute requests to different instances in sequence.
Advantages: simple to implement and relatively even distribution.
Disadvantages: cannot perceive GPU load or request complexity.
Weighted Round Robin
Distribute requests according to instance weights.
Suitable for clusters composed of different GPU models or instances of different specifications.
Least Connections
Send requests to instances with fewer current connections.
Effective for ordinary short requests, but long-lived streaming requests may cause the connection count to be inconsistent with the actual GPU workload.
Least Requests
Select instances currently handling fewer requests.
More adaptable to differences in request duration than round robin, but still cannot fully reflect Token computation volume and KV Cache pressure.
Shortest Queue
Prefer instances with shorter waiting queues.
Suitable for inference services with obvious request queuing.
Load-aware Routing
Comprehensively consider metrics such as GPU utilization, KV Cache usage, number of waiting Tokens, and instance health status.
Usually more suitable for production-grade large model inference services.
5.2 Load Balancing for AI Inference Services
flowchart TD
A["Model gateway receives request"] --> B["Get inference instance list"]
B --> C["Check instance health status"]
C --> D["Collect instance runtime metrics"]
D --> E["Calculate instance load score"]
E --> F["Filter out instances that do not meet resource requirements"]
F --> G{"Are there available instances?"}
G -->|"Yes"| H["Select the most suitable instance"]
G -->|"No"| I["Enter waiting queue"]
H --> J["Forward inference request"]
I --> K{"Is queue deadline exceeded?"}
K -->|"No"| B
K -->|"Yes"| L["Return overload error or degrade"]
J --> M["Monitor request execution status"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
5.3 Why Can't We Look Only at GPU Utilization?
Suppose there are two inference instances:
| Metric | Instance A | Instance B |
|---|---|---|
| GPU utilization | 80% | 50% |
| KV Cache usage | 95% | 60% |
| Waiting requests | 10 | 5 |
| Average TTFT | 2s | 500ms |
If we look only at GPU utilization, we might choose Instance B, but actual scheduling also needs to determine:
- Whether Instance B has loaded the target model.
- Whether there is enough KV Cache space.
- Whether the current request is compatible with existing requests.
- Whether KV Cache needs to be transferred across nodes.
- The request's priority and deadline.
Therefore, AI inference load balancing should use multidimensional state rather than a single hardware metric.
6. Inference Engine: The Core Execution Layer for High Concurrency
The inference engine is responsible for actually executing model computation and is the most important compute component of AI services.
Common inference frameworks include:
- vLLM.
- SGLang.
- TensorRT-LLM.
- Hugging Face TGI.
- llama.cpp.
They each have their own characteristics in scheduling policies, KV Cache management, model compatibility, quantization, and distributed execution.
6.1 Core Responsibilities of the Inference Engine
- Load model weights.
- Manage GPU memory.
- Execute Prefill and Decode.
- Manage KV Cache.
- Schedule concurrent requests.
- Perform dynamic batching.
- Support streaming Token output.
- Manage request cancellation and timeouts.
- Handle inference exceptions.
- Provide performance metrics.
6.2 Continuous Batching
Traditional static batching requires waiting for all requests in the entire batch to finish.
Continuous Batching, by contrast, allows requests to be rescheduled after each Decode round is completed.
This way, when a request finishes, its execution slot can be released immediately, and waiting requests can enter the batch.
Main benefits:
- Reduce GPU idle time.
- Increase Token throughput.
- Support requests of different lengths.
- Improve concurrent processing capacity.
But the scheduler still needs to handle issues such as mixing long and short requests, KV Cache capacity, and Prefill contention.
6.3 PagedAttention and KV Cache
KV Cache stores the Key and Value of historical Tokens, avoiding recomputation of the entire context during the Decode phase.
PagedAttention uses a block-based management approach, allocating the KV Cache into different GPU memory Blocks.
Its main value:
- Reduces memory fragmentation.
- Supports dynamic allocation and release.
- Improves KV Cache utilization.
- Supports more concurrent requests.
Continuous Batching and PagedAttention are complementary technologies:
- Continuous Batching solves the request scheduling problem.
- PagedAttention solves the KV Cache memory management problem.
6.4 Separating Prefill and Decode
Prefill and Decode place different demands on GPU resources.
Prefill needs to process longer input sequences and usually has a higher degree of computational parallelism.
Decode needs to generate Tokens continuously and usually focuses more on memory bandwidth and KV Cache access efficiency.
In high-concurrency scenarios, the two phases can be deployed to separate clusters:
- Prefill Worker: responsible for Prompt computation.
- Decode Worker: responsible for continuous generation.
- KV Transfer: responsible for passing context state.
- Scheduler: coordinates request execution.
This allows the two types of resources to be scaled independently, but it requires accounting for the bandwidth, latency, and extra GPU memory overhead of transferring the KV Cache across nodes.
VII. Request Queues and Backpressure Mechanisms
A high-concurrency service cannot accept requests without limit.
When requests arrive faster than the system can process them on a sustained basis, the waiting queue keeps growing, eventually leading to runaway latency and resource exhaustion.
Therefore, queue management and backpressure mechanisms need to be designed.
7.1 Role of the Request Queue
The request queue is used to temporarily hold requests that have not yet obtained inference resources.
Main responsibilities:
- Buffer burst traffic.
- Smooth the request arrival rate.
- Schedule requests by priority.
- Control the level of concurrency.
- Manage timeouts and cancellations.
- Record queueing time.
For online interactive requests, the queue cannot grow without bound.
Usually the following need to be set at the same time:
- Maximum queue length.
- Maximum queueing time.
- Maximum number of in-flight requests.
- Per-tenant queue quota.
- Request deadline.
7.2 Queue Scheduling Strategies
| Strategy | Description | Applicable Scenario |
|---|---|---|
| FIFO | First in, first out | Ordinary interactive requests |
| Priority Queue | Processed by priority | Multi-tenancy and paid tiers |
| Weighted Fair Queueing | Allocate resources by tenant weight | Multi-tenant platforms |
| Deadline-aware | Schedule according to request deadline | Real-time tasks |
| Token-aware | Schedule according to input and output Token budget | Large model inference |
| Length-aware | Schedule according to request length | Mixed long and short requests |
Long requests cannot simply be placed at the tail of the queue.
If long requests are constantly preempted by short ones, starvation can occur.
Therefore a balance must be struck among latency, throughput, and fairness.
7.3 Backpressure
Backpressure means that when downstream processing capacity is insufficient, the pressure is fed back upstream to limit the admission of new requests.
For example:
- The GPU queue has reached its limit.
- KV Cache space is insufficient.
- Inference instances are in an unhealthy state.
- Downstream model services are responding more slowly.
The system can then:
- Pause accepting some requests.
- Return 429 or 503.
- Delay processing of low-priority requests.
- Place requests into a bounded queue.
- Move tasks to an asynchronous queue.
- Enable fallback models or degradation strategies.
7.4 Queue and Backpressure Processing Flow
flowchart TD
A["Receive inference request"] --> B["Validate request"]
B --> C["Check concurrency and queue capacity"]
C --> D{"Is capacity available?"}
D -->|"Yes"| E["Enter scheduling queue"]
D -->|"No"| F["Apply backpressure policy"]
F --> G{"Is waiting allowed?"}
G -->|"Yes"| H["Enter bounded wait queue"]
G -->|"No"| I["Return 429 or 503"]
H --> J{"Is queue deadline exceeded?"}
J -->|"No"| K["Wait for inference resources"]
J -->|"Yes"| L["Return queue timeout"]
K --> E
E --> M["Allocate inference instance"]
M --> N["Run model inference"]
N --> O["Return result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style N fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
8. Cache Architecture: Reducing Duplicate Computation
Caching is an important means of improving the performance of high-concurrency AI services.
However, AI systems contain multiple types of caches, and their stored content, lifecycle, and hit conditions are not the same.
8.1 Cache Types
| Cache Type | Cached Content | Main Benefit |
|---|---|---|
| Response Cache | Complete request result | Avoid duplicate inference |
| Prompt Cache | Computation state related to the Prompt prefix | Reduce duplicate Prefill |
| KV Cache | Key and Value of historical Tokens | Avoid recomputing context |
| Embedding Cache | Text vectors | Reduce duplicate Embedding |
| Retrieval Cache | Retrieval results | Reduce duplicate vector retrieval |
| Model Cache | Model weights and files | Reduce model loading time |
| Tokenization Cache | Tokenization results | Reduce duplicate tokenization |
8.2 Response Cache
Response Cache caches the response result of a complete request.
For example:
If the same user question is submitted repeatedly within a short period, the previous result can be returned directly.
Suitable for:
- Fixed FAQs.
- Static knowledge queries.
- Classification tasks with the same parameters.
- Highly deterministic structured requests.
Not suitable for direct caching:
- Requests that strongly depend on real-time information.
- Requests with different user permissions.
- Generation requests with high randomness.
- Agent requests that need to perform side effects.
The cache key cannot consist only of user text; it also needs to consider:
- Model name and version.
- Prompt template version.
- Sampling parameters.
- User permissions or tenant scope.
- Retrieval data version.
- Tool configuration.
- Output format requirements.
Otherwise, cross-tenant data leakage may occur, or outdated results may be returned.
8.3 Prefix Caching
Prefix Caching is mainly used for prefix reuse in large-model inference.
For example, multiple requests use the same system prompt:
You are a professional technical assistant.
Please follow these response guidelines...
These shared prefixes can reuse the KV Cache when cache conditions are met, reducing duplicate Prefill.
Note that:
- The prefix must satisfy the matching rules of the inference engine.
- The model version and related configuration must be compatible.
- Cache entries may be evicted due to VRAM pressure.
- Prefix Cache is not the same as full response caching.
- A cache hit does not mean the entire request requires no computation.
8.4 Multi-Level Cache Architecture
flowchart TD
A["Receive AI Request"] --> B["Query L1 Local Cache"]
B --> C{"Hit?"}
C -->|"Yes"| D["Return Local Cache Result"]
C -->|"No"| E["Query L2 Distributed Cache"]
E --> F{"Hit?"}
F -->|"Yes"| G["Return Distributed Cache Result"]
F -->|"No"| H["Run RAG or Model Inference"]
H --> I["Write to Cache"]
I --> J["Return Result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
9. Rate Limiting and Quotas: Protecting Expensive GPU Resources
The resource cost of highly concurrent AI services is usually quite high, so you cannot simply rely on the backend GPU to absorb all traffic on its own.
Multi-dimensional rate limiting needs to be enforced at both the ingress and the model scheduling layers.
9.1 Common Rate-Limiting Dimensions
| Dimension | Description |
|---|---|
| IP rate limiting | Limit the request rate from a single source IP |
| User rate limiting | Limit the call volume of a single user |
| Tenant rate limiting | Control the total resource usage of an enterprise or organization |
| API Key rate limiting | Limit the call rate of a given access key |
| Model rate limiting | Limit concurrent requests to a specific model |
| Token rate limiting | Limit input and output token usage |
| GPU quota | Control the inference resources a tenant may occupy |
| Concurrency rate limiting | Limit the number of requests executing simultaneously |
For large model services, using QPS-based rate limiting alone is usually not enough.
Because two requests may have completely different computational costs:
- Request A: 100 input tokens, 50 output tokens.
- Request B: 10,000 input tokens, 4,000 output tokens.
If you bill or rate-limit purely by request count, you cannot accurately control compute resource consumption.
Therefore, a Token Budget can be introduced to control resources based on the input token budget, output token budget, and model cost.
9.2 Common Rate-Limiting Algorithms
Fixed Window
Time is divided into fixed windows, and each window allows at most a certain number of requests.
Advantages: simple to implement.
Disadvantages: burst traffic may occur at window boundaries.
Sliding Window
Counts the number of requests within the most recent time period.
Advantages: limits are smoother.
Disadvantages: requires maintaining more time-window state.
Token Bucket
The system replenishes tokens into a bucket at a fixed rate, and each request consumes a certain number of tokens.
Advantages:
- Supports burst traffic.
- Controls the long-term average rate.
- Allows different costs to be set for different requests.
Leaky Bucket
Requests flow out of the queue at a relatively steady rate.
Advantages: output traffic is smooth.
Disadvantages: burst requests may need to wait, and the queue must have a capacity limit.
9.3 Token Bucket Algorithm
Assume:
- The bucket capacity is 100 tokens.
- 20 tokens are replenished per second.
- A normal request consumes 1 token.
- Long requests to large models consume more tokens according to their budget.
When a request arrives:
- Enough tokens: the request is admitted.
- Not enough tokens: wait for replenishment or reject the request.
For token-based rate limiting, a request's token consumption can be tied to its estimated token count.
Note that output length often cannot be predicted accurately before a request begins; this can be handled through a maximum output token budget, dynamic deduction, and settlement based on actual usage after completion.
9.4 Multi-Tenant Quota Design
Enterprise-grade AI platforms typically need to configure different resource tiers for different tenants.
| Tenant tier | Concurrency limit | Request rate | Token quota |
|---|---|---|---|
| Free | 2 | 5 RPS | Low |
| Basic | 10 | 20 RPS | Medium |
| Pro | 50 | 100 RPS | High |
| Enterprise | Configured per contract | Configured per contract | Configured per contract |
The figures above are examples only; actual values must be determined through GPU capacity planning and business requirements.
The quota system should also support:
- Daily and monthly quotas.
- Burst allowance.
- Independent quotas per model.
- Billing for overages.
- Tenant-level concurrency isolation.
- Real-time quota adjustment.
10. Elastic Scaling: Letting Compute Resources Follow the Load
A large model inference cluster typically consists of expensive GPU instances.
If you always deploy for the highest peak, you incur high idle costs; if you deploy only for the average load, you may be unable to handle requests promptly during traffic spikes.
Therefore, an elastic scaling mechanism needs to be designed.
10.1 Horizontal and Vertical Scaling
Horizontal scaling (Scale Out)
Add more inference instances.
For example:
- Scale from 4 GPU workers to 8.
- Scale from 8 to 16.
Advantages:
- Increases the overall processing capacity of the cluster.
- Spreads out request pressure.
- Supports instance-level fault isolation.
Disadvantages:
- Requires loading model weights.
- Requires additional GPU resources.
- May increase model synchronization and scheduling costs.
Vertical scaling (Scale Up)
Increase the compute resources of a single instance.
For example:
- Use GPUs with more VRAM.
- Use more GPUs for model parallelism.
- Increase the CPU and memory configuration of a single machine.
Advantages: suitable for models that cannot easily be split.
Disadvantages: limited by the hardware specifications of a single machine, and resource costs are higher.
10.2 AI Service Scaling Cannot Look at CPU Alone
Ordinary web services often autoscale based on CPU utilization.
But the bottleneck of an AI inference service may lie in:
- GPU compute utilization.
- GPU VRAM usage.
- KV cache utilization.
- Waiting queue length.
- Request queueing time.
- TTFT.
- Token throughput.
- In-flight requests per instance.
Therefore, it is recommended to scale using a combination of multiple metrics.
10.3 Scaling Metrics
| Metric | Scale-out signal | Scale-in signal |
|---|---|---|
| GPU utilization | Above threshold for an extended period | Low for an extended period |
| KV cache utilization | Approaching the capacity limit | Consistently low usage |
| Waiting queue length | Continuously growing | Empty or low for an extended period |
| P95 TTFT | Persistently above target | Consistently below target |
| Output TPS | Single instance approaching its capacity limit | Resources idle for an extended period |
| Active requests | Exceeding safe concurrency | Below target for an extended period |
10.4 Autoscaling Workflow
flowchart TD
A["Periodically collect cluster metrics"] --> B["Calculate current inference load"]
B --> C["Analyze GPU and KV Cache utilization"]
C --> D["Analyze queue length and TTFT"]
D --> E{"Exceeds the scale-out threshold?"}
E -->|"Yes"| F["Calculate target instance count"]
F --> G["Request GPU resources"]
G --> H["Start inference instance"]
H --> I["Load model and complete warm-up"]
I --> J["Pass health check"]
J --> K["Join service discovery and load balancing"]
E -->|"No"| L{"Meets scale-in conditions?"}
L -->|"No"| A
L -->|"Yes"| M["Select low-load instance"]
M --> N["Stop accepting new requests"]
N --> O["Wait for in-flight requests to complete"]
O --> P["Release GPU instance"]
P --> A
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style N fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style O fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style P fill:#DCEFE5,stroke:#4F8B70,color:#24563F
10.5 Model Loading and Scale-Out Cold Start
Scaling out a large-model instance does not mean it can serve traffic immediately.
New instances usually also need to:
- Request GPU resources.
- Download or mount model weights.
- Load model parameters.
- Initialize CUDA and inference kernels.
- Allocate KV Cache.
- Perform model warm-up.
- Pass health checks.
- Join service discovery.
If the model weights are large, cold start may take a long time.
Therefore, you can adopt:
- A warm instance pool.
- Local caching of model weights.
- Tiered model storage.
- Advance scaling.
- Predictive scaling based on historical traffic.
- Prioritizing retention of hot models.
For businesses with high GPU costs, you also need to strike a balance between the number of reserved instances and cold start risk.
11. Asynchronous Task Architecture: Handling Long-Running AI Tasks
Not all AI requests are suitable for synchronous HTTP calls.
For example:
- Long document parsing.
- Batch data labeling.
- Long-running Agents.
- Large-scale Embedding.
- Multi-step report generation.
- Processing large numbers of images or videos.
These tasks may run for tens of seconds, several minutes, or even longer.
If they keep occupying client HTTP connections, they are prone to gateway timeouts, connection interruptions, and client retries.
11.1 Differences Between Synchronous and Asynchronous Calls
| Dimension | Synchronous Call | Asynchronous Task |
|---|---|---|
| Request method | Wait for the result after initiating | Return immediately after submitting the task |
| Connection occupancy | May last for a long time | Release the connection after submission |
| Task status | Usually reflected through the connection | Query through task status |
| Use cases | Short text generation, real-time Q&A | Long documents, batch processing, Agents |
| Error recovery | Depends on request retries | Can recover through task status and queues |
| Task management | Relatively simple | Requires persistent state and task scheduling |
11.2 Asynchronous Task Architecture
Core components:
- Task API: receives tasks.
- Task Store: saves task status.
- Message Queue: delivers tasks to be executed.
- Worker: executes AI tasks.
- Result Store: saves results.
- Callback or Polling: notifies or queries results.
flowchart TD
A["Client submits AI task"] --> B["Task API validates request"]
B --> C["Generate Task ID"]
C --> D["Save task status"]
D --> E["Send message to task queue"]
E --> F["Worker retrieves task"]
F --> G["Execute RAG / Agent / inference"]
G --> H{"Is the task successful?"}
H -->|"Yes"| I["Save task result"]
H -->|"No"| J{"Is retry allowed?"}
J -->|"Yes"| K["Enter retry queue"]
J -->|"No"| L["Mark task as failed"]
K --> F
I --> M["Update task as completed"]
L --> N["Record failure reason"]
M --> O["Client queries or receives notification"]
N --> O
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style L fill:#F7EAE7,stroke:#C98D7C,color:#754438
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
style O fill:#DCEFE5,stroke:#4F8B70,color:#24563F
11.3 Task Status Design
Asynchronous AI tasks usually need a clear state machine.
| Status | Meaning |
|---|---|
| PENDING | The task has been created and is waiting to be scheduled. |
| QUEUED | The task has entered the queue. |
| RUNNING | The Worker is executing it. |
| RETRYING | The task is waiting to be retried. |
| SUCCEEDED | The task executed successfully. |
| FAILED | The task execution failed. |
| CANCELLED | The task was cancelled. |
| EXPIRED | The task has exceeded its validity period. |
Note the following:
- Task status updates must be concurrency-safe.
- After a Worker restarts, unfinished tasks need to be recovered.
- The message queue may deliver messages more than once.
- Task execution needs idempotency control.
- Retries must be limited in number and time.
- Task results need a retention period.
For Agent tasks with side effects, you also need to consider the idempotency of tool calls and execution compensation.
12. Streaming Response Architecture: Ensuring a Real-Time Interactive Experience
Streaming responses are an important capability that distinguishes large-model services from traditional APIs.
The model does not need to wait until the entire answer has been generated before returning; instead, it can incrementally send generated tokens to the client.
12.1 SSE Streaming Responses
SSE is an HTTP-based server-side event push mechanism.
Typical response format:
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
data: {"token":"Hello"}
data: {"token":","}
data: {"token":" world"}
data: [DONE]
In actual protocols, events can also include:
- Request ID.
- Model name.
- Token usage.
- Error information.
- Finish reason.
- Tool call status.
12.2 Key Issues with Streaming Responses
Connection Lifecycle Management
Long-running generations may occupy a connection for an extended period, so connection timeouts and keep-alive mechanisms need to be configured appropriately.