I. Why AI Services Need Asynchronous Tasks and Queues
Traditional web services usually adopt a synchronous request model: the client sends a request, the server executes the logic, and returns the result after completion.
For ordinary CRUD APIs, this model is usually simple enough. But the execution process of AI services is often more complex.
An AI request may involve:
- Model inference: calling a large language model to generate content.
- RAG retrieval: querying vector databases and full-text search systems.
- Document parsing: extracting text from PDFs, Word documents, and images.
- Multimodal processing: image recognition, speech transcription, and video analysis.
- Agent execution: multi-turn model calls, tool calls, and result validation.
- Batch tasks: batch generation of embeddings, summaries, or classification results.
These tasks usually have the following characteristics:
- Uncertain execution time: Model inference may last several seconds or even several minutes.
- Uneven resource consumption: Different requests vary greatly in the GPU memory, CPU, and network resources they consume.
- Obvious burst traffic: Users may submit a large number of tasks in a short period.
- Complex execution process: Tasks may need to be retried, paused, resumed, or manually intervened.
- Results are not necessarily needed immediately: Users can submit tasks first and query the execution results later.
If all tasks are processed synchronously, the system is prone to request timeouts, thread exhaustion, GPU overload, and cascading failures.
Therefore, three core mechanisms need to be introduced:
| Mechanism | Core responsibility | Problem solved |
|---|---|---|
| Asynchronous tasks | Decouple submission from execution | Blocking by long-running requests |
| Message queues | Buffering, distribution, and scheduling | Traffic bursts and imbalance between production and consumption |
| Traffic control | Limit the load entering the system | Resource overload and service avalanche |
The three are not independent features, but a coordinated set of mechanisms in high-concurrency AI services.
II. Synchronous and Asynchronous Execution
2.1 Synchronous Execution
Synchronous execution means the requester must wait for the task to complete before obtaining the final result.
Typical process:
flowchart TD
A["Client submits request"] --> B["API receives request"]
B --> C["Execute model inference"]
C --> D["Process inference result"]
D --> E["Return final response"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Advantages of the synchronous model:
- Simple to implement.
- Requests and results are naturally linked.
- The client can directly obtain the execution result.
- No additional task state management is required.
But it also has obvious limitations:
- The request connection must remain occupied.
- Long-running inference increases connection and thread resource consumption.
- A client timeout does not mean the server-side task has stopped.
- Under high concurrency, server resources are easily exhausted.
- It is not suitable for complex tasks that take several minutes to execute.
2.2 Asynchronous Execution
Asynchronous execution separates task submission from task processing.
After the client submits a task, the server is only responsible for receiving, validating, persisting, and enqueuing it, and then immediately returns a task ID.
Background workers take tasks from the queue and execute them, and the client queries the status or receives notifications via the task ID.
flowchart TD
A["Client submits task"] --> B["API validates request"]
B --> C["Create task record"]
C --> D["Write task to message queue"]
D --> E["Return Task ID"]
D --> F["Worker fetches task"]
F --> G["Execute AI task"]
G --> H["Save execution result"]
H --> I["Update task status"]
I --> J["Client queries or receives notification"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Asynchronous mode moves the long-running execution work originally handled by the API request thread to a background task system.
It is especially suitable for:
- Long-form text generation.
- Batch Embedding computation.
- Large-scale document parsing.
- Long-running Agent tasks.
- Dataset generation and model evaluation.
- Multimodal processing such as video and audio.
2.3 How to Choose Between Synchronous and Asynchronous Execution
| Dimension | Synchronous Execution | Asynchronous Execution |
|---|---|---|
| Return behavior | Returns the final result | Returns the task ID first |
| Connection usage | Keeps waiting | Can be released after submission |
| Execution time | Suitable for shorter tasks | Suitable for long-running tasks |
| State management | Relatively simple | Requires task state management |
| Failure recovery | Usually relies on request retries | Can recover through queues and persistence |
| System complexity | Lower | Higher |
| Typical scenarios | Short-text Q&A, real-time conversation | Batch inference, document processing, Agent |
Note that asynchronous does not necessarily mean a message queue must be used. Background threads, process pools, and task schedulers can also implement asynchronous execution. However, when the system needs cross-service communication, persistence, peak shaving and load leveling, and failure recovery, a message queue is usually more appropriate.
III. Core Model of Asynchronous Tasks
Asynchronous tasks are not just about running functions in the background; they also require clearly defined task lifecycle, state, execution ownership, and result management.
3.1 Basic Composition of a Task
A complete task object usually includes:
| Field | Purpose |
|---|---|
| task_id | Unique task identifier |
| task_type | Task type |
| payload | Task input data or data reference |
| status | Current task status |
| priority | Task priority |
| created_at | Creation time |
| started_at | Execution start time |
| finished_at | Completion time |
| retry_count | Number of retries performed |
| max_retries | Maximum number of retries |
| timeout | Task timeout |
| idempotency_key | Idempotency key |
| result_ref | Execution result storage location |
| error | Error message |
Example task object:
```json { "task_id": "task_20261122_001", "task_type": "document_analysis", "payload": { "document_id": "doc_001" }, "status": "queued", "priority": 5, "retry_count": 0, "max_retries": 3, "timeout": 600, "idempotency_key": "document_analysis_doc_001_v1" } ```For large files and multimodal tasks, it is not advisable to put the full file content directly into the message queue.
A more reasonable approach is:
- Store the file in object storage.
- Keep the task metadata in the database.
- Pass only the task ID, file address and necessary parameters through the message queue.
This avoids oversized message bodies and reduces the network transfer and storage pressure on the queue.
3.2 Task Lifecycle
A task usually has to go through multiple states.
| State | Meaning |
|---|---|
| Pending | The task has been created but has not yet entered the executable queue |
| Queued | It has entered the queue and is waiting to be picked up by a Worker |
| Running | It is being executed |
| Retry Waiting | Execution failed and it is waiting to be executed again |
| Succeeded | Execution succeeded |
| Failed | The failure condition has been reached; it cannot continue |
| Cancelled | Cancelled by the user or the system |
| Timeout | The allowed execution time has been exceeded |
| Paused | Execution is paused, waiting to resume |
State transition diagram:
flowchart TD
A["Pending"] --> B["Queued"]
B --> C["Running"]
C --> D{"Execution Result"}
D -->|"Success"| E["Succeeded"]
D -->|"Retryable Failure"| F["Retry Waiting"]
D -->|"Non-retryable Failure"| G["Failed"]
D -->|"Timeout"| H["Timeout"]
F --> I{"Retry Limit Exceeded?"}
I -->|"No"| B
I -->|"Yes"| G
B --> J["Cancelled"]
C --> K["Cancelled"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
The state design must clearly define which transitions are legal.
For example:
- Succeeded must not become Running again.
- Cancelled should not be automatically re-executed by ordinary Workers.
- Retry Waiting must be constrained by the maximum retry count.
- Timeout and Failed can be defined as distinct terminal states depending on the business requirements.
- If a Worker crashes while a task is Running, there must be a recovery mechanism.
Task state must not rely solely on the message state in the message queue. The message queue is responsible for delivering tasks, while the database or persistent task store is responsible for maintaining business state. The two must work together through a consistency strategy.
3.3 The Difference Between Task State and Message State
The two are often confused.
| Task Status | Message Status |
|---|---|
| Describes business execution progress | Describes message delivery and consumption progress |
| Usually stored in a database | Usually managed by the message queue |
| Includes Running, Succeeded, Failed | Includes Ready, Unacked, Acknowledged, etc. |
| Oriented toward business queries | Oriented toward message transmission and consumption |
For example, the Worker has already fetched the message from the queue, but model inference has not yet finished:
- The message may be in an unacknowledged state.
- The business task is already in the Running state.
If the Worker executes successfully and acknowledges the message, but the task result is not reliably saved, the message may end up consumed while the business result is lost.
Therefore, the ordering among task status updates, result persistence, and message acknowledgment needs to be designed.
IV. The Core Role of Message Queues
A message queue is the intermediate buffer layer in an asynchronous task system.
It decouples task producers from task consumers, so that task submission speed and task processing speed need not match exactly.
4.1 Producers and Consumers
A typical architecture involves three roles:
- Producer: the task producer, responsible for creating and sending tasks.
- Broker: the message broker, responsible for storing, routing, and delivering messages.
- Consumer: the task consumer, responsible for fetching messages and executing tasks.
flowchart LR
A["API Service"] --> B["Producer"]
B --> C["Message Broker"]
C --> D["Worker 1"]
C --> E["Worker 2"]
C --> F["Worker 3"]
D --> G["Model Inference Service"]
E --> G
F --> H["Document Processing Service"]
G --> I["Result Storage"]
H --> I
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
4.2 What Problems Do Message Queues Solve
First, service decoupling.
The API service does not need to know which Worker actually executes a task, nor does it need to maintain connections to Workers.
Second, peak shaving and valley filling.
When a large number of requests arrive in a short period, the queue can temporarily hold the tasks and let consumers execute them gradually according to their own processing capacity.
Third, horizontal scaling.
Worker instances can be added based on the task backlog to increase overall processing capacity.
Fourth, fault isolation.
When one consumer fails, other consumers can continue processing tasks, and the queue can retain messages that have not yet been completed.
Fifth, task distribution.
Different types of tasks can go into different queues and be handled by different Worker clusters.
4.3 Common Message Queues
| Technology | Characteristics | Common Use Cases |
|---|---|---|
| Redis Streams | Lightweight, supports consumer groups and message acknowledgment | Small and medium-scale asynchronous tasks |
| RabbitMQ | Rich routing capabilities, supports acknowledgment mechanisms | Business tasks and complex message routing |
| Apache Kafka | High throughput, persistence, mature partitioning | Large-scale event streams and data processing |
| Apache Pulsar | Multi-tenancy, tiered storage, decoupling of messaging and compute | Large-scale distributed messaging systems |
| Amazon SQS | Managed queues, lower operational burden | Cloud-native asynchronous tasks |
When choosing a message queue, you cannot look only at throughput; you also need to consider:
- Message persistence capability.
- Message acknowledgment mechanism.
- How consumers recover after failure.
- Message ordering requirements.
- Support for delayed messages and dead-letter queues.
- Consumer group and partition scaling capability.
- Operational cost and observability.
4.4 Queue Models
Point-to-Point Queue
A message is usually successfully processed by one consumer, which suits task distribution.
flowchart TD
A["Task Producer"] --> B["Task Queue"]
B --> C["Worker A"]
B --> D["Worker B"]
B --> E["Worker C"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
Publish/Subscribe (Pub/Sub)
A single event can be consumed separately by multiple subscribers, making it suitable for event broadcasting.
flowchart TD
A["Event Producer"] --> B["Event Topic"]
B --> C["Subscriber A"]
B --> D["Subscriber B"]
B --> E["Subscriber C"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
Note that within the same Kafka consumer group, a specific message in a partition is usually processed by one consumer, while different consumer groups can independently consume data from the same topic. This is not exactly the same as simple broadcast-style Pub/Sub.
V. Queue Backlog and Peak Shaving / Valley Filling
One of the most important benefits of a message queue is to provide buffering when production and consumption rates are inconsistent.
5.1 Production Rate and Consumption Rate
Assume:
- 100 tasks are submitted per second.
- Each Worker processes 5 tasks per second.
- There are currently 10 Workers.
Overall processing capacity:
Production rate:
Queue backlog growth rate:
If this continues for 60 seconds:
The system will accumulate a backlog of about 3000 tasks.
A queue can only buffer traffic temporarily; it cannot infinitely absorb requests that exceed processing capacity.
5.2 Queue Length and Stability
Definitions:
- \lambda: task arrival rate.
- \mu: processing rate of a single consumer.
- c: number of consumers.
- Q: number of pending tasks in the queue.
The theoretical processing capacity of the system:
When:
The system can usually process tasks stably, but processing-time fluctuations, consumer failures, and resource contention must also be considered.
When:
The queue will keep growing.
When:
The system is in a critical state, and any short-term fluctuation may cause a backlog.
Therefore, production systems should not run for long periods at close to 100% of theoretical processing capacity.
5.3 Utilization and Queuing Delay
Define system utilization:
The closer \rho is to 1, the more likely queuing delay is to increase significantly.
For a simple M/M/1 queuing model:
where W is the average time spent in the system, including waiting and service time.
This formula applies only to specific idealized queuing assumptions. Real AI systems often have complex characteristics such as uneven task processing-time distributions, priority scheduling, and resource contention, so it cannot be used directly to accurately predict production latency.
In engineering practice, what matters more is continuous monitoring:
- Queue length.
- Wait time of the oldest message.
- Task arrival rate.
- Task completion rate.
- Worker busy ratio.
- P95/P99 queuing delay.
5.4 Estimating Queue Drain Time
If there are 10,000 tasks in the queue:
- Total Worker processing capacity: 200 tasks/s.
- New task arrival rate: 50 tasks/s.
Net drain rate:
Queue drain time:
This can be used to estimate the system recovery time.
However, if new tasks keep arriving, the drain rate must be computed as consumption capacity minus arrival rate, rather than simply using the consumer's total processing capacity.
6. Core Techniques of Flow Control
The goal of flow control is not simply to reject requests, but to let the system maintain predictable latency and stability as load changes.
Common techniques include:
- Rate limiting.
- Concurrency limiting.
- Backpressure.
- Circuit breaker.
- Degradation.
- Priority scheduling.
- Load balancing.
Each of these techniques controls pressure at a different layer.
6.1 Rate Limiting
Rate limiting caps the number of requests permitted to enter the system per unit of time.
For example:
- Each user may submit at most 10 requests per second.
- Each API key may submit at most 1000 tasks per minute.
- The service as a whole may receive at most 500 tasks per second.
Common rate limiting algorithms are as follows.
Fixed Window
Time is divided into fixed intervals, and each interval is counted independently.
For example, 100 requests are allowed per minute.
Advantages:
- Simple to implement.
- Low storage overhead.
- Suitable for basic API quota control.
Disadvantages:
- Bursts can occur at window boundaries.
For example:
- 100 submissions at 10:00:59.
- Another 100 submissions at 10:01:00.
In a short span of time, the system may actually receive 200 requests.
Sliding Window
Limiting is based on the number of requests within the most recent period of time.
It can be implemented with a sliding log or a sliding counter.
The advantage is that it is smoother than the fixed window, but a sliding log incurs higher storage and computation overhead.
Token Bucket
Tokens are generated at a fixed rate, and the bucket holds at most a certain number of tokens.
Each request must consume a token in order to pass.
Core parameters:
- r: token generation rate.
- B: bucket capacity.
- T: number of tokens.
- \Delta t: elapsed time.
Token update:
When a request arrives, if enough tokens are available, a token is consumed and the request is admitted; otherwise it is rejected or made to wait.
flowchart TD
A["Request arrives"] --> B["Check token bucket"]
B --> C{"Enough tokens?"}
C -->|"Yes"| D["Consume token"]
D --> E["Admit request"]
C -->|"No"| F{"Waiting allowed?"}
F -->|"Yes"| G["Enter wait queue"]
F -->|"No"| H["Return rate-limit response"]
G --> I["Wait for token refill"]
I --> B
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
The token bucket allows short bursts, but the long-term average rate is bounded by the token generation rate.
Leaky Bucket
Requests enter a buffer and flow out at a fixed rate.
It places more emphasis on smoothing the output rate and is well suited to limiting the processing rate of downstream services.
6.2 Comparison of Rate Limiting Algorithms
| Algorithm | Bursts Allowed | Main Features | Typical Scenarios |
|---|---|---|---|
| Fixed window | Allows concentrated requests within the window | Simple to implement | Basic quotas |
| Sliding window | Relatively smooth | Precise time-range control | API rate limiting |
| Token bucket | Allows limited bursts | Controls long-term average rate | API gateway |
| Leaky bucket | Relatively stable output rate | Smooths traffic | Downstream resource protection |
For AI services, rate limiting based solely on request count is often insufficient.
Because the cost of different requests can vary greatly:
- A short Q&A with 100 tokens.
- A long-context request with 100,000 tokens.
- A single multi-turn Agent execution.
- A batch document Embedding job.
Although each of these requests counts as only one API call, their actual resource consumption is completely different.
Therefore, traffic control needs to be introduced along the token and resource dimensions.
6.3 Token-level rate limiting
LLM services can limit request cost based on token count.
Common dimensions:
- Input token count.
- Output token count.
- Total input and output tokens.
- Tokens per minute (TPM).
- Tokens per second (TPS).
- Token quota for a user or tenant.
For example:
| User tier | RPM | TPM |
|---|---|---|
| Free | 10 | 20,000 |
| Pro | 60 | 100,000 |
| Enterprise | 300 | 1,000,000 |
Here, RPM stands for Requests Per Minute and TPM stands for Tokens Per Minute; the values are examples only.
Token-level rate limiting requires special attention:
- Input tokens can be estimated before inference using a tokenizer.
- The number of output tokens usually cannot be accurately determined before inference.
- A budget can be reserved based on max_tokens.
- Settle based on actual usage after execution completes.
- Separate limits should be set for ultra-long-context requests.
For streaming generation, the token budget can also be checked continuously during generation to prevent a task from exceeding the user's quota.
6.4 Concurrency Control (Concurrency Limiting)
Rate limiting controls how many requests are allowed to enter per unit time, while concurrency control limits the number of tasks executing simultaneously.
Suppose:
- 100 requests are allowed to enter per second.
- Each request takes an average of 10 seconds to execute.
If concurrency is not limited, theoretically about 1,000 requests could be executing simultaneously.
But AI services have limited GPU memory and inference capacity, so concurrency cannot be increased indefinitely.
Concurrency limits can be configured by different resource dimensions:
| Limit dimension | Example |
|---|---|
| Global concurrency | Execute at most 200 tasks simultaneously |
| Per-user concurrency | At most 5 tasks per user |
| Per-model concurrency | At most 64 concurrent requests for a model |
| GPU concurrency | Limit inference requests per GPU |
| Task-type concurrency | At most 20 document parsing tasks |
| Per-tenant concurrency | Independent resource quota per tenant |
Note that the concurrency count is not equal to the actual GPU batch size.
In LLM inference services, multiple requests can be dynamically combined by a continuous batching scheduler. What actually affects GPU resources is request length, KV Cache usage, batch token count, and model execution characteristics.
6.5 Backpressure
Backpressure means that when downstream processing capacity is insufficient, pressure is fed back to upstream to prevent upstream from continuing to produce tasks without limit.
For example:
- Worker processing speed decreases.
- Queue backlog continues to grow.
- Queue wait time exceeds the threshold.
- The system notifies the API to reduce its intake rate.
- The API delays submission, rejects new tasks, or lowers task priority.
Backpressure is not simply rate limiting.
Rate limiting usually restricts traffic according to preset rules, whereas backpressure emphasizes dynamically adjusting upstream traffic based on downstream real-time processing capacity.
flowchart TD
A["Client submits task"] --> B["API service"]
B --> C["Task queue"]
C --> D["Worker cluster"]
D --> E["Downstream model service"]
E --> F{"Is processing capacity sufficient"}
F -->|"Yes"| G["Consume normally"]
F -->|"No"| H["Generate backpressure signal"]
H --> I["Reduce production rate"]
I --> J["Delay acceptance or reject requests"]
J --> B
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
Common backpressure strategies:
- Pause production when the queue reaches a threshold.
- Reduce upstream concurrency when consumers process more slowly.
- Reject low-priority tasks when the queue backlog becomes severe.
- Return a queued status for tasks that can be delayed.
- Return an overload response directly for requests that cannot wait.
6.6 Circuit Breaker
When a downstream model service keeps failing, continuing to send requests only wastes resources.
Based on the failure rate or timeout conditions, the circuit breaker temporarily stops sending requests to the faulty service.
It usually has three states:
- Closed: requests pass through normally while failures are being counted.
- Open: the failure threshold is reached, so requests are temporarily rejected.
- Half-Open: after waiting a while, a small number of probe requests are let through.
stateDiagram-v2
[*] --> Closed
Closed --> Open: failure rate exceeds threshold
Open --> HalfOpen: circuit-breaker wait period ends
HalfOpen --> Closed: probe request succeeds
HalfOpen --> Open: probe request fails
A circuit breaker can protect service dependencies, but it needs to be configured sensibly:
- Failure rate threshold.
- Statistics window.
- Minimum number of request samples.
- Circuit-breaker duration.
- Number of probe requests in the half-open state.
If a downstream service is only briefly slower, it does not necessarily need to be tripped immediately. The decision can be made jointly with timeouts, concurrency limits, and load conditions.
6.7 Degradation
When system resources are insufficient, some service capabilities can be deliberately reduced.
Common degradation strategies for AI services:
| Failure or pressure | Degradation strategy |
|---|---|
| High-quality model overloaded | Switch to a lightweight model |
| Insufficient GPU resources | Delay execution of non-real-time tasks |
| Long-context overload | Limit the maximum input length |
| RAG retrieval anomaly | Answer using cached results or the base model |
| Agent tools unavailable | Return partial results or stop tool calls |
| Severe queue backlog | Pause low-priority tasks |
| Result generation timeout | Return the content generated so far or the task status |
Degradation must have clear business boundaries.
For example, when a user requires strict JSON output, you cannot casually switch to a model that lacks structured output capability.
7. Queue Tiering Design in AI Services
Task types in AI services vary widely, so it is usually best not to put all tasks into the same queue.
7.1 Why Multiple Queues Are Needed
Suppose the system has three kinds of tasks:
- Real-time conversation: requires low latency.
- Document parsing: takes a long time to execute.
- Batch embedding: throughput comes first.
If all three share a single FIFO queue, the following can happen:
- A large number of document parsing tasks occupy the queue.
- Real-time conversation tasks line up behind long tasks.
- User waiting time increases noticeably.
- GPU and CPU resources cannot be allocated according to business goals.
This is known as head-of-line blocking.
A more reasonable approach is to split queues by task type and priority.
flowchart TD
A["Task Entry"] --> B["Task Classifier"]
B --> C["Real-time Task Queue"]
B --> D["Normal Task Queue"]
B --> E["Batch Task Queue"]
C --> F["Real-time Inference Worker"]
D --> G["General Worker"]
E --> H["Batch Worker"]
F --> I["GPU Inference Resource Pool"]
G --> J["Shared Compute Resource Pool"]
H --> I
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
7.2 Scheduling by Priority
Common priorities:
| Priority | Task Type | Scheduling Strategy |
|---|---|---|
| P0 | System-critical tasks | Execute first |
| P1 | Real-time interactive requests | Low queuing latency |
| P2 | Normal user tasks | Normal scheduling |
| P3 | Batch processing tasks | Use idle resources |
| P4 | Offline analysis tasks | Execute during off-peak periods |
However, simple strict-priority scheduling can prevent low-priority tasks from being executed for a long time.
This is called starvation.
Solutions include:
- Priority aging: The longer a task waits, the more its priority gradually increases.
- Quota scheduling: Reserve processing capacity for each type of task.
- Weighted fair queuing: Allocate execution opportunities according to weights.
- Time-slice round-robin: Different task queues take turns receiving resources.
- Maximum waiting time: Raise priority after exceeding a threshold.
7.3 Resource Pool Isolation
Multiple queues do not necessarily mean complete physical resource isolation, but critical tasks should have explicit resource guarantees.
For example:
- Real-time inference uses an independent GPU resource pool.
- Offline batch processing uses the remaining GPU resources.
- Document parsing uses an independent CPU Worker.
- Embedding uses a dedicated batch processing service.
- Agent tool calls use an independent concurrency quota.
This reduces interference from long-running tasks with real-time tasks.
8. Reliability Design for Asynchronous Tasks
Queues can improve system resilience, but they do not automatically guarantee that tasks are executed only once or that tasks will never be lost.
Reliability must be designed at multiple levels, including message delivery, task execution, result persistence, and recovery mechanisms.
8.1 At-Least-Once Delivery
At-least-once delivery means messages may be delivered repeatedly, but as long as the system continues to recover, messages ultimately have a chance to be processed.
For example:
- Worker retrieves the message.
- Worker completes model inference.
- Worker crashes before acknowledging the message.
- The message re-enters a consumable state.
- Another Worker processes it again.
In this case, the same task may be executed twice.
Therefore, at-least-once delivery requires business processing to be idempotent.
8.2 At-Most-Once Delivery
At-most-once delivery means a message is delivered at most once, but it may be lost due to failures.
If a Worker acknowledges the message immediately after retrieving it and then executes the task:
- Worker retrieves the message.
- The message is acknowledged.
- The Worker crashes during execution.
- The message will not be delivered again.
This pattern may cause task loss.
8.3 Exactly-Once
Exactly-once is a concept that is easily misunderstood.
Some messaging systems can provide transactional or exactly-once processing semantics within a specific scope, but this does not mean that all side effects—such as external model inference, database writes, file storage, and tool calls—are naturally executed only once.
For example:
- The model has completed one paid inference.
- The Worker crashes before saving the result.
- The message is redelivered.
- The model is executed again.
Even if the messaging system has a transaction mechanism, it may not be able to roll back the first external model call.
Therefore, engineering practice commonly adopts:
At-least-once delivery + business idempotency + state persistence + deduplication mechanism.
8.4 Idempotency Design
Idempotency means that repeated execution of the same request does not produce duplicate business results.
Common methods:
Unique Task ID
Generate a unique ID for each business task.
Idempotency Key
Generate a stable idempotency identifier based on business parameters.
For example:
document_summary:doc_001:version_3
Database Unique Constraint
Create a unique index on the idempotency key to prevent duplicate result records.
Status Check
Before a Worker executes, it checks the task status first. If the task has already succeeded, it does not process it again.
Result Overwrite Protection
Use a version number or conditional update to prevent an old task from overwriting a new result.
Note that idempotency cannot rely only on an in-memory set in the Worker. Because after the Worker restarts, the in-memory state is lost, deduplication information must be persisted.
8.5 Message Acknowledgment Mechanism
Message acknowledgment (ACK) determines whether a message can transition from pending to completed.
Common strategies:
- Automatic acknowledgment: automatically acknowledge after the message is delivered; simple, but with a higher risk of task loss.
- Manual acknowledgment: acknowledge after the task is processed successfully; suitable for reliable tasks.
- Failure rejection: reject the message when processing fails.
- Requeue: requeue on temporary errors.
- Dead-letter transfer: move to the dead-letter queue when processing cannot continue.
For long-running AI tasks, it is usually necessary to properly configure the message visibility timeout, consumption acknowledgment timeout, or message lease.
If task execution time may exceed the message lease, the lease needs to be renewed periodically to prevent the task from being acquired again by another Worker while it is still running.
8.6 Transactional Messaging and the Outbox Pattern
A typical problem in asynchronous systems:
The business database write succeeds, but message sending fails.
For example:
- Create a task record.
- The database transaction commits successfully.
- The service sends a message to the queue.
- The message queue is temporarily unavailable.
- There is a task in the database, but no message in the queue.
This causes the task to never be executed.
Another case is that the message is sent successfully, but the database transaction rolls back, and the consumer receives a task that does not exist.
The Outbox Pattern can solve this kind of dual-write consistency problem.
Core idea:
- Write the business task record in the same database transaction.
- At the same time, write to the Outbox event table.
- A separate publisher scans the Outbox.
- Send the event to the message queue.
- After successful sending, mark the Outbox event as published.
- If publishing fails, continue retrying later.
flowchart TD
A["Receive task request"] --> B["Begin database transaction"]
B --> C["Write task record"]
C --> D["Write Outbox event"]
D --> E["Commit database transaction"]
E --> F["Outbox publisher scans events"]
F --> G["Send message to queue"]
G --> H{"Send successful"}
H -->|"Yes"| I["Mark event as published"]
H -->|"No"| J["Keep event and retry"]
J --> F
I --> K["Worker consumes task"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D
Outbox solves the problem of reliable publishing between the database and the message queue, but duplicate publishing still needs to be handled, so consumers still need idempotency.
9. Retries, Backoff, and Dead-Letter Queues
In AI services, failures are the norm. Failures need to be classified into different types, rather than immediately retrying on any exception.
9.1 Error Classification
| Error Type | Whether Retry Is Recommended | Handling Method |
|---|---|---|
| Network timeout | Usually yes | Delayed retry |
| Model service temporarily unavailable | Yes | Exponential backoff |
| HTTP 429 | Yes | Respect Retry-After |
| HTTP 500, 503 | Usually yes | Limit retry count |
| Parameter validation failure | No | Fail immediately |
| Unsupported file format | No | Return business error |
| Insufficient permissions | No | Terminate task |
| Token exceeds model limit | Should not retry as-is | Truncate or reprocess |
| GPU OOM | Depends | Reduce concurrency or switch resources |
| Task timeout | Depends on task semantics | Decide after checking execution status |