Back to series
· 19 min read

Asynchronous Tasks, Queues, and Flow Control


I. Why AI Services Need Asynchronous Tasks and Queues

Traditional web services usually adopt a synchronous request model: the client sends a request, the server executes the logic, and returns the result after completion.

For ordinary CRUD APIs, this model is usually simple enough. But the execution process of AI services is often more complex.

An AI request may involve:

  • Model inference: calling a large language model to generate content.
  • RAG retrieval: querying vector databases and full-text search systems.
  • Document parsing: extracting text from PDFs, Word documents, and images.
  • Multimodal processing: image recognition, speech transcription, and video analysis.
  • Agent execution: multi-turn model calls, tool calls, and result validation.
  • Batch tasks: batch generation of embeddings, summaries, or classification results.

These tasks usually have the following characteristics:

  1. Uncertain execution time: Model inference may last several seconds or even several minutes.
  2. Uneven resource consumption: Different requests vary greatly in the GPU memory, CPU, and network resources they consume.
  3. Obvious burst traffic: Users may submit a large number of tasks in a short period.
  4. Complex execution process: Tasks may need to be retried, paused, resumed, or manually intervened.
  5. Results are not necessarily needed immediately: Users can submit tasks first and query the execution results later.

If all tasks are processed synchronously, the system is prone to request timeouts, thread exhaustion, GPU overload, and cascading failures.

Therefore, three core mechanisms need to be introduced:

Mechanism Core responsibility Problem solved
Asynchronous tasks Decouple submission from execution Blocking by long-running requests
Message queues Buffering, distribution, and scheduling Traffic bursts and imbalance between production and consumption
Traffic control Limit the load entering the system Resource overload and service avalanche

The three are not independent features, but a coordinated set of mechanisms in high-concurrency AI services.

II. Synchronous and Asynchronous Execution

2.1 Synchronous Execution

Synchronous execution means the requester must wait for the task to complete before obtaining the final result.

Typical process:

flowchart TD
    A["Client submits request"] --> B["API receives request"]
    B --> C["Execute model inference"]
    C --> D["Process inference result"]
    D --> E["Return final response"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Advantages of the synchronous model:

  • Simple to implement.
  • Requests and results are naturally linked.
  • The client can directly obtain the execution result.
  • No additional task state management is required.

But it also has obvious limitations:

  • The request connection must remain occupied.
  • Long-running inference increases connection and thread resource consumption.
  • A client timeout does not mean the server-side task has stopped.
  • Under high concurrency, server resources are easily exhausted.
  • It is not suitable for complex tasks that take several minutes to execute.

2.2 Asynchronous Execution

Asynchronous execution separates task submission from task processing.

After the client submits a task, the server is only responsible for receiving, validating, persisting, and enqueuing it, and then immediately returns a task ID.

Background workers take tasks from the queue and execute them, and the client queries the status or receives notifications via the task ID.

flowchart TD
    A["Client submits task"] --> B["API validates request"]
    B --> C["Create task record"]
    C --> D["Write task to message queue"]
    D --> E["Return Task ID"]
    D --> F["Worker fetches task"]
    F --> G["Execute AI task"]
    G --> H["Save execution result"]
    H --> I["Update task status"]
    I --> J["Client queries or receives notification"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Asynchronous mode moves the long-running execution work originally handled by the API request thread to a background task system.

It is especially suitable for:

  • Long-form text generation.
  • Batch Embedding computation.
  • Large-scale document parsing.
  • Long-running Agent tasks.
  • Dataset generation and model evaluation.
  • Multimodal processing such as video and audio.

2.3 How to Choose Between Synchronous and Asynchronous Execution

Dimension Synchronous Execution Asynchronous Execution
Return behavior Returns the final result Returns the task ID first
Connection usage Keeps waiting Can be released after submission
Execution time Suitable for shorter tasks Suitable for long-running tasks
State management Relatively simple Requires task state management
Failure recovery Usually relies on request retries Can recover through queues and persistence
System complexity Lower Higher
Typical scenarios Short-text Q&A, real-time conversation Batch inference, document processing, Agent

Note that asynchronous does not necessarily mean a message queue must be used. Background threads, process pools, and task schedulers can also implement asynchronous execution. However, when the system needs cross-service communication, persistence, peak shaving and load leveling, and failure recovery, a message queue is usually more appropriate.

III. Core Model of Asynchronous Tasks

Asynchronous tasks are not just about running functions in the background; they also require clearly defined task lifecycle, state, execution ownership, and result management.

3.1 Basic Composition of a Task

A complete task object usually includes:

Field Purpose
task_id Unique task identifier
task_type Task type
payload Task input data or data reference
status Current task status
priority Task priority
created_at Creation time
started_at Execution start time
finished_at Completion time
retry_count Number of retries performed
max_retries Maximum number of retries
timeout Task timeout
idempotency_key Idempotency key
result_ref Execution result storage location
error Error message

Example task object:

```json { "task_id": "task_20261122_001", "task_type": "document_analysis", "payload": { "document_id": "doc_001" }, "status": "queued", "priority": 5, "retry_count": 0, "max_retries": 3, "timeout": 600, "idempotency_key": "document_analysis_doc_001_v1" } ```

For large files and multimodal tasks, it is not advisable to put the full file content directly into the message queue.

A more reasonable approach is:

  • Store the file in object storage.
  • Keep the task metadata in the database.
  • Pass only the task ID, file address and necessary parameters through the message queue.

This avoids oversized message bodies and reduces the network transfer and storage pressure on the queue.

3.2 Task Lifecycle

A task usually has to go through multiple states.

State Meaning
Pending The task has been created but has not yet entered the executable queue
Queued It has entered the queue and is waiting to be picked up by a Worker
Running It is being executed
Retry Waiting Execution failed and it is waiting to be executed again
Succeeded Execution succeeded
Failed The failure condition has been reached; it cannot continue
Cancelled Cancelled by the user or the system
Timeout The allowed execution time has been exceeded
Paused Execution is paused, waiting to resume

State transition diagram:

flowchart TD
    A["Pending"] --> B["Queued"]
    B --> C["Running"]
    C --> D{"Execution Result"}
    D -->|"Success"| E["Succeeded"]
    D -->|"Retryable Failure"| F["Retry Waiting"]
    D -->|"Non-retryable Failure"| G["Failed"]
    D -->|"Timeout"| H["Timeout"]
    F --> I{"Retry Limit Exceeded?"}
    I -->|"No"| B
    I -->|"Yes"| G
    B --> J["Cancelled"]
    C --> K["Cancelled"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#F7EAE7,stroke:#C98D7C,color:#754438

The state design must clearly define which transitions are legal.

For example:

  • Succeeded must not become Running again.
  • Cancelled should not be automatically re-executed by ordinary Workers.
  • Retry Waiting must be constrained by the maximum retry count.
  • Timeout and Failed can be defined as distinct terminal states depending on the business requirements.
  • If a Worker crashes while a task is Running, there must be a recovery mechanism.

Task state must not rely solely on the message state in the message queue. The message queue is responsible for delivering tasks, while the database or persistent task store is responsible for maintaining business state. The two must work together through a consistency strategy.

3.3 The Difference Between Task State and Message State

The two are often confused.

Task Status Message Status
Describes business execution progress Describes message delivery and consumption progress
Usually stored in a database Usually managed by the message queue
Includes Running, Succeeded, Failed Includes Ready, Unacked, Acknowledged, etc.
Oriented toward business queries Oriented toward message transmission and consumption

For example, the Worker has already fetched the message from the queue, but model inference has not yet finished:

  • The message may be in an unacknowledged state.
  • The business task is already in the Running state.

If the Worker executes successfully and acknowledges the message, but the task result is not reliably saved, the message may end up consumed while the business result is lost.

Therefore, the ordering among task status updates, result persistence, and message acknowledgment needs to be designed.

IV. The Core Role of Message Queues

A message queue is the intermediate buffer layer in an asynchronous task system.

It decouples task producers from task consumers, so that task submission speed and task processing speed need not match exactly.

4.1 Producers and Consumers

A typical architecture involves three roles:

  • Producer: the task producer, responsible for creating and sending tasks.
  • Broker: the message broker, responsible for storing, routing, and delivering messages.
  • Consumer: the task consumer, responsible for fetching messages and executing tasks.
flowchart LR
    A["API Service"] --> B["Producer"]
    B --> C["Message Broker"]
    C --> D["Worker 1"]
    C --> E["Worker 2"]
    C --> F["Worker 3"]
    D --> G["Model Inference Service"]
    E --> G
    F --> H["Document Processing Service"]
    G --> I["Result Storage"]
    H --> I
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

4.2 What Problems Do Message Queues Solve

First, service decoupling.

The API service does not need to know which Worker actually executes a task, nor does it need to maintain connections to Workers.

Second, peak shaving and valley filling.

When a large number of requests arrive in a short period, the queue can temporarily hold the tasks and let consumers execute them gradually according to their own processing capacity.

Third, horizontal scaling.

Worker instances can be added based on the task backlog to increase overall processing capacity.

Fourth, fault isolation.

When one consumer fails, other consumers can continue processing tasks, and the queue can retain messages that have not yet been completed.

Fifth, task distribution.

Different types of tasks can go into different queues and be handled by different Worker clusters.

4.3 Common Message Queues

Technology Characteristics Common Use Cases
Redis Streams Lightweight, supports consumer groups and message acknowledgment Small and medium-scale asynchronous tasks
RabbitMQ Rich routing capabilities, supports acknowledgment mechanisms Business tasks and complex message routing
Apache Kafka High throughput, persistence, mature partitioning Large-scale event streams and data processing
Apache Pulsar Multi-tenancy, tiered storage, decoupling of messaging and compute Large-scale distributed messaging systems
Amazon SQS Managed queues, lower operational burden Cloud-native asynchronous tasks

When choosing a message queue, you cannot look only at throughput; you also need to consider:

  • Message persistence capability.
  • Message acknowledgment mechanism.
  • How consumers recover after failure.
  • Message ordering requirements.
  • Support for delayed messages and dead-letter queues.
  • Consumer group and partition scaling capability.
  • Operational cost and observability.

4.4 Queue Models

Point-to-Point Queue

A message is usually successfully processed by one consumer, which suits task distribution.

flowchart TD
    A["Task Producer"] --> B["Task Queue"]
    B --> C["Worker A"]
    B --> D["Worker B"]
    B --> E["Worker C"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49

Publish/Subscribe (Pub/Sub)

A single event can be consumed separately by multiple subscribers, making it suitable for event broadcasting.

flowchart TD
    A["Event Producer"] --> B["Event Topic"]
    B --> C["Subscriber A"]
    B --> D["Subscriber B"]
    B --> E["Subscriber C"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49

Note that within the same Kafka consumer group, a specific message in a partition is usually processed by one consumer, while different consumer groups can independently consume data from the same topic. This is not exactly the same as simple broadcast-style Pub/Sub.

V. Queue Backlog and Peak Shaving / Valley Filling

One of the most important benefits of a message queue is to provide buffering when production and consumption rates are inconsistent.

5.1 Production Rate and Consumption Rate

Assume:

  • 100 tasks are submitted per second.
  • Each Worker processes 5 tasks per second.
  • There are currently 10 Workers.

Overall processing capacity:

\mu = 10 \times 5 = 50\ \text{tasks/s}

Production rate:

\lambda = 100\ \text{tasks/s}

Queue backlog growth rate:

\Delta Q = \lambda - \mu = 50\ \text{tasks/s}

If this continues for 60 seconds:

Q = 50 \times 60 = 3000

The system will accumulate a backlog of about 3000 tasks.

A queue can only buffer traffic temporarily; it cannot infinitely absorb requests that exceed processing capacity.

5.2 Queue Length and Stability

Definitions:

  • \lambda: task arrival rate.
  • \mu: processing rate of a single consumer.
  • c: number of consumers.
  • Q: number of pending tasks in the queue.

The theoretical processing capacity of the system:

C = c\mu

When:

\lambda < C

The system can usually process tasks stably, but processing-time fluctuations, consumer failures, and resource contention must also be considered.

When:

\lambda > C

The queue will keep growing.

When:

\lambda = C

The system is in a critical state, and any short-term fluctuation may cause a backlog.

Therefore, production systems should not run for long periods at close to 100% of theoretical processing capacity.

5.3 Utilization and Queuing Delay

Define system utilization:

\rho = \frac{\lambda}{c\mu}

The closer \rho is to 1, the more likely queuing delay is to increase significantly.

For a simple M/M/1 queuing model:

W = \frac{1}{\mu-\lambda}

where W is the average time spent in the system, including waiting and service time.

This formula applies only to specific idealized queuing assumptions. Real AI systems often have complex characteristics such as uneven task processing-time distributions, priority scheduling, and resource contention, so it cannot be used directly to accurately predict production latency.

In engineering practice, what matters more is continuous monitoring:

  • Queue length.
  • Wait time of the oldest message.
  • Task arrival rate.
  • Task completion rate.
  • Worker busy ratio.
  • P95/P99 queuing delay.

5.4 Estimating Queue Drain Time

If there are 10,000 tasks in the queue:

  • Total Worker processing capacity: 200 tasks/s.
  • New task arrival rate: 50 tasks/s.

Net drain rate:

R = 200-50=150\ \text{tasks/s}

Queue drain time:

T = \frac{10000}{150}\approx66.7\ \text{s}

This can be used to estimate the system recovery time.

However, if new tasks keep arriving, the drain rate must be computed as consumption capacity minus arrival rate, rather than simply using the consumer's total processing capacity.

6. Core Techniques of Flow Control

The goal of flow control is not simply to reject requests, but to let the system maintain predictable latency and stability as load changes.

Common techniques include:

  1. Rate limiting.
  2. Concurrency limiting.
  3. Backpressure.
  4. Circuit breaker.
  5. Degradation.
  6. Priority scheduling.
  7. Load balancing.

Each of these techniques controls pressure at a different layer.

6.1 Rate Limiting

Rate limiting caps the number of requests permitted to enter the system per unit of time.

For example:

  • Each user may submit at most 10 requests per second.
  • Each API key may submit at most 1000 tasks per minute.
  • The service as a whole may receive at most 500 tasks per second.

Common rate limiting algorithms are as follows.

Fixed Window

Time is divided into fixed intervals, and each interval is counted independently.

For example, 100 requests are allowed per minute.

Advantages:

  • Simple to implement.
  • Low storage overhead.
  • Suitable for basic API quota control.

Disadvantages:

  • Bursts can occur at window boundaries.

For example:

  • 100 submissions at 10:00:59.
  • Another 100 submissions at 10:01:00.

In a short span of time, the system may actually receive 200 requests.

Sliding Window

Limiting is based on the number of requests within the most recent period of time.

It can be implemented with a sliding log or a sliding counter.

The advantage is that it is smoother than the fixed window, but a sliding log incurs higher storage and computation overhead.

Token Bucket

Tokens are generated at a fixed rate, and the bucket holds at most a certain number of tokens.

Each request must consume a token in order to pass.

Core parameters:

  • r: token generation rate.
  • B: bucket capacity.
  • T: number of tokens.
  • \Delta t: elapsed time.

Token update:

T_{\text{new}}=\min(B,T_{\text{old}}+r\Delta t)

When a request arrives, if enough tokens are available, a token is consumed and the request is admitted; otherwise it is rejected or made to wait.

flowchart TD
    A["Request arrives"] --> B["Check token bucket"]
    B --> C{"Enough tokens?"}
    C -->|"Yes"| D["Consume token"]
    D --> E["Admit request"]
    C -->|"No"| F{"Waiting allowed?"}
    F -->|"Yes"| G["Enter wait queue"]
    F -->|"No"| H["Return rate-limit response"]
    G --> I["Wait for token refill"]
    I --> B
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

The token bucket allows short bursts, but the long-term average rate is bounded by the token generation rate.

Leaky Bucket

Requests enter a buffer and flow out at a fixed rate.

It places more emphasis on smoothing the output rate and is well suited to limiting the processing rate of downstream services.

6.2 Comparison of Rate Limiting Algorithms

Algorithm Bursts Allowed Main Features Typical Scenarios
Fixed window Allows concentrated requests within the window Simple to implement Basic quotas
Sliding window Relatively smooth Precise time-range control API rate limiting
Token bucket Allows limited bursts Controls long-term average rate API gateway
Leaky bucket Relatively stable output rate Smooths traffic Downstream resource protection

For AI services, rate limiting based solely on request count is often insufficient.

Because the cost of different requests can vary greatly:

  • A short Q&A with 100 tokens.
  • A long-context request with 100,000 tokens.
  • A single multi-turn Agent execution.
  • A batch document Embedding job.

Although each of these requests counts as only one API call, their actual resource consumption is completely different.

Therefore, traffic control needs to be introduced along the token and resource dimensions.

6.3 Token-level rate limiting

LLM services can limit request cost based on token count.

Common dimensions:

  • Input token count.
  • Output token count.
  • Total input and output tokens.
  • Tokens per minute (TPM).
  • Tokens per second (TPS).
  • Token quota for a user or tenant.

For example:

User tier RPM TPM
Free 10 20,000
Pro 60 100,000
Enterprise 300 1,000,000

Here, RPM stands for Requests Per Minute and TPM stands for Tokens Per Minute; the values are examples only.

Token-level rate limiting requires special attention:

  • Input tokens can be estimated before inference using a tokenizer.
  • The number of output tokens usually cannot be accurately determined before inference.
  • A budget can be reserved based on max_tokens.
  • Settle based on actual usage after execution completes.
  • Separate limits should be set for ultra-long-context requests.

For streaming generation, the token budget can also be checked continuously during generation to prevent a task from exceeding the user's quota.

6.4 Concurrency Control (Concurrency Limiting)

Rate limiting controls how many requests are allowed to enter per unit time, while concurrency control limits the number of tasks executing simultaneously.

Suppose:

  • 100 requests are allowed to enter per second.
  • Each request takes an average of 10 seconds to execute.

If concurrency is not limited, theoretically about 1,000 requests could be executing simultaneously.

But AI services have limited GPU memory and inference capacity, so concurrency cannot be increased indefinitely.

Concurrency limits can be configured by different resource dimensions:

Limit dimension Example
Global concurrency Execute at most 200 tasks simultaneously
Per-user concurrency At most 5 tasks per user
Per-model concurrency At most 64 concurrent requests for a model
GPU concurrency Limit inference requests per GPU
Task-type concurrency At most 20 document parsing tasks
Per-tenant concurrency Independent resource quota per tenant

Note that the concurrency count is not equal to the actual GPU batch size.

In LLM inference services, multiple requests can be dynamically combined by a continuous batching scheduler. What actually affects GPU resources is request length, KV Cache usage, batch token count, and model execution characteristics.

6.5 Backpressure

Backpressure means that when downstream processing capacity is insufficient, pressure is fed back to upstream to prevent upstream from continuing to produce tasks without limit.

For example:

  • Worker processing speed decreases.
  • Queue backlog continues to grow.
  • Queue wait time exceeds the threshold.
  • The system notifies the API to reduce its intake rate.
  • The API delays submission, rejects new tasks, or lowers task priority.

Backpressure is not simply rate limiting.

Rate limiting usually restricts traffic according to preset rules, whereas backpressure emphasizes dynamically adjusting upstream traffic based on downstream real-time processing capacity.

flowchart TD
    A["Client submits task"] --> B["API service"]
    B --> C["Task queue"]
    C --> D["Worker cluster"]
    D --> E["Downstream model service"]
    E --> F{"Is processing capacity sufficient"}
    F -->|"Yes"| G["Consume normally"]
    F -->|"No"| H["Generate backpressure signal"]
    H --> I["Reduce production rate"]
    I --> J["Delay acceptance or reject requests"]
    J --> B
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

Common backpressure strategies:

  1. Pause production when the queue reaches a threshold.
  2. Reduce upstream concurrency when consumers process more slowly.
  3. Reject low-priority tasks when the queue backlog becomes severe.
  4. Return a queued status for tasks that can be delayed.
  5. Return an overload response directly for requests that cannot wait.

6.6 Circuit Breaker

When a downstream model service keeps failing, continuing to send requests only wastes resources.

Based on the failure rate or timeout conditions, the circuit breaker temporarily stops sending requests to the faulty service.

It usually has three states:

  • Closed: requests pass through normally while failures are being counted.
  • Open: the failure threshold is reached, so requests are temporarily rejected.
  • Half-Open: after waiting a while, a small number of probe requests are let through.
stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure rate exceeds threshold
    Open --> HalfOpen: circuit-breaker wait period ends
    HalfOpen --> Closed: probe request succeeds
    HalfOpen --> Open: probe request fails

A circuit breaker can protect service dependencies, but it needs to be configured sensibly:

  • Failure rate threshold.
  • Statistics window.
  • Minimum number of request samples.
  • Circuit-breaker duration.
  • Number of probe requests in the half-open state.

If a downstream service is only briefly slower, it does not necessarily need to be tripped immediately. The decision can be made jointly with timeouts, concurrency limits, and load conditions.

6.7 Degradation

When system resources are insufficient, some service capabilities can be deliberately reduced.

Common degradation strategies for AI services:

Failure or pressure Degradation strategy
High-quality model overloaded Switch to a lightweight model
Insufficient GPU resources Delay execution of non-real-time tasks
Long-context overload Limit the maximum input length
RAG retrieval anomaly Answer using cached results or the base model
Agent tools unavailable Return partial results or stop tool calls
Severe queue backlog Pause low-priority tasks
Result generation timeout Return the content generated so far or the task status

Degradation must have clear business boundaries.

For example, when a user requires strict JSON output, you cannot casually switch to a model that lacks structured output capability.

7. Queue Tiering Design in AI Services

Task types in AI services vary widely, so it is usually best not to put all tasks into the same queue.

7.1 Why Multiple Queues Are Needed

Suppose the system has three kinds of tasks:

  • Real-time conversation: requires low latency.
  • Document parsing: takes a long time to execute.
  • Batch embedding: throughput comes first.

If all three share a single FIFO queue, the following can happen:

  • A large number of document parsing tasks occupy the queue.
  • Real-time conversation tasks line up behind long tasks.
  • User waiting time increases noticeably.
  • GPU and CPU resources cannot be allocated according to business goals.

This is known as head-of-line blocking.

A more reasonable approach is to split queues by task type and priority.

flowchart TD
    A["Task Entry"] --> B["Task Classifier"]
    B --> C["Real-time Task Queue"]
    B --> D["Normal Task Queue"]
    B --> E["Batch Task Queue"]
    C --> F["Real-time Inference Worker"]
    D --> G["General Worker"]
    E --> H["Batch Worker"]
    F --> I["GPU Inference Resource Pool"]
    G --> J["Shared Compute Resource Pool"]
    H --> I
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style C fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

7.2 Scheduling by Priority

Common priorities:

Priority Task Type Scheduling Strategy
P0 System-critical tasks Execute first
P1 Real-time interactive requests Low queuing latency
P2 Normal user tasks Normal scheduling
P3 Batch processing tasks Use idle resources
P4 Offline analysis tasks Execute during off-peak periods

However, simple strict-priority scheduling can prevent low-priority tasks from being executed for a long time.

This is called starvation.

Solutions include:

  • Priority aging: The longer a task waits, the more its priority gradually increases.
  • Quota scheduling: Reserve processing capacity for each type of task.
  • Weighted fair queuing: Allocate execution opportunities according to weights.
  • Time-slice round-robin: Different task queues take turns receiving resources.
  • Maximum waiting time: Raise priority after exceeding a threshold.

7.3 Resource Pool Isolation

Multiple queues do not necessarily mean complete physical resource isolation, but critical tasks should have explicit resource guarantees.

For example:

  • Real-time inference uses an independent GPU resource pool.
  • Offline batch processing uses the remaining GPU resources.
  • Document parsing uses an independent CPU Worker.
  • Embedding uses a dedicated batch processing service.
  • Agent tool calls use an independent concurrency quota.

This reduces interference from long-running tasks with real-time tasks.

8. Reliability Design for Asynchronous Tasks

Queues can improve system resilience, but they do not automatically guarantee that tasks are executed only once or that tasks will never be lost.

Reliability must be designed at multiple levels, including message delivery, task execution, result persistence, and recovery mechanisms.

8.1 At-Least-Once Delivery

At-least-once delivery means messages may be delivered repeatedly, but as long as the system continues to recover, messages ultimately have a chance to be processed.

For example:

  1. Worker retrieves the message.
  2. Worker completes model inference.
  3. Worker crashes before acknowledging the message.
  4. The message re-enters a consumable state.
  5. Another Worker processes it again.

In this case, the same task may be executed twice.

Therefore, at-least-once delivery requires business processing to be idempotent.

8.2 At-Most-Once Delivery

At-most-once delivery means a message is delivered at most once, but it may be lost due to failures.

If a Worker acknowledges the message immediately after retrieving it and then executes the task:

  1. Worker retrieves the message.
  2. The message is acknowledged.
  3. The Worker crashes during execution.
  4. The message will not be delivered again.

This pattern may cause task loss.

8.3 Exactly-Once

Exactly-once is a concept that is easily misunderstood.

Some messaging systems can provide transactional or exactly-once processing semantics within a specific scope, but this does not mean that all side effects—such as external model inference, database writes, file storage, and tool calls—are naturally executed only once.

For example:

  • The model has completed one paid inference.
  • The Worker crashes before saving the result.
  • The message is redelivered.
  • The model is executed again.

Even if the messaging system has a transaction mechanism, it may not be able to roll back the first external model call.

Therefore, engineering practice commonly adopts:

At-least-once delivery + business idempotency + state persistence + deduplication mechanism.

8.4 Idempotency Design

Idempotency means that repeated execution of the same request does not produce duplicate business results.

Common methods:

Unique Task ID

Generate a unique ID for each business task.

Idempotency Key

Generate a stable idempotency identifier based on business parameters.

For example:

document_summary:doc_001:version_3

Database Unique Constraint

Create a unique index on the idempotency key to prevent duplicate result records.

Status Check

Before a Worker executes, it checks the task status first. If the task has already succeeded, it does not process it again.

Result Overwrite Protection

Use a version number or conditional update to prevent an old task from overwriting a new result.

Note that idempotency cannot rely only on an in-memory set in the Worker. Because after the Worker restarts, the in-memory state is lost, deduplication information must be persisted.

8.5 Message Acknowledgment Mechanism

Message acknowledgment (ACK) determines whether a message can transition from pending to completed.

Common strategies:

  • Automatic acknowledgment: automatically acknowledge after the message is delivered; simple, but with a higher risk of task loss.
  • Manual acknowledgment: acknowledge after the task is processed successfully; suitable for reliable tasks.
  • Failure rejection: reject the message when processing fails.
  • Requeue: requeue on temporary errors.
  • Dead-letter transfer: move to the dead-letter queue when processing cannot continue.

For long-running AI tasks, it is usually necessary to properly configure the message visibility timeout, consumption acknowledgment timeout, or message lease.

If task execution time may exceed the message lease, the lease needs to be renewed periodically to prevent the task from being acquired again by another Worker while it is still running.

8.6 Transactional Messaging and the Outbox Pattern

A typical problem in asynchronous systems:

The business database write succeeds, but message sending fails.

For example:

  1. Create a task record.
  2. The database transaction commits successfully.
  3. The service sends a message to the queue.
  4. The message queue is temporarily unavailable.
  5. There is a task in the database, but no message in the queue.

This causes the task to never be executed.

Another case is that the message is sent successfully, but the database transaction rolls back, and the consumer receives a task that does not exist.

The Outbox Pattern can solve this kind of dual-write consistency problem.

Core idea:

  • Write the business task record in the same database transaction.
  • At the same time, write to the Outbox event table.
  • A separate publisher scans the Outbox.
  • Send the event to the message queue.
  • After successful sending, mark the Outbox event as published.
  • If publishing fails, continue retrying later.
flowchart TD
    A["Receive task request"] --> B["Begin database transaction"]
    B --> C["Write task record"]
    C --> D["Write Outbox event"]
    D --> E["Commit database transaction"]
    E --> F["Outbox publisher scans events"]
    F --> G["Send message to queue"]
    G --> H{"Send successful"}
    H -->|"Yes"| I["Mark event as published"]
    H -->|"No"| J["Keep event and retry"]
    J --> F
    I --> K["Worker consumes task"]
    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style K fill:#EAE8FA,stroke:#A49BD5,color:#51477D

Outbox solves the problem of reliable publishing between the database and the message queue, but duplicate publishing still needs to be handled, so consumers still need idempotency.

9. Retries, Backoff, and Dead-Letter Queues

In AI services, failures are the norm. Failures need to be classified into different types, rather than immediately retrying on any exception.

9.1 Error Classification

Error Type Whether Retry Is Recommended Handling Method
Network timeout Usually yes Delayed retry
Model service temporarily unavailable Yes Exponential backoff
HTTP 429 Yes Respect Retry-After
HTTP 500, 503 Usually yes Limit retry count
Parameter validation failure No Fail immediately
Unsupported file format No Return business error
Insufficient permissions No Terminate task
Token exceeds model limit Should not retry as-is Truncate or reprocess
GPU OOM Depends Reduce concurrency or switch resources
Task timeout Depends on task semantics Decide after checking execution status

9.2 Exponential Backoff