Back to series
· 15 min read

Caching, Retries, and Fallbacks in LLM Applications: Building Stable, Highly Available AI Services


1. Why Do LLM Applications Need Caching, Retry, and Fallback?

Compared with traditional business APIs, LLM applications have several distinct characteristics:

  • High response cost: A single request may involve a large number of input and output tokens, resulting in high inference costs.
  • Unstable response latency: Model inference, queuing, network transmission, and tool calls all affect response time.
  • Complex external dependencies: They depend on multiple systems such as model providers, vector databases, search services, and business APIs.
  • Limited service capacity: Model services have concurrency limits, token throughput limits, and rate limits.
  • Stochastic generation results: The same input does not necessarily produce exactly the same output.
  • Large context overhead: Multi-turn conversations may repeatedly carry a large amount of historical messages.

Without reliability design, production environments are prone to:

  • Repeatedly calling the model for the same question, wasting cost.
  • Brief network jitter causing requests to fail outright.
  • API rate limiting causing a large backlog of requests.
  • A model provider outage making the entire application unavailable.
  • Requests continuously retrying under high concurrency, further amplifying the failure.
  • A single slow request occupying connections and threads, affecting overall throughput.

Therefore, three types of mechanisms need to be established at the system level:

Mechanism Problem Solved Core Goal
Cache Duplicate requests and duplicate computation Reduce latency and cost
Retry Transient failures Improve request success rate
Fallback Primary service unavailable or timed out Ensure core functionality remains available

These three are not independent features; together they form the reliability system of an LLM application.

2. Caching Mechanisms: Reducing Duplicate Inference

2.1 What Is a Cache?

A cache is a technique for temporarily storing computation results or intermediate data so that subsequent requests can reuse them quickly.

In LLM applications, a cache can store:

  • Complete model responses.
  • Mappings between prompts and model outputs.
  • Embedding vectors.
  • RAG retrieval results.
  • Tool call results.
  • Context in multi-turn conversations.
  • Reusable prefix computation results from the model inference process.

Traditional API caches usually build cache keys based on information such as the request URL and parameters.

LLM applications need to consider more factors:

  • User input.
  • System prompt.
  • Conversation history.
  • Model name and version.
  • Generation parameters such as temperature and top_p.
  • Tool definitions and tool execution results.
  • RAG retrieval context.
  • User permissions and data isolation.
  • Output format and JSON Schema.

If the cache key design is incomplete, different requests may incorrectly reuse the same result.

2.2 Types of Caches in LLM Applications

Based on cache content and location, they can be divided into the following categories.

Cache Type Cached Content Primary Purpose
Response cache Complete model response Avoid repeated model calls
Prompt cache Reusable prompt prefix computation Reduce repeated prefill computation
Embedding cache Text vectors Avoid repeated vector computation
Retrieval cache RAG retrieval results Reduce repeated retrieval
Tool cache Tool call results Reduce repeated external requests
KV Cache K/V intermediate states of attention Reduce repeated computation in autoregressive generation
Session cache Session state and context Speed up multi-turn interaction

A key distinction must be made:

  • Response caches are usually managed by the application layer or service layer.
  • Prompt cache is usually managed by the model service or inference engine.
  • KV Cache is usually managed by the model inference engine during generation.

The objects they cache, their lifecycles, and their invalidation mechanisms are not the same.

2.3 Response Caching

Response caching is the most straightforward caching approach for LLM applications.

For example, a user asks multiple times:

What is a RESTful API?

If the request content, model configuration, and context are exactly the same, the system can directly return the previous result instead of calling the model again.

Basic flow:

flowchart TD
    A["Receive user request"] --> B["Build cache key"]
    B --> C["Query cache"]
    C --> D{"Cache hit?"}
    D -->|"Hit"| E["Read cached response"]
    D -->|"Miss"| F["Call LLM"]
    F --> G["Get model response"]
    G --> H["Write to cache"]
    H --> I["Return response"]
    E --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Response caching is suitable for:

  • Common knowledge Q&A.
  • Information extraction with a fixed format.
  • Highly repetitive classification tasks.
  • Summaries of the same document.
  • Stable explanations of business rules.
  • Reusable tool query results.

However, not all LLM requests are suitable for caching.

For example, the following requests are generally not suitable for direct global response caching:

  • Real-time stock quotes.
  • Current weather.
  • Real-time order status.
  • Personalized medical advice.
  • Account information subject to permission changes.
  • Conversations that need to be generated based on the latest context.

2.4 How to design the cache key?

The cache key determines whether two requests are considered the same request.

The simplest approach is to use the user input as the key:

cache_key = user_input

This design is usually not secure enough.

For example:

User A:

Summarize this document

User B:

Summarize this document

If the two users submit different documents, using only the user input as the key may cause an incorrect cache hit.

A more reasonable approach is to include the key parameters that affect the model result in the cache key.

For example:

import hashlib
import json

def build_cache_key(
    model,
    system_prompt,
    messages,
    temperature,
    schema_version
):
    payload = {
        "model": model,
        "system_prompt": system_prompt,
        "messages": messages,
        "temperature": temperature,
        "schema_version": schema_version
    }

    raw = json.dumps(
        payload,
        ensure_ascii=False,
        sort_keys=True
    )

    return hashlib.sha256(
        raw.encode("utf-8")
    ).hexdigest()

In production environments, you also need to add the following based on the capabilities actually used by the request:

  • Model version or model deployment identifier.
  • Generation parameters such as Top P and Max Tokens.
  • JSON Schema or output format version.
  • Tool definitions and their versions.
  • RAG document ID, version, or retrieval context summary.
  • Tenant ID and user permission scope.
  • Language, region, and business configuration.
  • Whether a specific security policy is enabled.

The principle is: all factors that may affect the response content or access permissions should be included in the cache isolation and consistency design.

For sensitive data, you cannot assume that different users can share a cache just because the request content is the same.

2.5 Cache expiration strategy

A cache cannot remain valid forever.

Common strategies:

Strategy Description Applicable Scenario
TTL Set a fixed expiration time General response caching
LRU Evict the least recently used data Limited memory capacity
LFU Evict less frequently accessed data Hot data caching
Version Invalidation Invalidate cache when dependent versions change Prompt, model, and Schema updates
Active Invalidation Trigger deletion via business events Order, account, and other state changes

TTL (Time To Live) is the most common cache expiration mechanism.

For example:

cache.set(
    key,
    response,
    ex=300
)

This means the cache is valid for 300 seconds.

Different businesses should set different TTLs:

Business TTL Design Approach
Fixed knowledge Q&A Can be longer
Product information Based on product update frequency
News summaries Based on content update frequency
Weather queries Shorter
Real-time orders Very short or event-driven invalidation
User permissions Requires strict control over invalidation and revalidation

TTL is not better when longer. The longer the cache time, the more likely it is to return stale information.

2.6 Cache Breakdown, Penetration, and Avalanche

A cache system also needs to handle classic cache problems.

Cache Breakdown

When a hot Key expires, a large number of requests access that Key simultaneously, and all of them penetrate to the model service.

Solutions:

  • Singleflight request coalescing.
  • Distributed mutex locks.
  • Proactive refresh of hot Keys.
  • Logical expiration and asynchronous update.

Cache Penetration

A large number of requests access nonexistent or uncacheable data, causing every request to reach the backend.

Solutions:

  • Parameter validation.
  • Short-term caching of empty results.
  • Bloom filter.
  • Rate limiting and anomalous request identification.

Cache Avalanche

A large number of caches expire at the same time, causing a sudden increase in backend traffic.

Solutions:

  • Add random jitter to TTL.
  • Expire in batches.
  • Prewarm hot data.
  • Multi-level caching.
  • Backend rate limiting.

2.7 Special Issues with LLM Caching

LLM response caching has several notable differences from ordinary database caching.

First, generated results are stochastic.

The same Prompt may generate different results under different Temperature, model versions, or inference configurations.

If requests use high randomness, response caching may reduce content diversity.

Second, Prompts may contain dynamic context.

For example:

Current time: 2026-11-20
User question: What important news is there today?

If the current time is ignored, the cache may return outdated results.

Third, tool calls may have side effects.

For example:

  • Create an order.
  • Send an email.
  • Transfer funds.
  • Delete data.

You cannot simply cache tool-call requests and treat an old execution result as if the operation had been successfully executed for the current request.

Fourth, cached content may involve sensitive data.

Tenant isolation, user permissions, encrypted storage, log redaction, and cache cleanup need to be considered.

3. Prompt Caching: Caching Reusable Context

3.1 What Is Prompt Caching?

Prompt Caching is an optimization mechanism in which a model service or inference system caches reusable Prompt prefixes.

When processing input Prompts, large models need to execute the Prefill phase, compute intermediate representations at each layer, and generate the K/V states required for Attention.

For repeatedly used Prompt prefixes, if the inference system can reuse previously computed results, it can reduce redundant computation.

For example, multiple requests all contain the same System Prompt:

You are a professional technical assistant.

You need to follow these rules:
1. Answer accurately.
2. Use Chinese.
3. Output structured content.
4. Do not fabricate facts.

After that, only the user question differs for each request.

Request A: Explain Redis.
Request B: Explain Kafka.
Request C: Explain PostgreSQL.

If the system supports Prompt Caching, it may reuse part of the computation results for the common prefix.

3.2 How Prompt Caching Works

flowchart TD
    A["Receive model request"] --> B["Identify Prompt prefix"]
    B --> C{"Is there a reusable cache"}
    C -->|"Hit"| D["Read cached prefix computation state"]
    C -->|"Miss"| E["Execute full Prefill"]
    E --> F["Save reusable state"]
    D --> G["Process remaining context"]
    F --> G
    G --> H["Enter Decode"]
    H --> I["Generate model response"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The specific caching strategy depends on the model provider and inference engine.

Note that Prompt Caching caches reusable model computation state, not simply the final answer.

3.3 Differences Between Prompt Caching and Response Caching

Comparison Dimension Prompt Caching Response Caching
Cached Content Reusable Prompt prefix computation state Complete model response
Where It Applies Model service or inference engine Application layer or cache layer
Whether to Regenerate Usually still needs generation No regeneration needed on a hit
Whether It Depends on Identical Input Depends on a matchable prefix and computation configuration Depends on the cache key
Main Benefit Reduce Prefill computation Reduce full model calls
Impact on Output Diversity Usually does not directly change the generation strategy May reuse a fixed output

3.4 Relationship Between KV Cache and Prompt Caching

KV Cache is an important optimization technique for Transformer autoregressive inference.

During the Decode phase, the model caches the Key and Value of historical Tokens, avoiding recomputing the K/V of all historical Tokens every time a new Token is generated.

Prompt Caching can be implemented in some inference systems by using reusable KV state.

However, the two are not entirely equivalent:

  • KV Cache is a K/V caching mechanism during inference.
  • Prompt Caching is a reuse strategy for repeated Prompt prefixes.
  • Prompt Caching can be implemented based on KV Cache and may also combine other internal mechanisms.
  • Application-layer response caching belongs to another layer.

4. Retry Mechanism: Handling Transient Failures

4.1 What Is Retry?

Retry means that when a request fails due to a transient failure, the system reissues the request according to a preconfigured strategy.

Large model API calls may fail for the following reasons:

  • Network connection interruption.
  • DNS resolution error.
  • Request timeout.
  • Server temporarily unavailable.
  • API rate limiting.
  • Insufficient concurrent capacity.
  • Brief failure of an upstream dependency.

Not all errors are suitable for retry.

For example:

Error Type Retryable Recommended Handling
Network timeout Usually yes Bounded retries
Connection reset Usually yes Bounded retries
HTTP 429 Usually yes Follow rate-limit information
HTTP 500 Depends Backoff retry
HTTP 502 Usually yes Backoff retry
HTTP 503 Usually yes Backoff retry
HTTP 504 Usually yes Backoff retry
HTTP 400 Usually no Fix request parameters
HTTP 401 Usually no Check authentication credentials
HTTP 403 Usually no Check permissions
HTTP 404 Usually no Check the resource address
Business validation failure Usually no Fix the business data

Ultimately, the semantics of the provider's error codes still take precedence.

4.2 Why can't we retry indefinitely?

Suppose a model service fails and 1,000 requests fail at the same time.

If every request retries indefinitely, you may end up with:

  • A continuously growing number of requests.
  • The connection pool being exhausted.
  • Even more pressure on the upstream service.
  • Longer request queueing times.
  • More requests timing out.
  • A further decline in overall system availability.

This kind of phenomenon can develop into a retry storm.

Therefore, retries must include:

  • A maximum number of retries.
  • A total request time budget.
  • Backoff waiting.
  • Random jitter.
  • Judgment of which errors are retryable.
  • Concurrency limits.
  • Idempotency control.

4.3 Fixed-interval retries

The simplest retry strategy is a fixed time interval.

For example:

1st failure: wait 1 second
2nd failure: wait 1 second
3rd failure: wait 1 second

This approach is simple to implement, but when a large number of clients fail at the same time, they may all resend their requests at the same moment.

That easily creates a request spike.

4.4 Exponential Backoff

Exponential backoff is a common retry strategy in production systems.

The wait time grows exponentially:

t_n = \min(t_{\max}, t_0 \times 2^n)

Where:

  • t_n: the wait time for the nth retry.
  • t_0: the initial wait time.
  • t_{\max}: the maximum wait time.
  • n: the retry count.

For example, with an initial wait time of 1 second:

Retry Count Wait Time
1st 1 second
2nd 2 seconds
3rd 4 seconds
4th 8 seconds
5th 16 seconds

This reduces the pressure on the upstream service during a sustained failure.

4.5 Random Jitter

Using exponential backoff alone still has problems.

If a large number of requests fail at the same time, they may use the same backoff time and retry at the same instant.

Jitter is used to add randomness to the wait time.

One common approach is Full Jitter:

t_n = U(0,\min(t_{\max},t_0 \times 2^n))

Here U(a,b) means a uniformly random value within the interval.

This spreads requests across different points in time and reduces the risk of concentrated request bursts.

4.6 Retry-After

For HTTP 429 or some service-unavailable responses, the server may use Retry-After to tell the client how long it should wait.

For example:

HTTP/1.1 429 Too Many Requests
Retry-After: 5

This means the client should wait at least 5 seconds before considering a retry.

If the server provides a valid Retry-After, the client should respect that information rather than mechanically applying its own short backoff time.

The overall request deadline must also be taken into account. If the wait time already exceeds the remaining time budget, the client should not keep retrying.

4.7 The Complete Retry Flow

flowchart TD
    A["Initiate model request"] --> B["Wait for response"]
    B --> C{"Is the request successful?"}
    C -->|"Yes"| D["Return result"]
    C -->|"No"| E["Determine error type"]
    E --> F{"Is it a retryable error?"}
    F -->|"No"| G["Return failure"]
    F -->|"Yes"| H{"Retry limit exceeded?"}
    H -->|"Yes"| G
    H -->|"No"| I["Calculate backoff time"]
    I --> J["Add random jitter"]
    J --> K["Check remaining time budget"]
    K --> L{"Is there time left?"}
    L -->|"No"| G
    L -->|"Yes"| M["Wait and retry"]
    M --> A

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#EAF3EC,stroke:#83A995,color:#315D49

V. Idempotency: Retries Must Not Re-execute Business Operations

This is an issue that is very easily overlooked in LLM applications.

5.1 What Is Idempotency?

Idempotency means that executing the same operation once or multiple times produces the same final business effect on the system.

For example:

Set the user nickname to Zhang San

No matter how many times it is executed, the final nickname is still Zhang San.

Such operations are usually idempotent.

Whereas:

Increase the account balance by 100 yuan

If it is executed twice, the balance will increase by 200 yuan.

This is not a naturally idempotent operation.

5.2 Why Do Model API Retries Produce Duplicate Operations?

Suppose the Agent needs to execute a transfer:

User: Transfer 100 yuan to Zhang San.

The model generates a tool call request, and the application executes the transfer.

But after the execution succeeds, the network connection suddenly drops, and the application does not receive the success response.

The application cannot determine:

  • Whether the transfer has already been executed.
  • Whether the tool failed to execute.
  • Whether the response was simply lost.

If the application retries directly, it may transfer the money twice.

5.3 Using an Idempotency Key

A unique idempotency key can be generated for each business operation.

For example:

POST /api/transfers
Idempotency-Key: transfer-20261120-000001
Content-Type: application/json

Request:

{
  "from_account": "A1001",
  "to_account": "A2001",
  "amount": 100
}

The server stores the idempotency key and the execution result.

When the same idempotency key is received again, the transfer is not executed a second time; instead, the previously stored result is returned.

Note the following:

  • The idempotency key should be generated by the business operation, not regenerated on every retry.
  • Idempotency records must be persisted, and a reasonable retention period should be designed.
  • The idempotency key must be bound to the request content, so that the same key is not used with different parameters.
  • Concurrent requests must be guaranteed to execute only once through a database unique constraint, transactions, or other atomic mechanisms.
  • Idempotency records and business execution results should be committed within the same transaction boundary wherever possible.

5.4 Idempotency Boundaries Between Model Generation and Business Execution

Two levels need to be distinguished:

Operation Requires idempotency control?
Calling a model to generate text Usually no business side effects, but may incur duplicate costs
Model generating tool-call parameters May generate duplicate results
Querying the database Usually safe to retry, but consistency must be considered
Creating an order Idempotency must be considered
Sending email Duplicate sending must be prevented
Transferring funds Strict idempotency control is required
Deleting data Business-level idempotency and permission control are required

The retry policy for model requests cannot be directly equated with the retry policy for tool operations.

6. Degradation mechanism: ensuring the system remains available during failures

6.1 What is degradation?

Fallback / degradation refers to the system proactively switching to a less capable but still functional processing method when the primary service is unavailable, responding too slowly, or short on resources.

Degradation is not simply returning an error; it is a deliberate sacrifice of some functionality to preserve core business capabilities.

For example:

  • When a high-performance model is unavailable, switch to a backup model.
  • When the model times out, return cached results.
  • When the RAG retrieval service is unavailable, answer using limited basic knowledge.
  • When advanced analytics features are unavailable, provide simplified results.
  • When external search fails, clearly state that real-time information cannot be obtained.

6.2 Common degradation methods

Degradation method Concrete approach Applicable scenario
Cache degradation Return historical cache Repetitive queries
Model degradation Switch to backup model Primary model unavailable
Capability degradation Disable complex reasoning or advanced features High load
Data degradation Use existing or static data External data source failure
Result degradation Return simplified content Insufficient latency budget
Human fallback Hand over to human handling High-risk tasks

6.3 Model degradation and model routing

Suppose the system is connected to multiple models at the same time:

  • Primary model: high capability, high cost.
  • Backup model: medium capability, medium cost.
  • Lightweight model: low cost, low latency.

Models can be selected based on request type and service status.

For example:

flowchart TD
    A["Receive AI request"] --> B["Classify and route request"]
    B --> C["Call primary model first"]
    C --> D{"Did primary model succeed?"}
    D -->|"Yes"| E["Return primary model result"]
    D -->|"No"| F["Determine whether switching is allowed"]
    F -->|"No"| G["Return explicit error or human fallback"]
    F -->|"Yes"| H["Call backup model"]
    H --> I{"Did backup model succeed?"}
    I -->|"Yes"| J["Return backup model result"]
    I -->|"No"| K["Use lightweight model or cache"]
    K --> L{"Is a usable result available?"}
    L -->|"Yes"| M["Return degraded result"]
    L -->|"No"| G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49
    style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Model degradation requires attention to differences in model capabilities.

A backup model may not be fully compatible with the primary model:

  • Prompt formats may differ.
  • Tool-calling protocols may differ.
  • JSON Schema support coverage may differ.
  • Context windows may differ.
  • Multimodal capabilities may differ.
  • Output quality and safety policies may differ.

Therefore, before switching models, a unified adaptation layer needs to be established and capability verification performed.

6.4 Degradation Does Not Mean Silently Lowering Quality

For example, the primary model supports complex reasoning, while the fallback model can only perform basic classification.

If the system still labels results after switching as having the same capabilities as the primary model, it may mislead users or downstream systems.

It is recommended to add status information to degraded results:

{
  "status": "degraded",
  "model": "fallback-model",
  "result": {
    "category": "technical_issue"
  },
  "message": "当前使用备用模型处理,结果可能存在能力差异。"
}

In practice, an API can indicate degraded status through response metadata, and does not necessarily need to mix these fields into the business data itself.

7. Circuit Breaker Mechanism: Preventing Ongoing Failure Propagation

7.1 What Is a Circuit Breaker?

A circuit breaker is a fault isolation mechanism.

When an external service continues to fail, the system temporarily stops sending requests to that service to avoid continuing to consume resources.

A circuit breaker mechanism usually has three states:

  • Closed: closed state; requests pass normally.
  • Open: open state; requests are rejected directly or routed to fallback logic.
  • Half-Open: half-open state; allows a small number of probe requests.

7.2 How a Circuit Breaker Works

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure rate or consecutive failures reach threshold
    Open --> HalfOpen: circuit breaker wait time ends
    HalfOpen --> Closed: probe request succeeds
    HalfOpen --> Open: probe request fails

    style Closed fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style Open fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style HalfOpen fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Closed:

  • Requests pass normally.
  • Track request success rate and failures.
  • When the failure rate reaches the threshold, trigger the circuit breaker.

Open:

  • No longer call the failing service normally.
  • Return a degraded result or error directly.
  • Wait for the circuit breaker timeout to end.

Half-Open:

  • Allow a small number of requests to enter.
  • Check whether the upstream service has recovered.
  • Decide whether to close or reopen the circuit breaker based on the probe result.

7.3 Why Do LLM Applications Need Circuit Breakers?

Suppose a model provider keeps returning HTTP 503.

If the application still retries each request multiple times:

  • Failed requests continue to increase.
  • The connection pool is occupied.
  • Request latency continues to rise.
  • Thread and coroutine resources are consumed.
  • Other normal business operations may also be affected.

After the system confirms that the service remains abnormal, the circuit breaker can proactively stop calling the failing upstream.

7.4 The Relationship Between Circuit Breaking and Retry

The two serve different purposes:

Mechanism Problem Solved How It Works
Retry Temporary failure of a single request Try again
Circuit Breaker Persistent upstream failure Pause calls
Timeout Request has no response for a long time Limit wait time
Fallback Primary path cannot complete Switch handling method
Rate Limiting Too many requests Control ingress rate

Without circuit breaking, retries may continue to amplify upstream failures.

Without retries, brief network jitter may directly cause request failures.

Without fallback, there may be no available service path after the circuit breaker trips.

They need to be used together.

8. Rate Limiting and Concurrency Control

In addition to caching, retries, and fallback, LLM applications must also manage request traffic.

8.1 Why Is Rate Limiting Needed?

Model providers usually impose limits on:

  • Requests per minute (RPM).
  • Tokens per minute (TPM).
  • Number of concurrent requests.
  • Daily call volume.
  • Account spending quota.

If the application has no rate limiting control, burst traffic may cause a large number of 429 errors.

8.2 Common Rate Limiting Algorithms

Algorithm Principle Characteristics
Fixed window Limits the number of requests within a fixed time window Simple, but has boundary bursts
Sliding window Counts request volume over a continuous time interval Smoother
Leaky bucket Processes requests at a fixed rate Stable output rate
Token bucket Generates tokens at a rate; requests consume tokens Allows a certain amount of burst traffic
Concurrency limit Limits the number of requests executed simultaneously Controls resource usage

For LLM applications, rate limiting solely by request count may not be enough.

Because token consumption varies greatly across different requests.

For example:

  • Request A: 100 tokens input, 50 tokens output.
  • Request B: 20,000 tokens input, 4,000 tokens output.

Although both count as a single request, the resource consumption differs significantly.

Therefore, rate limiting needs to be multi-dimensional, combining RPM, TPM, and concurrency.

8.3 Token Bucket Algorithm

The token bucket replenishes tokens into the bucket at a fixed rate.

Each time a request comes in, it must consume the corresponding number of tokens.

flowchart TD
    A["Request arrives"] --> B["Check token bucket"]
    B --> C{"Are tokens sufficient?"}
    C -->|"Yes"| D["Consume tokens"]
    D --> E["Allow request to execute"]
    C -->|"No"| F["Wait or reject"]
    G["Replenish tokens periodically"] --> B

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49

The token bucket allows a certain degree of burst traffic, but the long-term average request rate is still limited by the token generation rate.

8.4 Concurrency Control and Queues