1. Why Do LLM Applications Need Caching, Retry, and Fallback?
Compared with traditional business APIs, LLM applications have several distinct characteristics:
- High response cost: A single request may involve a large number of input and output tokens, resulting in high inference costs.
- Unstable response latency: Model inference, queuing, network transmission, and tool calls all affect response time.
- Complex external dependencies: They depend on multiple systems such as model providers, vector databases, search services, and business APIs.
- Limited service capacity: Model services have concurrency limits, token throughput limits, and rate limits.
- Stochastic generation results: The same input does not necessarily produce exactly the same output.
- Large context overhead: Multi-turn conversations may repeatedly carry a large amount of historical messages.
Without reliability design, production environments are prone to:
- Repeatedly calling the model for the same question, wasting cost.
- Brief network jitter causing requests to fail outright.
- API rate limiting causing a large backlog of requests.
- A model provider outage making the entire application unavailable.
- Requests continuously retrying under high concurrency, further amplifying the failure.
- A single slow request occupying connections and threads, affecting overall throughput.
Therefore, three types of mechanisms need to be established at the system level:
| Mechanism | Problem Solved | Core Goal |
|---|---|---|
| Cache | Duplicate requests and duplicate computation | Reduce latency and cost |
| Retry | Transient failures | Improve request success rate |
| Fallback | Primary service unavailable or timed out | Ensure core functionality remains available |
These three are not independent features; together they form the reliability system of an LLM application.
2. Caching Mechanisms: Reducing Duplicate Inference
2.1 What Is a Cache?
A cache is a technique for temporarily storing computation results or intermediate data so that subsequent requests can reuse them quickly.
In LLM applications, a cache can store:
- Complete model responses.
- Mappings between prompts and model outputs.
- Embedding vectors.
- RAG retrieval results.
- Tool call results.
- Context in multi-turn conversations.
- Reusable prefix computation results from the model inference process.
Traditional API caches usually build cache keys based on information such as the request URL and parameters.
LLM applications need to consider more factors:
- User input.
- System prompt.
- Conversation history.
- Model name and version.
- Generation parameters such as temperature and top_p.
- Tool definitions and tool execution results.
- RAG retrieval context.
- User permissions and data isolation.
- Output format and JSON Schema.
If the cache key design is incomplete, different requests may incorrectly reuse the same result.
2.2 Types of Caches in LLM Applications
Based on cache content and location, they can be divided into the following categories.
| Cache Type | Cached Content | Primary Purpose |
|---|---|---|
| Response cache | Complete model response | Avoid repeated model calls |
| Prompt cache | Reusable prompt prefix computation | Reduce repeated prefill computation |
| Embedding cache | Text vectors | Avoid repeated vector computation |
| Retrieval cache | RAG retrieval results | Reduce repeated retrieval |
| Tool cache | Tool call results | Reduce repeated external requests |
| KV Cache | K/V intermediate states of attention | Reduce repeated computation in autoregressive generation |
| Session cache | Session state and context | Speed up multi-turn interaction |
A key distinction must be made:
- Response caches are usually managed by the application layer or service layer.
- Prompt cache is usually managed by the model service or inference engine.
- KV Cache is usually managed by the model inference engine during generation.
The objects they cache, their lifecycles, and their invalidation mechanisms are not the same.
2.3 Response Caching
Response caching is the most straightforward caching approach for LLM applications.
For example, a user asks multiple times:
What is a RESTful API?
If the request content, model configuration, and context are exactly the same, the system can directly return the previous result instead of calling the model again.
Basic flow:
flowchart TD
A["Receive user request"] --> B["Build cache key"]
B --> C["Query cache"]
C --> D{"Cache hit?"}
D -->|"Hit"| E["Read cached response"]
D -->|"Miss"| F["Call LLM"]
F --> G["Get model response"]
G --> H["Write to cache"]
H --> I["Return response"]
E --> I
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Response caching is suitable for:
- Common knowledge Q&A.
- Information extraction with a fixed format.
- Highly repetitive classification tasks.
- Summaries of the same document.
- Stable explanations of business rules.
- Reusable tool query results.
However, not all LLM requests are suitable for caching.
For example, the following requests are generally not suitable for direct global response caching:
- Real-time stock quotes.
- Current weather.
- Real-time order status.
- Personalized medical advice.
- Account information subject to permission changes.
- Conversations that need to be generated based on the latest context.
2.4 How to design the cache key?
The cache key determines whether two requests are considered the same request.
The simplest approach is to use the user input as the key:
cache_key = user_input
This design is usually not secure enough.
For example:
User A:
Summarize this document
User B:
Summarize this document
If the two users submit different documents, using only the user input as the key may cause an incorrect cache hit.
A more reasonable approach is to include the key parameters that affect the model result in the cache key.
For example:
import hashlib
import json
def build_cache_key(
model,
system_prompt,
messages,
temperature,
schema_version
):
payload = {
"model": model,
"system_prompt": system_prompt,
"messages": messages,
"temperature": temperature,
"schema_version": schema_version
}
raw = json.dumps(
payload,
ensure_ascii=False,
sort_keys=True
)
return hashlib.sha256(
raw.encode("utf-8")
).hexdigest()
In production environments, you also need to add the following based on the capabilities actually used by the request:
- Model version or model deployment identifier.
- Generation parameters such as Top P and Max Tokens.
- JSON Schema or output format version.
- Tool definitions and their versions.
- RAG document ID, version, or retrieval context summary.
- Tenant ID and user permission scope.
- Language, region, and business configuration.
- Whether a specific security policy is enabled.
The principle is: all factors that may affect the response content or access permissions should be included in the cache isolation and consistency design.
For sensitive data, you cannot assume that different users can share a cache just because the request content is the same.
2.5 Cache expiration strategy
A cache cannot remain valid forever.
Common strategies:
| Strategy | Description | Applicable Scenario |
|---|---|---|
| TTL | Set a fixed expiration time | General response caching |
| LRU | Evict the least recently used data | Limited memory capacity |
| LFU | Evict less frequently accessed data | Hot data caching |
| Version Invalidation | Invalidate cache when dependent versions change | Prompt, model, and Schema updates |
| Active Invalidation | Trigger deletion via business events | Order, account, and other state changes |
TTL (Time To Live) is the most common cache expiration mechanism.
For example:
cache.set(
key,
response,
ex=300
)
This means the cache is valid for 300 seconds.
Different businesses should set different TTLs:
| Business | TTL Design Approach |
|---|---|
| Fixed knowledge Q&A | Can be longer |
| Product information | Based on product update frequency |
| News summaries | Based on content update frequency |
| Weather queries | Shorter |
| Real-time orders | Very short or event-driven invalidation |
| User permissions | Requires strict control over invalidation and revalidation |
TTL is not better when longer. The longer the cache time, the more likely it is to return stale information.
2.6 Cache Breakdown, Penetration, and Avalanche
A cache system also needs to handle classic cache problems.
Cache Breakdown
When a hot Key expires, a large number of requests access that Key simultaneously, and all of them penetrate to the model service.
Solutions:
- Singleflight request coalescing.
- Distributed mutex locks.
- Proactive refresh of hot Keys.
- Logical expiration and asynchronous update.
Cache Penetration
A large number of requests access nonexistent or uncacheable data, causing every request to reach the backend.
Solutions:
- Parameter validation.
- Short-term caching of empty results.
- Bloom filter.
- Rate limiting and anomalous request identification.
Cache Avalanche
A large number of caches expire at the same time, causing a sudden increase in backend traffic.
Solutions:
- Add random jitter to TTL.
- Expire in batches.
- Prewarm hot data.
- Multi-level caching.
- Backend rate limiting.
2.7 Special Issues with LLM Caching
LLM response caching has several notable differences from ordinary database caching.
First, generated results are stochastic.
The same Prompt may generate different results under different Temperature, model versions, or inference configurations.
If requests use high randomness, response caching may reduce content diversity.
Second, Prompts may contain dynamic context.
For example:
Current time: 2026-11-20
User question: What important news is there today?
If the current time is ignored, the cache may return outdated results.
Third, tool calls may have side effects.
For example:
- Create an order.
- Send an email.
- Transfer funds.
- Delete data.
You cannot simply cache tool-call requests and treat an old execution result as if the operation had been successfully executed for the current request.
Fourth, cached content may involve sensitive data.
Tenant isolation, user permissions, encrypted storage, log redaction, and cache cleanup need to be considered.
3. Prompt Caching: Caching Reusable Context
3.1 What Is Prompt Caching?
Prompt Caching is an optimization mechanism in which a model service or inference system caches reusable Prompt prefixes.
When processing input Prompts, large models need to execute the Prefill phase, compute intermediate representations at each layer, and generate the K/V states required for Attention.
For repeatedly used Prompt prefixes, if the inference system can reuse previously computed results, it can reduce redundant computation.
For example, multiple requests all contain the same System Prompt:
You are a professional technical assistant.
You need to follow these rules:
1. Answer accurately.
2. Use Chinese.
3. Output structured content.
4. Do not fabricate facts.
After that, only the user question differs for each request.
Request A: Explain Redis.
Request B: Explain Kafka.
Request C: Explain PostgreSQL.
If the system supports Prompt Caching, it may reuse part of the computation results for the common prefix.
3.2 How Prompt Caching Works
flowchart TD
A["Receive model request"] --> B["Identify Prompt prefix"]
B --> C{"Is there a reusable cache"}
C -->|"Hit"| D["Read cached prefix computation state"]
C -->|"Miss"| E["Execute full Prefill"]
E --> F["Save reusable state"]
D --> G["Process remaining context"]
F --> G
G --> H["Enter Decode"]
H --> I["Generate model response"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The specific caching strategy depends on the model provider and inference engine.
Note that Prompt Caching caches reusable model computation state, not simply the final answer.
3.3 Differences Between Prompt Caching and Response Caching
| Comparison Dimension | Prompt Caching | Response Caching |
|---|---|---|
| Cached Content | Reusable Prompt prefix computation state | Complete model response |
| Where It Applies | Model service or inference engine | Application layer or cache layer |
| Whether to Regenerate | Usually still needs generation | No regeneration needed on a hit |
| Whether It Depends on Identical Input | Depends on a matchable prefix and computation configuration | Depends on the cache key |
| Main Benefit | Reduce Prefill computation | Reduce full model calls |
| Impact on Output Diversity | Usually does not directly change the generation strategy | May reuse a fixed output |
3.4 Relationship Between KV Cache and Prompt Caching
KV Cache is an important optimization technique for Transformer autoregressive inference.
During the Decode phase, the model caches the Key and Value of historical Tokens, avoiding recomputing the K/V of all historical Tokens every time a new Token is generated.
Prompt Caching can be implemented in some inference systems by using reusable KV state.
However, the two are not entirely equivalent:
- KV Cache is a K/V caching mechanism during inference.
- Prompt Caching is a reuse strategy for repeated Prompt prefixes.
- Prompt Caching can be implemented based on KV Cache and may also combine other internal mechanisms.
- Application-layer response caching belongs to another layer.
4. Retry Mechanism: Handling Transient Failures
4.1 What Is Retry?
Retry means that when a request fails due to a transient failure, the system reissues the request according to a preconfigured strategy.
Large model API calls may fail for the following reasons:
- Network connection interruption.
- DNS resolution error.
- Request timeout.
- Server temporarily unavailable.
- API rate limiting.
- Insufficient concurrent capacity.
- Brief failure of an upstream dependency.
Not all errors are suitable for retry.
For example:
| Error Type | Retryable | Recommended Handling |
|---|---|---|
| Network timeout | Usually yes | Bounded retries |
| Connection reset | Usually yes | Bounded retries |
| HTTP 429 | Usually yes | Follow rate-limit information |
| HTTP 500 | Depends | Backoff retry |
| HTTP 502 | Usually yes | Backoff retry |
| HTTP 503 | Usually yes | Backoff retry |
| HTTP 504 | Usually yes | Backoff retry |
| HTTP 400 | Usually no | Fix request parameters |
| HTTP 401 | Usually no | Check authentication credentials |
| HTTP 403 | Usually no | Check permissions |
| HTTP 404 | Usually no | Check the resource address |
| Business validation failure | Usually no | Fix the business data |
Ultimately, the semantics of the provider's error codes still take precedence.
4.2 Why can't we retry indefinitely?
Suppose a model service fails and 1,000 requests fail at the same time.
If every request retries indefinitely, you may end up with:
- A continuously growing number of requests.
- The connection pool being exhausted.
- Even more pressure on the upstream service.
- Longer request queueing times.
- More requests timing out.
- A further decline in overall system availability.
This kind of phenomenon can develop into a retry storm.
Therefore, retries must include:
- A maximum number of retries.
- A total request time budget.
- Backoff waiting.
- Random jitter.
- Judgment of which errors are retryable.
- Concurrency limits.
- Idempotency control.
4.3 Fixed-interval retries
The simplest retry strategy is a fixed time interval.
For example:
1st failure: wait 1 second
2nd failure: wait 1 second
3rd failure: wait 1 second
This approach is simple to implement, but when a large number of clients fail at the same time, they may all resend their requests at the same moment.
That easily creates a request spike.
4.4 Exponential Backoff
Exponential backoff is a common retry strategy in production systems.
The wait time grows exponentially:
Where:
- t_n: the wait time for the nth retry.
- t_0: the initial wait time.
- t_{\max}: the maximum wait time.
- n: the retry count.
For example, with an initial wait time of 1 second:
| Retry Count | Wait Time |
|---|---|
| 1st | 1 second |
| 2nd | 2 seconds |
| 3rd | 4 seconds |
| 4th | 8 seconds |
| 5th | 16 seconds |
This reduces the pressure on the upstream service during a sustained failure.
4.5 Random Jitter
Using exponential backoff alone still has problems.
If a large number of requests fail at the same time, they may use the same backoff time and retry at the same instant.
Jitter is used to add randomness to the wait time.
One common approach is Full Jitter:
Here U(a,b) means a uniformly random value within the interval.
This spreads requests across different points in time and reduces the risk of concentrated request bursts.
4.6 Retry-After
For HTTP 429 or some service-unavailable responses, the server may use Retry-After to tell the client how long it should wait.
For example:
HTTP/1.1 429 Too Many Requests
Retry-After: 5
This means the client should wait at least 5 seconds before considering a retry.
If the server provides a valid Retry-After, the client should respect that information rather than mechanically applying its own short backoff time.
The overall request deadline must also be taken into account. If the wait time already exceeds the remaining time budget, the client should not keep retrying.
4.7 The Complete Retry Flow
flowchart TD
A["Initiate model request"] --> B["Wait for response"]
B --> C{"Is the request successful?"}
C -->|"Yes"| D["Return result"]
C -->|"No"| E["Determine error type"]
E --> F{"Is it a retryable error?"}
F -->|"No"| G["Return failure"]
F -->|"Yes"| H{"Retry limit exceeded?"}
H -->|"Yes"| G
H -->|"No"| I["Calculate backoff time"]
I --> J["Add random jitter"]
J --> K["Check remaining time budget"]
K --> L{"Is there time left?"}
L -->|"No"| G
L -->|"Yes"| M["Wait and retry"]
M --> A
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
V. Idempotency: Retries Must Not Re-execute Business Operations
This is an issue that is very easily overlooked in LLM applications.
5.1 What Is Idempotency?
Idempotency means that executing the same operation once or multiple times produces the same final business effect on the system.
For example:
Set the user nickname to Zhang San
No matter how many times it is executed, the final nickname is still Zhang San.
Such operations are usually idempotent.
Whereas:
Increase the account balance by 100 yuan
If it is executed twice, the balance will increase by 200 yuan.
This is not a naturally idempotent operation.
5.2 Why Do Model API Retries Produce Duplicate Operations?
Suppose the Agent needs to execute a transfer:
User: Transfer 100 yuan to Zhang San.
The model generates a tool call request, and the application executes the transfer.
But after the execution succeeds, the network connection suddenly drops, and the application does not receive the success response.
The application cannot determine:
- Whether the transfer has already been executed.
- Whether the tool failed to execute.
- Whether the response was simply lost.
If the application retries directly, it may transfer the money twice.
5.3 Using an Idempotency Key
A unique idempotency key can be generated for each business operation.
For example:
POST /api/transfers
Idempotency-Key: transfer-20261120-000001
Content-Type: application/json
Request:
{
"from_account": "A1001",
"to_account": "A2001",
"amount": 100
}
The server stores the idempotency key and the execution result.
When the same idempotency key is received again, the transfer is not executed a second time; instead, the previously stored result is returned.
Note the following:
- The idempotency key should be generated by the business operation, not regenerated on every retry.
- Idempotency records must be persisted, and a reasonable retention period should be designed.
- The idempotency key must be bound to the request content, so that the same key is not used with different parameters.
- Concurrent requests must be guaranteed to execute only once through a database unique constraint, transactions, or other atomic mechanisms.
- Idempotency records and business execution results should be committed within the same transaction boundary wherever possible.
5.4 Idempotency Boundaries Between Model Generation and Business Execution
Two levels need to be distinguished:
| Operation | Requires idempotency control? |
|---|---|
| Calling a model to generate text | Usually no business side effects, but may incur duplicate costs |
| Model generating tool-call parameters | May generate duplicate results |
| Querying the database | Usually safe to retry, but consistency must be considered |
| Creating an order | Idempotency must be considered |
| Sending email | Duplicate sending must be prevented |
| Transferring funds | Strict idempotency control is required |
| Deleting data | Business-level idempotency and permission control are required |
The retry policy for model requests cannot be directly equated with the retry policy for tool operations.
6. Degradation mechanism: ensuring the system remains available during failures
6.1 What is degradation?
Fallback / degradation refers to the system proactively switching to a less capable but still functional processing method when the primary service is unavailable, responding too slowly, or short on resources.
Degradation is not simply returning an error; it is a deliberate sacrifice of some functionality to preserve core business capabilities.
For example:
- When a high-performance model is unavailable, switch to a backup model.
- When the model times out, return cached results.
- When the RAG retrieval service is unavailable, answer using limited basic knowledge.
- When advanced analytics features are unavailable, provide simplified results.
- When external search fails, clearly state that real-time information cannot be obtained.
6.2 Common degradation methods
| Degradation method | Concrete approach | Applicable scenario |
|---|---|---|
| Cache degradation | Return historical cache | Repetitive queries |
| Model degradation | Switch to backup model | Primary model unavailable |
| Capability degradation | Disable complex reasoning or advanced features | High load |
| Data degradation | Use existing or static data | External data source failure |
| Result degradation | Return simplified content | Insufficient latency budget |
| Human fallback | Hand over to human handling | High-risk tasks |
6.3 Model degradation and model routing
Suppose the system is connected to multiple models at the same time:
- Primary model: high capability, high cost.
- Backup model: medium capability, medium cost.
- Lightweight model: low cost, low latency.
Models can be selected based on request type and service status.
For example:
flowchart TD
A["Receive AI request"] --> B["Classify and route request"]
B --> C["Call primary model first"]
C --> D{"Did primary model succeed?"}
D -->|"Yes"| E["Return primary model result"]
D -->|"No"| F["Determine whether switching is allowed"]
F -->|"No"| G["Return explicit error or human fallback"]
F -->|"Yes"| H["Call backup model"]
H --> I{"Did backup model succeed?"}
I -->|"Yes"| J["Return backup model result"]
I -->|"No"| K["Use lightweight model or cache"]
K --> L{"Is a usable result available?"}
L -->|"Yes"| M["Return degraded result"]
L -->|"No"| G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7EAE7,stroke:#C98D7C,color:#754438
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Model degradation requires attention to differences in model capabilities.
A backup model may not be fully compatible with the primary model:
- Prompt formats may differ.
- Tool-calling protocols may differ.
- JSON Schema support coverage may differ.
- Context windows may differ.
- Multimodal capabilities may differ.
- Output quality and safety policies may differ.
Therefore, before switching models, a unified adaptation layer needs to be established and capability verification performed.
6.4 Degradation Does Not Mean Silently Lowering Quality
For example, the primary model supports complex reasoning, while the fallback model can only perform basic classification.
If the system still labels results after switching as having the same capabilities as the primary model, it may mislead users or downstream systems.
It is recommended to add status information to degraded results:
{
"status": "degraded",
"model": "fallback-model",
"result": {
"category": "technical_issue"
},
"message": "当前使用备用模型处理,结果可能存在能力差异。"
}
In practice, an API can indicate degraded status through response metadata, and does not necessarily need to mix these fields into the business data itself.
7. Circuit Breaker Mechanism: Preventing Ongoing Failure Propagation
7.1 What Is a Circuit Breaker?
A circuit breaker is a fault isolation mechanism.
When an external service continues to fail, the system temporarily stops sending requests to that service to avoid continuing to consume resources.
A circuit breaker mechanism usually has three states:
- Closed: closed state; requests pass normally.
- Open: open state; requests are rejected directly or routed to fallback logic.
- Half-Open: half-open state; allows a small number of probe requests.
7.2 How a Circuit Breaker Works
stateDiagram-v2
[*] --> Closed
Closed --> Open: failure rate or consecutive failures reach threshold
Open --> HalfOpen: circuit breaker wait time ends
HalfOpen --> Closed: probe request succeeds
HalfOpen --> Open: probe request fails
style Closed fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style Open fill:#F7EAE7,stroke:#C98D7C,color:#754438
style HalfOpen fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
Closed:
- Requests pass normally.
- Track request success rate and failures.
- When the failure rate reaches the threshold, trigger the circuit breaker.
Open:
- No longer call the failing service normally.
- Return a degraded result or error directly.
- Wait for the circuit breaker timeout to end.
Half-Open:
- Allow a small number of requests to enter.
- Check whether the upstream service has recovered.
- Decide whether to close or reopen the circuit breaker based on the probe result.
7.3 Why Do LLM Applications Need Circuit Breakers?
Suppose a model provider keeps returning HTTP 503.
If the application still retries each request multiple times:
- Failed requests continue to increase.
- The connection pool is occupied.
- Request latency continues to rise.
- Thread and coroutine resources are consumed.
- Other normal business operations may also be affected.
After the system confirms that the service remains abnormal, the circuit breaker can proactively stop calling the failing upstream.
7.4 The Relationship Between Circuit Breaking and Retry
The two serve different purposes:
| Mechanism | Problem Solved | How It Works |
|---|---|---|
| Retry | Temporary failure of a single request | Try again |
| Circuit Breaker | Persistent upstream failure | Pause calls |
| Timeout | Request has no response for a long time | Limit wait time |
| Fallback | Primary path cannot complete | Switch handling method |
| Rate Limiting | Too many requests | Control ingress rate |
Without circuit breaking, retries may continue to amplify upstream failures.
Without retries, brief network jitter may directly cause request failures.
Without fallback, there may be no available service path after the circuit breaker trips.
They need to be used together.
8. Rate Limiting and Concurrency Control
In addition to caching, retries, and fallback, LLM applications must also manage request traffic.
8.1 Why Is Rate Limiting Needed?
Model providers usually impose limits on:
- Requests per minute (RPM).
- Tokens per minute (TPM).
- Number of concurrent requests.
- Daily call volume.
- Account spending quota.
If the application has no rate limiting control, burst traffic may cause a large number of 429 errors.
8.2 Common Rate Limiting Algorithms
| Algorithm | Principle | Characteristics |
|---|---|---|
| Fixed window | Limits the number of requests within a fixed time window | Simple, but has boundary bursts |
| Sliding window | Counts request volume over a continuous time interval | Smoother |
| Leaky bucket | Processes requests at a fixed rate | Stable output rate |
| Token bucket | Generates tokens at a rate; requests consume tokens | Allows a certain amount of burst traffic |
| Concurrency limit | Limits the number of requests executed simultaneously | Controls resource usage |
For LLM applications, rate limiting solely by request count may not be enough.
Because token consumption varies greatly across different requests.
For example:
- Request A: 100 tokens input, 50 tokens output.
- Request B: 20,000 tokens input, 4,000 tokens output.
Although both count as a single request, the resource consumption differs significantly.
Therefore, rate limiting needs to be multi-dimensional, combining RPM, TPM, and concurrency.
8.3 Token Bucket Algorithm
The token bucket replenishes tokens into the bucket at a fixed rate.
Each time a request comes in, it must consume the corresponding number of tokens.
flowchart TD
A["Request arrives"] --> B["Check token bucket"]
B --> C{"Are tokens sufficient?"}
C -->|"Yes"| D["Consume tokens"]
D --> E["Allow request to execute"]
C -->|"No"| F["Wait or reject"]
G["Replenish tokens periodically"] --> B
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
The token bucket allows a certain degree of burst traffic, but the long-term average request rate is still limited by the token generation rate.