I. Why AI Model Serving Needs High Availability Design
Compared with traditional web services, AI model serving has a more complex runtime environment and resource dependencies.
Traditional web services can usually scale horizontally by adding application instances, whereas AI model serving must also account for model weights, GPU memory, inference engines, KV Cache, model loading time, and coordination among compute resources.
A complete AI model serving system may depend on:
- API Gateway: receives requests and performs authentication, routing, and rate limiting.
- Model Router: selects inference instances based on model version, load, and resource status.
- Model Server: loads models and executes inference.
- GPU Worker: provides GPU compute resources.
- Cache: caches model weights, Prefix Cache, KV Cache, or inference results.
- Model Registry: manages model files, versions, and deployment configurations.
- Object Storage: stores model weights, datasets, and inference results.
- Scheduler: handles task scheduling and resource allocation.
- Monitoring: collects runtime metrics, logs, and distributed tracing data.
A failure in any one of these critical links can affect user requests.
1.1 Typical Failures in AI Model Serving
| Failure Type | Typical Symptom | Impact |
|---|---|---|
| GPU failure | GPU Xid, device dropout, compute errors | Inference instance unavailable |
| Out of GPU memory | CUDA Out of Memory | Request failure or process crash |
| Model loading failure | Corrupted weights, download timeout | Instance fails to start |
| Inference process crash | Segmentation Fault, Runtime Error | Current instance stops serving |
| Network failure | Connection interruption between nodes | Request timeout or lost results |
| Cache failure | Redis unavailable, cache avalanche | Increased latency, higher compute pressure |
| Traffic surge | Concurrency exceeds processing capacity | Request backlog, increased timeouts |
| Scheduler failure | Unable to allocate GPU or Worker | New requests cannot be executed |
| Model version failure | New model has compatibility issues | Inference errors or performance degradation |
| Dependency service failure | Tokenizer, vector database, etc. unavailable | Inference pipeline interrupted |
High availability design does not mean guaranteeing that no component ever fails; rather, it ensures that after a failure occurs:
- The failure can be detected promptly.
- The scope of the failure can be isolated.
- Automatic failover or recovery can occur.
- The remaining healthy resources can be protected.
- Core business capabilities can be maintained as far as possible.
1.2 Core Metrics of High Availability
High availability design usually requires defining several key metrics.
Availability
Indicates the proportion of time within a specified period that the service can provide normal service.
It can also be calculated from downtime:
Common availability targets:
| Availability Target | Theoretical Annual Downtime |
|---|---|
| 99% | About 87.6 hours |
| 99.9% | About 8.76 hours |
| 99.95% | About 4.38 hours |
| 99.99% | About 52.56 minutes |
| 99.999% | About 5.26 minutes |
The downtime figures here are theoretical budgets; whether they are actually met depends on the definition of the service level objective (SLO) and the statistical window.
SLO (Service Level Objective)
A service level objective defines the level of reliability the system must achieve.
For example:
- Monthly availability of no less than 99.95%.
- Online inference request success rate of no less than 99.9%.
- P95 TTFT of no more than 2 seconds.
- P99 inference request error rate below a specified threshold.
SLI (Service Level Indicator)
A service level indicator is the actual observed data used to measure an SLO.
For example:
- Successful requests / total requests.
- Normal service time / total service time.
- P95 and P99 request latency.
- Timeout rate.
- Model inference failure rate.
RTO (Recovery Time Objective)
Recovery time objective, the maximum recovery time allowed after a failure occurs.
RPO (Recovery Point Objective)
Recovery point objective, the tolerable range of data loss when a failure occurs.
For stateless inference instances, it is usually unnecessary to restore historical inference progress; however, model configurations, task states, session records, and business data may require a strict RPO.
II. High Availability Architecture for AI Model Services
2.1 Problems with Single-Instance Deployment
The simplest model service architecture is an API service connected to a single model inference instance.
flowchart TD
A["User Request"] --> B["API Service"]
B --> C["Single Model Inference Instance"]
C --> D["GPU"]
D --> E["Return Result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#E5F0E9,stroke:#83A995,color:#315D49
This architecture is simple, but it has obvious single points of failure:
- If the API service crashes, all requests cannot enter.
- If the GPU fails, the model cannot continue inference.
- If the model process runs OOM, the entire service becomes unavailable.
- During model upgrades, the service may need to be taken offline.
- When traffic exceeds the capacity of a single instance, all requests are affected.
This deployment is suitable for development, testing, and low-traffic internal applications, but not for production environments with clear SLA requirements.
2.2 Multi-Instance Redundant Architecture
The first step to improving availability is to deploy multiple independent model instances.
flowchart TD
A["User Request"] --> B["API Gateway"]
B --> C["Load Balancer"]
C --> D["Model Server A"]
C --> E["Model Server B"]
C --> F["Model Server C"]
D --> G["GPU Node A"]
E --> H["GPU Node B"]
F --> I["GPU Node C"]
G --> J["Inference Result"]
H --> J
I --> J
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style J fill:#E5F0E9,stroke:#83A995,color:#315D49
Multiple instances can achieve:
- Switching to other instances when an instance fails.
- Distributing traffic across multiple GPU nodes.
- Supporting rolling upgrades.
- Supporting dynamic scaling based on load.
- Reducing the impact of a single-node failure on the overall service.
However, note that multiple instances do not necessarily mean true high availability.
If all three model instances are deployed on the same physical server, a server failure will still render all instances unavailable.
Therefore, redundancy must cover different fault domains.
2.3 Multi-Layer High Availability Architecture
Production-grade AI model services typically adopt a multi-layer redundancy design.
flowchart TD
A["Client"] --> B["DNS / Global Traffic Management"]
B --> C["Region A Entry"]
B --> D["Region B Entry"]
C --> E["API Gateway Cluster"]
D --> F["API Gateway Cluster"]
E --> G["Region A Model Router"]
F --> H["Region B Model Router"]
G --> I["Model Inference Instance Pool A"]
H --> J["Model Inference Instance Pool B"]
I --> K["GPU Resource Pool A"]
J --> L["GPU Resource Pool B"]
K --> M["Shared Model Artifacts and Configuration Management"]
L --> M
E --> N["Monitoring and Alerting"]
F --> N
G --> N
H --> N
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
This architecture eliminates single points of failure at multiple levels:
| Layer | High Availability Measures |
|---|---|
| DNS / Traffic Entry | Multi-region entry, health checks, traffic failover |
| API Gateway | Multiple instances, stateless deployment |
| Model Router | Multiple replicas, externalized state |
| Model Server | Multi-instance redundancy |
| GPU Nodes | Cross-host and cross-rack deployment |
| Model Artifacts | Multi-replica storage, verification, and version management |
| Metadata Storage | Primary-replica replication, clustering, or multi-region disaster recovery |
| Monitoring System | Independent deployment, fault alerting |
Two concepts need to be distinguished:
- High Availability (HA): When a failure occurs, the system can continue to provide service or recover quickly.
- Disaster Recovery (DR): When a region-level, data center-level, or large-scale infrastructure failure occurs, the system can recover business operations.
Multi-region deployment is a higher level of disaster recovery capability, but it also increases costs for data synchronization, networking, model distribution, and operations and maintenance.
3. Failure Domains and Redundancy Design
The key to a highly available architecture is not merely increasing the number of instances, but localizing failures as much as possible.
3.1 Failure Domains
A failure domain refers to a group of resources that may be affected simultaneously by the same failure cause.
Common failure domains include:
| Failure Domain | Typical Failure |
|---|---|
| Process level | Python process crash, memory leak |
| Container level | Container exit, health check failure |
| Host level | Operating system crash, GPU failure |
| Rack level | Power, switch, or network failure |
| Availability zone level | Data center network, power, or infrastructure failure |
| Region level | Large-scale cloud service or network failure |
If multiple model instances share the same failure domain, their failures are not independent.
Therefore, during deployment, they should be distributed as much as possible:
- Different physical hosts.
- Different GPU nodes.
- Different racks.
- Different availability zones.
- Different regions.
3.2 N+1 Redundancy
N+1 redundancy means the system requires N instances for normal operation, while one additional standby instance is prepared.
For example:
- Normally requires 4 model instances.
- 5 instances are actually deployed.
- When one instance fails, the remaining 4 instances continue to handle the workload.
However, N+1 does not necessarily apply to all GPU services.
If GPU resources are very expensive, the following may be adopted:
- N+1 hot standby.
- Partial capacity reservation.
- Multiple instances share redundant capacity.
- Cross-region cold standby.
The choice should be based on the recovery time objective and cost.
3.3 Active-Active: Dual-Active Deployment
Active-Active means multiple instances process requests at the same time.
For example:
Region A: handles 50% of requests
Region B: handles 50% of requests
Advantages:
- All nodes can provide actual service.
- Resources are not left idle for long periods.
- Failover is usually faster.
- Traffic can be shared across regions.
Disadvantages:
- Model versions need to remain compatible.
- Data and cache consistency must be considered.
- Cross-region traffic may increase network latency.
- Multi-region resource scheduling is more complex.
For stateless inference services, Active-Active is usually relatively easy to implement.
However, for stateful sessions, task execution progress, and distributed inference, additional state synchronization and recovery mechanisms need to be designed.
3.4 Active-Standby: Primary-Standby Deployment
Active-Standby means the primary instance handles requests, and the standby instance takes over upon failure.
Normal state:
Primary -> handles all requests
Standby -> waits to take over
Failure state:
Primary -> unavailable
Standby -> takes over requests
Advantages:
- The architecture is relatively simple.
- Standby resources can be configured on demand.
- Suitable for services with low traffic but requiring fast recovery.
Disadvantages:
- Standby resources may remain idle for long periods.
- Health checks and failover mechanisms are required.
- The standby model may not yet be loaded onto the GPU.
- Failover may involve significant cold-start time.
For large models, loading weights on the standby node may take a long time, so services that truly require a low RTO usually need to pre-warm standby instances.
4. Health Checks for Model Instances
A high-availability system must be able to determine whether a model instance can actually process requests.
Traditional web services can usually make a basic determination through HTTP health checks, but the health status of AI model services is more complex.
4.1 Three Basic Health Checks
Liveness Probe: Liveness Check
Checks whether the process is still alive.
For example:
- Whether the HTTP service can respond.
- Whether the main loop is still running.
- Whether critical threads are functioning normally.
- Whether a deadlock has occurred.
If the liveness probe fails, the platform can restart the container.
However, the liveness probe should not rely too heavily on model inference; otherwise, a brief period of GPU busyness may cause the container to restart repeatedly.
Readiness Probe: Readiness Check
Checks whether the instance is already capable of handling real requests.
Typical checks include:
- Whether model weights have finished loading.
- Whether the tokenizer is available.
- Whether the GPU can be accessed normally.
- Whether the inference engine has finished initializing.
- Whether critical dependencies meet the requirements.
- Whether it is currently draining or upgrading.
When readiness fails, the instance should be removed from the traffic pool, but it does not necessarily need to be restarted.
Startup Probe: Startup Check
Used to protect services with long startup times.
Large models may take a long time to:
- Download model weights.
- Load weights onto the GPU.
- Initialize CUDA.
- Create the KV Cache management structure.
- Perform warm-up inference.
The startup probe can prevent the service from being prematurely judged as abnormal during startup.
4.2 Health Status of AI Model Services
It is recommended to divide the states of a model instance into:
| State | Meaning | Accepts New Requests |
|---|---|---|
| STARTING | Starting | No |
| LOADING | Loading model | No |
| WARMING | Warming up | Usually no |
| READY | Can serve normally | Yes |
| DEGRADED | Performance degraded or some capabilities are abnormal | Depends on policy |
| DRAINING | Draining requests | No |
| UNHEALTHY | Health check failed | No |
| TERMINATED | Stopped | No |
Health state transitions:
flowchart TD
A["STARTING"] --> B["LOADING"]
B --> C["WARMING"]
C --> D{"Warm-up successful?"}
D -->|Yes| E["READY"]
D -->|No| F["UNHEALTHY"]
E --> G{"Operating normally?"}
G -->|Yes| E
G -->|Partially abnormal| H["DEGRADED"]
H --> I{"Recovered?"}
I -->|Yes| E
I -->|No| F
E --> J["DRAINING"]
H --> J
J --> K["TERMINATED"]
F --> L["Restart or reschedule"]
style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F7EAE7,stroke:#C98D7C,color:#754438
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style J fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style K fill:#E5F0E9,stroke:#83A995,color:#315D49
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
4.3 Why health checks cannot rely solely on HTTP 200
Model services may encounter the following situations:
- The HTTP process is healthy, but the GPU is already unavailable.
- The model has finished loading, but a deadlock occurs inside the inference engine.
- The service can return health check results, but the inference queue is severely backlogged.
- The model can perform inference, but actual outputs remain abnormal.
- The GPU is normal, but a critical Tokenizer file is missing.
Therefore, health checks should be split into multiple layers:
- Process health.
- Service health.
- Model health.
- GPU health.
- Inference capability health.
- Performance health.
Performance health, in particular, can be judged by latency, queue backlog, timeout rate, and resource pressure, rather than relying only on a boolean state.
V. Traffic Management and Load Balancing
After multiple model instances are deployed, you still need to decide which instance each request is sent to.
5.1 Load Balancing Strategies
Common strategies are as follows:
| Strategy | How it works | Use cases |
|---|---|---|
| Round Robin | Distributes requests in turn | Instances have similar capacity |
| Weighted Round Robin | Distributes requests by weight | Instances have different specifications |
| Least Connections | Selects the instance with fewer connections | Request durations vary widely |
| Least Outstanding Requests | Selects the instance with fewer outstanding requests | Inference durations differ significantly |
| Latency-aware | Selects instances based on historical latency | Low latency is the goal |
| Resource-aware | Routes based on GPU, VRAM, and queue status | AI inference services |
| Consistent Hashing | Maps stably by Key | Cache locality is required |
For AI model services, ordinary Round Robin is often insufficient.
This is because the computational cost of different requests can vary greatly.
For example:
- Request A: 100 input tokens, 50 output tokens.
- Request B: 10000 input tokens, 2000 output tokens.
If the two requests are treated as having the same workload, simple round-robin may cause some GPU nodes to accumulate many long requests.
A more reasonable approach is to schedule based on the following metrics:
- Current number of active requests.
- Number of queued requests.
- GPU compute utilization.
- GPU memory usage.
- Available KV Cache space.
- Number of input tokens per request.
- Estimated number of output tokens.
- Estimated execution time.
- Prefix Cache hit status.
5.2 Request Routing Architecture
flowchart TD
A["Request Incoming"] --> B["API Gateway"]
B --> C["Authentication and Quota Check"]
C --> D["Model Router"]
D --> E["Get Instance Health Status"]
E --> F["Get Resource and Queue Metrics"]
F --> G["Check Model Version and Request Capabilities"]
G --> H{"Is There an Available Instance?"}
H -->|Yes| I["Calculate Routing Score"]
H -->|No| J["Rate Limit, Queue, or Return Error"]
I --> K["Select Target Instance"]
K --> L["Forward Inference Request"]
L --> M{"Is the Request Successful?"}
M -->|Yes| N["Return Inference Result"]
M -->|No| O["Determine Whether Retry Is Allowed"]
O --> P{"Can It Be Safely Retried?"}
P -->|Yes| D
P -->|No| Q["Return Error or Degraded Result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#E5F0E9,stroke:#83A995,color:#315D49
style O fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style P fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style Q fill:#F7EAE7,stroke:#C98D7C,color:#754438
5.3 Request Retries and Duplicate Execution
After a model instance fails, the Router may resend the request to another instance.
However, note: a request timeout does not mean the model did not execute.
For example:
- The request has already been sent to GPU node A.
- Node A completes inference.
- The network is interrupted while the result is being returned.
- API Gateway considers the request timed out.
- The request is resent to node B.
- The same task may be executed again.
This may lead to:
- Repeated consumption of GPU resources.
- Repeated execution of tool calls.
- Repeated generation of business side effects.
- Different instances returning different results.
Therefore, duplicate execution needs to be controlled through Request ID, Idempotency Key, and task state management.
For plain text inference, duplicate execution may mainly cause cost and latency issues; for Agent tool calls, database writes, and external API operations, it may create real business risks.
6. Timeout, Retry, and Circuit Breaker Mechanisms
A high-availability system cannot wait indefinitely for failure recovery, nor can it continue sending requests when a dependent service is failing.
6.1 Timeout: Timeout Control
Timeout means that once a request exceeds a preset time, it stops waiting for the current operation.
AI model services need to distinguish timeouts at different stages:
| Timeout Type | Meaning |
|---|---|
| Connect Timeout | Connection establishment timeout |
| Queue Timeout | Request queueing timeout |
| Model Load Timeout | Model loading timeout |
| Prefill Timeout | Input processing timeout |
| First Token Timeout | First token timeout |
| Idle Timeout | Streaming output has no data for an extended period |
| Total Request Timeout | Total request duration timeout |
| Dependency Timeout | Dependency service call timeout |
For streaming inference, you cannot simply use a very short total request timeout.
Because generating longer text may take tens of seconds or even longer.
A more reasonable approach is:
- Control connection establishment time.
- Control queueing time.
- Control first-token time.
- Control the maximum idle time between two token outputs.
- Set a reasonable upper limit on total execution time.
6.2 Retry: Retry Mechanism
Retries are suitable for failures that may recover temporarily.
Common retryable errors:
- Temporary network connection failure.
- GPU Worker is temporarily unavailable.
- Service returns 503.
- Downstream dependency experiences a brief timeout.
- Requests are rejected due to excessive load.
Errors that should not be blindly retried:
- Invalid parameter format.
- Model version does not exist.
- Authentication failed.
- Insufficient user permissions.
- Token count exceeds a hard limit.
- Unsupported input format.
A common retry strategy is exponential backoff:
For example:
| Retry count | Base wait time |
|---|---|
| 1st retry | 1 second |
| 2nd retry | 2 seconds |
| 3rd retry | 4 seconds |
| 4th retry | 8 seconds |
In practice, systems should also add random jitter to avoid a large number of requests being retried at the same time.
6.3 Circuit Breaker: Circuit Breaker Mechanism
A circuit breaker prevents continued access to a service that has already failed.
It generally has three states:
- CLOSED: normal state, allowing requests to pass through.
- OPEN: circuit-breaker state, directly rejecting requests or failing fast.
- HALF-OPEN: half-open state, allowing a small number of probe requests.
Workflow:
flowchart TD
A["Request received"] --> B["Circuit breaker"]
B --> C{"Current state"}
C -->|CLOSED| D["Call model service"]
C -->|OPEN| E["Fail fast or degrade"]
C -->|HALF-OPEN| F["Allow a small number of probe requests"]
D --> G{"Call result"}
G -->|Success| H["Success count and state recovery"]
G -->|Failure| I["Increment failure count"]
I --> J{"Has circuit-breaker threshold been exceeded?"}
J -->|Yes| K["Enter OPEN"]
J -->|No| L["Continue observing"]
F --> M{"Is probe successful?"}
M -->|Yes| N["Restore CLOSED"]
M -->|No| K
K --> O["Wait for circuit-breaker window"]
O --> F
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7EAE7,stroke:#C98D7C,color:#754438
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style K fill:#F7EAE7,stroke:#C98D7C,color:#754438
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style N fill:#E5F0E9,stroke:#83A995,color:#315D49
style O fill:#EAF3EC,stroke:#83A995,color:#315D49
Circuit breakers need to be isolated by dependency or model instance.
For example:
- A failure of Model A should not affect Model B.
- A failure of GPU node A should not trip the circuit breaker for the entire inference cluster.
- A Redis failure should not directly make all model services unavailable.
6.4 Differences Between Retry and Circuit Breaker
| Mechanism | Problem Solved | Core Role |
|---|---|---|
| Timeout | Requests wait too long | Limit waiting time |
| Retry | Temporary failures | Attempt recovery |
| Circuit Breaker | Persistent downstream failures | Block invalid requests |
| Fallback | Primary service unavailable | Provide alternative capability |
| Bulkhead | Resource interference | Isolate fault scope |
These mechanisms need to work together rather than replace one another.
7. Fault Isolation and Bulkhead Pattern
The Bulkhead Pattern originates from the compartment design concept of ships: when one area fails, it should not bring down the entire system.
In AI model services, the Bulkhead Pattern is especially important because different models and tasks vary greatly in resource consumption.
7.1 Isolation by Model
Different models use dedicated inference resource pools.
For example:
Model A
-> GPU Pool A
-> Worker A
Model B
-> GPU Pool B
-> Worker B
Model C
-> GPU Pool C
-> Worker C
This prevents an OOM or inference anomaly in one model from affecting other models.
7.2 Isolation by Task Type
Different types of AI tasks use different resource pools:
| Task Type | Recommended Isolation Method |
|---|---|
| Online LLM Inference | Dedicated GPU Pool |
| Embedding | Dedicated Inference Queue or Resource Pool |
| OCR | Dedicated CPU/GPU Worker |
| Document Parsing | CPU and Memory Resource Pool |
| Batch RAG Indexing | Dedicated Asynchronous Task Pool |
| Model Training | Dedicated Training Cluster |
In particular, avoid large-scale offline tasks preempting online inference resources.
7.3 Isolation by Tenant
A multi-tenant AI platform needs to control resource usage by different tenants.
You can set:
- Maximum concurrency per tenant.
- Requests per minute.
- Daily token quota.
- Maximum context length.
- Maximum number of output tokens.
- GPU resource budget.
- Priority and service level.
This prevents abnormal traffic from a single tenant from affecting other tenants.
7.4 Complete Structure of Resource Isolation
flowchart TD
A["AI Model Service"] --> B["Request Classification"]
B --> C["Tenant A"]
B --> D["Tenant B"]
B --> E["Background Tasks"]
C --> F["Online Inference Resource Pool"]
D --> G["Dedicated Inference Resource Pool"]
E --> H["Offline Task Resource Pool"]
F --> I["GPU Worker A"]
G --> J["GPU Worker B"]
H --> K["CPU / GPU Worker C"]
I --> L["Dedicated Queues and Quotas"]
J --> M["Dedicated Queues and Quotas"]
K --> N["Dedicated Queues and Quotas"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#E5F0E9,stroke:#83A995,color:#315D49
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
style N fill:#E5F0E9,stroke:#83A995,color:#315D49
8. Failover and Automatic Recovery
Failover refers to transferring requests to other available instances after the primary service or node fails.
8.1 Basic Process of Failover
A typical process includes:
- The monitoring system detects an anomaly.
- Health checks confirm that the instance is unavailable.
- Mark the anomalous instance as Unhealthy.
- Remove it from the load balancing pool.
- Route new requests to healthy instances.
- Retry requests that can be safely retried.
- Start a replacement instance.
- After the instance finishes warming up, add it back to the traffic pool.
flowchart TD
A["Model instance running normally"] --> B["Health checks and metric monitoring"]
B --> C{"Is the instance healthy?"}
C -->|Yes| A
C -->|No| D["Mark instance as Unhealthy"]
D --> E["Remove from traffic pool"]
E --> F["Route to other healthy instances"]
F --> G["Check whether the request can be retried"]
G --> H{"Is retry allowed?"}
H -->|Yes| I["Resend the request"]
H -->|No| J["Return an error or degrade"]
D --> K["Start a replacement instance"]
K --> L["Load model weights"]
L --> M["Perform warm-up and health checks"]
M --> N{"Is it ready?"}
N -->|Yes| O["Rejoin the traffic pool"]
N -->|No| P["Continue recovery or isolate"]
I --> Q["Monitor recovery status"]
J --> Q
O --> Q
P --> Q
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F7EAE7,stroke:#C98D7C,color:#754438
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#F7EAE7,stroke:#C98D7C,color:#754438
style K fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#EAF3EC,stroke:#83A995,color:#315D49
style N fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style O fill:#E5F0E9,stroke:#83A995,color:#315D49
style P fill:#F7EAE7,stroke:#C98D7C,color:#754438
style Q fill:#E5F0E9,stroke:#83A995,color:#315D49
8.2 Failure Detection Should Not Be Too Sensitive
If health checks are too sensitive, false positives may occur:
- The GPU is briefly busy.
- The network has a momentary jitter.
- Model inference queue times become longer.
- Occasional request timeouts.
These situations do not necessarily mean that the instance has actually failed.
It is recommended to use:
- Number of consecutive failures.
- Multiple time windows.
- Combinations of metric thresholds.
- Active probe requests.
- Passive request failure rate.
For example, remove an instance only after multiple consecutive health check failures.
You can also set a recovery observation period to prevent an instance from immediately receiving a large number of requests right after it recovers.
8.3 Automatic Recovery and Manual Intervention
Not all failures should trigger automatic restarts.
For example:
- A GPU driver anomaly may require a node restart.
- Corrupted model weights need to be downloaded again.
- A VRAM leak may require replacing the process.
- Incorrect configuration may cause new instances to keep crashing.
- Data corruption may require manual repair.
It is recommended to establish a tiered recovery strategy:
| Failure Level | Handling Method |
|---|---|
| Temporary network failure | Automatic retry |
| Single-request inference failure | Isolate the request and log the error |
| Model process anomaly | Restart the process |
| GPU device anomaly | Remove the node and reschedule |
| Corrupted model file | Re-pull and verify the model |
| Configuration release error | Roll back the configuration |
| Regional failure | Start cross-region disaster recovery |
9. High Availability Design for Model Deployment and Version Upgrades
Model upgrades are a very important source of risk in AI services.
A new model may exhibit the following:
- Incompatible model weights.
- Tokenizer mismatch.
- The inference engine does not support new operators.
- Degraded output quality.
- Increased inference latency.
- Increased VRAM usage.
- Specific inputs cause the process to crash.
Therefore, model deployment needs to support progressive rollout and fast rollback capabilities.
9.1 Rolling Update
A rolling update is a deployment method that gradually replaces old instances.
For example, suppose there are originally 4 instances:
- Start a new-version instance first.
- Wait for the new instance to finish model loading and warm-up.
- Remove one old instance from the traffic pool.
- Wait for the old instance to finish processing in-flight requests.
- Update the next instance.
- End the release after all updates are complete.
This maintains a certain level of service capacity during the upgrade.
Note that during a rolling update, old and new versions may coexist, so the API protocol, model inputs and outputs, and dependent services must remain compatible.
9.2 Blue-Green Deployment
Blue-green deployment prepares two complete environments:
- Blue: the current production environment.
- Green: the new environment to be released.
After the new version is deployed, warmed up, and validated in the Green environment, traffic is switched over.
flowchart TD
A["Current traffic"] --> B["Traffic ingress"]
B --> C["Blue environment: old version"]
D["Deploy new version"] --> E["Green environment"]
E --> F["Load model and warm up"]
F --> G["Run validation tests"]
G --> H{"Validation passed?"}
H -->|No| I["Stop release and keep Blue"]
H -->|Yes| J["Switch production traffic to Green"]
J --> K["Monitor new version metrics"]
K --> L{"Any anomalies?"}
L -->|No| M["Release complete"]
L -->|Yes| N["Roll back to Blue"]
N --> O["Restore old version traffic"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#F7EAE7,stroke:#C98D7C,color:#754438
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style M fill:#E5F0E9,stroke:#83A995,color:#315D49
style N fill:#F7EAE7,stroke:#C98D7C,color:#754438
style O fill:#EAF3EC,stroke:#83A995,color:#315D49
The advantage of blue-green deployment is a clear rollback path, but the disadvantage is that an additional environment must be prepared.
For large models, the additional GPU resources and model weight storage costs can be high.
9.3 Canary Release
Canary release first routes a small amount of traffic to the new version, then gradually increases the traffic proportion.
For example:
Stage 1: 1% of traffic goes to the new version
Stage 2: 5% of traffic goes to the new version
Stage 3: 20% of traffic goes to the new version
Stage 4: 50% of traffic goes to the new version
Stage 5: 100% of traffic goes to the new version
At each stage, monitor:
- Request success rate.
- P95/P99 latency.
- TTFT and TPOT.
- GPU memory usage.
- Number of OOMs.
- Model output quality.
- Token cost.
- User feedback.
For generative AI, you cannot monitor only system performance; you must also monitor model quality.
For example, although a new model may run normally, it may still exhibit:
- Lower answer accuracy.
- Higher hallucination rate.
- Unstable JSON output format.
- Lower tool-call success rate.
- Degraded performance in specific languages.
Therefore, model releases need to combine automated evaluation with online metrics.
9.4 Comparison of Release Strategies
| Strategy | Advantages | Disadvantages | Use Cases |
|---|---|---|---|
| Rolling Update | Relatively low resource overhead | Rollback is relatively complex | Routine model upgrades |
| Blue-Green | Clear switching and rollback | High resource overhead | High-reliability model releases |
| Canary | Risk is released gradually | Requires traffic allocation and metric analysis | Large-scale online model upgrades |
| Shadow Deployment | Allows comparison of old and new models | Increases compute cost | Model quality validation |
Shadow Deployment copies production requests to the new model but does not return the new model's results to users.
This allows observing the new model's latency, resource usage, and output differences under real traffic.