1. What Is LLM Deployment?
LLM deployment is the process of turning a trained or already released model into a service that applications can call, through compute resources, an inference engine, and service interfaces.
In simple terms, model deployment addresses three core questions:
- Where does the model run? Local devices, enterprise servers, cloud servers, or a third-party model platform.
- How does the model provide service? A local inference interface, an HTTP API, a container service, or a distributed inference cluster.
- How does the model run reliably? How to manage GPU memory, concurrency, latency, throughput, failure recovery, and resource costs.
Several easily confused concepts need to be distinguished:
| Concept | Meaning |
|---|---|
| Model training | Adjusting model parameters using data and algorithms |
| Model fine-tuning | Continuing training on top of an existing model so that it adapts to a specific task |
| Model deployment | Placing a model into a runtime environment to provide inference capability |
| Model inference | Generating results with an already trained model |
| Model serving | Packaging inference capability into a service that other systems can call |
| Model hosting | A platform or provider takes charge of running the model and managing the infrastructure |
Model deployment does not necessarily require buying your own GPUs, nor does it necessarily require managing an inference framework yourself. Calling a third-party LLM API is likewise a way of using an already deployed model service.
2. The Overall Architecture of LLM Deployment
A complete LLM deployment system usually consists of five parts: model files, the inference engine, compute resources, service interfaces, and operations monitoring.
1. Core Components
Model Layer
Provides the model weights and related configuration.
Mainly includes:
- Model weight files.
- Model configuration files.
- Tokenizer.
- Model architecture definition.
- Quantization parameters.
- Model version information.
Inference Layer (Inference Engine)
Executes the model's forward computation.
Main functions:
- Model loading.
- Tokenization.
- Prefill.
- Decode.
- KV Cache management.
- Batch scheduling.
- GPU memory management.
- Quantized computation.
Common inference frameworks include:
- vLLM
- SGLang
- Hugging Face Transformers
- TensorRT-LLM
- llama.cpp
- Ollama
Compute Layer
Provides the hardware resources the model needs in order to run.
Mainly includes:
- CPU.
- GPU.
- NPU.
- GPU memory.
- System memory.
- High-speed storage.
- GPU interconnect network.
Serving Layer
Exposes inference capability to upper-layer applications.
Mainly includes:
- HTTP API.
- OpenAI-compatible interface.
- Request queue.
- Concurrency control.
- Streaming responses.
- Authentication.
- Rate limiting and load balancing.
Operations Layer
Ensures that the model service runs stably over the long term.
Mainly includes:
- Service monitoring.
- Log management.
- Performance analysis.
- Model version management.
- Autoscaling.
- Failure recovery.
- Resource cost management.
2. LLM Deployment Architecture Diagram
flowchart TB
A["User Application"] --> B["API Gateway"]
B --> C["Request Scheduling and Load Balancing"]
C --> D["Model Serving Layer"]
D --> E["Inference Engine"]
E --> F["Model Loading and Computation"]
F --> G["Model Weights"]
F --> H["Tokenizer"]
F --> I["KV Cache"]
E --> J["GPU / CPU / NPU"]
J --> K["Compute and GPU Memory Resources"]
D --> L["Streaming Response"]
L --> M["Return Result to Application"]
N["Monitoring and Logging System"] -.-> D
N -.-> E
N -.-> J
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style K fill:#EAF3EC,stroke:#83A995,color:#315D49
style L fill:#EAF3EC,stroke:#83A995,color:#315D49
style M fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style N fill:#EAE8FA,stroke:#A49BD5,color:#51477D
3. How Is a Single Model Inference Request Completed?
Take a locally deployed chat model as an example:
flowchart TD
A["Receive User Request"] --> B["Tokenizer Encoding"]
B --> C["Build Model Input"]
C --> D["Prefill Stage"]
D --> E["Generate First Token"]
E --> F["Decode Stage"]
F --> G{"Generation Finished?"}
G -->|"No"| H["Update KV Cache"]
H --> F
G -->|"Yes"| I["Tokenizer Decoding"]
I --> J["Return Generated Result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
It is important to note here that model deployment is not simply loading model files into GPU memory. What really affects service performance also includes request scheduling, batching, KV Cache management, and GPU compute efficiency.
III. Method 1: Calling a Cloud LLM API
Cloud APIs are one of the simplest and most common ways to use large models.
Developers do not need to download model weights themselves or purchase and maintain GPU servers; they only need to call a third-party model service through HTTP requests.
1. How It Works
flowchart TD
A["Business application"] --> B["Construct API request"]
B --> C["Authentication"]
C --> D["Cloud model API"]
D --> E["Model inference cluster"]
E --> F["Generate model output"]
F --> G["API returns response"]
G --> H["Business application processes result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The application is only responsible for sending requests; model loading, GPU scheduling, inference computation, and underlying operations and maintenance are all handled by the service provider.
2. Typical Invocation Method
Take calling an OpenAI-compatible API with Python as an example:
from openai import OpenAI
client = OpenAI(
api_key="YOUR_API_KEY",
base_url="https://api.example.com/v1"
)
response = client.chat.completions.create(
model="example-model",
messages=[
{
"role": "user",
"content": "解释什么是大模型部署"
}
],
temperature=0.7
)
print(response.choices[0].message.content)
The domain name and model name here are only examples; in actual use, they should be replaced with the API address and model identifier provided by the service provider.
3. Advantages
- No need to configure a GPU yourself.
- Low integration cost.
- A wide choice of models.
- Usually able to use larger-scale models directly.
- The service provider handles model operation and infrastructure maintenance.
- Easy to scale the business quickly.
4. Disadvantages
- Depends on external networks.
- Requires bearing API call costs.
- May be subject to request rate and token quota limits.
- Data must be processed according to the service provider's API and data policies.
- Model versions and service policies may change.
- Limited control over the underlying inference process.
5. Use Cases
- Rapid development of AI applications.
- Startup projects and MVPs.
- Internal enterprise intelligent assistants.
- Teams with low operations capability.
- Businesses that do not need full control over model infrastructure.
Common services include OpenAI API, Anthropic API, Google Gemini API, and other model platforms that provide compatible APIs.
IV. Method 2: Local Deployment (Local Deployment)
Local deployment means downloading the model to your own device or server and performing inference on local computing resources.
Local deployment does not necessarily mean it can only run on a personal computer; it can also be deployed on enterprise internal servers or in a LAN environment.
1. How It Works
flowchart TD
A["Download model files"] --> B["Prepare local runtime environment"]
B --> C["Install inference framework"]
C --> D["Load model weights"]
D --> E["Allocate CPU / GPU memory"]
E --> F["Start local inference service"]
F --> G["Local application sends request"]
G --> H["Model performs inference"]
H --> I["Return generated result"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
2. Common Local Deployment Tools
Ollama
Suitable for individual developers and local model experimentation.
Features:
- Simple to install and start.
- Supports common open-source models.
- Provides a local API.
- Supports model download and management.
- Suitable for local development and lightweight applications.
llama.cpp
Suitable for CPU/GPU hybrid inference and quantized model deployment.
Features:
- Supports the GGUF model format.
- Supports multiple hardware backends.
- Can run quantized models on ordinary computers.
- Supports CPU and partial GPU acceleration.
- Suitable for resource-constrained environments.
Transformers
Suitable for model development, experimentation, and custom inference.
Features:
- Integrated with the Hugging Face ecosystem.
- Flexible model loading methods.
- Easy to modify inference logic.
- Supports a large number of model architectures.
- Suitable for algorithm validation and secondary development.
3. Local Deployment Example
Using the Ollama command-line tool:
# Download the model
ollama pull qwen3
# Start model interaction
ollama run qwen3
# List local models
ollama list
You can also call the model through a local HTTP interface:
import requests
response = requests.post(
"http://localhost:11434/api/generate",
json={
"model": "qwen3",
"prompt": "解释什么是本地大模型部署",
"stream": False
},
timeout=120
)
print(response.json()["response"])
The specific model names and API parameters should be based on the official documentation for the locally installed version.
4. Advantages
- Model data can remain in the local environment.
- No need to pay third-party model services per token.
- You can control the model version.
- Inference parameters can be customized.
- Suitable for offline or network-restricted environments.
- Convenient for development and debugging.
5. Disadvantages
- Requires preparing compute resources.
- Model loading may consume a large amount of memory.
- Large models may not be able to run on ordinary devices.
- Concurrency capability depends on local hardware.
- You need to maintain model files and the runtime environment yourself.
- Performance optimization requires some technical experience.
6. Use Cases
- Personal learning and experimentation.
- Local AI development environments.
- Small-scale applications within enterprises.
- Offline inference.
- Scenarios that require local data processing.
V. Method 3: Cloud Server Deployment
Cloud server deployment means running the model on a virtual machine or GPU instance provided by a cloud vendor.
Unlike directly calling a cloud model API, cloud server deployment usually requires users to manage the model, inference engine, and service process themselves.
1. How It Works
flowchart TD
A["User Request"] --> B["Cloud API Gateway"]
B --> C["Cloud Server"]
C --> D["Model Service Process"]
D --> E["Inference Framework"]
E --> F["GPU Compute"]
F --> G["Model Output"]
G --> H["Return API Response"]
I["Model Storage"] --> D
J["Monitoring System"] -.-> C
J -.-> D
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style J fill:#EAE8FA,stroke:#A49BD5,color:#51477D
2. Deployment Steps
Usually includes:
- Select a cloud server instance.
- Configure the GPU driver and CUDA environment.
- Install the inference framework.
- Download model weights.
- Configure model parameters.
- Start the model service.
- Configure API and network access.
- Set up monitoring and security policies.
3. Example: Starting a Model with vLLM
On a server with a compatible GPU environment, you can use vLLM to start a model service.
vllm serve Qwen/Qwen3-8B \
--host 0.0.0.0 \
--port 8000
Once started, the service can be accessed through the OpenAI-compatible API.
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1"
)
response = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[
{
"role": "user",
"content": "介绍云服务器部署方式"
}
]
)
print(response.choices[0].message.content)
The command here shows a basic way to start the service. A real deployment also needs GPU memory, context length, concurrency and network access policies configured according to the model's requirements.
4. Advantages
- Compute resources can be chosen on demand.
- No need to purchase physical servers yourself.
- GPU instances can be used.
- Supports public or private network access.
- Easy to scale storage and compute resources.
- You keep full control over the model inference service.
5. Disadvantages
- GPU instance costs can be high.
- You have to manage the operating system and drivers.
- You must handle service stability yourself.
- Network and storage can affect inference performance.
- Idle costs arise when resource utilization is low.
6. Use cases
- Enterprises that need independent control over model services.
- Teams with some operations and maintenance capability.
- Businesses that need to run model APIs long-term.
- Scenarios with custom requirements for model versions and inference parameters.
VI. Approach 4: Containerized deployment
Containerized deployment means packaging the model inference service, runtime dependencies and startup configuration into a container image, then starting the service through a container runtime.
Docker is one of the most common containerization tools.
Containerization is not a separate type of compute resource but a way of delivering and running an application. It can run on a local machine, a cloud server, or a Kubernetes cluster.
1. Why containerize?
Traditional deployment approaches easily lead to environment inconsistency:
- It runs in the development environment but fails to start in production.
- Python dependency versions differ.
- CUDA or driver configurations differ.
- Migrating the service is difficult.
- The deployment process is hard to reproduce.
Containerization can package application dependencies and runtime configuration together, improving environment consistency and deployment efficiency.
2. Containerized deployment architecture
flowchart TD
A["Model inference code"] --> B["Dockerfile"]
C["Runtime dependencies"] --> B
D["Service configuration"] --> B
B --> E["Build Docker Image"]
E --> F["Image registry"]
F --> G["Target server"]
G --> H["Start container"]
H --> I["Load model"]
I --> J["Start inference service"]
J --> K["Expose API"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F
3. Docker deployment example
Here is a simplified Dockerfile:
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["python", "server.py"]
Build the image:
docker build -t llm-service:1.0 .
Start the container:
docker run -d \
--name llm-service \
-p 8000:8000 \
llm-service:1.0
If the inference service needs to use NVIDIA GPUs, you also need to configure a GPU-enabled container runtime environment and ensure compatibility among the host drivers, container runtime, and CUDA-related dependencies.
4. Advantages of Containerized Deployment
- Good environment consistency.
- Repeatable deployment process.
- Convenient version management.
- Supports rapid migration.
- Suitable for automated deployment.
- Easy to integrate with Kubernetes.
5. Considerations for Containerized Deployment
Model weights can reach several GB or even larger, so it is not advisable to re-download the model every time a container starts.
Production environments typically:
- Use model cache volumes.
- Store models in shared storage.
- Use a dedicated model repository.
- Manage application images and model files separately.
- Control model loading and container startup time.
VII. Method 5: Kubernetes Cluster Deployment
When a single server cannot meet the concurrency, availability, or resource requirements of model services, you can use Kubernetes to manage multiple model service instances.
Kubernetes itself is not responsible for model inference; rather, it handles container orchestration, resource scheduling, service discovery, and failure recovery.
1. Core Architecture
A typical Kubernetes model deployment includes:
- Deployment: manages model service replicas.
- Pod: hosts the inference service container.
- Service: provides a stable access endpoint.
- Ingress: handles external request traffic.
- PersistentVolume: provides persistent storage.
- ConfigMap: stores non-sensitive configuration.
- Secret: stores sensitive configuration.
- HPA: performs automatic scaling based on metrics.
2. Kubernetes Deployment Architecture
flowchart TB
A["User Request"] --> B["Ingress"]
B --> C["Kubernetes Service"]
C --> D["Load Balancing"]
D --> E["Pod 1"]
D --> F["Pod 2"]
D --> G["Pod 3"]
E --> H["Inference Engine"]
F --> I["Inference Engine"]
G --> J["Inference Engine"]
H --> K["GPU Node"]
I --> L["GPU Node"]
J --> M["GPU Node"]
N["HPA"] --> D
O["Monitoring System"] -.-> E
O -.-> F
O -.-> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style E fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style F fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style G fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style H fill:#EAF3EC,stroke:#83A995,color:#315D49
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#EAF3EC,stroke:#83A995,color:#315D49
style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style M fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style N fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style O fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
3. Kubernetes Deployment Example
The following is a simplified Deployment configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-service
spec:
replicas: 2
selector:
matchLabels:
app: llm-service
template:
metadata:
labels:
app: llm-service
spec:
containers:
- name: llm
image: llm-service:1.0
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: 1
This configuration assumes that the cluster has already installed and correctly configured the NVIDIA GPU device plugin or the corresponding resource exposure mechanism.
Production environments also need to configure:
- GPU node labels and scheduling constraints.
- Readiness Probe.
- Liveness Probe.
- Resource requests and limits.
- Model weight mounting.
- Service rolling update strategy.
- Node failure handling.
- GPU resource monitoring.
4. Advantages
- Supports multi-instance deployment.
- Supports service discovery and load balancing.
- Can automatically recover unhealthy containers.
- Supports elastic scaling.
- Facilitates unified management of multiple model services.
- Suitable for large enterprise-level systems.
5. Disadvantages
- High architectural complexity.
- GPU scheduling requires additional configuration.
- Model startup and loading take time.
- High requirements for resource management and monitoring.
- Not suitable for all small projects.
VIII. Method Six: Multi-GPU and Distributed Deployment
When the model is too large for a single GPU to hold the complete model, or when single-card throughput cannot meet business requirements, multi-GPU or distributed inference is required.
Two goals need to be distinguished here:
- Model scale expansion: Split a model across multiple GPUs so that the model can run.
- Inference throughput expansion: Use multiple GPUs or multiple instances to improve overall processing capacity.
The technical solutions for the two are not exactly the same.
1. Data Parallelism
The core idea of data parallelism is: multiple GPUs or GPU groups process different requests or input data separately.
In a model inference service, multiple model replicas can process different user requests.
flowchart TD
A["Request Entry"] --> B["Request Scheduler"]
B --> C["GPU 1 Model Replica"]
B --> D["GPU 2 Model Replica"]
B --> E["GPU 3 Model Replica"]
C --> F["Result Aggregation"]
D --> F
E --> F
F --> G["Return to User"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Advantages:
- Requests can be processed independently.
- Suitable for improving concurrent throughput.
- Cross-GPU communication overhead for a single request is low.
- Service capacity can be expanded by adding model replicas.
Disadvantages:
- Each model replica usually requires a copy of the model weights.
- GPU memory usage may be high.
- Request scheduling and load balancing are required.
- It cannot directly solve the problem that a single model cannot fit into a single GPU.
2. Tensor Parallelism
Tensor parallelism splits the matrix computations in the model across multiple GPUs.
For example, the weight matrix of a linear layer can be split by column or row, with different GPUs computing different parts.
Suppose a matrix is:
Use two GPUs for column splitting:
Each GPU is responsible for part of the matrix computation, after which necessary result communication and merging are performed according to the computation method.
Schematic diagram:
flowchart TD
A["Input Tensor"] --> B["Tensor Parallel Scheduling"]
B --> C["GPU 1 Computes W1"]
B --> D["GPU 2 Computes W2"]
C --> E["Communication and Result Merging"]
D --> E
E --> F["Output Tensor"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#EAE8FA,stroke:#A49BD5,color:#51477D
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Advantages:
- Can distribute model parameters and part of the computation.
- Helps run models that cannot fit on a single GPU.
- Can utilize the computing power of multiple GPUs.
Disadvantages:
- Frequent communication is required between GPUs.
- Sensitive to interconnect bandwidth and communication latency.
- Configuration and tuning are complex.
- Increasing the number of GPUs does not necessarily bring linear performance improvement.
Common communication technologies include NVIDIA NVLink, PCIe, InfiniBand, etc.
3. Pipeline Parallelism
Pipeline parallelism assigns different layers of a model to different GPUs.
For example:
- GPU 1 runs the first half of the model.
- GPU 2 runs the middle part of the model.
- GPU 3 runs the second half of the model.