Back to series
· 12 min read

CUDA and GPU Parallel Computing


I. What Is CUDA?

CUDA (Compute Unified Device Architecture) is NVIDIA's general-purpose GPU computing platform and programming model.

Before CUDA, GPUs were mainly used for graphics rendering, and developers typically used GPUs through graphics APIs.

CUDA opened GPU computing power to general-purpose programs, enabling developers to write GPU parallel programs in languages such as C and C++.

CUDA is not just a programming language, nor a standalone library, but a complete GPU computing ecosystem.

It mainly includes:

  • CUDA programming model: defines how GPU threads and computing tasks are organized.
  • CUDA C/C++: used to write GPU kernels.
  • CUDA Runtime: provides interfaces for device management, memory allocation, and kernel launch.
  • CUDA Driver API: provides lower-level GPU control capabilities.
  • CUDA Toolkit: includes compilers, development tools, and related libraries.
  • CUDA Libraries: provide high-performance computing capabilities such as matrix computation and deep learning.

1. Why Is CUDA Needed?

CPUs and GPUs have different design goals.

CPUs focus more on single-thread performance, complex control logic, and low-latency tasks.

GPUs focus more on throughput, executing similar computing tasks simultaneously through a large number of compute units.

For example, suppose you need to perform the same squaring operation on an array of one million elements:

A CPU can use multiple cores to process in parallel, but the number of cores is relatively limited.

A GPU, by contrast, can launch a large number of threads and have different threads process different elements.

flowchart TD
    A["Input one million data elements"] --> B{"Select compute device"}
    B -->|"CPU"| C["A few high-performance cores"]
    B -->|"GPU"| D["A large number of parallel compute threads"]
    C --> E["Process data in batches"]
    D --> F["Multiple threads process different elements simultaneously"]
    E --> G["Merge computation results"]
    F --> G
    G --> H["Obtain output array"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

2. The Core Value of CUDA

CUDA mainly addresses three problems:

First, it enables GPUs to perform general-purpose computing.

Developers are no longer limited to graphics rendering; they can perform matrix multiplication, image processing, scientific computing, and neural network computation on GPUs.

Second, it provides a unified parallel programming model.

Developers can split tasks into a large number of threads, and the GPU automatically schedules and executes them.

Third, it provides a mature high-performance computing ecosystem.

CUDA provides libraries such as cuBLAS, cuDNN, and NCCL, reducing the work developers need to do to repeatedly implement low-level computations.

II. GPU Hardware Execution Architecture

To understand CUDA, you first need to understand how GPUs organize computing resources.

A GPU is not a processor with a large number of independent CPU cores; rather, it consists of multiple Streaming Multiprocessors (SMs).

Each SM contains multiple compute units, registers, shared memory, and scheduling resources.

1. Main Components of a GPU

A typical NVIDIA GPU includes:

Component Function
SM GPU's main computing and scheduling unit
CUDA Core Performs general-purpose computation such as integer and floating-point operations
Tensor Core Accelerates tensor operations such as matrix multiplication
Warp Scheduler Schedules Warp execution
Register File Stores thread-private registers
Shared Memory Provides high-speed storage shared by threads within an SM
L1 Cache Caches locally accessed data
L2 Cache Cache shared by multiple SMs on the GPU
Global Memory Global data storage in GPU memory

The exact hardware quantities and organization vary depending on GPU architecture and model.

2. What Is an SM?

An SM is the core hardware unit for CUDA parallel execution.

A GPU usually contains multiple SMs; each SM can host multiple thread blocks at the same time and schedule different warps according to resource availability.

During execution, a thread block is handled by a single SM and is not split across multiple SMs for simultaneous execution.

An SM can manage multiple thread blocks at the same time, but how many thread blocks it can host simultaneously depends on:

  • The number of registers available to the SM.
  • Shared memory capacity.
  • The number of threads per thread block.
  • The thread and thread block limits of the GPU architecture.

For example, suppose a GPU has 80 SMs and the program launches 1000 thread blocks.

The GPU does not place all thread blocks into hardware execution at once; instead, it schedules them in batches according to the resource conditions of the SMs.

flowchart TD
    A["GPU"] --> B["SM 0"]
    A --> C["SM 1"]
    A --> D["SM 2"]
    A --> E["SM ..."]
    B --> F["Warp Scheduler"]
    C --> G["Warp Scheduler"]
    D --> H["Warp Scheduler"]
    F --> I["CUDA Core / Tensor Core"]
    G --> J["CUDA Core / Tensor Core"]
    H --> K["CUDA Core / Tensor Core"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49

3. CUDA Core and Tensor Core

CUDA Core is the basic computing unit in a GPU that performs general-purpose computation.

Tensor Core, by contrast, is specially optimized for operations such as matrix multiplication and matrix multiply-add.

In deep learning, a common form of computation is:

C=A\times B

or matrix multiply-add:

D=A\times B+C

Tensor Core can execute operations corresponding to the supported data types and matrix shapes with higher throughput.

Commonly supported precisions include:

  • FP16
  • BF16
  • TF32
  • FP8
  • INT8
  • some lower-bit-width integer formats

The specific supported range depends on the GPU architecture.

Note that Tensor Core is not a simple replacement for CUDA Core. Ordinary operators, control logic, and other computations may still be handled by CUDA Core and other execution units.

III. CUDA Programming Model: Thread, Block, and Grid

The most important concept in CUDA is the thread organization model.

Developers write a GPU Kernel, define what work each thread is responsible for, and then organize a large number of threads through Grids and Blocks.

CUDA uses three core concepts:

  • Thread: a thread.
  • Block: a thread block.
  • Grid: a collection of thread blocks.

1. Thread: Thread

Thread is the most basic execution unit in the CUDA programming model.

Each thread has its own:

  • Thread ID.
  • Registers.
  • Local variables.
  • Execution state.

For example, when performing an array squaring computation, each thread can be made to process one array element.

__global__ void square(float* input, float* output, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    if (idx < n) {
        output[idx] = input[idx] * input[idx];
    }
}

Where:

  • threadIdx.x: the index of the thread within the current Block.
  • blockIdx.x: the index of the Block within the Grid.
  • blockDim.x: the number of threads per Block along the x dimension.

Through the global index idx, a thread determines the position of the data it processes.

2. Block: Thread Block

A Block is a group of threads that can execute cooperatively.

Threads within the same Block can:

  • Share data through Shared Memory.
  • Coordinate using synchronization primitives.
  • Perform cooperative computation through Warp-level instructions.

For example, a Block contains 256 threads:

dim3 block(256);

This means a one-dimensional thread block contains 256 threads.

A Block is the basic organizational unit of GPU scheduling.

A Block is assigned to one SM for execution, and the threads within the Block are not split across multiple SMs.

3. Grid: Grid

A Grid is all the Blocks contained in a single Kernel launch.

Suppose you need to process 10,000 elements, with 256 threads per Block:

N_{\text{blocks}}= \left\lceil\frac{10000}{256}\right\rceil =40

Therefore, 40 Blocks need to be launched.

The last Block may have some threads with no corresponding data, so boundary checks are needed to avoid out-of-bounds access.

4. Three-Level Organizational Relationship

flowchart TD
    A["Grid"] --> B["Block 0"]
    A --> C["Block 1"]
    A --> D["Block 2"]
    A --> E["Block ..."]
    B --> F["Thread 0"]
    B --> G["Thread 1"]
    B --> H["Thread ..."]
    C --> I["Thread 0"]
    C --> J["Thread 1"]
    C --> K["Thread ..."]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#EAF3EC,stroke:#83A995,color:#315D49
    style K fill:#EAF3EC,stroke:#83A995,color:#315D49

These can be understood as:

Level Meaning Main Role
Grid All thread blocks Describes the complete computational task
Block A group of threads Organizes collaboration and data sharing
Thread A single thread Performs the specific computation

IV. Warp: The GPU's True Execution Scheduling Unit

After understanding Thread, Block, and Grid, you also need to understand Warp.

Warp is an important unit by which GPU hardware schedules threads.

In NVIDIA CUDA GPUs, a Warp typically contains 32 threads.

Threads in the same Warp use the SIMT (Single Instruction, Multiple Threads) execution model.

This means threads in a Warp typically execute the same instruction but can operate on different data.

1. What Is SIMT?

SIMT is one of the core execution models of CUDA GPUs.

Suppose a Warp has 32 threads, and each thread is responsible for one element in an array.

All threads execute the same square-operation instruction but access different data locations.

Thread 0  -> input[0]  -> output[0]
Thread 1  -> input[1]  -> output[1]
Thread 2  -> input[2]  -> output[2]
...
Thread 31 -> input[31] -> output[31]

This approach enables the GPU to efficiently process large amounts of computations of the same type.

2. Relationship Between Warp and Block

Suppose a Block contains 256 threads:

N_{\text{warps}}=\frac{256}{32}=8

This Block can be divided into 8 Warps.

If a Block has 128 threads, it contains 4 Warps.

If the number of threads is not divisible by 32, some threads in the last Warp may be inactive.

3. Warp Divergence: Branch Divergence

Warp Divergence is a very important concept in CUDA performance optimization.

Suppose the 32 threads in a Warp execute a conditional:

if (threadIdx.x % 2 == 0) {
    result = a + b;
} else {
    result = a * b;
}

Half of the threads perform addition, and the other half perform multiplication.

Because these threads belong to the same Warp, the hardware may need to execute the different branches separately and use thread masks to control which threads are active.

This reduces execution efficiency.

Note that modern GPUs have more complex optimization mechanisms for branch execution, and the actual impact depends on the architecture, branch conditions, and code structure.

Common situations that can easily cause performance issues:

  • Different threads within the same Warp frequently execute different branches.
  • The branches contain complex computations.
  • Thread execution paths differ significantly.

Optimization directions:

  • Try to make threads in the same Warp follow similar execution paths.
  • Reduce unnecessary conditional branches.
  • Group data for different computation paths appropriately.

4. Warp Shuffle

Warp Shuffle is a mechanism for threads within a Warp to exchange register data.

For example, one thread can read the register value of another thread in the same Warp.

Common uses:

  • Reduction within a Warp.
  • Prefix sum computation.
  • Data rearrangement.
  • Cooperative computation within a Warp.

Compared with using Shared Memory, Warp Shuffle can reduce memory access and synchronization overhead in scenarios where it applies.

V. The CUDA Memory System

A GPU does not have only one kind of storage space, namely video memory.

CUDA provides a multi-level memory system, and different memories differ in access speed, capacity, scope, and lifetime.

Understanding the memory system is the key to writing high-performance CUDA programs.

1. Memory Hierarchy

Memory Type Location Access Scope Characteristics
Register Inside the SM A single thread Fast, limited capacity
Local Memory GPU memory system A single thread Thread-private, usually located in video memory
Shared Memory Inside the SM The same Block Shared by threads, fast
L1 Cache Inside the SM Accessed by threads in the SM Caches local data
L2 Cache GPU level Shared by multiple SMs Caches global data
Global Memory Video memory All GPU threads Large capacity, relatively high latency
Constant Memory GPU memory system Read-only for all threads Suitable for broadcast access
Texture Memory GPU cache system GPU threads Has a specific cache access mechanism

2. Register: Registers

Registers are thread-private high-speed storage space.

Each thread can use registers to store:

  • Intermediate computation results.
  • Local variables.
  • Indices.
  • Temporary values.

Register access is extremely fast, but the total number of registers per SM is limited.

If a Kernel uses a large number of registers per thread, it may reduce the number of Warps that can reside simultaneously in an SM.

This affects the GPU's Occupancy.

3. Shared Memory

Shared Memory is a high-speed storage space that threads within the same Block can access together.

It resides inside the SM and is suitable for holding data that needs to be accessed repeatedly by multiple threads.

For example, in matrix multiplication, part of the matrix data can be loaded into Shared Memory and then reused by multiple threads.

This reduces repeated access to Global Memory.

Example:

__shared__ float tile[16][16];

This declares a shared two-dimensional array within a Block.

Note the following:

  • The scope of Shared Memory is the Block.
  • Different Blocks cannot directly share the same block of Shared Memory.
  • Thread synchronization must be considered before use.
  • Shared Memory has limited capacity.

4. Global Memory

Global Memory is usually the GPU's video memory.

It has a large capacity, and all GPU threads can access it through pointers.

For example:

__global__ void add(float* a, float* b, float* c, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    if (idx < n) {
        c[idx] = a[idx] + b[idx];
    }
}

Here, a, b, and c usually point to data in Global Memory.

Global Memory has higher access latency than registers and Shared Memory, so access efficiency should be improved as much as possible.

5. Memory Access Performance

The CUDA memory system can be understood from three aspects: access latency, capacity, and sharing scope.

flowchart TD
    A["CUDA Memory Hierarchy"] --> B["Thread-Private Storage"]
    A --> C["Block-Shared Storage"]
    A --> D["GPU Global Storage"]
    B --> E["Register"]
    B --> F["Local Memory"]
    C --> G["Shared Memory"]
    C --> H["L1 Cache"]
    D --> I["L2 Cache"]
    D --> J["Global Memory"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Note that although Local Memory has “Local” in its name, it is not high-speed on-chip storage within an SM. It usually resides in the device memory hierarchy, may be cached, and its access cost is often significantly higher than that of registers.

When register pressure is too high, the compiler may place some variables in Local Memory; this situation is called Register Spilling.

6. CUDA Kernel Execution Mechanism

A Kernel is a function in CUDA that runs on the GPU.

Developers declare a Kernel using __global__ and invoke it with a special launch syntax.

1. Kernel Function

A simple vector addition:

__global__ void vector_add(
    float* a,
    float* b,
    float* c,
    int n
) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    if (idx < n) {
        c[idx] = a[idx] + b[idx];
    }
}

Here:

  • __global__: indicates that this is a GPU Kernel.
  • a, b: input arrays.
  • c: output array.
  • idx: the data index handled by the current thread.
  • if (idx < n): prevents threads from accessing locations outside the array bounds.

2. Kernel Launch

CUDA uses the following syntax to launch a Kernel:

vector_add<<<gridSize, blockSize>>>(a, b, c, n);

Where:

  • gridSize: the number of Blocks in the Grid.
  • blockSize: the number of threads per Block.

For example:

int n = 10000;
int blockSize = 256;
int gridSize = (n + blockSize - 1) / blockSize;

vector_add<<<gridSize, blockSize>>>(a, b, c, n);

The calculation gives:

gridSize = \left\lceil\frac{10000}{256}\right\rceil =40

The GPU schedules threads to execute the Kernel according to the organization of the Grid and Blocks.

3. Kernel Execution Process

flowchart TD
    A["CPU prepares input data"] --> B["Allocate GPU memory"]
    B --> C["Copy input data to GPU"]
    C --> D["Configure Grid and Block"]
    D --> E["Launch CUDA Kernel"]
    E --> F["GPU schedules thread blocks to SMs"]
    F --> G["Warp schedules threads to execute instructions"]
    G --> H["Read and write GPU memory and complete computation"]
    H --> I["Check execution results"]
    I --> J["Copy results back to CPU"]
    J --> K["Release GPU resources"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style D fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style J fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style K fill:#DCEFE5,stroke:#4F8B70,color:#24563F

There is an important concept here: kernel launches are usually asynchronous.

After the CPU submits a kernel launch request, it does not necessarily wait for the GPU to finish the computation before continuing to execute subsequent code.

If you need to ensure the computation is complete, you can use a synchronization operation.

For example:

cudaDeviceSynchronize();

In actual development, you also need to check for kernel launch errors and asynchronous execution errors.

7. CUDA Thread Synchronization Mechanisms

Many threads on the GPU may access shared data at the same time.

If there are data dependencies between multiple threads, synchronization mechanisms are needed to ensure execution order and data correctness.

1. __syncthreads()

__syncthreads() is a block-level synchronization primitive.

It causes threads participating in execution within the same block to wait at a specified point until all threads in the block have reached the synchronization point.

For example:

__global__ void shared_example(float* input, float* output) {
    __shared__ float cache[256];

    int tid = threadIdx.x;
    int idx = blockIdx.x * blockDim.x + tid;

    cache[tid] = input[idx];

    __syncthreads();

    output[idx] = cache[tid] * 2.0f;
}

Here, data is first written to shared memory, then synchronization ensures the data is ready, and finally the computation is performed.

Note the following:

  • It can only synchronize threads within the same block.
  • Different blocks cannot directly rely on __syncthreads() for synchronization.
  • Threads in the same block must reach the synchronization point with correct control flow.
  • If some threads skip the synchronization point, it may lead to incorrect program behavior or deadlock.

2. Atomic Operations

When multiple threads modify the same variable at the same time, a data race may occur.

For example:

counter += 1;

If multiple threads perform this operation simultaneously, updates may be lost.

CUDA provides atomic operations:

atomicAdd(&counter, 1);

It can guarantee that the addition operation at the specified memory location is atomic.

Common atomic operations:

  • atomicAdd
  • atomicSub
  • atomicExch
  • atomicMin
  • atomicMax
  • atomicCAS

Atomic operations can ensure correctness, but when many threads contend for the same address, they can also become a performance bottleneck.

3. Stream: CUDA Streams

A CUDA stream is an ordered queue of GPU operations.

Operations in the same stream are executed in the order they are submitted.

Operations in different streams can execute concurrently when dependencies and hardware conditions permit.

Common uses:

  • Overlapping data transfer and computation.
  • Executing multiple independent kernels in parallel.
  • Multi-request task scheduling.
  • Improving GPU utilization.

For example:

cudaStream_t stream1;
cudaStream_t stream2;

cudaStreamCreate(&stream1);
cudaStreamCreate(&stream2);

kernel_a<<<grid, block, 0, stream1>>>();
kernel_b<<<grid, block, 0, stream2>>>();

These two kernels are submitted to different streams, but whether they actually execute concurrently still depends on GPU resources, kernel characteristics, and dependencies.

8. Key Optimization Techniques for CUDA Parallel Computing

Writing a CUDA program is not just about splitting a task into a large number of threads; it also requires threads, memory, and compute resources to work together efficiently.

1. Coalesced Memory Access: Coalescing Memory Accesses

The GPU memory system tries to coalesce memory accesses from multiple threads into fewer memory transactions.

Suppose a warp contains 32 threads, and each thread reads contiguous 4-byte data:

Thread 0  -> data[0]
Thread 1  -> data[1]
Thread 2  -> data[2]
...
Thread 31 -> data[31]

This kind of contiguous access is usually conducive to memory transaction coalescing.

If the addresses accessed by threads are highly scattered, more memory transactions may be required, reducing bandwidth utilization.

Therefore, contiguous access and a reasonable data layout are very important.

2. Shared Memory Tiling

Matrix multiplication requires repeatedly reading data from matrices.

If every thread reads data directly from global memory, a large number of duplicate accesses may occur.

Tiling splits the matrix into small tiles, first loads them into shared memory, and then allows multiple threads within a block to reuse them.

For example:

C=A\times B

The matrix can be split into multiple submatrices:

C_{tile} = \sum_k A_{tile,k}\times B_{k,tile}

This approach improves data reuse and reduces global memory accesses.

3. Occupancy: Occupancy Rate

Occupancy is the ratio of the number of warps actually resident on an SM to the maximum number of resident warps allowed by the hardware.

Formula:

Occupancy = \frac{\text{Active Warps}} {\text{Maximum Active Warps}}

Factors affecting occupancy include:

  • The number of registers used by each thread.
  • The shared memory used by each block.
  • The number of threads per block.
  • SM hardware resource limits.

Increasing occupancy helps hide memory access latency, but it does not mean that higher occupancy necessarily leads to better performance.

If increasing occupancy reduces data reuse or causes more redundant computation, final performance may actually decrease.

4. Kernel Fusion: Operator Fusion

Suppose a computational task consists of three operations:

y = (x \times a)+b

If multiplication and addition are executed separately, multiple kernels may need to be launched, and intermediate results may be read and written multiple times.

Kernel Fusion combines multiple operations into a single kernel for execution.

flowchart TD
    A["Input data x"] --> B["Execute multiplication kernel separately"]
    B --> C["Write intermediate result"]
    C --> D["Launch addition kernel"]
    D --> E["Write final result"]
    A --> F["Fused kernel"]
    F --> G["Execute multiplication and addition once"]
    G --> H["Write final result"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style G fill:#EAE8FA,stroke:#A49BD5,color:#51477D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F