Sharding in LLMs: How to Distribute Computations Across GPUs

Scaling large language models requires more than one GPU or TPU. One of the key techniques for this is sharding: distributing matrices and vectors across multiple accelerators so that computations are performed efficiently and consistently.

Why Matrices Don't Fit in Memory

When working with truly large neural networks, weight and activation matrices almost never fit entirely into the memory of a single GPU or TPU. The parameters of modern LLMs are measured in hundreds of billions, and each parameter requires several bytes of memory. A single accelerator simply cannot handle this. Therefore, it becomes necessary to divide—to shard—these matrices across multiple chips.

How Sharding Works

Imagine a matrix A of size [I, J]. It can be divided in several ways:

• Row-wise — the first accelerator takes rows 1–N, the second takes rows N+1–2N, and so on
• Column-wise — similarly, but vertically
• Block-wise partitioning — the matrix is divided into rectangular blocks and distributed across a grid of chips
• Diagonal and other patterns — more complex schemes for specific operations

The role of each accelerator is to perform operations on its portion of the matrix, then synchronize the results with neighboring accelerators.

What Quality Sharding Affects

The choice of a proper sharding strategy affects:

• Training speed — minimizing communication time between GPUs
• Memory efficiency — proper buffer allocation on each chip
• Throughput — the amount of data that needs to be transferred between accelerators
• Compute locality — how much work can be performed without synchronization

Poor sharding can result in accelerators spending most of their time waiting for each other or underutilizing the bandwidth of inter-accelerator connections. Good sharding aims to minimize these losses and keep all chips occupied simultaneously.

What This Means

Sharding is more than just a technical optimization—it is the foundation of scaling economics. Proper distribution of computations enables clusters of hundreds or thousands of GPUs to achieve near-ideal aggregate speeds, preventing computational resources from being wasted on communication delays.