Sharding in LLMs: How to Split Computation Across GPUs

Scaling large language models takes more than one GPU or TPU. One of the key techniques for this is sharding: distributing matrices and vectors across several accelerators so that computation runs efficiently and in a coordinated way.

Why matrices don't fit in memory

With truly large neural networks, the weight and activation matrices almost never fit entirely in the memory of a single GPU or TPU. A modern LLM can have hundreds of billions of parameters, each of which needs several bytes of memory. A single accelerator simply can't cope. So these matrices have to be split, or sharded, across several chips.

How sharding works

Take a matrix A of size [I, J]. It can be split in several ways:

• Row-wise — the first accelerator takes rows 1–N, the second takes rows N+1–2N, and so on
• Column-wise — the same idea, but vertically
• Block partitioning — the matrix is divided into rectangular blocks and distributed across a grid of chips
• Diagonal and other patterns — more complex schemes for specific operations

Each accelerator's job is to perform the operation on its part of the matrix, then synchronize the results with its neighbors.

What the quality of sharding affects

The choice of sharding strategy determines:

• Training speed — minimizing the time spent on communication between GPUs
• Memory efficiency — filling the buffers on each chip correctly
• Bandwidth — the amount of data that has to be transferred between accelerators
• Computation locality — how much work can run without synchronization

Poor sharding can leave accelerators spending most of their time waiting for each other or wasting the bandwidth of the links between them. Good sharding tries to minimize these losses and keep all chips working at the same time.

What this means

Sharding is not just a technical optimization; it underpins the economics of scaling. Distributing computation well makes it possible to use clusters of hundreds or thousands of GPUs so that total throughput approaches the ideal and compute isn't wasted waiting on communication.