Sharding in LLM: how to distribute computations between GPUs
Large neural networks require distributing matrices across multiple accelerators. This is called sharding. How well data is partitioned determines the speed and
◐ Listen to article
Large neural networks require distributing matrices across multiple accelerators. This is called sharding. How well data is partitioned determines the speed and efficiency of LLM training.