UC Berkeley builds mKernel: a single library for GPU synchronization in clusters

The UCCL team at UC Berkeley has released mKernel, a CUDA library that aims to overhaul GPU synchronization in large clusters. The library combines local communication between GPUs, inter-server synchronization and computation in a single persistent kernel, avoiding constant context switches.

The bottleneck in large-scale clusters

In large data centers, synchronization between GPUs is one of the main throughput bottlenecks. Engineers training large models in a distributed setup run into a problem that has traditionally been handled very inefficiently.

The old approach was layered: engineers used three separate tools and libraries. The first handled fast communication inside a single server (NVLink, which runs over high-speed links between GPUs). The second handled synchronization between servers over the network (RDMA, which uses specialized network adapters). The third handled the computation itself.

Every handoff between these three systems freezes the whole GPU pipeline: the GPU has to stop, switch context, flush part of its memory, load new data, and only then resume work. On clusters with thousands of GPUs, these microsecond delays add up to minutes of lost performance.

How mKernel solves the problem

mKernel changes the approach radically. Instead of three separate systems, all operations (local communication, network synchronization and computation) run in a single persistent kernel. This is a program that lives permanently in GPU memory and is never offloaded to the CPU.

The architecture combines three components:

• NVLink for communication between GPUs on the same server — delivers speeds 10-20x higher than PCIe thanks to direct high-speed channels between GPUs
• RDMA over network adapters for synchronization between servers — avoids CPU overload and operating-system latency when moving data between machines
• Dense computation built directly into the kernel — the GPU works on local data without switching, synchronizes with its neighbors and moves straight on to the next task

This means a GPU can move seamlessly from local NVLink communication to global RDMA synchronization to its own computation, all within one piece of code and without stopping.

A concrete example: how it works in practice

In distributed training, one GPU cannot move forward until the other GPUs on other servers finish their computations and synchronize gradients. In the old approach, the GPU simply waits with a completely empty pipeline.

With mKernel the process is different: the GPU keeps computing on data it has already loaded, synchronizes with its neighbors over NVLink and RDMA at the same time, and moves straight on to the next wave of training without any pauses. It is like an automotive assembly line that does not stop while a part travels to the next station.

Why this matters for data centers

Distributed training of large models is one of the most demanding computational tasks in modern AI development. When you use 1000 GPUs at once (and large companies work with even more), even a small slowdown in synchronization can waste 20-30% of all resources.

mKernel promises to eliminate this overhead entirely.

In early tests, researchers already see results: on multi-node clusters the library speeds up synchronization by 2-3x on typical operations. This is especially important for the attention mechanism in transformers, where gradient synchronization between GPUs is the most expensive part of the workload.

What it means

mKernel is a sign that GPU programming is entering the era of integrated systems. Engineers used to write code in layers: first computation, then synchronization, then transfer. Now the boundary between them is blurring. That means faster computation in data centers and more accessible, faster training of large models. Above all, the next generation of distributed systems will be designed in a completely new way.