KTransformers Moves Part of the MoE Computation to the CPU and Eliminates Weight Transfers During Fine-Tuning

The KTransformers project proposes computing MoE experts on the CPU during fine-tuning instead of shuttling their weights to the GPU. The authors are targeting the open flagship models DeepSeek-V4, Qwen-3.8, Kimi-K3 and GLM-5.3. Each such release raises the quality bar sharply, but most developers still work with these models only through an API: tuning them for their own tasks runs into limits on memory and hardware cost.

Where fine-tuning large MoE models breaks down

An MoE model stores hundreds of billions of parameters, sometimes a trillion, but activates only a few experts per token. Inference on such an architecture is cheap in compute but heavy on memory. All the experts have to live somewhere, and they do not fit into the VRAM of one or two cards.

The standard solution is offloading. The weights sit in system RAM, and at every step the required experts are copied to the GPU, processed and sent back. Fine-tuning costs more than inference here: it needs both a forward and a backward pass, plus a parameter update.

The proportions are unfavorable. PCIe 5.0 x16 delivers a theoretical 64 GB/s or so in one direction. The HBM on an H100 SXM delivers about 3.35 TB/s, roughly 50 times more. The accelerator sits idle waiting for data. The announcement's headline points at exactly this: computation should not have to wait while weights are moved.

What bringing in the CPU changes

The KTransformers idea is that if the expert weights already sit in RAM, it is easier to compute them there. A server CPU with 12 channels of DDR5 provides on the order of 400+ GB/s of memory bandwidth per socket. That is an order of magnitude more than the PCIe bus. In raw compute the CPU is weaker than a GPU, but only a few experts are active per token, and this workload suits a CPU better than it might seem.

The project was already built around a hybrid CPU+GPU scheme. Developers from the MADSys lab at Tsinghua University have shown large MoE models running on a single GPU with a large amount of RAM. The new work carries that approach over from inference to training, where memory and synchronization requirements are stricter.

The announcement's headline and introduction point to the following parameters:

• Division of work: the GPU handles the dense parts of the model, while the MoE experts are computed on the CPU.
• What is removed: transferring expert weights over PCIe at every step.
• Target models: DeepSeek-V4, Qwen-3.8, Kimi-K3, GLM-5.3.
• Scenario: fine-tuning on resources that fall short of a full GPU cluster.
• License and code: KTransformers is developed as an open project.

What the announcement does not yet show

The available excerpt gives no throughput in tokens per second, no RAM requirements and no comparison with classic offloading. It does not state the tuning method (LoRA or full), the supported processors or the degree of accuracy loss. Without these data, it is impossible to say how practical the scheme is for work beyond a demo.

The main question is scaling. If the CPU computes experts faster than the bus can transfer them, the gain will be noticeable. With long batches the CPU load grows, and the advantage may disappear. This needs to be tested on real configurations.

Who will find this useful, and when

A team that has a server with 1–2 TB of RAM and one or two GPUs, but no budget for an eight-GPU node, should take a look at KTransformers. For such projects, fine-tuning a flagship MoE on their own data could become a real task rather than a pipe dream. Three things to check first: which tuning method is supported, how much RAM your model needs, and what speed you get on your processor. A small task is enough for a pilot, for example domain adaptation via LoRA. If the numbers add up, you can think about scaling. If not, the fallback is renting GPU hours.