University of Twente cuts LLM training energy use by 14%

Researchers at the University of Twente in the Netherlands have developed a method that saves up to 14% of the energy used to train large language models, without slowing training down. The technique relies on dynamic control of the GPU clock frequency and is the first to be applied at such a fine level of granularity.

How DVFS works

DVFS (dynamic voltage-frequency scaling) is a well-known technique that changes the GPU clock frequency depending on the current computational load. Every operation in the chip is triggered by a clock pulse, and the frequency of those pulses determines both the GPU's speed and its energy consumption.

Modern GPUs have two independent clock systems: one for the compute core and one for the memory block. When the core is computing intensively, its clock runs at a high frequency, while the memory clock can be slowed down. When the core is waiting for data from memory, the opposite holds: the core can slow down and the memory can speed up. This balance lowers overall energy consumption without losing performance.

Why earlier methods fell short

DVFS has existed since the 1990s, but applying it to LLM training turned out to be harder than it looked. Earlier attempts either slowed computation too much or were not flexible enough. The main problem: most methods adjusted frequency only at the level of whole training iterations (forward pass and backpropagation), which was too coarse for effective optimization.

Innovation at the kernel level

Jeffrey Spaan's team decided to change the frequency at a much finer level: individual kernels (elementary computational blocks). GPU computation is split into tiny operations; for example, a single vector multiplication is one kernel. Training a single neural network layer launches roughly 40 such kernels. By tuning the frequency for each kernel separately, the team was able to find much larger energy savings:

• Adjustment at the level of individual computational blocks instead of whole iterations
• Predicting the next kernel makes it possible to set the right frequency in advance
• Average energy savings of 14% with a slowdown of only 0.6%
• Automatic GPU DVFS performs worse because it cannot anticipate the next steps

Results and limitations

The experiment was run on the GPT-3-XL model (1.3 billion parameters) on an Nvidia RTX 3080 Ti GPU. The result: 14% energy savings with a slowdown of only 0.6%.

"We optimize energy savings without losing performance.

In the real world, performance is the holy grail," said Jeffrey Spaan.

One limitation: switching frequencies takes time, though less than fully powering a core off and on. The researchers' calculations did not account for this, so 14% is a best-case scenario. Newer GPUs, such as Nvidia Blackwell, switch much faster and should be able to capture more of these savings.

What this means

If Spaan's method is adopted in industry, billions of watt-hours of energy could be saved when training frontier models. That would reduce the AI industry's carbon footprint and operating costs, without requiring investment in new hardware.