AI mathematical tricks are useless for scientific computing

The artificial intelligence revolution has changed not only how we interact with computers, but also how computers count at the most fundamental level — the level of number representation. In recent years, the industry has spawned dozens of new numerical formats optimized for machine learning tasks. But attempts to apply these formats outside of AI exposed a fundamental problem: what works great for neural networks turns out to be completely unsuitable for scientific computing.

For decades, the computer industry lived by a simple rule: each number is represented by 64 bits, and that was more than enough. Users bought new machines every few years and gained performance improvements, essentially for free. But about ten years ago, that era ended. Moore's Law slowed down, while the appetites of AI models grew exponentially. Companies began looking for any way to save computational resources and energy, and one of the most effective turned out to be reducing the bit width of numbers. If neural networks don't need all 64 bits of precision, why waste them? Thus formats of 16, 8, and even 2 bits appeared, allowing models to be trained and run faster and cheaper.

The problem is that the IEEE 754 standard, which defines the representation of 64-bit floating-point numbers, is inherently poorly scalable to smaller bit widths. Its architecture is redundant for small numbers of bits, and direct truncation leads to the loss of important properties. Therefore, specialized formats like Google's bfloat16 and NVIDIA's FP8 were developed for AI, tailored to the distribution of numbers typical for neural networks. In machine learning, values usually concentrate around a specific range, and ultra-high precision at the edges is not required.

But scientific computing lives by completely different rules. Computational physics, hydrodynamics, biological modeling, and engineering simulations operate with numbers scattered across a giant range — from subatomic scales to cosmic distances. And the same high precision is needed for both very large and very small quantities. It was precisely this gap between the needs of AI and science that became the starting point for László Hunhólz's work, who recently defended a doctoral dissertation in computer science at the University of Cologne and joined the Barcelona startup Openchip as an AI accelerator engineer.

Hunhólz developed a numerical format called takum, based on the earlier posit format. Posit distributes number representations unevenly: values that are used more often get more bit combinations and, consequently, higher precision. For AI, this works wonderfully — posit concentrates representation density around unity, where typical neural network weight values are concentrated. But for scientific computing, this approach is catastrophic: precision drops sharply when transitioning to large or small numbers, and it's precisely these that are critical for modeling physical processes.

Takum solves this problem elegantly. Hunhólz analyzed the actual value ranges used in scientific computing across all major disciplines and designed the format so that as the number of bits decreases, the dynamic range doesn't narrow. This means that scientists and engineers could potentially switch to more compact number representations, saving energy and computational time without sacrificing the ability to work with extreme magnitudes. According to Hunhólz, even a ten percent gain in numerical format efficiency translates to a ten percent savings for all applications, which on the scale of global computing power means enormous energy savings.

The significance of this work goes far beyond an academic exercise. As supercomputers and research clusters consume increasingly more electricity, optimization at the level of number representation becomes one of the few remaining levers for improving efficiency without increasing hardware capacity. Notably, Hunhólz points out: over recent years, dozens of new numerical formats have been proposed, but takum remains the only one deliberately designed specifically for scientific computing. All other innovations in this field serve exclusively the machine learning industry.

The story of takum is a reminder that the AI boom, for all its transformative power, should not overshadow the needs of the rest of computational science. Neural networks are not the only programs that need efficiency. Physicists modeling climate, engineers designing bridges, and biologists simulating protein folding deserve the same innovations in basic arithmetic. And if the takum format gains widespread adoption, it could become that invisible foundation on which scientific computing of the next decade will become faster, cheaper, and more environmentally friendly.