The battle for memory: why AI infrastructure is no longer limited to GPUs alone

In recent years, when it comes to artificial intelligence infrastructure, attention has been consistently focused on graphics processing units (GPUs), primarily those from Nvidia. However, as AI models become increasingly complex and large-scale, it becomes clear that computational power is only one side of the coin. Equally, and possibly even more importantly, memory has become a factor determining the efficiency of modern AI systems.

High-performance memory with high bandwidth (High Bandwidth Memory, HBM) is transforming from a secondary component into a critical infrastructure element, since the exponential growth in the number of parameters in AI models requires colossal volumes of data for their instantaneous processing. This places memory manufacturers at the center of the technology boom, shifting industry focus from a simple race for computational power to comprehensive optimization of data storage and transfer systems within servers.

The context of this transformation lies in the very nature of modern deep learning architectures. Models such as GPT-3, GPT-4 and their analogues operate with trillions of parameters. Each of these parameters is a numerical value that must be loaded from memory into GPU computational cores to perform mathematical operations.

The larger the model, the more data must be constantly moved between memory and processor. If the data transfer speed (memory bandwidth) does not match the computation speed, the GPU will idle waiting for the next batch of information. This is a clear "bottleneck" that limits performance and increases training and inference time (the application of a model to obtain results).

Traditional memory types such as DDR4 or DDR5 simply cannot provide the necessary speed and volume for such tasks.

A deep dive into technical details shows that HBM offers a fundamentally different approach. Instead of placing memory chips separately from the GPU and connecting them through the motherboard, HBM is integrated much closer to the computational cores, often as multiple layers "stacked" on top of or alongside the GPU. This drastically reduces the physical distance that data must travel and allows for a significant increase in data bus width, which directly impacts bandwidth.

Current HBM3 and HBM3e standards provide bandwidth in terabytes per second, which is orders of magnitude higher than conventional memory modules. It is this ability to quickly "feed" giant models with data that makes HBM indispensable for cutting-edge AI applications such as training large language models, image generation and complex scientific analysis.

The consequences of this shift in priorities are colossal. First, it changes the landscape of manufacturers. Whereas GPU-producing companies once dominated, memory manufacturers such as SK Hynix, Samsung and Micron are now coming to the fore.

These are the companies that possess the technologies and manufacturing capabilities to produce HBM, which is complex to manufacture and expensive. Second, it affects data center architecture. Now, when designing servers for AI, equal attention must be paid to memory layout, cooling systems for densely packed HBM chips and overall input/output system bandwidth.

The cost of the entire AI infrastructure now consists of a more balanced share of GPUs and memory. Third, it stimulates further innovation in materials science and chip engineering aimed at increasing memory density, reducing power consumption and improving heat dissipation.

In conclusion, the battle for dominance in AI infrastructure is no longer exclusively a battle for computational power. It is becoming a complex task of optimizing the entire system, where memory plays a role no less important than the processor. The ability to quickly move enormous volumes of data is the new "gold standard" for AI, and companies that can solve this problem efficiently will take leading positions in the next wave of technological progress. Memory manufacturers, thanks to their cutting-edge developments in HBM, are becoming new undisputed players on this arena, determining the future of artificial intelligence.