Memory Shortage Is the Biggest Bottleneck in AI Infrastructure, Says Huang

At Dell World in Las Vegas, Nvidia CEO Jensen Huang addressed a problem critical to the entire AI industry: the shortage of high-bandwidth memory. In a conversation with Dell CEO Michael Dell and Bloomberg journalist Ed Ludlow, Huang said bluntly that the memory shortage has proved more acute than the bottlenecks in chip manufacturing itself. It is a candid admission that the pace of AI infrastructure deployment is slowing not because of a lack of GPUs, but because the industry cannot produce enough memory.

Why memory became scarce

HBM (High Bandwidth Memory) is a special type of memory built directly into the accelerator. It stores neural network parameters and intermediate computations, letting the GPU run at full speed without the latency of reaching out to slower system memory. Nvidia's H100 and H200 carry between 80 and 141 gigabytes of it.

The problem is that demand for HBM is growing exponentially, while production cannot even keep up with linear growth. A year ago, large companies ordered hundreds of GPUs; now they talk in thousands; soon they will need millions. Memory manufacturing runs on a multi-year technology cycle: new fabs take five to seven years to build, the technology keeps getting more complex, and there are very few competitors in this segment.

Who is hit hardest

The main memory makers, South Korea's Samsung and SK Hynix and the US company Micron, are failing to keep up with demand growth. All three are investing tens of billions in new production capacity, but even that is not enough: the shortfall is already compounding geometrically. This creates a cascade of problems across the supply chain:

• Rising prices: GPUs with sufficient memory cost 1.5–2 times more on the secondary market
• Strategic uncertainty: companies do not know whether they will receive the hardware they need on time
• Architectural sacrifices: teams have to use smaller models and spread computation across more GPUs
• Market stratification: those who sign memory supply contracts first will gain a significant competitive advantage

Even Dell, one of the largest server integrators, depends on memory supply. The company cannot sell more high-performance AI servers without GPUs, and GPUs remain incomplete without enough memory.

How the industry is responding

Nvidia is actively working on new generations of memory: HBM3e and HBM4 are already in development, with projected high bandwidth and performance. However, developing, qualifying and mass-producing new memory standards takes years, not months.

In parallel, the company and its partners are looking for software workarounds: software optimization and new neural network architectures that need less memory per unit of performance. But this has hard limits too, since not every model can be trimmed without losing output quality.

Huang said this is not a short-term problem that will be solved in months.

The memory shortage will remain a serious limiting factor for at least the next several years.

What this means for the AI industry

The memory shortage is becoming a new ceiling on how fast AI infrastructure can be deployed. Not every company will be able to deploy large language models at the scale and on the timeline it needs. This creates a clear market divide: those who secure memory access first will move faster, and those who are late will fall behind.

For venture investors and startups, this means that getting the necessary hardware is becoming as critical to success as the idea and the team. It recalls the era of silicon shortages, when simply having chips gave a competitive advantage.