Nvidia unveils Groq 3, betting on dedicated chips for AI inference

Nvidia has shown Groq 3, the company's first chip designed specifically for AI inference rather than model training. It is an important signal: the market is shifting from a race for ever-larger models to a race over how quickly and cheaply those models answer users.

Why Nvidia is changing course

At the GTC conference, Nvidia's chief executive announced not only the Vera Rubin lineup but also a separate class of processors for running models. Groq 3 LPU is built on technology licensed from the startup Groq, with which Nvidia struck a deal late last year. The fact that only about two and a half months passed between the licensing and the product announcement shows how fast demand for inference in data centers is growing.

"AI is finally able to do useful work, and the inflection point of

inference has already arrived."

Training and inference solve different problems, so they need different hardware. During training, a system spends weeks crunching huge volumes of data and updating the model's weights. Inference happens at the moment a user makes a request, and for reasoning models a single session can include many internal passes before the person sees an answer. What matters here is not peak FLOPS but latency, a steady data flow and predictable token generation time.

How Groq 3 is built

Groq's approach differs from the familiar GPU design. Instead of relying on separate high-bandwidth HBM memory next to the graphics processor, the chip uses SRAM embedded directly in the compute block. This simplifies data movement: data flows through the processor in a line, without extra trips out and back. As a result, the architecture gives up versatility but wins where the fastest possible response is needed. For inference, where a model generates tokens sequentially rather than computing everything in one big batch, this design is especially useful.

The difference shows in the specs too. Rubin GPU remains a machine for heavy computation and large models, while Groq 3 was built for a different goal: minimal latency in the decode stage, when the answer is already being assembled token by token. In overall compute and memory capacity the LPU is noticeably more modest, but it wins on bandwidth and suits final output better. So Nvidia is not replacing the GPU with a new class of chip; it is complementing it with a specialized accelerator.

• Rubin GPU has 288 GB of HBM; Groq 3 has about 500 MB of on-chip SRAM
• Rubin delivers up to 50 petaflops in 4-bit compute; Groq 3 delivers 1.2 petaflops in 8-bit
• Groq 3 reaches 150 TB/s of memory bandwidth versus 22 TB/s for Rubin
• Groq 3 focuses not on versatility but on fast, low-latency token generation

The market is moving into inference

Over the past couple of years, inference chips have seen a real explosion of startups. D-Matrix is betting on digital in-memory compute, Etched on ASICs for transformers, RainAI on neuromorphic circuits, EnCharge on analog in-memory compute, and FuriosaAI on an architecture for tensor operations. With this announcement, Nvidia did not simply add another product; it effectively confirmed that the niche is too big to ignore inside the GPU empire.

The bet is not only on a separate chip but also on splitting inference into stages. AWS recently showed a system with Trainium and Cerebras CS-3, where prefill and decode run on different types of hardware. Nvidia is heading the same way: the new Groq 3 LPX module will include eight LPUs alongside the Vera Rubin system. Prefill and the heavier part of decode will stay on Rubin, while the final output stage will run on Groq 3. This hybrid lets each processor play to its strengths instead of settling for a compromise.

What this means

The main news is not that Nvidia released yet another accelerator, but that the market's largest player has publicly recognized inference as a separate class of computing. For AI products this is good news: if such architectures really scale in production, model responses will get faster and the economics of mass use more predictable. The next stage of AI competition will be fought not only over model quality but also over the cost of a million useful answers.