Stanford unveils Onyx, a sparse-AI chip that is faster than CPUs and 70x more energy-efficient

Stanford researchers have unveiled Onyx, an accelerator that can extract value from the "zeros" inside AI models. The idea is to stop spending energy on multiplications and additions that change nothing in advance, and so speed up computation without giving up large models.

Why zeros matter

In neural networks, data, weights and activations are stored as arrays of numbers: vectors, matrices and tensors. In many cases a significant share of these numbers is zero, or so close to zero that they can be treated as zeros without a noticeable loss of quality. This property is called sparsity.

If more than half the values are zeros, a model can already benefit from specialized algorithms: instead of storing and processing the whole matrix, the system keeps only the useful values and skips the empty operations.

This matters for the industry because models are growing faster than infrastructure is getting cheaper. More parameters mean higher quality, but also higher cost to run in time, energy and carbon footprint. The paper cites the example of Meta Llama with 2 trillion parameters. The researchers also recall Cerebras results: on Llama 7B, 70–80 percent of parameters were zeroed out without loss of accuracy. That means large models already hide a reserve for speedups; the task is to learn to use it.

Where efficiency is lost

The problem is that popular hardware was originally built for dense computation, not sparse structures. When data is compressed, metadata must be stored alongside the non-zero values: row indices, column indices and segments. Access to such data becomes indirect and unpredictable: the processor first has to look up coordinates, and only then the value itself. As a result, part of the time goes not to math but to wandering through memory and housekeeping operations.

• GPUs handle dense matrices well, but with random sparsity they often parallelize useless operations on zeros.
• Structured sparsity does not always help, because it requires a rigid zeroing pattern, for example two zeros out of every four adjacent parameters.
• CPUs are more flexible, but often run into prefetcher misses and unpredictable memory accesses.
• Even sparse libraries do not cut all the overhead, because part of the resources goes to maintaining the data itself.

Chipmakers are already looking for workarounds, but so far they are narrow. Apple sped up indirect memory accesses in the A14 and M1 chips, Cerebras promotes a sparse approach in its Wafer Scale Engine, and Meta is developing MTIA. But limits remain here too: some solutions work only with weight sparsity, others support only individual operations such as matrix multiplication. That is not enough for real AI workloads, because models consist not of one operation but of a long chain of different layers and transformations.

How Onyx works

The Stanford team started from the lowest level and built Onyx, a programmable accelerator that handles sparse and dense computation equally well. It is based on a CGRA architecture, an intermediate option between a CPU and an FPGA: noticeably more flexible than a classic processor, yet more efficient than fully bit-configurable circuits.

Onyx consists of compute blocks and memory blocks that store compressed matrices and process them directly in that form, without unpacking them back into dense format unless necessary. The compiler is also important: it translates expressions such as sparse-matrix-by-vector multiplication into a memory-and-compute graph, then maps it onto the chip's blocks.

According to Stanford, Onyx consumed on average 70x less energy than a CPU and ran computations roughly 8x faster. On the energy-delay product metric, the gain reached 565x relative to a 12-core Intel Xeon running sparse libraries. The next generation of Onyx is expected to add support for nonlinear layers, normalization, softmax and more convenient switching between sparse and dense modes.

What this means

The paper's main point is not that yet another AI chip has appeared, but that developers are starting to optimize models not only through lower precision or smaller size, but through the structure of the computation itself. If the sparsity approach takes hold, large models could be run cheaper and faster, which means the next leap in AI may come not only from new models but from a new class of hardware.

*Meta is recognized as an extremist organization and banned in the Russian Federation.