HyenaND: субквадратичный оператор вместо attention для многомерных данных

HyenaND: A

Subquadratic Operator to Replace Attention for Multidimensional Data

Researchers have introduced HyenaND — a subquadratic operator for neural networks that replaces the attention mechanism when working with multidimensional data, processing images, volumes, and partial differential equations (PDEs) in their native geometry, without unrolling them into a one-dimensional sequence. The preprint appeared on arXiv in July 2026, with the key promise being O(L log L) complexity instead of attention's quadratic O(L²).

What's wrong with attention

The attention mechanism scales quadratically: for an input of length L, its cost grows as O(L²), making long context expensive. According to the authors, all existing subquadratic alternatives are forced into trade-offs on multidimensional data.

Ordinary convolutions lack a global receptive field and input dependence, while recurrent models require unrolling an image or a three-dimensional volume into an artificial one-dimensional traversal order — and that order destroys the original spatial structure of the data.

How HyenaND works

HyenaND is a global, input-dependent operator that acts directly on the native geometry of multidimensional data via convolutions with implicitly parameterized multidimensional kernels. Unlike an ordinary convolution, the kernel here is global and adapts to the specific input.

Practical speed is delivered by the CUDA implementation nSubQ: it fuses the convolution path via the fast Fourier transform (FFT) into a single kernel, turning the theoretical O(L log L) complexity into real wall-clock speedup.

• HyenaND — a subquadratic, global, input-dependent operator
• Works with images, 3D volumes, and PDEs without unrolling into 1D
• The nSubQ CUDA kernel fuses the FFT convolution path into a single operation
• Complexity O(L log L) versus O(L²) for attention
• Validated across 4 domains: genomics, vision, medical imaging, PDEs

What the experiments showed

Pure HyenaND stacks matched the accuracy of strong attention baselines across four types of tasks at once: long-context genomics, computer vision, medical imaging, and modeling of partial differential equations. In other words, dropping quadratic attention cost no quality.

Hybrid configurations proved even stronger: alternating HyenaND and attention layers outperformed both pure attention and strong recurrence-based hybrids.

"Hybrid configurations interleaving

HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids," the abstract of the arXiv preprint states.

What it means

The race to replace attention is shifting from one-dimensional text to multidimensional data. HyenaND shows that it's possible to preserve transformer-level accuracy while removing the quadratic cost where data is inherently two- or three-dimensional — in medicine, biology, and physical modeling.

Frequently asked questions

What is HyenaND?

HyenaND is a subquadratic operator for neural networks, an alternative to attention. It processes multidimensional data (images, volumes, PDEs) in their native geometry through global, input-dependent convolutions and scales as O(L log L).

How does HyenaND differ from attention?

Attention scales quadratically — O(L²) with input length. HyenaND provides the same global receptive field and input dependence, but with O(L log L) complexity, and does not require unrolling multidimensional data into a one-dimensional sequence.

Where is HyenaND used?

According to the preprint, HyenaND has been tested on long-context genomics, computer vision, medical imaging, and modeling of partial differential equations.