STILL from HIT Shenzhen: Hybrid Attention Within a Layer Picks Tokens by Content

The quadratic complexity of softmax attention remains the main constraint for long-context models: cost grows as O(n²) with sequence length. A team at Harbin Institute of Technology (Shenzhen campus) is presenting STILL at NeurIPS 2026, an architecture it calls the first content-aware intra-layer hybrid attention. The first author is PhD student Meng Weikang, who works on efficient architectures and on linear and hybrid attention. The corresponding author is Professor Zhang Zheng. Jiqizhixin reported on the research.

We have only the introductory excerpt of the source material. It contains no concrete quality or speed metrics, so below we examine the idea, not the results.

Why hybrids are called "pseudo-hybrids"

Linear attention is cheap: cost grows linearly with context length. But it compresses history into a fixed-size state and handles exact retrieval of details worse. So the industry combines both mechanisms. The most common design is inter-layer: most layers are linear, and a full softmax attention layer appears every few layers. Jamba, MiniMax-01 and Qwen3-Next, for example, follow this principle.

Intra-layer hybrids split the work inside a single layer. A typical variant uses exact attention in a local window around the current token and leaves everything else to a linear mechanism. Judging by the title, the STILL authors regard this design as positional "pseudo-hybridity": the decision about which tokens deserve exact processing is made by distance, not by what the tokens actually say. An important figure or name at the start of a long document ends up in the "cheap" branch alongside noise.

What STILL proposes

The key claim: the split between exact and linear attention within a layer is determined by content. The available excerpt does not reveal how the selection mechanism works. It could be a learned router, a token-importance score or some other scheme. The details will have to come from the paper.

Here is what the public data confirms so far:

• Conference: NeurIPS 2026.
• Architecture name: STILL.
• Type: intra-layer hybrid of softmax and linear attention.
• Difference from predecessors: selection by token content, not by position.
• Institution: Harbin Institute of Technology (Shenzhen).
• Perplexity, context length and speedup metrics: not provided in the available excerpt.

What to check when reading the paper

The intra-layer approach has a practical advantage. It does not require changing the structure of the layer stack, and in theory it gives every layer access to both modes. But content-based token selection has a cost: dynamic routing is harder to accelerate cleanly on a GPU than a static window. Here, kernels and throughput matter as much as theoretical complexity.

Who should care right now: teams that train or fine-tune models for long context and compare hybrid designs. What to look for in the paper and code:

• How STILL performs on exact-retrieval tasks (needle-in-a-haystack, retrieval) compared with purely linear models.
• Real-world speed on long sequences, not just asymptotics.
• Comparison with inter-layer hybrids at the same parameter and training budget.
• Whether an open implementation exists and whether it is compatible with existing frameworks.

For now, this is a bid for the next step in long-context architectures. The verdict depends on numbers that the introductory excerpt does not contain.