Nous Research unveils Lighthouse Attention to speed up LLM training

Nous Research has published Lighthouse Attention, a new method for optimizing the training of large language models on long contexts. The mechanism works exclusively during pre-training and is fully switched off once that process ends, so it does not affect the architecture or the forward-pass behavior of the finished model.

How Lighthouse Attention works

Lighthouse Attention is a selective hierarchical attention mechanism that wraps standard scaled dot-product attention during model pre-training. In practice, at each pass through an attention layer the model uses a dedicated selective mechanism instead of full attention over every token in the context.

The key difference from earlier approaches (such as NSA and HISA) is symmetric pooling of all components of the attention mechanism. Previous methods reduced only keys and values (K and V) and ignored queries, whereas Lighthouse pools queries, keys and values (Q, K and V) at the same time through a multi-level resolution pyramid. This gives a more balanced and efficient reduction in computation at every level.

Technically, this cuts the computational complexity of the attention operation from O(N·S·d) to O(S²·d), where N is the full context length, S is the size of the selected compact subsequence, and d is the model's hidden dimension. After selection, standard FlashAttention runs on the small dense subsequence, which saves significantly on both compute and the GPU memory required.

Impressive results

Nous Research tested Lighthouse Attention on a 530-million-parameter Llama-3-style model with a 98,000-token context, which is already a fairly long context for testing. The results showed a significant and consistent improvement in training performance:

• A 1.40–1.69x end-to-end training speedup over the baseline cuDNN SDPA implementation on GPU
• Comparable or lower final training loss, which indicates no loss in model quality or accuracy
• Full compatibility with existing FlashAttention infrastructure and standard frameworks such as PyTorch

This means organizations could train large models 40–70 percent faster without compromising on quality or accuracy. For large models trained on huge datasets, that translates into weeks of saved compute time on expensive GPU clusters.

Practical use and scalability

Lighthouse Attention's main advantages are its ease of adoption and its lack of any effect on the behavior of the finished model. The mechanism is used only during pre-training and switches off automatically once that critical stage is complete. A model trained with Lighthouse is therefore fully compatible with existing applications, services and workflows, with no changes to code, infrastructure or deployment.

The speedup is especially valuable for organizations training large models on contexts of tens and hundreds of thousands of tokens. Typical uses include analysis of long documents and reports, full-text search across large knowledge bases, writing and analyzing code with contexts of 100K+ tokens, handling conversations with a deep message history, and working with scientific papers and patents. Every percent of compute saved means concrete savings in electricity and significant cuts in cloud computing bills.

What it means for research and the industry

Optimizing transformer training remains an active and fertile area of research, despite twenty years of investment in the core architectural mechanisms. Lighthouse Attention shows that even well-studied, polished attention architectures leave room for innovation, improvements and unexpected optimizations.

If the research community adopts such methods and they are widely built into popular open frameworks such as PyTorch, HuggingFace Transformers and others, this could significantly lower the barrier to entry for organizations, startups and research groups that want to train their own large language models without needing huge compute resources and budgets.