Quanta Magazine examines why attention became the core of LLMs after a 2017 paper

Eight authors, one paper from 2017, and a one-line formula: softmax(QK^T/√d)V. This operation sits underneath most large language models. A Quanta Magazine column dated October 12, 2026 asks why attention, of all things, turned out to be the key to processing language at scale, and whether it can be considered the essence of intelligence.

The column's announcement offers no answer, only the question, so below we lay out the technical background against which this debate is taking place. Read the original for the author's conclusions.

Where the formula came from

The idea of attention is older than the transformer. In 2014, Bahdanau, Cho and Bengio used a similar mechanism in a recurrent model for machine translation: while generating each word, the decoder could "look at" different parts of the source sentence. The recurrent network remained the foundation, and attention served as an add-on.

In 2017, the authors of Attention Is All You Need at Google removed recurrence entirely. What remained were layers in which each token builds a query, compares it with the keys of all other tokens, and takes a weighted sum of their values. On the WMT 2014 translation benchmark, the large version of the model scored 28.4 BLEU for English–German and 41.8 for English–French.

Why attention scales

The main advantage is that the entire sequence is processed in parallel. A recurrent network reads text step by step, which leaves GPUs idle. Attention boils down to large matrix multiplications, which is exactly what modern accelerators are built for. More data, more parameters, more chips: this recipe reliably delivered quality gains.

The second reason is flexibility. Attention weights are computed from the input itself, so the link between a word at the start of a paragraph and a word at the end does not fade along the way, as it did in older recurrent architectures.

Key figures from the original paper:

• 8 authors, NeurIPS 2017;
• base model: about 65 million parameters, trained for roughly 12 hours on 8 NVIDIA P100 GPUs;
• large model: about 213 million parameters, 3.5 days on the same 8 GPUs;
• results: 28.4 BLEU (EN→DE) and 41.8 BLEU (EN→FR);
• for comparison, GPT-3 in 2020 had 175 billion parameters and a context of 2048 tokens.

The price of universality

Attention has a weak spot: every token is compared with every other token. Time and memory grow in proportion to the square of the context length. Double the context, and compute for this part quadruples.

The industry is responding with engineering. FlashAttention (2022) does not change the math. It restructures how the GPU handles memory and delivers the exact same result faster. Sparse and linear attention are developing in parallel, as are hybrids with state space models. These approaches save compute but often lose to classical attention in quality on hard tasks.

How to read this debate in practice

The claim that "attention is the essence of intelligence" is worth splitting into two. The first is testable: attention has proven an exceptionally successful computational primitive for training on text. The second is philosophical: what follows from this for our understanding of the mind remains a matter of debate, and Quanta puts it in the headline.

For engineers, the takeaway is down to earth. If you are choosing an architecture or evaluating infrastructure, count not only parameters but also context length: the quadratic part determines the cost of long requests, RAG pipelines, and agents with large memory. If you follow research, look at work that compares alternatives to attention against the transformer at an equal compute budget. For now, the transformer remains the baseline.