A Team of AI Researchers Found a Linear Attention Design That Beat GDN2 at 1.3B

Autonomous agents can already propose ideas, edit code and run experiments. The authors of a new paper went a step further. Instead of a single "AI scientist," they built a team of persistent researchers, each grounded in its own scientific discipline. They used this team to search for linear attention architectures. At 1.3B parameters, the result beat GDN2 on language modeling, common-sense reasoning, and both synthetic and real retrieval tasks.

Linear attention remains one of the main candidates to replace the quadratic attention of the classic transformer on long contexts. The space of possible architectures is huge, and every test costs GPU-hours. As a result, the search easily gets stuck in local minima: agents circle around the same tricks and cycle through minor variations of known designs.

Researchers with different schooling and personal memory

The authors' core idea is that diversity should be organizational. If all agents think alike, they will propose similar experiments. So each agent was given a profile in a specific discipline, and the team explores the space in several directions at once.

Disciplinary profiles alone are not enough, though. The agents in this system are persistent: they retain context between iterations. Each has a private research history that holds its hypotheses, failures and interim conclusions. The experimental data, however, is shared. Any result, positive or negative, becomes available to the whole team and shapes the next round of proposals.

The result is a division of labor that resembles a real lab. The shared evidence gives everyone a single picture of what has already been tested. Personal histories let each agent develop its own line of reasoning instead of retelling someone else's.

How to split a limited experiment budget

The second problem is a practical one. Experiments are expensive, so the team can test only a small fraction of its ideas. Someone has to decide which researcher gets a slot for a run. If you always pick the one with the best recent results, the search narrows. If you rotate through everyone in turn, resources go to weak directions.

The authors use adaptive allocation of opportunities. It balances exploring new directions against building on ones that already work, the classic exploration/exploitation dilemma. Each result changes every researcher's chance of getting the next run.

Key parameters of the experiment:

• verification scale: models with 1.3B parameters;
• baseline: GDN2;
• tasks: language modeling, common-sense reasoning, synthetic and real retrieval;
• team composition: persistent agents with different disciplinary profiles;
• memory: private histories for each agent, shared experimental data for all;
• budget allocation: adaptive, with a fixed cap on runs.

What breaks when you remove any element

The most convincing part of the work is the controlled comparisons. The authors switched off individual components of the organization one at a time, and each time the findings were noticeably weaker.

Without disciplinary profiles, the search quickly plateaus. Agents without different training arrive at similar ideas, and novelty runs out early. Without private histories, researchers make less progress even though they see the same shared results. So it is not only about access to data: the accumulated personal trajectory of reasoning also affects the quality of the next proposals. And choosing researchers uniformly does worse than adaptive selection on the same budget.

The authors' conclusion applies to the whole field. Sustained autonomous discovery depends on more than the strength of the base model. What matters is who does the research, what they remember, and how research effort is distributed. For the industry, this shifts the focus from the race for the best agent to designing an organizational structure out of agents.

For R&D teams already trying agents to automate experiments, this suggests three working rules. Give agents different expertise profiles instead of cloning a single role. Give them their own memory plus a shared log of results, including failed ones. And don't split the compute budget equally: reallocate it based on the runs already completed. If the authors' assessment is correct, the gain comes from this kind of research organization, not simply from buying a stronger model.