Huginn Beats Ouro on All 10 Benchmarks in Looped LLM Comparison

Looped language models save parameters, but until now it was unclear which architectural decisions actually produce the gain. A new paper, the first part of the series "Towards Looped Models Done Right," separates these decisions and tests each one on its own. The result: at 730M parameters and 336B tokens, the full Huginn-style architecture beat Ouro on all ten benchmarks. In the 8BA0.8B MoE version (32BA3B when unrolled), trained on 500B tokens, Huginn is again ahead almost everywhere. At equal FLOPs in training and inference, it uses 75% fewer resident parameters than a 112-layer feedforward MoE.

Three axes that used to be entangled

A looped model reuses the same physical Transformer blocks across several steps of logical depth. One set of weights runs multiple times, so the network "thinks" deeper without adding parameters. In existing architectures, three decisions are tied together. First: where in the stack the recurrence sits. Second: how the original input is mixed back in at each iteration. Third: how the recurrent state is organized. As a result, comparing two finished models does not reveal what actually worked.

The authors pulled these axes apart. They trained models from scratch at the same parameter scale, the same logical depth, and equal token budgets. They then tested whether the findings carry over to MoE models, where both resident and active parameters match.

What the experiments showed

The "sandwich" performed best: an input block (prelude), a recurrent block, and an output block (coda). It consistently improves retrieval from context and tasks that require reasoning. Input injection, meaning re-feeding the input inside the loop, mainly helps with knowledge retrieval and better use of context. Random state initialization and separate recurrent states H and L produced mixed or negative results. The combination of these findings is the Huginn design.

Key numbers from the paper:

• 730M parameters, 336B tokens: the full Huginn architecture beats Ouro on all 10 benchmarks.
• MoE 8BA0.8B (32BA3B when unrolled), 500B tokens: Huginn outperforms Ouro on almost all tests of knowledge, reasoning, and coding.
• Huginn's expert routing is better balanced than Ouro's.
• At equal FLOPs in training and inference, Huginn uses 75% fewer resident parameters than a 112-layer feedforward MoE.
• Against this baseline, the looped model wins on DROP and GSM8K, holds parity on MATH500, and stays competitive on the remaining tests.

The transfer to MoE matters separately. Looped architectures are usually tested on dense models, and a gain at that scale does not guarantee the design survives the move to experts. Here it does, and that noticeably strengthens the case for Huginn.

What to change in practice

The main benefit is memory. Resident parameters determine how many weights must be kept on the accelerator, and repeated passes spend compute, not memory. If inference is limited by memory capacity rather than FLOPs, a looped MoE with a quarter of the resident parameters looks like a serious alternative to a deep feedforward stack. This applies especially where arithmetic and reading tasks like GSM8K and DROP matter.

There are caveats. The abstract has no data on latency or throughput, and on some benchmarks the looped model is only "competitive," not better than the baseline. The scale is limited to 8B resident parameters and 500B tokens, so conclusions for models an order of magnitude larger need separate verification. For teams building looped models, the paper offers a short checklist: use the sandwich topology, keep input injection, do not complicate the state with separate H/L without proven benefit, and watch expert routing balance. This is only the first part of the series, and the follow-up will show how robust this set of decisions is.