Huginn Beats Ouro on All 10 Benchmarks: The Design Space of Recurrent LLMs

The Architecture of Recurrent Models: Three Design Axes

Recurrent language models reuse the physical blocks of a Transformer to achieve logical depth, saving on compute. Existing architectures, however, tangle together three critical design axes: where recurrent blocks are placed, how input signals are injected, and how recurrent state is organized. Researchers at the Institute of Foundation Models ran a controlled comparison of these axes using models trained from scratch at matched parameter counts, logical depth, and token budgets.

The paper's authors are Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, and Xuezhe Ma. The study found that a prelude-recurrent block-coda sandwich architecture (introduction → recurrent blocks → conclusion) consistently outperforms alternative recurrence-placement configurations. This approach substantially improves performance on context-retrieval tasks and tasks requiring complex logical reasoning.

Input

Injection and Recurrent State: What Works and What Doesn't

Input injection — feeding input data at every loop step — mainly improves knowledge retrieval and effective context use. Random initialization of the hidden state and a scheme using separate recurrent states for the H and L parameters, however, produce mixed or outright negative results, failing to justify the added complexity.

Huginn vs. Ouro: A Full Benchmark Comparison

At the base scale (730M parameters, 336B training tokens), the full Huginn architecture outperforms Ouro on all ten test benchmarks. The advantage holds when scaling up to Mixture-of-Experts (MoE) models. At the 8B×0.8B parameter scale (32B×3B when deployed), trained on 500B tokens, Huginn models outperform Ouro on nearly every benchmark, spanning knowledge, reasoning, and coding domains.

Specific results on named benchmarks:

• DROP (text information extraction): Huginn wins • GSM8K (word-problem math reasoning): Huginn wins • MATH500 (complex math problems): Huginn matches the baseline MoE model

Expert routing in Huginn is more balanced, reducing the problem of expert specialization and collapse — where most examples get routed to a single expert.

Parameter Efficiency and Practical Applications

At matched training and inference FLOPs, the Huginn architecture uses 75% fewer resident parameters than a baseline 112-layer feedforward MoE model. Despite its smaller physical parameter count, the model outperforms the baseline on DROP and GSM8K, matches it on MATH500, and stays competitive on the remaining seven benchmarks. That makes Huginn a promising architecture for parameter-efficient large language models, particularly under memory constraints or fast-inference requirements. The research is fully open-sourced in the xllm-loop GitHub repository, letting the community reproduce the results and build on these design principles.