Mixture of Agents

Mixture of Agents (MoA) is an ensemble framework in which multiple large language models independently generate responses to a query, and one or more aggregator models synthesize those outputs into a single, higher-quality answer.

Mixture of Agents is a multi-model collaboration architecture proposed by Wang et al. at Together AI in 2024. It is distinct from Mixture of Experts (MoE), which routes tokens through specialized sub-networks inside a single model at the weight level. In MoA, multiple complete, independent LLM instances—called proposers—each answer the input query in parallel, and their outputs are then passed to an aggregator model that synthesizes a final response. The core empirical observation motivating the design is that language models reliably improve their output quality when shown diverse reference answers, even when those references come from weaker or peer-level models.

In a standard MoA pipeline, proposer models receive only the original query and respond independently. The aggregator receives the original query plus all proposer outputs and is prompted to synthesize, correct, and extend them into a single coherent answer. Multiple aggregation rounds can be applied sequentially, progressively distilling quality. The proposers and aggregator can be drawn from different providers or model families, allowing complementary strengths—such as pairing a model with strong factual recall against one with stronger structured reasoning.

MoA allows practitioners to improve answer quality without access to model weights or the ability to run fine-tuning, by combining publicly available APIs. In benchmarks published with the 2024 paper, a MoA configuration using open-weight models surpassed GPT-4o on AlpacaEval 2.0 at the time of release. It also offers a natural path to combining specialized models for domain-specific tasks.

As of 2026, MoA has influenced ensemble patterns in production agent systems and contributed to research on collaborative inference and LLM debate. The primary practical constraint is cost and latency: running multiple full models per query multiplies inference expenses and response time. Optimizations including speculative aggregation, model caching, and routing queries to smaller proposers for low-complexity tasks have been explored to reduce overhead in commercial deployments.