LASAGNA Tests 450 RAG Configurations: The Promised +47% Gain Turns Into Just +17%

A new paper, Retrieval Augmented Generation Compositionality, challenges the usual way of comparing RAG techniques. The authors released LASAGNA, an open-source framework for controlled evaluation, and ran 450 configurations through it on eight benchmarks, with corpora from 5 thousand to 200 thousand documents. The most telling result: three retrieval techniques individually predicted a combined gain of +47%, but when assembled into a single pipeline they delivered only +17%.

Why isolated measurements inflate expectations

Modern RAG stopped being a simple "embeddings plus vector search" setup long ago. A typical pipeline includes an embedding model, chunking, retrieval, reranking, query expansion, context compression, and agentic answer verification. Each of these components is usually described in its own paper and tested against its own baseline, changing one component while holding everything else constant.

The LASAGNA authors asked whether these gains survive combination. The answer is an uncomfortable one. Three strong techniques from different pipeline stages do not add up arithmetically: the expected +47% shrank to +17%. It is reasonable to assume that some techniques solve overlapping problems, so the second method simply picks up what the first has already fixed. But the explanation matters less than the gap itself, which is nearly threefold.

The methodology deserves a separate mention. The authors declared three hypotheses in advance, before running the experiments. In a field where conclusions are often fitted to lucky runs, that earns some extra trust.

Retrieval and faithfulness pull in different directions

The second finding concerns the metrics themselves. Component gains combine differently for retrieval quality and for faithfulness, meaning how closely an answer sticks to the sources it found. Optimizing one metric can hurt the other.

In practice, a team that pushes recall up and watches only retrieval quality risks quietly losing reliability in the final answers. And reliability is exactly why RAG is deployed in the first place: so the model relies on external knowledge instead of making things up.

The winner at 5K doesn't have to win at 200K

The third finding is about scale. The best combination changes as the corpus grows: a setup that won on a small set of documents can lose on a large one. Many academic and internal experiments run on small samples because it is fast and cheap, and then carry their conclusions over to production, where there are orders of magnitude more documents.

What the paper records:

• Framework: LASAGNA, open source • Configurations tested: 450 • Benchmarks: 8 • Corpus size: from 5K to 200K documents • Hypotheses declared in advance: 3 • Key gap: expected +47% versus actual +17% for three retrieval techniques

How to rebuild your own RAG evaluation

The practical takeaway for engineers is straightforward: don't add up gains from papers, and don't assume that three good ideas will deliver the sum of three improvements. Test a new technique inside the full stack you actually run, and compare it against your current configuration as a whole.

Second, measure retrieval and faithfulness separately, and watch both at the same time. If a retrieval gain comes with a drop in faithfulness, that is a reason to stop, not to celebrate.

Third, test on a corpus close to production size. Results on 5 thousand documents are a poor predictor of behavior on 200 thousand, so choosing an architecture based on a small benchmark may turn out to be a mistake.

The authors urge researchers to evaluate new RAG methods in the full environment, and they offer LASAGNA to make such experiments less costly. For the industry, it is a chance to re-examine which numbers underpin its architecture decisions for retrieval pipelines.