Deep Fusion Memory cuts retrieval 32x and beats baselines by 12.7%

Researchers have proposed Deep Fusion Memory (DFM), a parametric memory for large language models that embeds external knowledge into hidden states and does not need a database lookup while the model generates an answer. Building the training data for this memory is 32 times cheaper in retrieval queries and 2048 times cheaper in retrieved items than in existing parametric approaches. On six QA benchmarks, DFM beats the strongest baseline by 12.71% on Mistral-7B-v0.3 and by 11.19% on Qwen3-8B-Base.

Three ways to teach a model something new, each with its own price

Anyone who has taken an LLM into a production application knows the problem: the model has to know things that were not in its pretraining data. That means a company's private documents, medical literature, fresh data. The classic approaches all involve trade-offs. Fine-tuning on a new corpus risks catastrophic forgetting, where the model learns the new material and loses the old. RAG pulls in documents reliably, but every request turns into a search and the context balloons, which directly costs latency and tokens.

The third approach is external parametric memory. Knowledge lives in separate trainable parameters, and there is no retrieval at runtime. But this approach has two flaws, and DFM targets both. First, preparing data to train such a memory requires massive offline retrieval, because the targets are built at the level of individual tokens. Second, the memory is usually mixed with the model's output by interpolating at the probability-distribution level. That is a shallow fusion: the knowledge does not influence how the model "thinks" inside its layers.

A memory predictor instead of retrieval

DFM moves the work from tokens to chunks. A causal memory predictor generates a memory vector from the preceding context. It is trained on embeddings of chunks found in the corpus, which serve as the supervision signal. The predicted memory is shared by all tokens within a chunk, so far fewer queries to the corpus are needed. That is where the 32x savings in queries and the 2048x savings in retrieved items come from.

The second part of the design is deep fusion. Memory enters the hidden states through gated cross-attention, meaning a trainable gate that regulates how much external knowledge to let through at each point. The predictor and the fusion module are trained end to end, while the base model itself stays frozen. Its original capabilities are left untouched, which directly addresses the forgetting problem. At inference there is no explicit retrieval: the memory is predicted from the context.

What the measurements showed

• Six QA tasks: +12.71% over the strongest baseline on Mistral-7B-v0.3.
• The same tasks on Qwen3-8B-Base: +11.19%.
• Domain adaptation: on average +6.51 points for Qwen3-4B-Base and +4.38 for Qwen3-8B-Base.
• Cost of building training data: 32 times fewer queries and 2048 times fewer retrieved items than methods with token-level objectives.
• The backbone is frozen, and no retrieval is needed at inference.

Note the difference in domain adaptation: the 4B model gains more than the 8B. That makes sense if the memory compensates for the model's own lack of knowledge. Smaller models, which are often run on limited hardware, benefit more from this kind of add-on.

Who can use this in practice

For teams building domain-specific assistants on open models, DFM looks like a middle ground between RAG and fine-tuning. It does not add a retrieval call to every request, and it does not require unfreezing the weights. That matters most where latency is critical and the knowledge corpus is stable: internal documentation, technical support, professional reference works.

The limitations are also clear from the design itself. The corpus still has to be processed offline once to get the chunk embeddings for training. If knowledge changes daily, retraining the predictor costs more than updating a RAG index. So DFM is more likely to complement retrieval than replace it: for fast-aging data, search will remain more convenient, while for stable subject areas, parametric memory with deep fusion becomes a serious alternative. The main takeaway from the work is that cutting data-preparation cost by several orders of magnitude did not cost any quality, and on QA, quality actually rose by double-digit percentages.