Google Taught Language Models to "Sleep" and Move Knowledge from Memory into Weights

Google

Taught Language Models to "Sleep" and Move Knowledge from Memory into Weights

Google Research has published a framework called SLEEP — a method for teaching language models new information without erasing what they already knew. The idea grew out of two long-standing pain points in continual learning.

The first is catastrophic forgetting: when a model is fine-tuned on a new task, weight updates can overwrite information that mattered for earlier tasks. The second is that knowledge pulled from the context window never becomes permanent memory. As long as information sits in the prompt, the model can use it. Once the context is gone, so is the knowledge. And keeping a long context around is expensive — it has to be reprocessed and held in memory for every single request.

Wakefulness, sleep, and "dreaming"

SLEEP divides a model's operation into two modes.

• Wakefulness — the model takes in inputs and stores information in short-term memory, as usual.
• Sleep — new input is paused. The model activates additional MoE (mixture-of-experts) memory blocks and transfers accumulated knowledge into their weights, trying to minimize disruption to information that's already locked in. Once the transfer is complete, short-term memory is cleared.

There's a separate "dreaming" phase as well: random combinations of experts generate their own training examples, and the system selects the most useful ones to use for further training. In other words, the model partly prepares its own consolidation material rather than relying on an external dataset for every cycle.

One important detail: "sleep" is a controlled fine-tuning loop governed by rules the researchers define, not the model restructuring itself autonomously. The architecture and code stay fixed — only the weights of designated MoE blocks change.

What the experiments showed

The team tested the approach on three different tasks.

• BABILong — a benchmark for handling extremely long context. The model retained information from sequences up to 10 million tokens, not by expanding the context window, but by locking knowledge directly into its parameters.
• Sequential translation training — a classic test for forgetting. Languages were learned one after another, and on such benchmarks, quality on earlier languages typically drops noticeably once new ones are introduced. With SLEEP, that drop was nearly zero.
• Math (AIME) — 79.2% accuracy on AIME 2024 and 69% on AIME 2025. The gap between the two years isn't a bombshell — similar spreads show up for many models across these two problem sets, and it's too early to draw conclusions about causes from a single paper. What matters more is that transferring knowledge into weights didn't break the model's capacity for mathematical reasoning.

What this means for engineering teams

The main question is cost. Moving knowledge into weights requires extra fine-tuning cycles, and therefore compute that has to be scheduled: while the model is "sleeping," it isn't serving requests. Teams will need to decide how often to run this cycle, how much memory to accumulate beforehand, and how many MoE blocks to allocate for storage — all of these are parameters set by the developers' own rules, and tuning them for a specific workload will be up to each team.

The potential payoff lies in context economics. If a corporate knowledge base, a long conversation history, or frequently updated documents can be locked into weights, there's no need to run millions of tokens through the model every time and pay for reprocessing them. For agents with long operating histories, that translates directly into savings on memory and response latency.

There's also a risk the framework's description doesn't resolve: writing information into weights is harder to track and undo than a document sitting in context or in an external database. For regulated industries, teams will need to separately figure out how to audit — and, if necessary, remove — whatever the model locked in during "sleep." At this stage, SLEEP is best understood as a strong research result and a candidate for pilot testing in projects with constantly updated data, not as a ready-made production component.