MemT-1.0 builds memory into the model itself: 66x smaller storage at 128K

The authors of MemT-1.0 propose that language models stop leaning on external stores and instead make memory part of the network's own computation. At a 128K-token history, the smaller model in the family, MemT-1.0-Nano, shows roughly 1.86x faster end-to-end inference and cuts persistent storage 66x compared with a full KV-cache. Two models have been released in total, Nano and Flash, and training used 34B tokens of data.

Memory as part of the computation

Most current systems handle long context in one of two ways. The first is to carry the entire KV-cache, which grows with the history. The second is retrieval: search an external database for relevant fragments and mix them into the prompt. Both approaches have a cost: memory or latency in the first case, architectural complexity and dependence on search quality in the second.

MemT-1.0 takes a third path. The authors call it memory-native evolution: memory should be a built-in capability of the foundation model, allowing it to accumulate information continuously and use it. The architecture is built on Metis and uses Gated DeltaNet-2, a mechanism with stable token-level memory updates within each turn of a conversation. To scale capacity, the team added Mixture-of-Storage, so native memory grows without inflating the context.

How it was trained

Before the main training, the team prepared the backbone separately. A two-stage pipeline followed: continued pre-training and post-training. The total data volume was 34B tokens, which is modest for this class of work and suggests the model was further trained on an existing base rather than grown from scratch. The authors also mention specialized infrastructure for efficient training and inference, but the abstract does not disclose details.

What the measurements showed

The evaluation covered both memory capabilities and the model's general skills. Key results from the description of the work:

• MemT-1.0 outperforms parametric-memory baselines on average;
• it remains competitive with retrieval-augmented generation methods;
• it mostly preserves general capabilities, meaning that training for memory did not destroy basic skills;
• at a 128K-token history, MemT-1.0-Nano speeds up end-to-end inference by roughly 1.86x relative to full context with a KV-cache;
• persistent storage shrinks 66x;
• on agentic tasks, the share of successful runs rises because the model absorbs extracted procedural experience.

The wording "competitive with RAG" accurately reflects the situation: the authors do not claim superiority over retrieval. That matters, because retrieval remains the workhorse in production. The abstract gives no figures for specific benchmarks or model sizes, so the size of the gap over the baselines can only be judged from the full paper.

Where this could be useful

The most obvious use case is agents that run for a long time and need to remember what they have already tried and what worked. If procedural experience is baked into the model's state, there is no need to pull it from a database every time and spend context tokens on it. For developers of assistants and coding agents, the 66x storage savings means a user's personal history can be kept more compactly and cheaply, and in long conversations the speed gain grows with context length.

Caution is also warranted. The results come from two small models, and the claim about preserving general capabilities is phrased as "mostly." Before replacing a retrieval pipeline, it is worth waiting for independent verification, looking at behavior when memory holds erroneous or outdated entries, and assessing how native memory fits with audit and data-deletion requirements. But the direction is clear: memory is ceasing to be an add-on and becoming a property of the architecture.