NGT Memory module shows how to give an LLM persistent memory without a vector database

The open-source project NGT Memory offers a more practical answer to an old LLM problem: how to avoid forgetting important facts about a user between sessions. Instead of endlessly expanding the context, the module saves facts, builds a profile and mixes memory into the model's response.

How the memory works

The project's author starts from a simple idea: keeping the whole dialogue history in the context window is convenient only until the first token limit. After that come truncation of old messages, summarization that loses detail, or connecting an external vector database with unnecessary infrastructure. NGT Memory takes a different route: memory lives directly in the Python process, is exposed through a REST API and Docker, and does not require a separate database at the start. The idea is to retrieve not the whole conversation, but only the facts the model actually needs for the current request.

• Cosine similarity finds facts that are close in meaning
• An associative graph links concepts such as "vegetarian" and "restaurants"
• Hierarchical consolidation promotes frequently used facts into long-term memory
• A structured profile moves age, city, diet and allergies into separate slots

On top of the text memory, the module also builds a user profile. If a person writes in fragments, the system can stitch pieces like "I" + "30" + "years old" into one meaningful fact and store it with reduced confidence. If a conflict arises later, for example the age suddenly decreases, the update is blocked until an explicit correction such as "I was wrong". As a result, the memory behaves not like a passive log but like a validation layer that tries to tell real user data from random noise.

Test results

The most interesting part of the article is not the architecture itself but the check on practical scenarios. In one experiment with a medical assistant, a personal assistant and tech support, the memory mode raised factual accuracy from 1.22 to 2.44 points out of 3. In a more realistic A/B test, memory won 17 of 18 ratings, and the average answer score was 0.889 versus 0.056 without memory. Separately, the author ran 54 edge-case checks and got 51 passes, about 94%.

"A response to garbage is also garbage."

This formula explains another important layer of the system: the quality filter. Short meaningless messages, bare numbers, special characters and even the assistant's replies to such garbage do not make it into memory. Otherwise useful facts would quickly drown in noise. Such a filter matters especially for chatbots and agent systems, where users often write in fragments and the model itself tends to generate polite but empty phrases that only clutter the results of the next memory search.

Main limitations

Despite the good results, NGT Memory cannot yet be called a ready, universal production memory without caveats. The current version keeps everything in the RAM of a single process, so state is lost when the container restarts. When several workers run, another problem appears: each holds its own session store, so a save request may land in one process and a retrieval request in another. In tests, local memory is fast, around 2–3 ms on CPU, but most of the latency still comes from external embedding and chat calls, which take hundreds of milliseconds.

The author also describes more mundane bugs. An overly soft system prompt led the model to see the memory but ignore it as an optional hint. After the instruction was tightened, answers became more stable. Another unpleasant case came from a regex for extracting names: a pattern like I'm + name began treating words like allergic as a name. Such details show that LLM memory depends not only on retrieval but on a mass of small rules, without which a demo quickly falls apart in a real dialogue.

What this means

NGT Memory shows a useful shift in how LLM applications are built: memory is not enough if it is merely "smart"; it must be verifiable, cheap and resistant to garbage. For developers of bots and AI agents, it is a good sign that a long-term context layer can already be built as a separate engineering module rather than a set of scattered hacks.