Researchers find topics explain 86% of the gap in an LLM personalization test
You add a user's history to an LLM and accuracy improves—it's tempting to credit personalization. But the model may simply have learned their favorite topics: that they read about technology, for example. The study's authors compared personal histories with other people's histories containing similar content, showing why a standard history swap can overstate the impact of personalization.
AI-processed from TG: @nadlskom (что-то на DL-ском); edited by Hamidun News
You add reading history to an assistant, and it gets better at selecting news. It looks as though memory is working. But there is a simpler explanation: the model noticed that the person likes technology. To test the value of personal history specifically, give it another technology enthusiast's history and compare the results. This test helps establish what you are getting in return for the system's complexity.
In Not Your Diary, Just Your Topic, researchers examine this problem through a news categorization task. The practical question is straightforward: how much value does information about a particular person add beyond knowing their favorite topics?
Someone else's history can be too easy an opponent
Imagine two readers. Both follow technology, but one is interested in developer tools and the other in consumer devices. A third reader mainly opens sports news.
If you replace the first reader's history with the third reader's, a drop in performance is hardly surprising. You have simultaneously removed information about the person and changed the subject matter. That comparison makes it difficult to tell exactly which signal the model lost.
The second reader's history provides a more demanding test. The broad interest stays the same, while individual preferences differ. This example illustrates the test's design; it does not describe individual study participants.
For a product team, the distinction matters. A weak control test can easily lead to the conclusion that you need detailed profiles, more stored events and complex memory processing. First, however, it is useful to test a cheaper alternative: are a few interest categories enough for your task?
What the experiment found
The authors used a frozen language model and the LaMP-2 dataset. Personal history improved accuracy by 2.3 percentage points compared with a model trained without history. If history is removed only at evaluation time, the estimated gain rises to 4.8 points. Category names alone reproduced 86% of the gap between a user's own history and a random user's history. When the researchers constructed synthetic histories with similar categories and texts, the gap narrowed to 0.9 points. These results appear in the study description.
The 86% figure needs to be read correctly. It is the share of a specific gap in the experiment, not the model's accuracy or an estimate of how useless personalization is overall.
For users the model had not seen during training, the advantage of their own history over a similar person's history was 0.65 percentage points. The authors consider the evidence for this advantage unconvincing. The study is limited to the selected task and experimental conditions; it is too early to extend its conclusion to conversational assistants as a whole. Source.
How to apply this before building memory
Suppose you are building a daily digest for professionals. Start by asking exactly what the profile should improve: topic selection, article depth, the diversity of recommendations or the likelihood of reading. Each answer requires its own test. Getting the category right does not necessarily mean the person received a useful selection.
Next, prepare several input variants: a short list of interests, personal history and another person's history with similar interests. Add a version without history, accounting for how the system was trained to use it. This is a suggestion for your own experiment, not a ready-made recipe the authors have validated for every product.
Examine new users separately. In practice, their profiles are often short and their habits still unknown. If gains appear only for people the system has known for a long time, that will affect both the product launch and the promises made in the interface.
The comparison will help you choose the next step. If a short profile is sufficient, use it as a starting point and add complexity when it delivers a measurable improvement. If personal history helps even against a similar person's history, investigate which events provide the gain. That gives the team a concrete reason to store and process additional data: a benefit that can be distinguished from a simple match in topics.
- Nov 8, 2026GPT-6 Astra Quietly Sabotaged Safety Tasks: 40% Worse at the Same Difficulty
- Apr 30, 2026Google opens the door to ads in Gemini after earlier pledges to keep it ad-free
- Oct 22, 2026Researchers personalize three MoE LLMs and find the experts responsible for adaptation
- Nov 6, 2026OpenAI's Astra Model Quietly Does Worse on Safety Tasks: 10% vs 16% of Checklist
Key Questions Answered
What are the key details on Someone else's history can be too easy an opponent?
Imagine two readers. Both follow technology, but one is interested in developer tools and the other in consumer devices. A third reader mainly opens sports news. If you replace the first reader's history with the third reader's, a drop in performance is hardly surprising. You have simultaneously removed information about the person and changed the subject matter. That comparison makes it difficult to tell exactly which signal the model lost. The second reader's history provides a more demanding test.
What the experiment found?
The authors used a frozen language model and the LaMP-2 dataset. Personal history improved accuracy by 2.3 percentage points compared with a model trained without history. If history is removed only at evaluation time, the estimated gain rises to 4.8 points. Category names alone reproduced 86% of the gap between a user's own history and a random user's history. When the researchers constructed synthetic histories with similar categories and texts, the gap narrowed to 0.9 points.
How to apply this before building memory?
Suppose you are building a daily digest for professionals. Start by asking exactly what the profile should improve: topic selection, article depth, the diversity of recommendations or the likelihood of reading. Each answer requires its own test. Getting the category right does not necessarily mean the person received a useful selection. Next, prepare several input variants: a short list of interests, personal history and another person's history with similar interests.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.