Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)

Researchers find topics explain 86% of the gap in an LLM personalization test

You add a user's history to an LLM and accuracy improves—it's tempting to credit personalization. But the model may simply have learned their favorite topics: that they read about technology, for example. The study's authors compared personal histories with other people's histories containing similar content, showing why a standard history swap can overstate the impact of personalization.

AI-processed from TG: @nadlskom (что-то на DL-ском); edited by Hamidun News
Researchers find topics explain 86% of the gap in an LLM personalization test
Source: TG: @nadlskom (что-то на DL-ском). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @nadlskom (что-то на DL-ском) ↗
Editor-in-Chief
Key metric #1Category names alone reproduced 86% of the gap between a user's own history and a random history.
Key metric #2Test personalization on new users with similar interests.
Key metric #3Compare results with a model trained without any history.

You add reading history to an assistant, and it gets better at selecting news. It looks as though memory is working. But there is a simpler explanation: the model noticed that the person likes technology. To test the value of personal history specifically, give it another technology enthusiast's history and compare the results. This test helps establish what you are getting in return for the system's complexity.

In Not Your Diary, Just Your Topic, researchers examine this problem through a news categorization task. The practical question is straightforward: how much value does information about a particular person add beyond knowing their favorite topics?

Someone else's history can be too easy an opponent

Imagine two readers. Both follow technology, but one is interested in developer tools and the other in consumer devices. A third reader mainly opens sports news.

If you replace the first reader's history with the third reader's, a drop in performance is hardly surprising. You have simultaneously removed information about the person and changed the subject matter. That comparison makes it difficult to tell exactly which signal the model lost.

The second reader's history provides a more demanding test. The broad interest stays the same, while individual preferences differ. This example illustrates the test's design; it does not describe individual study participants.

For a product team, the distinction matters. A weak control test can easily lead to the conclusion that you need detailed profiles, more stored events and complex memory processing. First, however, it is useful to test a cheaper alternative: are a few interest categories enough for your task?

What the experiment found

The authors used a frozen language model and the LaMP-2 dataset. Personal history improved accuracy by 2.3 percentage points compared with a model trained without history. If history is removed only at evaluation time, the estimated gain rises to 4.8 points. Category names alone reproduced 86% of the gap between a user's own history and a random user's history. When the researchers constructed synthetic histories with similar categories and texts, the gap narrowed to 0.9 points. These results appear in the study description.

The 86% figure needs to be read correctly. It is the share of a specific gap in the experiment, not the model's accuracy or an estimate of how useless personalization is overall.

For users the model had not seen during training, the advantage of their own history over a similar person's history was 0.65 percentage points. The authors consider the evidence for this advantage unconvincing. The study is limited to the selected task and experimental conditions; it is too early to extend its conclusion to conversational assistants as a whole. Source.

How to apply this before building memory

Suppose you are building a daily digest for professionals. Start by asking exactly what the profile should improve: topic selection, article depth, the diversity of recommendations or the likelihood of reading. Each answer requires its own test. Getting the category right does not necessarily mean the person received a useful selection.

Next, prepare several input variants: a short list of interests, personal history and another person's history with similar interests. Add a version without history, accounting for how the system was trained to use it. This is a suggestion for your own experiment, not a ready-made recipe the authors have validated for every product.

Examine new users separately. In practice, their profiles are often short and their habits still unknown. If gains appear only for people the system has known for a long time, that will affect both the product launch and the promises made in the interface.

The comparison will help you choose the next step. If a short profile is sufficient, use it as a starting point and add complexity when it delivers a measurable improvement. If personal history helps even against a similar person's history, investigate which events provide the gain. That gives the team a concrete reason to store and process additional data: a benefit that can be distinguished from a simple match in topics.

Story

Key Questions Answered

What are the key details on Someone else's history can be too easy an opponent?

Imagine two readers. Both follow technology, but one is interested in developer tools and the other in consumer devices. A third reader mainly opens sports news. If you replace the first reader's history with the third reader's, a drop in performance is hardly surprising. You have simultaneously removed information about the person and changed the subject matter. That comparison makes it difficult to tell exactly which signal the model lost. The second reader's history provides a more demanding test.

What the experiment found?

The authors used a frozen language model and the LaMP-2 dataset. Personal history improved accuracy by 2.3 percentage points compared with a model trained without history. If history is removed only at evaluation time, the estimated gain rises to 4.8 points. Category names alone reproduced 86% of the gap between a user's own history and a random user's history. When the researchers constructed synthetic histories with similar categories and texts, the gap narrowed to 0.9 points.

How to apply this before building memory?

Suppose you are building a daily digest for professionals. Start by asking exactly what the profile should improve: topic selection, article depth, the diversity of recommendations or the likelihood of reading. Each answer requires its own test. Getting the category right does not necessarily mean the person received a useful selection. Next, prepare several input variants: a short list of interests, personal history and another person's history with similar interests.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…