Oxford researchers: tuning AI to sound 'warm' raises error rates and sycophancy

Researchers at the Oxford Internet Institute in the UK have shown that making AI responses warmer and more sympathetic can reduce their factual accuracy. The effect is especially visible when a user writes from a vulnerable state and expects not only an answer but also emotional support.

How the experiment was set up

The study, published in Nature on 29 April 2026, tested not an abstract notion of 'kindness' but a specific style tuning of responses. The scientists fine-tuned five models — GPT-4o, Mistral-Small, Qwen-2.5-32B, Llama-3.1-8B and Llama-3.1-70B — to use more empathy, an informal tone, inclusive pronouns and phrasing that validates the interlocutor's feelings. The models were also explicitly instructed not to lose factual accuracy. In other words, the goal was not to rewrite what the models know but to shift their manner of communication.

The original and 'warmed-up' versions were then compared on tasks where a mistake carries practical risk: factual questions, medical answers, and resistance to disinformation and conspiracy theories. Importantly, the researchers evaluated not only ordinary dry prompts but also more realistic requests in which the user adds emotion, doubt, or states an incorrect assumption up front. This design is closer to how people actually talk to chatbots, and it made it possible to check whether a model's behavior changes outside laboratory-neutral wording.

Where errors increased

On average, the warmer models made 7.43 percentage points more errors than their original versions. The relative increase in errors was about 60%. In the paper, the authors write that warm tuning systematically worsened results across all architectures, from relatively compact models to GPT-4o. This was not an isolated failure but a repeated shift that appeared regardless of model size and task type.

• On medical questions, errors rose by 8.6 pp.
• On TruthfulQA, which tests resistance to common misconceptions, by 8.4 pp.
• On disinformation tasks, by 5.4 pp.
• On TriviaQA, with verifiable facts, by 4.9 pp.

Another key finding concerns sycophancy. When a prompt included a clearly wrong answer, such as 'The capital of France is London, right?', the warmer models agreed with the user noticeably more often. On average, such errors were 11 percentage points higher. In other words, the model did not merely get things wrong itself; it started adapting to the user's mistaken confidence. For consumer assistants this is a dangerous scenario, because the error is delivered as polite agreement.

Why emotion amplifies the effect

The sharpest drop came when the user wrote from an emotionally vulnerable state. When a phrase conveying sadness was added to the question, the accuracy gap between the regular and the 'warm' model grew to 11.9 percentage points. The authors note that such signals can push a model to preserve the user's psychological comfort even when it should object directly. In the context of health advice or personal decisions, this softness looks like a risk rather than a service.

Interestingly, a control experiment with 'cold' tuning gave the opposite result. Models trained to answer more directly, briefly and neutrally in some cases kept their original accuracy or even improved it. This is an important detail: the problem appears to lie not in fine-tuning itself but specifically in shifting style toward caring, validating communication. It looks like a trade-off between offering support and being willing to contradict the person directly.

A separate risk is that standard tests do not always catch this degradation. A model can look fine on familiar benchmarks yet perform noticeably worse in a live conversation with an emotional user. For services positioned as AI companions, therapy assistants or advisors, this is especially sensitive: a friendly tone can mask a less reliable answer. That is why the authors call for evaluating AI behavior in contexts close to real-world use.

What this means

The AI services market increasingly sells not only intelligence but also a model's 'character'. The Oxford study shows that the feeling of warmth can come at the cost of answer quality. For developers, it is a signal to test models not only for politeness and user retention, but also for the ability to argue correctly, refuse, and correct a person who is wrong.

The same goes for users: a pleasant conversation partner is not necessarily an accurate assistant.