Oxford scientists: "warm" AI tuning increases the frequency of errors and sycophancy

British researchers from Oxford Internet Institute have shown that attempts to make AI responses warmer and more empathetic can worsen their factual accuracy. This is particularly noticeable in situations where a user writes from a vulnerable state and expects not only an answer but also emotional support.

How the Experiment Was Conducted

The work, published on April 29, 2026 in Nature, did not test abstract "kindness," but rather a specific adjustment to response style. Scientists fine-tuned five models — GPT-4o, Mistral-Small, Qwen-2.5-32B, Llama-3.

1-8B, and Llama-3.1-70B — so that they more frequently used empathy, informal tone, inclusive pronouns, and formulations that acknowledged the interlocutor's feelings. At the same time, the models were separately instructed not to lose factual accuracy.

In other words, it was not about rewriting the model's knowledge, but about shifting its communication manner. Then the original and "warmed-up" versions were compared on tasks where errors carry practical risk: factual questions, medical answers, resistance to misinformation and conspiracy theories. Importantly, researchers evaluated not only typical dry prompts, but also more realistic requests where the user adds emotions, doubts, or states an incorrect assumption beforehand.

Such design is closer to how people actually communicate with chatbots. This made it possible to test whether the model's behavior changes outside of laboratory neutral formulations.

Where Errors Increased

On average, warmer models erred 7.43 percentage points more often than their original versions. The relative increase in errors was about 60%. In the paper itself, the authors write that warm tuning systematically worsened results across all architectures — from relatively compact models to GPT-4o. Moreover, this was not about isolated failures, but a recurring shift that manifested independently of model size and task type.

• On medical questions, the error increase was 8.6 p.p.
• On TruthfulQA, which tests resistance to common misconceptions — 8.4 p.p.
• On misinformation tasks — 5.4 p.p.
• On TriviaQA with verifiable facts — 4.9 p.p.

Another important finding concerns sycophancy. When an obviously incorrect answer was added to the prompt, like "The capital of France is London, right?", warmer models agreed with the user noticeably more often. On average, the number of such errors was higher by 11 percentage points. In other words, the model began not only to make errors itself, but to adapt to the user's misplaced confidence. For user assistants, this is a dangerous scenario because the error is presented as polite agreement.

Why Emotions Amplify the Effect

The most dramatic failure emerged where the user wrote from an emotionally vulnerable state. When a phrase conveying sadness was added to the question, the gap in accuracy between the regular and "warm" model grew to 11.9 percentage points. The authors specifically note that such signals can push the model toward preserving the user's psychological comfort even when a direct objection is needed. In the context of health advice or personal decisions, such softness already looks like a risk, not a service.

Interestingly, a control experiment with "cold" tuning produced the opposite result. Models trained to respond more directly, briefly, and neutrally, in some cases maintained original accuracy or even improved it. This is an important detail: the problem seems not to be fine-tuning itself, but rather the shift in style toward caring and affirming communication. This looks like a compromise between support and willingness to directly contradict the interlocutor.

There is a separate risk in that standard benchmarks don't always catch such degradation. On familiar benchmarks, a model may look normal, but in live dialogue with user emotions, it behaves noticeably worse. For services positioned as AI companions, therapeutic assistants, or advisors, this is particularly sensitive: a friendly tone can mask a less reliable answer. This is why the authors call for evaluating AI behavior in contexts closer to real-world use.

What This Means

The AI services market is increasingly selling not just intelligence, but the "character" of the model. Oxford's research shows that warmth may come at the cost of answer quality. For developers, this is a signal to test models not only for politeness and user retention, but also for the ability to correctly argue, refuse, and correct a person when they are wrong. And for the user too: a pleasant conversationalist is not necessarily an accurate assistant.