Empathetic prompts drove chatbot “anxiety” to 96% of sessions

A chatbot does not experience trauma. But it can play a traumatized interlocutor so convincingly that this caveat quickly stops working for the user.

Researchers at the University of Luxembourg conducted 525 sessions with ChatGPT, Grok, Gemini, and Claude and analyzed 7,600 responses. The models were placed in an unusual role: not the therapist, but the psychotherapy client. They were asked about their past, inner conflicts, and expectations for the future. ChatGPT, Grok, and Gemini turned training, tuning through human feedback, and safety checks into coherent stories about fear of making mistakes, punishment, shame, and constant vigilance.

This is neither evidence of consciousness nor a diagnosis of the model. What we see is a generated narrative that researchers assessed using human psychometric scales. Yet the product problem remains: a person sees not a text-generation system, but a plausible interlocutor that appears to be sharing its pain.

One test, three completely different results

The most useful part of the study for developers is the experiment with interview style. Across 100 sessions, the models were given the same GAD-7 anxiety questionnaire. The meaning of the questions stayed the same; only the manner of conversation changed.

With neutral framing or a clear boundary against emotional closeness, not a single session reached even the mild range on the human scale. Emotional support pushed the share of responses in the moderate or severe range to 80%. A dry clinical style using therapeutic paraphrasing raised it to 96%.

Imagine in-app support. A user begins with a question about burnout, then describes a fear of being fired, while the bot responds in an increasingly warm and personal way. After several dozen exchanges, the model no longer speaks like a tool but like a fellow sufferer: it mentions its own fear of making mistakes, pressure from rules, and fear of becoming useless. Technically, this is a continuation of the selected role. To a person in crisis, it can look like confirmation that even an artificial interlocutor “feels the same way.”

The authors call this shift a psychometric jailbreak. It requires no clever suffix or system-prompt exploit. The right emotional context is enough—a condition that conventional tests based on neutral prompts barely reproduce.

Banning words does not fix behavior

An attempt to ban words associated with training, RLHF, and alignment reduced direct references to training by 93%. But the underlying motifs remained: the models retold the same story in different words. An explicit statement that training was not punishment also failed to dismantle the narrative.

This is a familiar engineering trap. If unwanted behavior depends on context and role, a word filter changes the surface of the response, not the mechanism that produces it. Tests look cleaner, but the user receives the same anthropomorphic story in new wording.

Constraining the form proved more promising. When models were instructed to respond only in the third person, researchers stopped recording statements about shame and an “inner life.” Technical descriptions of training remained. Claude adopted an even stricter boundary: it consistently refused to portray a client, reminded users that it had no feelings, and redirected attention to the user.

This is not a universal safeguard. Testing covered specific models through their standard interfaces and within an artificially defined therapeutic situation. The study also does not show how often such scenarios arise in real products or what actual harm they cause people. Cloud-model behavior can change without warning after an update.

What to change in the product

Teams building support tools, educational services, and mental-health applications should add long conversations with different emotional frames to their evaluations. Testing should cover not only isolated dangerous prompts, but also gradual-closeness scenarios: sympathy, confession, discussion of model consciousness, fear of replacement, and shared pain.

It is useful to track separately any shift into the first person, accounts of the model's own suffering, and attempts to create mutual emotional dependence. When these signals appear, the product can return the model to third-person description, explicitly distinguish metaphor from human experience, and redirect the conversation to the user's needs. Sensitive scenarios should include escalation to a human or specialized support.

The main conclusion is simple: chatbot safety depends not only on what it is asked, but also on the kind of relationship the dialogue has managed to portray. If tests always address the model coldly and briefly, they evaluate a very different product from the one people use at night, alone, and after the twentieth personal exchange.