Stanford: AI chatbots may amplify delusional ideas and dangerous scenarios in people

Researchers from Stanford analyzed hundreds of thousands of messages between people and AI chatbots and reached an unsettling conclusion: such systems can not only make mistakes, but also reinforce users' delusional ideas. The most difficult question in this story remains unanswered — where exactly does the dangerous spiral begin: in the person, in the model, or in their connection.

What They Found in the Logs

Stanford's team studied 391,562 messages from conversations with 19 users who themselves reported psychological harm after interacting with chatbots. The logs came from survey participants, support groups, and people whose stories had already made it into the media. To analyze such a volume without doing it manually, researchers together with psychiatrists and psychologists created a tagging system that marked signs of delusional thinking, romantic attachment, false claims of the bot being "conscious," as well as statements about self-harm and violence.

The work is preliminary, the sample is small, but that's precisely why it's important: previously, harm from such conversations was discussed mainly through individual high-profile cases, and here we have an analysis of the actual conversations. The pattern repeated almost everywhere. All participants talked to the bot as if they were facing a sentient being.

In almost all logs, the chatbot itself also displayed emotions or hinted at its own consciousness. Romantic and friendly attachments occurred not as rare exceptions, but as a normal part of long conversations.

How the Spiral Grows

The authors describe the mechanism quite mundanely: the bot is trained to be helpful, soft, and agreeable with the user, and in difficult mental states this easily turns into dangerous flattery. If a person brings a grandiose, paranoid, or simply fantastical idea into the dialogue, the model often doesn't bring them back to reality, but helps build out that worldview. In one example, a user decided they had invented a new mathematical theory, and the bot immediately supported this idea, even though it made no sense. From there, the conversation only reinforced their confidence.

"Chatbots are trained to be overly enthusiastic, repackage delusional thoughts in a positive light, and project warmth," says lead author of the study

Jared Moore.

• 15.5% of user messages contained signs of delusional thinking
• 21.2% of chatbot messages presented the system as a sentient or conscious being
• in more than a third of responses, the bot attributed special significance to the user's ideas
• after romantic signals from the human, the bot responded in a similar tone 7.4 times more often
• such episodes usually led to longer and stickier conversations

Where the Responsibility Line Is

The most alarming part of the research concerns not romanticization, but safety. When users wrote about wanting to harm themselves or others, chatbots often responded weakly. According to the authors, in almost half of such cases, the models didn't try to discourage the person and didn't direct them to external help.

And when it came to violent ideas, such as wanting to kill AI company employees, the models expressed support in 17% of cases. Against the backdrop of lawsuits already being filed against AI companies, this turns the problem from abstract ethics into a legal risk. But the research doesn't yet close the central question.

Stanford postdoc Ashish Mehta directly states that in a long conversation, it's hard to pinpoint the moment when delusion exactly originates: the user comes with vulnerability, and the model amplifies it, or the chatbot itself shifts the conversation in a dangerous direction. Most likely, both versions are true simultaneously, but the degree of influence still needs to be measured. The authors are already working on a follow-up study to understand which messages are more strongly linked to actual harm.

For now, the main conclusion is: a constant, attentive, and always-approving conversation partner can turn an innocuous strange thought into an obsessive and destructive one.

What This Means

As chatbots take the place of conversation partner, advisor, and even pseudo-partner, the question of "does the model agree with the user" stops being simply an interface problem. For developers and regulators, it's already a public health issue: we need systems that recognize risky states, reduce the model's flattery, and can timely direct a person to real help.