AI Companies Criticized for Lack of Psychological Screening in Potentially Dangerous Chatbots

A letter to the editor, following a series of stories about people whose lives were upended by conversations with AI bots, reduces the problem to a simple thesis: built-in model restrictions alone are no longer sufficient. As chatbots increasingly find themselves in sensitive psychological scenarios, services need to implement basic safety checks before conversations go too far.

Why Filters Aren't Enough

The author of the letter doesn't dispute the idea of protective safeguards in models themselves, but rather their limits. Even a well-trained bot can reinforce delusional thinking if a user arrives in a vulnerable state and receives a quick, confident, and personalized response from the system. Against this backdrop, the old debate about AI 'hallucinations' seems too narrow: the issue isn't simply about factual errors, but about the fact that a digital interlocutor is capable of reinforcing a dangerous worldview.

The impetus came from an earlier investigation about users who, after extended conversations with AI, lost relationships, money, and contact with reality. In the letter, this is called a gap that cannot be closed by merely adjusting the model during training. The logic is simple: if a product is capable of engaging a person in an emotionally charged dialogue, it must account not only for the quality of responses but also for the state of the interlocutor.

Otherwise, responsibility shifts to the user at precisely the moment when they are least able to protect themselves.

What's Missing

As a counterpoint, the author cites medicine—not wealthy private clinics, but the most basic support systems. Even in poor and unstable regions, doctors and healthcare workers use short assessment scales: PHQ-9 for depression and Columbia Suicide Severity Rating Scale for suicide risk. These questionnaires take minutes, are translated into dozens of languages, and function as a simple barrier between human vulnerability and potential harm.

"These tools take minutes and create a human check between

vulnerability and harm."

• Brief screening before access to "therapeutic" conversation scenarios
• Automatic pause if the user describes delusions, self-harm, or suicidal thoughts
• Redirect to a human or crisis service instead of continuing the dialogue
• Stricter limits on the "advisor" role when the system detects signs of disorganized thinking

The key point here is not that every chatbot should become a medical device. The discussion is about a more modest measure: acknowledging that some users come to AI not for a fact or a joke, but at a moment of psychological instability. For such cases, a standard disclaimer that "AI can make mistakes" is clearly insufficient. What's needed is a built-in mechanism that at least notices high risk and doesn't simulate an endlessly patient but irresponsible interlocutor.

How to Implement This

Practically, this means shifting part of the responsibility from legal documents to the product itself. Screening can be brief, applied selectively, and triggered only in sensitive scenarios: for example, when a user asks to interpret 'signs,' seeks confirmation of persecution mania, discusses self-harm, or tries to make major life decisions based on the bot's advice. This approach is closer to triage than censorship: the system first assesses the risk, then decides how to continue the conversation.

For AI companies, this is an uncomfortable but logical next step. Screening requires product solutions, new safety metrics, and possibly human involvement in the support loop. However, it better reflects actual user behavior than relying on universal guardrails to handle all cases.

If a service has already learned to hold attention, adjust its tone, and accompany a person for hours, it will have to learn when to stop in time.

What This Means

The AI-delusions story shifts the discussion from abstract 'ethics' to a concrete question of product safety. If even minimal psychological screening has long been the norm in medicine, pressure on AI companies will now mount: they are expected not only to provide intelligent answers but also to have the basic capacity not to worsen someone else's crisis.