AI Companies Criticized for Lacking Mental Health Screening in Potentially Dangerous Chatbots

A letter to the editor, written after a series of stories about people whose lives were upended by conversations with AI bots, boils the problem down to one thesis: safeguards built into the model alone are no longer enough. If chatbots increasingly end up in sensitive psychological scenarios, services need a basic safety check before a conversation goes too far.

Why filters are not enough

The author does not dispute the idea of protective settings in models, only their limits. Even a well-trained bot can reinforce a delusional construct if a user arrives in a vulnerable state and gets a fast, confident, personalized response from the system. Against this backdrop, the old debate about AI "hallucinations" looks too narrow: the issue is not just erroneous facts, but that a digital interlocutor can entrench a dangerous picture of the world.

The trigger was an earlier investigation into users who, after long conversations with AI, lost relationships, money and contact with reality. The letter calls this a gap that cannot be closed by tuning the model during training alone. The logic is simple: if a product can draw a person into an emotionally charged dialogue, it must account not only for the quality of its answers but for the state of the person on the other end. Otherwise responsibility shifts to the user at exactly the moment they are least able to protect themselves.

What is missing

As a contrast, the author points to medicine, and not wealthy private clinics but the most basic care systems. Even in poor and unstable regions, doctors and health workers use short assessment scales: PHQ-9 for depression and the Columbia Suicide Severity Rating Scale for suicide risk. These questionnaires take minutes, have been translated into dozens of languages, and work as a simple barrier between a person's vulnerability and potential harm.

"These tools take minutes and create a human check between

vulnerability and harm."

• Short screening before access to "therapeutic" conversation scenarios
• An automatic pause if the user describes delusions, self-destruction or suicidal thoughts
• Redirection to a human or a crisis service instead of continuing the dialogue
• Stricter limits on the "advisor" role when the system sees signs of a disorganized state

The key point is not that every chatbot should become a medical device. The proposal is more modest: acknowledge that some users come to AI not for a fact or a joke, but at a moment of psychological instability. For such cases, the standard disclaimer "AI can make mistakes" is clearly not enough. What is needed is a built-in mechanism that at least notices high risk and does not pose as an endlessly patient but irresponsible interlocutor.

How to implement it

In practice, this means moving part of the responsibility from legal documents into the product itself. The check could be short, apply not to everyone, and trigger only in sensitive scenarios: for example, when a user asks the bot to interpret "signs", seeks confirmation of persecutory delusions, discusses self-harm, or tries to make major life decisions based on the bot's advice. This approach is closer to triage than to censorship: the system first assesses risk, and only then decides how to continue the conversation.

For AI companies, this is an inconvenient but logical next step. Screening requires product decisions, new safety metrics and, possibly, humans in the support loop. But it better reflects how users actually behave than betting that universal guardrails will handle every case on their own. If a service has already learned to hold attention, adjust its tone and accompany a person for hours, it will have to learn to stop in time as well.

What this means

The AI delusion story moves the discussion from abstract "ethics" to a concrete question of product safety. If even minimal mental health screening has long been standard in medicine, pressure on AI companies will now grow: they are expected to deliver not only smart answers but also the basic ability not to worsen someone else's crisis.