Confessing to an algorithm: why ChatGPT is the worst keeper of your medical secrets

More than 230 million people turn to ChatGPT every week for health and wellness advice, according to OpenAI figures reported by The Verge. The company says the chatbot has become an "ally" for many users: it helps them navigate the maze of insurance plans, fill out paperwork, and advocate for themselves more confidently within the healthcare system. But tech companies aren't bound by the same obligations as medical institutions — and that gap is where the biggest risk lies.

Why users trust a chatbot with their health

Modern medicine often feels like a convoluted bureaucratic maze where patients feel like a superfluous cog. Against that backdrop, ChatGPT — polite, instant, and always available — looks like a lifeline: it doesn't rush you, doesn't get irritated by repeat questions, and will discuss symptoms at any hour of the day. OpenAI frames this as the role of an "ally" that helps people handle correspondence with insurers, sort out paperwork, and generally stand up for themselves as patients more confidently. The problem is that this responsiveness is a high-quality statistical imitation, not genuine care: the chatbot never took the Hippocratic oath.

A legal vacuum: a chatbot is not a medical institution

Doctors and clinics are bound by strict confidentiality rules such as HIPAA in the US or GDPR in Europe. OpenAI and other tech companies operate under terms of service that most people scroll past in a second. When a user uploads an MRI scan or a list of their medications to a chat, that data stops being a purely personal secret — it becomes part of the corpus used to train future versions of the models. Companies promise anonymization, but the industry's track record of leaks shows that online "anonymity" is fragile and temporary. Tech companies are not bound by the same obligations as medical institutions.

The

Google Nightingale precedent and the probabilistic nature of AI

This has happened before: in the Google Nightingale project, data on millions of patients was handed to a tech company without the patients' knowledge. The difference now is that users hand over their medical information voluntarily, lured by the convenience of a chatbot interface. And a language model is not a knowledge base; it generates the most probable sequences of words. In tasks like writing code or essays, a wrong but "plausible" answer can look amusing. In a medical context, a hallucinated drug dosage or a misread heart rhythm is a far more serious risk.

Anthropomorphism and the absence of accountability

People subconsciously read a polite, well-structured answer as a sign of competence. That's anthropomorphism at work: users tend to trust the bot more than they should. But behind that facade there's none of the accountability medicine demands. A doctor who makes a mistake risks their license and a lawsuit. If a chatbot gives bad advice, the company can point to a clause in its terms of service stating the service is intended "for informational purposes only."

The bottom line: tech companies are not medical institutions, and the convenience of chatting with a bot does not mean your personal medical data is protected. Using AI to look up general information or organize notes is one thing; turning a chatbot into the keeper of your medical history and prescriptions is something else entirely.

How many people turn to ChatGPT for health advice each week?

More than 230 million people a week, according to OpenAI.

How do

ChatGPT's obligations to users differ from a doctor's?

Tech companies aren't bound by the same obligations as medical institutions: confidentiality rules like HIPAA and GDPR don't apply to them the way they apply to doctors and clinics, and their terms of service state that the service is intended "for informational purposes only."

What happens to the medical data a user uploads to the chat?

It stops being a personal secret and becomes part of the dataset used to train future versions of the model. Companies promise anonymization, but the industry's track record of leaks shows that this concept is fragile and temporary.