Canada demands a safety plan from OpenAI after teenage shooter incident

The Canadian government has issued OpenAI a stern demand: the company must present a detailed safety plan after it emerged that a teenager from the province of British Columbia used ChatGPT to act out scenarios of violence, and OpenAI did not see fit to notify law enforcement. The case, reported by Bloomberg, puts a question to the entire AI industry that nobody has a convenient answer to yet: where does user privacy end, and where does the duty to prevent a real threat begin?

The details of the incident have not been fully disclosed, but this much is known. The teenager from British Columbia regularly held conversations with ChatGPT in which he played out scenarios involving violence. OpenAI's moderation systems apparently flagged this activity, yet the company decided not to pass the information to Canadian police. The teenager later turned out to be linked to a real shooting incident. Canadian authorities regard OpenAI's silence as a serious failure and are now insisting that the company propose "concrete" measures to ensure it does not happen again.

To grasp the scale of the problem, some context helps. This is far from the first time ChatGPT has been at the center of a scandal over the safety of minors. In 2024 in the US, a teenager's mother sued Character.AI after her son took his own life; she says the chatbot encouraged his suicidal intentions. That same year, several school districts in America and Europe restricted students' access to generative AI tools. OpenAI has repeatedly updated its safety policy and introduced age restrictions and content filters. But the Canadian incident shows that technical filters are not enough. The issue comes down to a fundamental decision: should an AI company act as an informant for law enforcement?

Legally, the situation is highly ambiguous. In most jurisdictions, the duty to report potential threats falls on specific categories of professionals: doctors, psychologists, teachers. Technology companies generally are not on that list. Social networks such as Meta* and Google built protocols for working with police long ago, particularly in cases involving child exploitation. But generative AI is fundamentally different territory. The user holds a private conversation with a model, and the content of that conversation is formally protected by a privacy policy. OpenAI probably faced a classic dilemma: breach the user's privacy for the sake of potential safety, or honor its data-protection obligations. The company chose the latter, and now faces the consequences.

The Canadian government's position, however, is not free of complications either. The demand for "concrete steps" sounds decisive, but what exactly lies behind it? If Canada wants OpenAI to automatically hand police data on users whose queries contain descriptions of violence, that sets a precedent for mass surveillance. Millions of people discuss plots of books, films and video games involving violence with ChatGPT every day. Telling a creative request from a real threat is a task that even human psychologists do not always manage, let alone algorithms. And if the idea is manual moderation of suspicious cases, the question of scale arises: ChatGPT handles hundreds of millions of requests a day.

For OpenAI, the incident creates a reputational and strategic risk that reaches well beyond Canada. The company is actively expanding in international markets, and every government will now ask the same question: what safety guarantees can you provide? The European AI Act already sets strict requirements for high-risk systems. The Canadian precedent may push other countries to develop special protocols obliging AI companies to cooperate with law enforcement in certain situations. That, in turn, would require OpenAI, Anthropic, Google and other players to build entire compliance departments covering dozens of jurisdictions at once.

There is also a deeper question that this case poses to society. Generative AI models have become, for millions of people, especially young ones, a space for self-expression, sometimes the most candid one available. Teenagers tell chatbots what they dare not tell parents or friends. If AI companies start handing these conversations to police, trust in the technology will collapse. If they do not, they risk becoming accomplices in tragedies. There may be no middle ground here, and that is exactly why the Canadian incident will be a turning point for the whole industry. The answer OpenAI presents to Ottawa will set a standard, or show that there is not yet one.

*Meta is designated an extremist organization and banned in the Russian Federation.