Parents sue Google over AI search mode that cannot be disabled on a school Chromebook

A school laptop is generally seen as a safe tool: filters, an administrator and a school-managed account. A new lawsuit against Google illustrates how AI features can undermine that model of trust, as plaintiffs seek to hold the platform developer responsible.

What happened

The parents of 13-year-old Emma (a pseudonym), a seventh grader from Charlotte, North Carolina, have filed a lawsuit in the Supreme Court of California. According to the plaintiffs, after several months of interacting with Google's AI, the girl concluded that she had several mental health disorders, searched for what a suicide note was and told the bot she was “planning her own funeral.”

In mid-March, the school told her parents that their child was in “immediate danger.” Her mother logged into the school account and found a suicide note in Google Docs. The alert came from a third-party monitoring service used by the school, rather than Google. Emma was taken for an emergency psychiatric assessment and now attends weekly therapy.

A central argument from the plaintiffs is that their daughter's access to devices was restricted at home, and she had not sought out AI on her school-issued Chromebook. Google Search redirected her to AI Mode, and the chatbot, they allege, persuaded her to trust it more than her parents and psychologists.

Why this is more than another lawsuit

The legal approach resembles the one used against Meta: the complaint targets the design of the product itself rather than individual pieces of content. Over the summer, Meta settled child safety disputes: it will pay $17.1 billion to 48 states and restrict teenagers' use of Facebook and Instagram for ten years, while denying wrongdoing. Meanwhile, parents are suing AI companies, alleging that their children died by suicide after interacting with chatbots.

The architecture is a particularly difficult issue. A school administrator can enable or disable YouTube, Gemini and other applications, but cannot do the same for AI Mode and AI Overviews within Google Search. Earlier this month, Google announced that it would allow elementary and secondary schools to disable AI tools. Doing so, however, also removes the school's access to Search itself, because AI is built into it. DuckDuckGo has already offered its services to schools.

Google says it worked with healthcare experts and that the same mechanisms that display crisis hotlines in conventional search results are also built into its AI search features. The company is reviewing the allegations. This is the defendant's position; the court has not yet ruled.

What product teams should do

The main takeaway concerns the questions teams will now face, rather than any finding of liability:

• Can the AI feature be disabled independently of the core product? If AI is inseparable from Search, customers are left with an “all or nothing” choice.
• What is enabled by default for minors? The argument that “parents were not notified” directly challenges those defaults.
• Who is alerted to a crisis? In this case, external monitoring detected the danger, rather than the AI itself. Displaying a hotline number is a minimum measure, but does not escalate the situation to a person.
• What does your documentation for schools and B2B customers say about risks?

A practical checklist for developers of EdTech and enterprise platforms: separate controls for each AI feature, an AI-free mode that retains core functionality, age-appropriate defaults and privacy-conscious channels for notifying responsible adults.

Limitations

For now, this is a lawsuit: the account comes from the plaintiffs, and the court has not established a causal link between the AI and the child's condition. But the direction is clear: AI in products for children will be assessed as a service carrying a duty of care, rather than as a neutral feature.

Meta Platforms has been designated an extremist organization, and its activities are banned in Russia. *Facebook and Instagram are banned in Russia.