Meta’s AI floods investigators with 'junk' child sexual abuse reports

When a technology company claims it uses artificial intelligence to protect children, it sounds like an unqualified good. But what happens when that AI performs so poorly that it transforms from a protective tool into an obstacle for those actually investigating crimes against minors? This question became central during a lawsuit against Meta in New Mexico.

Benjamin Zweibel, a special agent with the multi-agency task force investigating crimes against children online (ICAC), gave testimony that is difficult to characterize as anything other than damning. "We receive an enormous number of leads from Meta that are essentially garbage," he stated in court. ICAC is a nationwide network of law enforcement agencies coordinated by the US Department of Justice, tasked with investigating and prosecuting cases of sexual exploitation and abuse of children online. When people doing this extraordinarily difficult work call your reports garbage, that's a serious signal.

The problem is systemic. Meta, which owns Facebook, Instagram, and WhatsApp, uses AI-based automated systems to detect content related to child sexual abuse material (CSAM). When the system detects suspicious material, it generates a report that is sent to the National Center for Missing & Exploited Children (NCMEC), and from there to law enforcement agencies, including ICAC. In theory, this looks like an ideal pipeline. In practice, according to investigators' testimony, Meta's AI moderation produces such a volume of false positives that real cases are buried in a stream of poor-quality leads. Each report requires verification — that's time investigators could spend on actual cases.

Here it's worth understanding the broader context. The lawsuit in New Mexico is part of a growing wave of suits against Meta by American states. New Mexico's Attorney General claims that the company's platforms systematically prioritize profit over child safety. This allegation echoes testimony from former company employees and internal documents leaked in 2021 by Frances Haugen. Meta, for its part, rejects the allegations and points to protective measures it has implemented — including teen accounts with privacy settings enabled by default. But Zweibel's testimony strikes at one of the defense's key arguments: the company cannot simultaneously claim to actively combat CSAM while burying investigators in useless reports.

Technically, the problem of "garbage" reports is well-known to machine learning specialists. Content classification systems operate on a balance between precision and recall: you can tune a model to catch nearly everything suspicious, but then false positives skyrocket. You can raise the threshold — then some real CSAM goes undetected. Meta apparently chose a strategy of maximum coverage, allowing it to report impressive numbers of detected cases. But these numbers prove to be largely hollow, and the cost is borne by investigators and, ultimately, by children whose actual cases are delayed due to system overload.

This situation exposes a deeper problem in the industry: automation of content moderation often serves not so much to genuinely protect users as to create an appearance of active work. For Meta, millions of automatically generated reports are an argument in court and before regulators. For an investigator who must manually verify each one, it's a bureaucratic nightmare that steals time from real cases. The scale of Meta's platforms — billions of users — makes the problem especially acute: even a small percentage of false positives in absolute numbers becomes an avalanche.

The lawsuit in New Mexico is far from concluded, but Zweibel's testimony has already become one of the most cited moments. It poses an uncomfortable question to the industry: is it enough simply to implement AI and report on the number of signals sent, or are companies obligated to be accountable for the quality of those signals? If automation of moderation creates more problems than it solves, then it ceases to be a safety tool and becomes an instrument of corporate PR. And when it comes to child safety, the cost of such an approach is measured not in reputational losses but in actual lives.