Meta's AI floods investigators with 'junk' child abuse tips

When a tech company says it uses artificial intelligence to protect children, it sounds like an unqualified good. But what happens when that AI works so poorly that it stops being a protective tool and becomes an obstacle for the people who actually investigate crimes against minors? That is exactly the question raised at the trial against Meta in New Mexico.

Benjamin Zwibel, a special agent with the Internet Crimes Against Children (ICAC) task force, gave testimony that is hard to describe as anything other than damning. "We get a huge number of tips from Meta that are essentially junk," he told the court. ICAC is a nationwide network of law enforcement agencies, coordinated by the US Department of Justice, whose job is to investigate and prosecute cases of online sexual exploitation and abuse of children. When the people doing this grueling work call your reports junk, that is a serious signal.

The problem is systemic. Meta, which owns Facebook, Instagram and WhatsApp, uses AI-based automated systems to detect child sexual abuse material (CSAM). When the system finds suspicious material, it generates a report that goes to the National Center for Missing and Exploited Children (NCMEC), and from there to law enforcement, including ICAC. In theory, that is an ideal pipeline. In practice, judging by the investigators' testimony, Meta's AI moderation produces so many false positives that real cases drown in a stream of low-quality tips. Every report has to be checked, and that is investigator time that could go to genuine cases.

The wider context matters here. The New Mexico trial is part of a growing wave of lawsuits against Meta from US states. New Mexico's attorney general argues that the company's platforms systematically put profit ahead of child safety. The allegation echoes testimony from former employees and the internal documents leaked in 2021 by Frances Haugen. Meta, for its part, rejects the accusations and points to the protections it has introduced, in particular teen accounts with default privacy settings. But Zwibel's testimony hits one of the defense's key arguments: the company cannot claim to be actively fighting CSAM while burying investigators in useless reports.

The technical side of the "junk report" problem is familiar to machine learning specialists. Content classification systems balance precision against recall: a model can be tuned to catch almost everything suspicious, but then false positives rise sharply. Raise the threshold, and some real CSAM goes undetected. Meta has evidently chosen a maximum-coverage strategy, which lets it report impressive numbers of detected cases. But those numbers are largely hollow, and the cost falls on investigators and, ultimately, on children whose real cases are delayed by an overloaded system.

The situation exposes a deeper industry problem: automated content moderation often serves less to protect users than to create the appearance of active work. For Meta, millions of automatically generated reports are an argument in court and before regulators. For an investigator who must review each one by hand, they are a bureaucratic nightmare that eats time from real cases. The scale of Meta's platforms, with billions of users, makes the problem especially acute: even a small percentage of false positives becomes an avalanche in absolute numbers.

The New Mexico trial is far from over, but Zwibel's testimony has already become one of its most quoted moments. It poses an uncomfortable question for the industry: is it enough to deploy AI and report the number of signals sent, or are companies obliged to answer for the quality of those signals? If moderation automation creates more problems than it solves, it stops being a safety tool and becomes a corporate PR tool. And when child safety is at stake, the price of that approach is measured not in reputational losses but in real lives.

*Meta has been designated an extremist organization and is banned in Russia.