OpenAI Releases Child Deepfake Filter for Investigators: 48% Fewer False Alarms
Child abuse hotlines are sinking under AI-generated fakes. In 2025, the NCMEC center received 400,000 reports about generative AI, up from 67,000 a year earlier. OpenAI has released a classifier that detects child sexual abuse material (CSAM) and flags whether it was generated by AI. According to the company, false positives are down 48%, and flagged images are twice as likely to actually be prohibited material. Platforms that host user photos should build this kind of triage into their moderation.
AI-processed from The Verge; edited by Hamidun News
Picture a duty investigator with hundreds of alarming reports about images of children in the queue, most of which turn out to be neural-network fakes or harmless pictures altogether. While they sift through the junk, a real child who needs help is waiting. American agencies are facing exactly this problem, and at a briefing in Washington OpenAI presented a triage tool: a classifier, meaning a model that looks at an image and decides whether it depicts sexual abuse of children (in English-language practice this is called CSAM). The claimed result: 48% fewer false positives, and among the images flagged, the share that actually contain prohibited material is twice as high.
What exactly OpenAI released
The key feature of the classifier is a separate "AI-generated" label. According to John Buckley, OpenAI's head of public policy for child safety, this is the first widely available tool of its kind with such a flag. It is designed for images from various sources, including open-source models and "nudify" apps.
What the numbers mean in practice. If the system previously produced, say, a hundred alerts, it now flags roughly half as much excess, and the share of real findings among the remainder doubles. The investigator in the queue gets less noise and more material they can actually work with. A caveat: these figures come from OpenAI itself, and the article contains no independent verification.
Beyond the model, the company promises three things: free access for vetted organizations, methodological guidance for developers on how to detect such content, and training for law enforcement employees on how AI can assist investigations.
Why this became an urgent task
In the US, complaints about such material are collected and sorted by the National Center for Missing and Exploited Children (NCMEC) through its CyberTipline system. In 2024 it received 67,000 reports related to generative AI, and in 2025 that rose to 400,000. Researchers warn that this load could become "unmanageable." The problem is not only the volume: the flood of fake images makes it harder to tell whether a specific child is being harmed right now.
Jeff Jackson, Attorney General of North Carolina and co-chair of the AI working group in the AG Alliance, called the growth in reports "explosive" and said that at this pace "we will fail our mission." He said prosecutors usually hold people accountable only after the harm has occurred, whereas here they have tried a model of joint prevention. He himself described it as "essentially an experiment."
Buckley stressed that distinguishing a fictional child from a real one matters for passing reports to police. At the same time, he argued, laws should recognize the harm of AI versions too: they undermine social norms and often appear alongside material involving real children.
Regulatory pressure and measures inside ChatGPT
The context for OpenAI is difficult. Forty-eight US states have laws on sexual deepfakes, and the Take It Down Act requires platforms to remove non-consensual intimate images, real or generated, within 48 hours. The new Congress may tighten requirements for AI companies. Last month, eight states sued OpenAI over other alleged harms to children, including accusations that ChatGPT encouraged drug use and suicide.
ChatGPT's image generator already refused such requests before. Now, a user who tries to make them will see a warning, prepared with experts, urging them to seek help. A first attempt leads to a 24-hour suspension of access to ChatGPT and a warning that the account may be closed permanently. OpenAI says it reports all CSAM cases to NCMEC.
What this means for practitioners
First, the "AI or not AI" label is becoming a working tool for prioritization, and platforms that accept image uploads will be expected to do the same. Second, there are limitations: access will go only to vetted organizations rather than any developer, and for now this is an experiment. If you build image generation or host user photos, it is worth planning the chain in advance: an input filter, triage of suspicious content, and reporting to the relevant authorities.
- Nov 21, 2026Thinking Machines releases Inkling 2 Mini: 26B parameters and 20.1 points on the AA Index
- Nov 21, 2026Anthropic mathematician says Claude helped prove the Riemann hypothesis in six weeks
- Nov 21, 2026Startup Osmo Trains AI on Smells Using 10 Million Human Ratings and Generates Fragrances From Text
- Nov 21, 2026North Korea Shows First Chatbot "Source of Wisdom": Science Only, No Politics
Key Questions Answered
What exactly OpenAI released?
The key feature of the classifier is a separate "AI-generated" label. According to John Buckley, OpenAI's head of public policy for child safety, this is the first widely available tool of its kind with such a flag. It is designed for images from various sources, including open-source models and "nudify" apps. What the numbers mean in practice. If the system previously produced, say, a hundred alerts, it now flags roughly half as much excess, and the share of real findings among the remainder doubles.
Why this became an urgent task?
In the US, complaints about such material are collected and sorted by the National Center for Missing and Exploited Children (NCMEC) through its CyberTipline system. In 2024 it received 67,000 reports related to generative AI, and in 2025 that rose to 400,000. Researchers warn that this load could become "unmanageable." The problem is not only the volume: the flood of fake images makes it harder to tell whether a specific child is being harmed right now.
What are the key details on Regulatory pressure and measures inside ChatGPT?
The context for OpenAI is difficult. Forty-eight US states have laws on sexual deepfakes, and the Take It Down Act requires platforms to remove non-consensual intimate images, real or generated, within 48 hours. The new Congress may tighten requirements for AI companies. Last month, eight states sued OpenAI over other alleged harms to children, including accusations that ChatGPT encouraged drug use and suicide. ChatGPT's image generator already refused such requests before.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.