OpenAI releases CSAM detector: moderators get 48% fewer images to review
OpenAI has released a CSAM detector. CSAM stands for child sexual abuse material, including AI-generated images. The model gives each new image a score, and a moderator makes the final decision. In tests with partners, it flagged 48% fewer images while finding the same number of real violations, so moderators see fewer harmless pictures. For platforms that handle photo uploads, this saves moderator time.
AI-processed from TechCrunch; edited by Hamidun News
Any platform where users upload images or generate them with AI sooner or later runs into CSAM, the abbreviation for child sexual abuse material. Generative models have made the problem more acute: such images can now not only be distributed but also created. OpenAI has announced a package of measures in four areas: a new image classifier, warnings in ChatGPT, a guide for developers, and tools for the police.
The package's headline number is 48%. In early tests with partners, the classifier flagged 48% fewer images than the previous approach, while the number of real violations it found stayed the same. Put simply, moderators get fewer harmless pictures that they have to look at by eye, and real cases are not lost.
Classifier: less unnecessary manual work
The new model looks for visual signs of abuse in images, including AI-generated ones. It also works on pictures it has never seen before, so it does not rely solely on matching files against a database of known material. The output is a score. The model does not deliver a verdict; it indicates what to look at first and what to do next.
OpenAI developed the tool together with the National Center for Missing and Exploited Children (NCMEC). The company provides access through the API for free, but for now only to selected organizations. Distribution is being handled together with Tech Coalition and ROOST.
The second figure from the tests: images flagged by the classifier are twice as likely to be real violations. For moderators, this is also a question of psychological load: every extra questionable image is one a person has to view.
Warnings in ChatGPT and instructions for developers
ChatGPT already blocked CSAM and related requests. Now, when such an attempt is made, the chat will show a warning and direct the person to specialized help to stop the harmful behavior. After that, the account will be suspended for 24 hours. For repeated violations, it may be deleted permanently.
OpenAI prepared the warning texts with the organizations Protect Children and the Lucy Faithfull Foundation, and with researcher Joel Scanlan from the University of Tasmania. Scanlan independently evaluated the pilot with messages meant to deter people from such requests.
Separately, the company published a guide for developers building on its tools. It describes how to prevent and detect CSAM and how to help users report it. According to OpenAI, in early tests the guide helped several developers integrate child protection tools and register for reporting with NCMEC.
For law enforcement working on child safety cases, the company is adding AI features: plugins, investigation tools, and analysis of complex data sets. Access is controlled by the program's administrators. Investigators are offered training and a field guide on working with AI.
Context and limitations
The announcement comes amid pressure on the company. Lawsuits have been filed against OpenAI, accusing the technology of harming users, especially children. The organization Common Sense Media rated OpenAI's products for teenagers as an "unacceptable risk." Other chatbots, including Grok, have been used to create sexualized images without consent, including of children. Over the past 12 months, OpenAI has already released a safety plan, updated its content rules, retrained its models, and launched a teen version of ChatGPT.
Caveats are necessary. The 48% and "twice as likely" figures come from early tests with partners, and so far only the company itself has reported them. The article contains no independent verification. Not everyone will get access to the classifier: it is granted to selected companies. The score remains a hint, and the final decision is made by a human.
For product teams, the takeaway is simple. If a service has image upload or generation, it needs a pipeline planned in advance: automated scoring, manual review, and reporting to the relevant organization. OpenAI's guide can serve as a starting point, and if your company does moderation work, it's worth asking about classifier access through Tech Coalition or ROOST.
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
- Nov 21, 2026A swimming team captain handed ChatGPT three scheduled tasks and cut her routine work by a multiple
- Nov 21, 2026OpenAI GPT-6 Cyber Found About 35 Flaws in iPhone, Mac and Other Apple Devices
- Nov 21, 202622 Claude Agents Spent Six Weeks on Black Hole Problems — and Solved Two
Key Questions Answered
What are the key details on Classifier: less unnecessary manual work?
The new model looks for visual signs of abuse in images, including AI-generated ones. It also works on pictures it has never seen before, so it does not rely solely on matching files against a database of known material. The output is a score. The model does not deliver a verdict; it indicates what to look at first and what to do next. OpenAI developed the tool together with the National Center for Missing and Exploited Children (NCMEC).
What are the key details on Warnings in ChatGPT and instructions for developers?
ChatGPT already blocked CSAM and related requests. Now, when such an attempt is made, the chat will show a warning and direct the person to specialized help to stop the harmful behavior. After that, the account will be suspended for 24 hours. For repeated violations, it may be deleted permanently. OpenAI prepared the warning texts with the organizations Protect Children and the Lucy Faithfull Foundation, and with researcher Joel Scanlan from the University of Tasmania.
What are the key details on Context and limitations?
The announcement comes amid pressure on the company. Lawsuits have been filed against OpenAI, accusing the technology of harming users, especially children. The organization Common Sense Media rated OpenAI's products for teenagers as an "unacceptable risk." Other chatbots, including Grok, have been used to create sexualized images without consent, including of children.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.