Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseAPI / Product · ActionableTechCrunch→ original

OpenAI releases CSAM detector: moderators get 48% fewer images to review

OpenAI has released a CSAM detector. CSAM stands for child sexual abuse material, including AI-generated images. The model gives each new image a score, and a moderator makes the final decision. In tests with partners, it flagged 48% fewer images while finding the same number of real violations, so moderators see fewer harmless pictures. For platforms that handle photo uploads, this saves moderator time.

AI-processed from TechCrunch; edited by Hamidun News
OpenAI releases CSAM detector: moderators get 48% fewer images to review
Source: TechCrunch. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTechCrunch ↗
Editor-in-Chief
Key metric #1Flagged images are twice as likely to be real violations
Key metric #2ChatGPT will show a warning and suspend the account for 24 hours
Key metric #3API access is free, but for now only for selected companies

Any platform where users upload images or generate them with AI sooner or later runs into CSAM, the abbreviation for child sexual abuse material. Generative models have made the problem more acute: such images can now not only be distributed but also created. OpenAI has announced a package of measures in four areas: a new image classifier, warnings in ChatGPT, a guide for developers, and tools for the police.

The package's headline number is 48%. In early tests with partners, the classifier flagged 48% fewer images than the previous approach, while the number of real violations it found stayed the same. Put simply, moderators get fewer harmless pictures that they have to look at by eye, and real cases are not lost.

Classifier: less unnecessary manual work

The new model looks for visual signs of abuse in images, including AI-generated ones. It also works on pictures it has never seen before, so it does not rely solely on matching files against a database of known material. The output is a score. The model does not deliver a verdict; it indicates what to look at first and what to do next.

OpenAI developed the tool together with the National Center for Missing and Exploited Children (NCMEC). The company provides access through the API for free, but for now only to selected organizations. Distribution is being handled together with Tech Coalition and ROOST.

The second figure from the tests: images flagged by the classifier are twice as likely to be real violations. For moderators, this is also a question of psychological load: every extra questionable image is one a person has to view.

Warnings in ChatGPT and instructions for developers

ChatGPT already blocked CSAM and related requests. Now, when such an attempt is made, the chat will show a warning and direct the person to specialized help to stop the harmful behavior. After that, the account will be suspended for 24 hours. For repeated violations, it may be deleted permanently.

OpenAI prepared the warning texts with the organizations Protect Children and the Lucy Faithfull Foundation, and with researcher Joel Scanlan from the University of Tasmania. Scanlan independently evaluated the pilot with messages meant to deter people from such requests.

Separately, the company published a guide for developers building on its tools. It describes how to prevent and detect CSAM and how to help users report it. According to OpenAI, in early tests the guide helped several developers integrate child protection tools and register for reporting with NCMEC.

For law enforcement working on child safety cases, the company is adding AI features: plugins, investigation tools, and analysis of complex data sets. Access is controlled by the program's administrators. Investigators are offered training and a field guide on working with AI.

Context and limitations

The announcement comes amid pressure on the company. Lawsuits have been filed against OpenAI, accusing the technology of harming users, especially children. The organization Common Sense Media rated OpenAI's products for teenagers as an "unacceptable risk." Other chatbots, including Grok, have been used to create sexualized images without consent, including of children. Over the past 12 months, OpenAI has already released a safety plan, updated its content rules, retrained its models, and launched a teen version of ChatGPT.

Caveats are necessary. The 48% and "twice as likely" figures come from early tests with partners, and so far only the company itself has reported them. The article contains no independent verification. Not everyone will get access to the classifier: it is granted to selected companies. The score remains a hint, and the final decision is made by a human.

For product teams, the takeaway is simple. If a service has image upload or generation, it needs a pipeline planned in advance: automated scoring, manual review, and reporting to the relevant organization. OpenAI's guide can serve as a starting point, and if your company does moderation work, it's worth asking about classifier access through Tech Coalition or ROOST.

Story

Key Questions Answered

What are the key details on Classifier: less unnecessary manual work?

The new model looks for visual signs of abuse in images, including AI-generated ones. It also works on pictures it has never seen before, so it does not rely solely on matching files against a database of known material. The output is a score. The model does not deliver a verdict; it indicates what to look at first and what to do next. OpenAI developed the tool together with the National Center for Missing and Exploited Children (NCMEC).

What are the key details on Warnings in ChatGPT and instructions for developers?

ChatGPT already blocked CSAM and related requests. Now, when such an attempt is made, the chat will show a warning and direct the person to specialized help to stop the harmful behavior. After that, the account will be suspended for 24 hours. For repeated violations, it may be deleted permanently. OpenAI prepared the warning texts with the organizations Protect Children and the Lucy Faithfull Foundation, and with researcher Joel Scanlan from the University of Tasmania.

What are the key details on Context and limitations?

The announcement comes amid pressure on the company. Lawsuits have been filed against OpenAI, accusing the technology of harming users, especially children. The organization Common Sense Media rated OpenAI's products for teenagers as an "unacceptable risk." Other chatbots, including Grok, have been used to create sexualized images without consent, including of children.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…