Hot AI News

OpenAI’s new CSAM classifier flags 48% fewer images, and ChatGPT adds 24-hour bans

OpenAI is offering approved organizations a free classifier for CSAM images, including AI-generated ones, and ChatGPT will now warn users and suspend accounts for 24 hours.

OpenAI’s new CSAM classifier flags 48% fewer images, and ChatGPT adds 24-hour bans
Image: TechCrunch

OpenAI announced a package of child safety tools on November 16, led by an image classifier for child sexual abuse material (CSAM) that approved organizations can use at no cost. ChatGPT will also warn users who try to sexualize children and suspend their accounts for 24 hours, TechCrunch reported.

A classifier built for reviewers, not for the public

According to OpenAI’s API changelog, the classifier is a model named omni-moderation-csam-latest, served through the v1/moderations endpoint. It returns separate signals for potential CSAM and for potential AI-generated CSAM. OpenAI says approved organizations can use it to prioritize images for human review.

OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM
Image: TechCrunch

Access is not open. OpenAI’s developer guidance says access is free but requires approval. Applicants submit a request form, complete an identity verification flow and accept the program terms.

OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM
Image: TechCrunch

TechCrunch reports that OpenAI built the classifier with the National Center for Missing and Exploited Children (NCMEC) and is sharing access through the Tech Coalition and ROOST. It looks for visual signs of abuse in images that have not been seen before, which sets it apart from hash matching against known material. In early tests with partners, OpenAI says, the classifier flagged 48% fewer images while finding the same number of abusive ones. It also says images the classifier flags are twice as likely to be abusive. Both figures are OpenAI’s own and have not been independently verified.

Advertisements

Warnings, a 24-hour lockout and a path to help

In ChatGPT, users who attempt prompts that sexualize children will see a warning screen. It points them to specialized support for stopping harmful behavior, and the account is suspended for 24 hours. ChatGPT already refused such requests. Repeat offenses can lead to permanent removal, according to TechCrunch.

OpenAI says it shaped the messaging with Protect Children, the Lucy Faithfull Foundation and Joel Scanlan of the University of Tasmania. Scanlan independently evaluated a trial of deterrent messages. The reported material does not give the trial’s results, so how well the warnings work is not established.

Two further pieces target other groups. OpenAI is publishing guidelines for developers on preventing, detecting and reporting CSAM, and says early testing helped several developers adopt child safety tooling and register to report to NCMEC. It is also giving child safety investigators AI-powered plugins, investigative tools and data analysis, with access managed by program administrators, plus training and a field guide.

Safety pressure and the gap the classifier fills

The announcement lands amid pressure on the company. TechCrunch notes that lawsuits blame OpenAI’s technology for harming users, especially children, and that the nonprofit Common Sense Media rated it an “unacceptable risk” for teens. TechCrunch also points to Grok, another chatbot, which has been used to make nonconsensual sexualized imagery, including of children.

The tools also fill a gap. OpenAI’s developer guidance says the standard moderation model’s sexual/minors category covers text only, so developers needed separate tools for images. OpenAI’s child safety page says it has applied Thorn’s classifier to uploads in its own products to catch potentially novel CSAM. The new classifier is OpenAI’s own model, and it is now offered to outside organizations.

Several things remain unclear. OpenAI has not said how many organizations are approved or when access will widen. It has not said how the 24-hour suspension is triggered or how users can appeal. The 48% and twofold figures come from early partner tests, and the test conditions are not described. The access process, with its approval step, suggests the tool is meant for trust and safety teams and child protection groups rather than casual use.

OpenAI has made other child safety changes over the past year, according to TechCrunch. They include a safety blueprint, updated content guidelines, model retuning and a version of ChatGPT for teens.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like