
OpenAI has released an image-classification tool designed to help vetted organisations identify potential child sexual abuse material (CSAM), including suspected AI-generated images, and is adding warnings in ChatGPT that interrupt attempts to seek such content. The measures, announced on 16 November, reflect the growing difficulty of detecting newly created abusive imagery that does not match databases of previously identified material.
The classifier is available through OpenAI’s API at no cost to approved child-safety organisations and other vetted users. It assigns risk scores to images so trained teams can prioritise review. One score indicates suspected CSAM and another indicates imagery that may be AI-generated. Neither is a legal finding or proof of an image’s origin, according to the company’s model card.
Conventional hash-matching systems are useful when an abusive image is already known. They can miss novel material or manipulated images. OpenAI says its classifier looks for visual signals and is intended to complement, not replace, hash matching, contextual detection and human review. The National Centre for Missing and Exploited Children helped the company develop and evaluate the tool.
In OpenAI’s evaluation of user-uploaded images, the new system flagged 48% fewer images for review while identifying roughly the same number of abusive images as the existing tool. Images it did flag were about twice as likely to contain abuse. These are company-reported results from its own traffic, not a guarantee that the classifier will perform identically on other services. OpenAI says deployers must test suitable thresholds against their own data.
The model card is explicit about limitations. False positives and missed cases remain possible, and the AI-generated signal cannot verify provenance. OpenAI recommends that trained, authorised people assess flagged images before reports are submitted to child-safety hotlines. That safeguard matters for both victims and people whose lawful images could be wrongly identified.
OpenAI is also introducing product warnings when users try to find abusive material through ChatGPT. The warning explains the harm, including when the requested image would be synthetic, and points users towards specialist help. OpenAI says it will remove the flagged conversation and pause access to ChatGPT for 24 hours. Repeated violations may result in longer restrictions, account closure or reports to authorities.
The company is also publishing guidance for developers and supporting child-safety investigators. The policy change goes beyond a simple content refusal: it combines intervention, account enforcement and a tool that other qualified organisations can use. Whether the approach makes a measurable difference will depend on deployment, independent scrutiny and how accurately safety teams can identify genuine cases without overwhelming reviewers.
For AI platforms, the challenge is becoming more complex as image generation improves and abusive material can be created without copying an existing photograph. TechBooky has previously covered OpenAI’s safeguards for teenage ChatGPT users. Today’s announcement addresses a different, wider part of the child-safety problem: how to detect suspected abuse and intervene when the technology is misused.







