Amid growing safety concerns regarding the use of AI tools, OpenAI today announced a new tool to combat child sexual abuse material (CSAM). These features include a new image classifier to detect CSAM, warnings and temporary blocks to prevent users from creating or accessing content that sexualizes children, new guidelines for developers, and a set of tools for law enforcement.
The company announced a new image classifier that can detect CSAM images, including those generated by AI. OpenAI offers some companies access to its classifiers for free through its API. The tool looks for visual signs of abuse in images that have not been seen before and assigns a score that can guide reviewers on how to address CSAM.
OpenAI said it developed the classifier in collaboration with the National Center for Missing and Exploited Children (NCMEC) and is working with the Tech Coalition and ROOST to share access to the tool. In initial testing with partners, the AI company found that the classifier flagged 48% fewer images while detecting the same number of inappropriate images, reducing the work of reviewers. Furthermore, images flagged by the classifier are twice as likely to be abusive images.
Additionally, the company is adding a new warning screen to ChatGPT to prevent users from accessing content that sexualizes children. ChatGPT will display a warning to any user attempting such a prompt, direct the user to professional support to stop the harmful activity, and suspend the account for 24 hours as a warning. The app already blocks CSAM content and related requests. Repeat violations may result in permanent account deletion.
OpenAI noted that it had worked with organizations and researchers, including Protect Children, the Lucy Faithful Foundation, and Joel Scanlan from the University of Tasmania, who independently evaluated the deterrence message trial, to develop messages and guidelines around the practice.
The company also publishes guidelines for developers to use OpenAI tools to prevent, detect, and help users report CSAM content. The guidelines said that initial testing helped many developers deploy child safety tools and register to report CSAM to NCMEC. OpenAI also provides law enforcement agencies working on child safety with AI-powered capabilities such as proprietary plugins, investigative tools, and complex data analysis, access of which is controlled by program administrators. We also provide training to investigators, along with field guides on how to use AI.
OpenAI has filed multiple lawsuits against the company, alleging that its technology harms users, especially children. Common Sense Media, a non-profit organization focused on safety, has labeled OpenAI’s technology an “unacceptable risk” to teenagers. Other AI chatbots, particularly Grok, have been used to create non-consensual sexual images, including of children.
The AI startup has been making changes to its technology over the past 12 months. This includes releasing a safety blueprint, updating our content guidelines to prohibit inappropriate content, recalibrating our model to prevent the generation of harmful content, and releasing a special version of ChatGPT for teens.
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
