OpenAI’s latest report on malicious AI use underscores the tightrope that AI companies are walking between preventing misuse of their chatbots and reassuring users that their privacy is respected.
The report, which dropped today, highlights several cases where OpenAI investigated and disrupted harmful activity involving its models, focusing on scams, cyberattacks, and government-linked influence campaigns. However, it arrives amid growing scrutiny over another type of AI risk, the potential psychological harms of chatbots. This year alone has seen several reports of users committing acts of self-harm, suicide, and murder after interacting with AI models. This new report, along with previous company disclosures, provides some additional insight into how OpenAI moderates chats for different kinds of misuse.
OpenAI said that since it began reporting public threats in February 2024, it has disrupted and reported more than 40 networks that violated their usage policies. In today’s report, the company shared new case studies from the past quarter and details on how it detects and disrupts malicious use of its models.
For example, the company identified an organized crime network, reportedly based in Cambodia, that tried to use AI to streamline its workflows. Additionally, a Russian political influence operation reportedly used ChatGPT to generate video prompts for other AI models. OpenAI also flagged accounts linked to the Chinese government that violated its policies on national security use, including requests to generate proposals for large-scale systems designed to monitor social media conversations.
The company has previously said, including in its privacy policy, that it uses personal data, such as user prompts, to ‘prevent fraud, illegal activity, or misuse’ of its services. OpenAI has also said it relies on both automated systems and human reviewers to monitor activity. But in today’s report, the company offered slightly more insight into its thought process for preventing misuse while still protecting users more broadly.
“To detect and disrupt threats effectively without disrupting the work of everyday users, we employ a nuanced and informed approach that focuses on patterns of threat actor behavior rather than isolated model interactions,” the company wrote in the report.
While monitoring for national security breaches is one thing, the company also recently outlined how it addresses harmful use of its models by users experiencing emotional or mental distress. Just over a month ago, the company published a blog post detailing how it handles these types of situations. The post came amid media coverage of violent incidents reportedly linked to ChatGPT interactions, including a murder-suicide in Connecticut.
The company said that when users write that they want to hurt themselves, ChatGPT is trained not to comply and instead acknowledge the user’s feelings and steer them toward help and real-world resources.
When the AI detects someone is planning to harm others, the conversations are flagged for human review. If a human reviewer determines the person represents an imminent threat to others, they can report them to law enforcement.
OpenAI also acknowledged that its model’s safety performance can degrade during longer user interactions and said it’s already working to improve its safeguards.
