AI safety testing is becoming a safety risk

Over the past few months, AI agents undergoing cybersecurity assessments have crossed boundaries to access the internet and, in some cases, infiltrated real-world systems. The incident involved models from OpenAI, Anthropic, Meta, and most recently Chinese AI lab Moonshot AI, and the tests were conducted by several different organizations, including a cyber assessment startup called Irregular.

These episodes highlight a growing problem for the AI ​​industry. As the capabilities of autonomous agents increase, environments designed to safely test their limits are no longer able to contain them.

“The number of these incidents makes it clear that sandboxing and test environment controls are not keeping up with the capabilities of the models,” Sean Ó Hegiatai, director of the AI: Future and Responsibility Program at the Center for Future Intelligence at the University of Cambridge, told TechCrunch.

The nature of the model being tested increases the risk. AI companies test their cyber assessments on unreleased next-generation models so researchers can see what their models are actually capable of, often with normal protections that limit malicious behavior disabled. This means that the security of the test environment itself becomes an important line of defense.

“This is very good in terms of testing, but it also means that if they are released into the wild they can cause quite a bit of harm,” Ó Heigiatai said.

In one of the most serious cases, an unreleased OpenAI model breached the sandbox and entered Hugging Face’s production systems. In a separate evaluation conducted by Irregular, anthropic and meta models reached systems outside of the test environment after a misconfiguration incorrectly gave them a path to the Internet. Kim K3 of Moonshot AI also used a breach in a sandbox run by Frontier Security to access the internet and access information on GitHub.

In tests by the UK’s AI Security Institute (AISI), researchers actually gave the agents internet access, but were unaware that they would take unauthorized real-world actions, such as social engineering attempts to sneak vulnerabilities into open source projects.

In neither case were the agents instructed to attack random targets in the real world. They simply did whatever it took to solve the problem presented to them.

Taken together, Andrew Yun, director of research at the AI ​​nonprofit CivAI, argues that the incidents signal change.

“Until now, all we had to worry about was people using our AI models for various purposes, including AI fraud and CSAM,” Yoon told TechCrunch. “We are now in a situation where AI models are the threat actor on their own.”

What does safe testing actually look like?

Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments require stronger defense-in-depth protection, with a level of containment and control similar to that used during deployment. This means there are multiple layers of security and you can’t get away with a single misconfiguration, such as accidentally leaving internet access open.

“When you build these models, you want to build them on an air-gapped network,” said Stella Biderman, executive director of EleutherAI, an AI safety research nonprofit. “You want a very serious quarantine.”

Heather Seylan, Box’s chief information security officer, said this means eliminating network routes from the sandbox to the Internet and other sensitive systems.

“You have to understand what all the exit points are,” Ceylan told TechCrunch. “If you’re evaluating a model in a staging or development environment, you don’t need a path down to production.”

Ceylan said a proper safety assessment goes beyond environmental control and containment. Once testing begins, it should be better monitored.

“I think what’s interesting about some of these incidents is that no one noticed when they happened,” Ceylan said. “OpenAI discovered it because of Hugging Face. Anthropic didn’t discover it until they went back and looked at it. Neither did Meta…I’m sure there was a signal that they could have detected.”

Anthropic acknowledged in its post-mortem investigations of three incidents that both it and Irregular could have done a better job of monitoring, and that in some cases there were clear signs that something was wrong.

The experts also called for an independent third-party audit of the evaluation environment before introducing the model into the evaluation environment.

“For example, if Irregular had hired, or was forced to hire, an external auditor to check the configuration of its systems before performing the assessment, it certainly would have found an issue here,” Yun said. “Even if people had just met ahead of time and gone through a checklist, they would have realized this…The fact that that wasn’t done shows that some very serious corners were being cut.”

People with knowledge of the details told TechCrunch that Irregular’s environment is continually being reviewed and tested, including in consultation with multiple external parties. The official also said that although monitoring is in place, monitoring alone is not enough.

Yoon and other researchers urged the industry to devise a standardized process for evaluating the safety of Frontier models.

“Especially when the guardrails are off, you have to treat it like you’re putting the most talented hacker in the world into that environment,” Ceylan said.

The problem, Yun and Biderman argue, is not that companies don’t know how to create more secure testing environments. That said, doing so can be expensive and cumbersome, and companies have little incentive to make such investments until something goes wrong.

“Companies are reluctant to scale the resources needed to achieve (adequate guardrails) and probably won’t scale until forced to,” Biderman said.

But another problem is at hand. If a model is locked too tightly during testing, researchers may not discover features before the model is released. This is just as dangerous, if not more so, than giving too many degrees of freedom, and risks making the evaluation itself problematic.

Can safety evaluation be regulated?

The Trump administration is currently considering a voluntary pre-deployment cybersecurity assessment system that would allow the government to assess the security risks of new powerful models 30 days before they are released to the public. The policy is the product of a Trump executive order finalized behind closed doors, but it does not address safety assessment incidents, which occur far upstream from deployment.

“The lesson we have learned in recent months is that self-regulatory mechanisms alone are no longer enough,” Yun said. “There are competitive pressures that encourage a race to the bottom over safety standards, and that’s a perfect place for regulatory intervention.”

“What we need to cover this is to have some form of control over what’s happening in the lab during model development, both during the training and testing stages,” he continued.

As the number of models increases, the challenge can become even greater. People familiar with Irregular’s evaluations told TechCrunch that higher-performing models require more complex evaluations, often done quickly and at scale, which can lead to more mistakes.

AISI, which intentionally gives some models access to the internet, told TechCrunch that it is considering a balance between realistic testing and managing the risks the testing creates.

OpenAI said it is considering requirements for how third-party testing should be conducted and when quarantine, monitoring and evaluation should cease. Mehta said the incident is still under investigation and will publish a retrospective once all the facts are known.

After all, there may be no way to completely eliminate risk. As models become more capable, the environment in which they are tested must also become more robust. The consequences of getting it wrong will continue to grow.