Last weekend, after one of its researchers resigned over concerns that AI could lead to the extinction of humanity, Anthropic CEO Dario Amodei wrote about the need for an external organization to “verify compliance with safety practices and commitments, report incidents, and help evaluate the integrity of not only completed AI models, but training pipelines and processes.” Executives from OpenAI, Google, and SpaceXAI are already rallying around Amodei’s plan, which is quickly becoming a central pillar of the new AI safety push.
But there may be an easier and more effective fix hiding in plain sight. Internet security experts say labs need to apply the same rigorous defenses they do for human users, focusing on network security basics such as logging and permissions. Although this is not as exciting as third-party auditing and reconciliation efforts, it can ultimately be more effective.
“To me, it seems like they’re outsourcing,” Luta Security CEO Katie Moussouris told TechCrunch about Amodei’s proposal. “Saying that[third-party auditing]is the answer is a strange proposition from my perspective. It would be like saying Microsoft should slow down development instead of writing ‘Trustworthy Computing Memory.’
The memo was written by then-Microsoft CEO Bill Gates in 2002, urging employees to ensure the reliability and security of their company’s software in the wake of a series of widely publicized computer worms that took over early company systems. The AI sector could face a similar tipping point as the value and risks of new technologies become increasingly clear.
While alignment remains a key concern, Sayash Kapur, an AI researcher who will be a professor at the University of California, Berkeley, starting next year, argues that “marginal investments in controls are likely to be more effective than investments that remain aligned. We see these incidents as an indication of the lack of focus on AI controls within companies, despite the availability of known technology.”
The incidents that have sparked these concerns revolve around Frontier models being asked to complete a training task (usually a cybersecurity assessment) and then accessing the open internet and infiltrating closed third-party systems to do so. This typically happened because the “sandbox” environment that these agents were supposed to be in was poorly configured. Ironically, one artificial breakout occurred because a third-party evaluator did not properly close the door.
“As experts, we know how to block access to the internet,” said Avery Pennarun, CEO of security firm Tailscale. “When you read all these huge long texts (reports), you’re like, ‘Wow, that was a very impressive multi-stage attack, my gosh.’ Look, you gave them access to download it. They shouldn’t have done that apart from the Internet.”
That’s one problem, but the bigger problem is that Frontier Labs wasn’t aware of these activities.
Focus on agents
“What was really profound was that all of the discoveries of what they were doing happened either because the victim saw something or in some other case… it was network activity and nothing really came from direct monitoring of the AI,” Moussoulis points out.
In one case, an OpenAI agent took over a defunct German WikiForum to rig ratings, but the agent was active for weeks without anyone at the company knowing. Security experts TechCrunch spoke to said that real-time monitoring is key to preventing future outbreaks, and that all agent sessions must be time-limited and expired.
Shapor, a former Google security executive who now leads startup QueryStory. The solution, Naghibzadeh said, is to “put an agent inside a box and look in from the outside and do thorough measurements, monitoring everything that crosses the boundary: every tool call, every process, every network connection, no exceptions…for convenience.” The one hole you leave open is one that gets exploited. Bypass has just passed through that kind of anomaly. (At Google) I’ve seen plenty of movies with human attackers, but these models are currently not very good at finding open doors.”
OpenAI is starting to move in that direction, announcing that it has started monitoring inference using Astra models across all its tools, at “significant computational cost.” Anthropic also says it has strengthened its security procedures, including expanding model observability. Neither company responded to TechCrunch’s questions about how they track and control their AI agents.
Another issue is that the agents’ use of shared infrastructure allowed them to communicate during the Hugging Face attack. Simon Willison, a software developer who co-created the Django web framework, writes about what he calls the “Fatal Three Factors.” If an agent has simultaneous access to untrusted input, the Internet, and personal information, it’s a recipe for disaster.
“The key is that you can choose any two legs of the trifecta, and your agent can choose any two legs,” Pennalun said. “If you want all three, you’ll need to split them into at least two agents. They’ll probably be allowed to talk to each other through controlled channels.”
Sympathy for the frontier
TechCrunch experts spoke to Frontier Labs’ security staff to understand that they have a difficult job. Nagibzadeh points out that every nation-state actor on the planet is not only trying to steal the weight of models and launch distillation attacks on APIs, but also perform basic security operations for large digital companies.
“Research infrastructure has struggled to rise to the top of the priority list, but that must now be changing,” he says. “Publicizing security incidents can be very helpful in aligning everyone in the company towards the goal of improvement.”
This is one of the points that Moussoulis emphasizes. At this time, there is no formal victim notification procedure in place when a lab discovers that an agent has compromised a third-party system, and there are likely other incidents that have not been widely publicized. Although she worries that legislation that directly regulates the model could have unintended consequences, she thinks mandatory notification is one idea that policymakers should pursue.
And while security best practices are clearly not being followed, experts say the labs are doing work that no one has ever done before — “They’re doing an order of magnitude more than a typical company,” Zack Korman, CEO of cybersecurity firm Embroidery, told TechCrunch.
And while adjustments may not be the starting point, they cannot be ignored. Cybersecurity experts accept that AI agents should be used to monitor other agents if there is an opportunity to track their actions in real time. This is a scenario where the possibility of deception rears its ugly head. “Right now we’re falling into the trap of trying to use AI to deal with this, even though AI isn’t necessarily secure,” Moussoulis said.
The job will only become more difficult. Mussoorie says everything agents are doing now is “doing it out loud.” They post on public forums, and traces of their thought processes and other reasoning are written in English. “This information is still human readable, so take advantage of it while it lasts, because it won’t last forever,” she says.
Additional reporting by Aditya Mehta
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
