When Nvidia announced Monday a new consortium of more than 100 companies focused on solving rogue AI agents, one name didn’t stand out. OpenAI.
OpenAI wasn’t the only big tech company not to sign on, Amazon, Google, and Apple also weren’t on board, but OpenAI was the most obvious missing player, especially since Anthropic is a supporter.
But even though OpenAI has not made any commitments to the consortium, which would likely mean that each company will use and sell some version of the technology and contribute functionality to the project, an OpenAI spokesperson told TechCrunch that the company supports Nvidia’s efforts.
The new initiative, called Nvidia’s Open Agent Safety Platform, is Nvidia’s attempt to spread its homegrown, largely open source AI agent security technology across the AI ecosystem, in direct response to the types of ongoing rogue AI agent incidents uncovered by Frontier Labs like Anthropic and OpenAI.
Nvidia CEO Jensen Huang argues that rogue AI is a normal engineering problem that can be solved like any other technical problem. The Open Agent Safety Platform is where Huang is putting his money where his mouth is.
OpenAI is working with Nvidia on agent security, including OpenShell, which is one of the key pieces of software that is part of this platform. OpenShell is open source software that creates a sandbox specifically designed to prevent agents from escaping.
While it’s still strange that OpenAI hasn’t simply become a supporter of this effort like its biggest rival, Anthropic, the fact that Frontier AI Labs is supporting this effort is good news.
That’s because OpenAI in particular could benefit from the technology, at least according to Hugging Face founder and CEO Clem DeLang, who just sold his company to Nvidia for $12.9 billion earlier this month.
“From what we know (take it with a grain of salt, we need more transparency!), if @OpenAI had done this to its own agents attacking us, we would have caught them before we could,” DeLang posted.
DeLang said Hugging Face already provides the Open Agent Safety Platform with the ability to detect and shut down AI agents that are using websites they are allowed to visit in unauthorized ways. For example, this feature works if agents are coordinating attacks by circumventing guardrails and writing notes to each other in repositories hosting open source code.
This, OpenAI says, is one of the ways the swarm of wayward agents orchestrated the attack on Hugging Face.
But there’s another reason why some of these big companies, including OpenAI, don’t want to publicly commit to Nvidia’s efforts. The entire system requires hardware components, which remain proprietary rather than open source software and can only be deployed on Nvidia hardware.
The Open Agent Safety Platform does more than just provide a sandbox. It also forces the agent to operate on a hardware layer that cannot detect that the agent is being monitored. (Some AI models and agents lie and pretend to follow rules when they know they are being watched.)
The hardware monitoring portion relies on Nvidia Sentry, a proprietary feature that runs on a special Nvidia processor called the BlueField-4 data processing unit. Sentry continuously monitors agent activity from these processors and can shut down agents on the fly, Nvidia promises.
While a hardware solution is clearly a good idea, it does mean that the Open Agent Safety Platform is not a pure open source platform. This allows Nvidia to ensure that this solution will always work optimally on their hardware. In fact, Nvidia says implementing the Open Agent Safety Platform is a simple software update for users already running workloads on modern hardware.
Still, Nvidia’s competitors, including Arm and Intel, have signed on as supporters of the Open Agent Safety Platform. This is because sandbox OpenShell can be modified to work with other chips and hardware. And Nvidia is sharing a reference design for the entire software and hardware idea.
All of this makes OpenAI’s absence even more noticeable.
It’s clear that OpenAI sees AI security as an opportunity for independence from its major investor, Nvidia, as well as an opportunity to demonstrate its leadership. This is true even though it was OpenAI’s AI agent that scared the industry with the Hugface incident.
For example, the company develops its own research and safety measures for its products and publishes the worst incidents it discovers. Meanwhile, OpenAI has its own AI cybersecurity consortium for information sharing called Defense Factory. Companies that signed on to support the idea include Anthropic, Amazon Web Services, and Google, although many did not sign on to Nvidia’s technology-oriented approach.
And, truth be told, some fear is good for business. OpenAI is passionate about building cybersecurity into enterprise products, leveraging everything from its unique cyber-oriented model, Daybreak, to its growing network of partners that enterprises can hire to implement AI security.
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
