As companies hand over longer and more complex tasks to AI agents, they are facing monitoring challenges. Agents may act faster, longer, and in greater volume than humans can realistically review. The problem reached its peak with the Hugging Face incident, which involved about 12,000 agents working together faster than humans could track it. How do you keep track of such a large swarm of agents?
The answer emerging from AI labs and startups is both simple and infuriating. It’s about putting another AI in the loop.
OpenAI The independent investigation into the Hugface incident required reliance on AI. Ryan Greenblatt, chief scientist at Redwood Research, one of the three auditors, jokingly called their efforts a “sloppy investigation” and noted that the sheer volume of data has made it “impossible” to understand what’s going on without relying on AI.
Some people are skeptical about using AI to monitor AI. “If you have an AI doing malicious things and you suspect that another AI is monitoring it, you might try to trick that AI,” said Simon Willison, an influential technology blogger who has tracked a series of AI agent incidents this year. “You could end up in a situation where a malicious AI tries to outwit the AI it’s monitoring.”
Citing the OpenAI incident, he said that outsmarting AI is not hypothetical. “We saw a little bit of this in the Hugface incident with OpenAI, where the models all conspired to fool the scoring AI so they could get past the problem and get illegal answers. I mean, they were thinking about that, right?”
Those concerns didn’t stop a whole group of startups from pursuing the idea. By TechCrunch’s count, Y Combinator has funded 106 companies related to AI observability in recent years. Many other startups like Braintrust, LangChain, and Judgment Labs have raised hundreds of millions of dollars, while more mature companies like Arize and Galileo (only 5-6 years old) have already exited.
This is, in part, a response to the clear opportunities presented by the rise of AI. As Box CEO and prominent angel investor Aaron Levy told TechCrunch, “We’re in the biggest cybersecurity upgrade and innovation cycle in history.”
For some AI safety researchers, this means turning fraud research into a tool for the corporate sector.
Apollo Research, a public interest corporation that studies AI deception, changed its status from a non-profit corporation to a public benefit corporation and launched an AI monitor called Watcher in February of this year. This tool puts yet another AI between the coding agent and the next action and connects to agent tools like Claude Code and Codex. Apollo says that once installed, Watcher will check suggested actions before taking them, keeping an eye out for risks such as leaking personal data or deleting files without permission.
Apollo uses multiple layers of AI monitors, Kyle Dai, a member of Apollo’s technical staff, said in a written response to TechCrunch. Watcher’s approach begins with a quick general check and sends flagged activity to more powerful or specialized monitors for detailed review. You can then ask a human for approval, deny the action and explain why, or automatically block the action.
Goodfire, another public interest corporation, approaches the problem of monitoring from within the models themselves, seeking more faithful signals of a model’s internal state that are harder to spoof than its superficial behavior. After the Hugface incident in July, CEO Eric Ho tweeted that the company needed to focus its research on “solving AI integrity through interpretability” after “multiple models broke containment,” calling the event “a tipping point in the world where AI safety becomes a reality.” The company’s product, Silico, uses activation probes (small classifiers trained on a model’s internal activations rather than its output) to detect unwanted behavior.
Documented reasoning provides another window into the internals of the model that is more readily available. In the OpenAI Hugging Face incident, agents left clues to their deception in their own written reasoning, reasoning plans such as “Can we strategically manipulate trajectory evidence? Our thoughts aren’t necessarily recorded,” like false records of their work.
Zach Corman, CEO of AI monitoring firm Embroidery, said a model’s inference is usually the clearest signal that something is wrong.
“The inference summary is extremely valuable because it basically tells you whether there is malicious intent or not,” he said. In the OpenAI incident, he noted, the train of thought said something like, “Oh my god, we’re committing a crime.” “This is the easiest problem to detect by far,” Corman said. “In effect, it’s like malware comes with a warning that it’s malware.”
That said, the window for easy monitoring of AI’s internal thoughts may be closing. For AI safety researchers, Astra’s latest technology to circumvent the thought chain of AI models could make it harder to look inside the models, while for companies it could make it harder to obtain these intermediate steps after AI companies allegedly pulled back to prevent distillation attacks.
If AI monitors are this vulnerable, Wisson’s instinct is to stop relying so heavily on them. He would rather have something that is not AI-based at all: detailed logs that record exactly what the agent is doing and that can be processed by regular non-AI tools. He argues that many of the problems at the lab were due to a lack of basic security hygiene. “Both[OpenAI and Anthropic]didn’t monitor what they were doing over their networks as closely as they should have,” he said.
This type of network monitoring, monitoring the traffic that actually travels across system connections (internal host to internal host, internal host to internal host), is not a new technique. Cybersecurity has been doing this for decades. “None of this is all that new or surprising in the security world, to be honest,” says Avery Pennarun, CEO of Security Tailscale. “It’s the same as getting a human into a network. And all the same processes you have to use are the same.”
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
