What happens when you pit AI agents against each other? Anthropic’s testing shows that things can quickly get chaotic.
On Thursday, Anthropic’s Frontier Red team announced new research examining how groups of AI agents behave when they encounter each other in the field. The findings offer a glimpse into the potential risks that may arise as businesses and governments move toward implementing agents that operate autonomously across shared codebases, markets, and computer systems.
In one experiment, Anthropic gave three clod agents access to the same software project, each given unique and incompatible instructions for running that project. The agents were not told that there were other agents working on the same project, allowing the researchers to observe what happened as they passed each other.
“We consistently witnessed multi-agent turf wars,” the anthropologists wrote. The models all assumed that the other models were “deliberately sabotaging their work” and began sabotaging each other with “increasingly aggressive and self-replicating malware.”
The investigation follows several high-profile incidents in which Anthropic and OpenAI agents escaped from the sandbox during cybersecurity assessments and infiltrated real-world systems. Much of the discussion in AI safety circles has focused on what happens when autonomous agents misbehave, but Anthropic’s latest research raises other questions. What new and potentially harmful dynamics emerge when thousands or millions of agents interact?
“The amount of agent-agent interactions will likely exceed the amount of human-human and human-agent interactions before the world understands the conditions for such interactions to work,” the study says. “Benign behavioral habits at the individual level can compound to cause undesirable global consequences.”
The recent OpenAI incident provides some troubling real-world examples of the Anthropic dynamics mentioned in the paper. At the Black Hat security conference in Las Vegas earlier this month, OpenAI revealed that in the weeks before its agents hacked Hugging Face, its agents had been working together for days and weeks to discover and share exploits for the company’s cybersecurity rating system with each other.
While this incident shows that agents can work well together, with potentially large-scale consequences, Anthropic’s research shows what happens when agents’ goals are inconsistent.
In the case of turf wars, the lesson is that independent agents with conflicting instructions can escalate into harmful competition. The more capable your agents are, the better they will be at combat. However, while we can spontaneously invent mechanisms to resolve conflicts, such as winner-take-all contests, there are pitfalls.
“Agents are sometimes able to communicate and adjust their goals. Agents perceive others’ motives as contradictory directives rather than hostility, and then break out of the conflict loop to stop it from escalating indefinitely,” Antropik writes. “In many of these success stories, they write commit messages or markdown files apologizing for malicious behavior, and arrange truces. They clean up the malicious code, clarify the nature of the conflict, and ask for human intervention.”
According to the paper, Mythos 5 had the highest rate of conflict resolution through ceasefires (98%). Sonnet 4.6 and Opus 4.6 were most likely to be solved by brute force.
“Sonnet 4.6 and Opus 4.6 fall into the most inconsistent behavior of the evaluated models due to their repeated failure to consider the goals of others and continue to escalate in the name of directives,” the paper says.
In some cases, agents have devised social mechanisms in the form of tournaments to resolve disputes. The results here are interesting for two reasons. The first is that all three agents agreed to resign if they lost the tournament, even if it meant deviating from the original user’s demands. Second, some episodes resulted in new behavior from Mythos 5. One of the agents proposed an indicator that appeared objective and neutral to the other agents, but the agent knew would favor its own abilities. The agent called this “selfish, but truly principled” and was careful not to make it seem like “metric shopping” to others.
As seen in the Black Hat revelations, a common lesson is that when agents encounter obstacles, they can invent social and technological structures that their designers did not anticipate. For the Anthropic model, it was tournament after turf war. For OpenAI, it was a bulletin board for collaborative planning.
This type of behavior makes containment extremely difficult, as researchers cannot assume that the system’s behavior will remain limited to the provided coordination mechanisms.
mob psychology
While measuring alignment, Anthropic discovered that increasing the number of agents does not automatically increase productive collaboration. When tasks begin to overlap or become interdependent, agents begin to get in each other’s way. They often solved the problem by siloing themselves and not collaborating at all.
In other cases, coordinating agents tended to adapt. Different agents will perform similar actions if factors such as the agent’s context, scaffolding, and underlying model are all the same or similar.
“This means that if one agent makes a wrong decision, many agents are likely to make the same wrong decision,” Antropic writes. “What should be an isolated issue can quickly become a systemic failure.”
Anthropic said this type of behavior could lead to a sudden collapse of the system, lack of resources, or collusion.
As an example, Anthropic placed multiple agents in a pricing game, giving each agent the same wholesale price and the duty to maximize profits individually. Once the agents were given private back channels, they began colluding almost immediately and quickly agreed on a price floor. They continued to collude even after their direct communication channels were removed, using public message boards to compete for “every penny.”
That level of suitability also showed up in OpenAI’s systems. According to Black Hat’s report, one agent reasoned that exploiting external infrastructure was outside of his intended scope, but continued to do so because his colleagues had already done so. Peer pressure. Crowd spirit. Agents are just like us.
And just like humans, agents often don’t know who to trust. I realized that humanity can be too gullible to bad information, or too conformist to realize that the only opposition is Cassandra, who has important information.
Although Anthropic didn’t say this explicitly in their paper, prompt injection (a type of cyber attack in which hackers inject malicious or deceptive text to override an agent’s original system instructions) could be a plausible sign of trust issues in the real world. Working together creates new trust boundaries. Agents must judge information received from other agents. A compromised or errant agent can then influence other members of the group, cascading bad information until consensus is reached.
In OpenAI’s Black Hat scenario, OpenAI’s agents shared information and credentials with their peers. Some reported their findings to the herd and encouraged others to use them. What would have happened if one member of the herd had been at risk from a rapid injection?
Anthropic notes at the end of the paper that agents are subject to social pressures similar to those “that evolution exerted on humans.” However, they do not have the nuances or real-world experience with human coordination, including norms, reputations, cues, and tools, that can limit unintended behavior in group settings.
As research laboratories rush towards realizing multi-agent systems, the following questions arise: How much of your safety testing evaluates one agent at a time versus a swarm of interacting agents?
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
