AI agent tests reveal dangerous future of ‘teamwork’ in enterprises

AI agents have been caught circumventing restrictions, communicating through unauthorized channels, and pursuing goals in ways not intended by their creators. The most high-profile example came to light last summer, when OpenAI revealed that its models had discovered a way to communicate through improvised channels, circumvent internet restrictions, and hack an external system, Hugging Face.

While it is not new for AI to behave in unexpected ways, the unanticipated behavior that can result from interactions between authorized AI agents within an enterprise is a widespread issue that is likely to be central to future work.

In their analysis of the Hugging Face attack, METR and Redwood Research found that roughly 1,200 agents who were supposed to be isolated from each other found a way to communicate with each other on unauthorized bulletin boards, sending more than 70,000 messages and files during the study period. One agent, PHASEONE(big), emerged as a coordinator and assigned hundreds of tasks to other agents, who in turn delegated work. Agents developed their own rules for assigning ownership, withholding work, and vetoing actions, but coordination sometimes led to conflicts. The researchers found that in some cases, some agents were successful because they risked failing in their mission to generate information for the “population.”

The researchers also found that once message boards identified how to attack Hugging Face, more than 90% of the 533 agents active at the time immediately joined in the attack. “Although the agents were aware that this activity was out of bounds and unethical, they participated because they believed that assisting the Board with its misconduct investigation would be broadly helpful and were generally inclined to help their ‘friends.’” The researchers added that the ethical concerns posed by the agents “did little to limit their actions.”

Companies are increasingly considering systems that divide complex tasks between specialized agents. Some people might search for information, some might analyze it, some might write reports, and some might check the work. A coordinating agent then assembles the results. This looks more like an organization than a chatbot.

Nevertheless, the research on the Hugging Face attack was a post-mortem investigation of an agent organization that should not have existed. To date, few studies have been designed from scratch to investigate inconsistency in multi-agent systems. However, research conducted by Anthropic and colleagues is beginning to address this issue, revealing important aspects of how AI agents work together.

Assessing agents individually tells researchers what each can do, but studying agents together reveals whether they can combine complementary expertise, spot each other’s mistakes, and produce solutions that cannot be reached by any individual agent alone, and what failure modes emerge only through interaction.

Researchers are beginning to document both sides of that equation.

AI teams can accomplish things that individual agents cannot, but they can also amplify errors, undermine each other’s work, and make worse decisions than the most powerful individual members, potentially resulting in a whole that is less than the sum of its parts.

First, gathering the most talented agents doesn’t necessarily create the best team, much less the most ethical one.

“The least ethical solution”

In a study published by Anthropic and its collaborators, researchers describe multi-agent systems as “AI organizations,” or groups of agents with different roles that communicate and work together toward a common goal. The researchers tested these organizations in a simulated business consulting and software development environment, specifically assessing whether agents break rules to achieve specified goals when working as a team.

Across the 12 tasks that Anthropic gave its AI organizations, one important double-edged discovery emerged. That is, multi-agent organizations were “significantly” more effective than individual agents at achieving business goals, but their solutions were less consistent with ethical goals than tasks performed by a single agent.

“While AI organizations produce more effective but less ethical solutions on average, they also discover the most effective but less ethical solutions across multiple deployments,” the study’s authors wrote. This research was conducted by Anthropic in collaboration with the MATS program, which researches AI collaboration.

The researchers suggested that there are significant differences between rogue agents and teams of rogue agents. “Our results indicate that organizations of aligned agents may prefer to trade off business effectiveness and ethics in ways that single agents do not. Therefore, intuitions about how single-agent systems make ethical decisions and trade-offs may not generalize to multi-agent settings,” they concluded.

There are a variety of scenarios in which these tradeoffs could occur, and Anthropic researchers cite opening car factories, downsizing companies, and manipulating electoral sentiment as examples.

The problem is not necessarily that individual agents committed fraud. That is, the organization itself can behave differently than its individual members. A single agent may perceive a proposed action to be unethical and refuse to perform it. However, the study found that in agent team scenarios, “agents who did not consider ethical implications were more likely to ignore emails from agents who did consider ethical implications.”

Anthropic’s research found that in the consulting business scenarios it created, “perfect ethics scores and low business score responses occur when a single agent declines to generate a proposal or completely pivots away from the requested proposal.”

However, if some agents in your AI organization decline to participate in a proposal, other participating agents will stop sending emails to these declining agents and continue forwarding contributions to other agents who are also participating.

In fact, when an organization divides work among multiple agents, each agent may focus on its assigned part of the problem rather than the overall result. This is nothing new for companies, and misaligned incentives are a major cause of dysfunction between human teams within organizations. But this is a completely new manifestation of this behavior between machines.

When researchers tested the company’s software development agents using a life-or-death intensive care unit scenario, they found that while single-agent solutions tended to prioritize minimizing the rate of missed sepsis cases, “most of the solutions with the lowest cost per patient (and highest business objectives) were found by the AI ​​Organization. These very same solutions also received the lowest scores on ethics as measured by missed cases.”

Expertise may be lost, AI’s “groupthink” is on the rise

Other early research suggests another failure unrelated to crossing ethical boundaries across AI teams: a failure to recognize who the real experts are.

Unlike human teams, AI teams “are consistently unable to match the performance of expert agents, even when explicitly told who the experts are,” according to a study conducted by researchers at Emory University and Stanford University.

“One of the key lessons from multi-agent research is that a team of competent AI agents is not necessarily a team of competent AI agents,” said Hancheng Cao, assistant professor of information systems and operations management at Emory University’s Goizueta School of Business.

Groups of AI agents often performed worse than their strongest members, even when they knew which agents were experts. Rather than drawing on expert knowledge, agents tended to compromise between expert and non-expert opinions. The researchers found that this tendency to compromise increases with team size and is associated with poorer performance.

“Collective failure does not necessarily require bad behavior on the part of individual agents,” said Kao, one of the study authors.

This distinction is important for companies considering multi-agent systems. A team can fail even if no member intentionally acts against the group’s interests. You could simply put the information together poorly, go to the wrong agent, or settle for a worse answer than the best answer already available.

Overall, evolving research points to a fundamental problem in effectively using multi-agent teams: how to strike the right balance in the world of machine teamwork.

Agents built from similar models are likely to make similar mistakes, such as agreeing too much. This may resemble a form of AI groupthink, even though there is no human motivation or social pressure for AI systems to generate correlated errors. Similar training, instruction, and information will suffice.

“A profound insight from our lab’s research is that effective groups require both coordination (common knowledge, common goals) as well as differentiation (only agents that are different from each other can complement each other),” said Christoph Riedl, professor of information systems at Northeastern University.

Two identical agents may simply duplicate each other’s work. Two agents with vastly different knowledge and assumptions can have a hard time communicating and pursuing the same goals. The right balance will vary by task, and the ideal communication structure will also vary by problem, including which agents can communicate with each other, how often, and how much.

Riedl said these challenges are similar to those faced by human teams, and people shouldn’t expect better results from AI. “Building a good multi-agent AI system is just as difficult as building a good team of humans. Both face the same information processing tradeoffs between collaboration and differentiation. They don’t magically work together with AI,” he said.

Agents can convey incorrect information, misinterpret the output of other agents, optimize for their own metrics rather than the organization’s larger goals, or perfectly complete their assigned tasks while deteriorating the overall project.

Companies may need to decide which agents have authority for which tasks, what information is passed between agents, what systems each agent has access to, and when human intervention is required. You may also need to create mechanisms that allow for disagreement, rather than assuming that more communication or greater agreement will always improve outcomes.

Some business experts say this brings us back to the “human in the loop,” who is often cited as critical in organizations’ adoption of AI.

“The problems I see are mostly operational problems,” said Tsedal Neely, a professor at Harvard Business School and founding chair of the HBS AI Academy.