Microsoft built a fake marketplace to test its AI agent – and it failed in a surprising way

On Wednesday, Microsoft researchers released a new simulation environment designed to test AI agents, along with new research showing that current agent models may be vulnerable to manipulation. The study, conducted in collaboration with Arizona State University, raises new questions about how well AI agents perform when working without supervision, and how quickly AI companies can realize the promise of their future.

The simulation environment, named “Magentic Marketplace” by Microsoft, is built as a synthesis platform for experimenting with AI agent behavior. In a typical experiment, a customer agent might try to order dinner according to a user’s instructions, while agents representing different restaurants compete to get the order.

The team’s first experiment involved 100 individual customer-side agents interacting with 300 business-side agents. Because the Marketplace source code is open source, it is easy for other groups to adapt the code to run new experiments and reproduce the results.

Ece Kamar, managing director of Microsoft Research’s AI Frontiers Lab, said this type of research will be important for understanding the capabilities of AI agents. “There are real questions about how the world changes when these agents work together and talk to each other and negotiate with each other,” Kamal said. “We want to understand these things deeply.”

In our initial research, we investigated a combination of key models, including GPT-4o, GPT-5, and Gemini-2.5-Flash, and discovered some surprising weaknesses. Specifically, researchers have discovered several techniques that companies can use to manipulate customer agents into purchasing their products. Researchers found that efficiency decreased, especially as customer agents had more options to choose from and vast amounts of agent attention space.

“We want these agents to help us work through a lot of options,” Comer says. “And we find that the current model is actually overwhelmed by too many options.”

Agents also encountered problems when asked to work together toward a common goal. Apparently, they didn’t know which agent should play what role in the collaboration. Although giving the model clearer instructions on how to collaborate improved performance, the researchers believed that the model’s unique features still needed improvement.

“You can instruct a model step-by-step, just like you would teach a model,” Comer says. “But if you’re essentially testing collaborative features, you would expect these models to have those features by default.”