AI agents still need humans for real-world tasks

The next time someone’s powerful AI agent tries to call you to negotiate your cable bill, chances are it’s a real person negotiating customer service, not software.

That’s because our AI agents leverage state-of-the-art underlying models developed by companies such as: meta and OpenAIit is not yet possible to reliably perform actions on the web, especially actions that involve a phone call. Marissa Mayer, former CEO of Yahoo and now at the helm of agent startup Dazzle AI, says that while current top-level AI models are great at delivering online home improvement services to consumers, they can reach their limits when you need to book and schedule rooftop upgrades.

“We rely on the Frontier model to give us a good proposal and a good shortlist, but we check it thoroughly to make sure it’s really providing the kind of service we can support,” Mayer said, noting how Dazzle focuses certain tasks on human contractors.

Digital assistants powered by artificial intelligence are currently gaining renewed interest thanks to Meta, OpenAI, and others. These technologies are touted as capable of tasks such as finding shoes on the Internet.

Meta’s Muse, OpenAI’s Dots, and a host of other agents can accomplish many web-browsing feats, but the technology is still in its infancy and often hits digital roadblocks. Meghan Joyce, CEO of digital agent startup Duckbill, said these barriers typically include tasks like making a phone call to an AI agent, scheduling a doctor’s appointment on someone’s behalf, or booking a local cleaning service to clean a sofa.

Joyce said humans are still better than AI when it comes to tasks that require real-world judgment, and companies developing digital assistants for consumers will need to patch in real humans to take over. That’s why Duckbill recently debuted an application programming interface (API) aimed at companies building personal agents to connect with human contractors.

“We are working on a model that looks like this: UberThat’s where you boil down tasks into atomic bite-sized pieces,” Joyce said, referring to the ride-hailing company he used to work for.

Companies that create agents can use Duckbill’s API to delegate tasks that agents are likely to fail at. Duckbill turns these tasks into bite-sized tasks that the average adult can perform consistently and reliably.

“And we’ll follow up on what happened,” Joyce said.

Mayer said AI models may still lack common sense. She said one Dazzle AI super user tried to have the technology handle a series of home improvement projects at the same time, including replacing a roof, breaking a window, and buying a glass table top.

But Mayer said the AI ​​agent confused the window glass with the tabletop glass, which caused the digital assistant to “try to figure out if the tabletop could fit into the window.”

Mayer said the incident didn’t last long because Dazzle’s technology flagged the work and referred it to a person to complete.

“A human would have resolved the situation within seconds, whereas an agent would have burned a ridiculous number of tokens,” Mayer said. “Or maybe we just didn’t come up with a good solution.”

Shivani Poddar, CEO of personal AI agent startup Wajo AI, is also incorporating human workers to answer phone calls and handle CAPTCHA tests to prevent bots from coming through the site.

“I think we need AI and humans, collaborative AI, to actually get the job done,” Poddar said, stressing that people will stop using technology if agents can’t complete the task.

Interactions between chatbots and agents can be very personal, and users have good reason to be concerned about human intervention, especially if the company doesn’t explain how it works.

Meta employees previously raised privacy concerns about internal tests that used human contractors to handle calls on behalf of Muse AI agents. A Meta spokesperson said in a statement at the time that this type of testing is “core to our product development process.”

Meta currently does not use humans to complete your Muse requests.

Duckbill’s Mr. Joyce said the company does “full background checks” on the 1,099 contractors it uses and compares them to Uber and DoorDash drivers, “people who live here in the United States and are looking for an additional source of income.” Duckbill also typically employs contract workers from companies that provide contact center labor to businesses.

Meyer said Dazzle conducts background checks and looks for contractors with the same qualities it looks for in human personal assistants.

“We generally screen people who we think are very careful and have good taste and judgment, both in terms of how they interact with users and just their ability to forget and compartmentalize that information later on,” Mayer said. “I understand that it’s a privilege, but I’ve had a personal assistant for a while and I know how I want them and I know how to coach them.”

Mayer said that while the underlying models are still being improved, the market is in a “transitional period” until the technology can perform more complex web tasks and phone calls as efficiently as humans.

“We’re not even trying to hide it. We see human involvement as an advantage,” Mayer said.

Attention: Who is responsible if an AI agent breaks the rules?