A big trend in robotics is passing keys to generative AI models, but that comes with problems. Its architecture is not predictable unlike traditional algorithms. How can you be sure your brand new humanoid is safe?
Dr. Ding Zhao, director of the Safe AI Lab at Carnegie Mellon University, has been working on this problem for almost his entire career. Now, he and veteran startup executive Kyle Wong and machine learning engineer Simo Rashidi have founded Safeworld, a company aimed at solving just that.
“The safety challenge we’re talking about is a combination of probabilistic evaluation of very advanced generative AI. How do you take on the risks of a probabilistic system?” Zhao says. “The second part that’s really difficult is the trust part, and you need both parts to deploy a robot.”
Safeworld is emerging from stealth today with a more than $12 million seed round led by Shine Capital and a16z Speedrun, with additional investment from Box Group, Carnegie Mellon University Endowment, Innovation Endeavors, and SV Angel.
“As robots are designed and deployed, now is the time to create industry safety standards,” a16z Speedrun partner Jonathan Lai told TechCurnch. “By the time a robot collides with a child or causes a safety incident in the home, it is already too late.”
Safeworld’s specialty is evaluating robot control systems in simulations using realistic human body models. This is similar to the challenge faced by companies like Tesla and Wayve, who need to ensure their vehicles can respond appropriately to a variety of unexpected events that may be encountered on the road. But that will be more difficult for robots, Zhao argues, because they work in unstructured environments and each facility they’re in has different safety standards.
“One of the most common areas is whether there are blind spots in this particular factory,” Wong says. “What speed or stopping distance does this robot need to avoid colliding with a particular human? For example, if a human is carrying a box, will the robot detect the human?”
To answer that question, Safeworld builds a digital version of its corner with models like Genesis and MuJoCo, inserts simulations of the robot under evaluation driven by real software, and runs thousands of scenarios in which the human model encounters the robot. According to Zhao, this is harder than it seems because people are unpredictable.
“Tripping and falling is another good example of something we test over and over again in simulation,” Wong said. “Otherwise you’re going to go and stumble and end up falling in love with robots. It’s like a difficult thing to do all the time.”
There are clear similarities between the platform that Safeworld is building and the tools that robot builders use internally. But the founders believe that beyond specific expertise, robot makers want third parties to verify their work, if only to share information about safety practices among competitors.
“A lot of people underestimate how difficult some of these edge cases are to solve,” Chao said. “What we’re concerned about is not robots in a vacuum or in demonstrations. It’s robots being deployed at scale with people who may have never operated a robot before.”
Vishal Dugar, CTO of Gritt Robotics, is currently developing AI brains for robots that aim to help workers install solar panels at industrial-scale solar power plants and take on more complex construction tasks. His company partners with Safeworld to develop safety simulations.
“The challenge with most of our systems is that it’s very difficult to do the math, write the equations, and formally prove that the system is confirmed to be secure,” Dugar says. “It must necessarily be done empirically.”
His robots work alongside human workers, and it’s clear that keeping the robot arm from hitting humans is a top priority. To actually verify it, you need to consider all sorts of potential scenarios.
“Humans come in many different looks,” Dugar points out. “Their bodies can take on different shapes. They can be kneeling, they can be standing. They can potentially trip and fall. They can crouch, they can run. You have to account for all these behaviors that humans might exhibit in these places, and the different variations in human appearance, you know, clothing, size, shape, height, skin color, and everything else.”
Both Safeworld and generative AI in robotics are still in their early stages, and the company is still figuring out the best model for its product (a platform for external users or a service-based approach?), but the team is confident they’re addressing the right problem.
“We are probably the first company in this field to make a profit,” Zaho said. “Because if someone wants to deploy, they’re going to have to pay us to deal with the situation.”
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
