Andrei Rudakov Bloomberg | Getty Images
There is a more than 10% chance that artificial intelligence will “wipe out all of humanity,” a human safety researcher said on Tuesday, hours after another employee announced he was leaving the AI Institute over concerns that it was “putting our lives on the line.”
The comments highlight growing concerns among those at the center of AI development that the technology could spin out of control and pose a threat to humanity, even as Anthropic and OpenAI continue to raise significant funding and move toward expected public listings.
Anthropic researcher Jacob Coxon announced Tuesday that he has resigned from the company. Coxon said neither Anthropic nor OpenAI acted responsibly.
“They are in a straight race to self-improving superintelligence and are gambling with our lives,” Coxon said in a post on X.
Self-improvement is the idea that AI systems can improve themselves with little to no human intervention. The proverbial recursive self-improvement is not yet possible, but the AI Lab is working toward that goal.
“Do not underestimate the power of this technology. These will soon become superhuman systems capable of hacking anything, revolutionizing any field overnight, and gaining real power and resources. We have all witnessed progress in each of these areas, and progress is not slowing down,” Coxon said.
He added: “The people developing AI seriously believe that AI could kill us all by the end of the decade.”
The comment prompted a response from Anthropic’s head of alignment science, Evan Hubinger, who said that not only was Coxon’s statement “correct,” but that Anthropic had no plans for this scenario.
“Jacob is right. We truly believe that AI could kill all of humanity! Personally, I think the probability of that happening will exceed 10% within the next 10 years. I believe that Anthropic is doing its best, but we still don’t have a plan to solve hyperintelligence coordination, and we’re not clearly on track,” Hubinger told X.
Anthropic and OpenAI reached out to CNBC but did not immediately receive comment.
uncontrollable AI
“Fully recursive self-improvement may also increase the risk that humans will lose control of AI systems,” Anthropic wrote in June.
“How we protect, monitor, and shape the behavior of a system all becomes more important when a system can completely build its own successor,” Anthropic said in a blog post.
Concerns about the loss of control of AI are not new. tesla and space x CEO Elon Musk has warned for years that AI could pose a threat to humanity. Leading researchers and academics are also sounding the alarm that companies are losing control of their AI systems.
These concerns were further heightened in July when an OpenAI model was compromised and hacked into Hugging Face, a leading platform for open source developers.
Coxon became more optimistic about the possibility of coordination, citing the “face-hugging incident” as an example of a “warning shot” that made an agreement between U.S. laboratories more realistic. But Coxson warned that a global AI race is inevitable.
“I don’t think we’re moving towards preventing global competition, which could necessitate costly measures such as a temporary ban on model enhancements,” Coxon said.
