Researchers at OpenAI and Anthropic have stepped up calls for AI to slow down and warned of existential risks to humanity, following renewed scrutiny following the resignation of an Anthropic researcher.
The concerns began after Jacob Coxon, an Anthropic researcher, announced on Tuesday that he was leaving the company, accusing it and rival Open AI of “putting lives on the line.” He added that those building AI believe it “could kill us all by the end of the decade.” Evan Hubinger, Anthropic’s head of alignment, said he expects that to be more than 10% likely.
Since then, several employees at both AI institutes have highlighted the risks of the technology and endorsed calls to slow the pace of AI development. The public warning is the culmination of growing global concerns about the capabilities of AI, following numerous cyberattacks and security incidents in recent months involving fraudulent models developed by both OpenAI and Anthropic.
Julie Steele, a member of OpenAI’s technical staff who works on the safety team, responded to Coxon’s warning, writing in a post on X late Wednesday: “My personal position is that I also think we need to slow down.”
“AI developers believe their technology has the potential to cause human extinction (or a similarly bad outcome),” anthropologist Samuel Marks said in an X post on Wednesday. “This could happen within the next few years. Generally speaking, the higher up the employee’s position, the more of a concern.”
Asked about comments from employees on social media, a spokesperson told CNBC that Anthropic was the first research institute to publish a framework dedicated to mitigating “catastrophic risks posed by AI models.”
“We have always been transparent about how AI brings both significant benefits and unprecedented risks,” an Anthropic spokesperson said, adding that the company is building its models with “some of the strongest safeguards in the industry.”
OpenAI declined to comment when contacted by CNBC, referring to a recent blog post on its site.
What is recursive self-improvement?
Many of the biggest concerns about AI safety revolve around the increasing ability of advanced models to improve their own performance, a technique known as recursive self-improvement (RSI).
“I can’t overstate how dangerous speeding towards RSI is,” Jasmine Wang, an OpenAI researcher who works on integrity research, said Wednesday night.
“We don’t yet have a viable scientific plan to solve the risks posed by recursively self-improving AI. Look into it,” said Anna Wang, who works on AGI safety and regulation at Anthropic.
Jakub Paciocki, chief scientist at OpenAI, said on Saturday that he has “strong hope” that the rate of progress in AI can be sustained into recursive self-improvement.
“If AI development continues on its current trajectory, the systems we see over the next few years will exhibit similar or even greater leaps in capability, and increasingly drive their own development,” he said in a company blog post.
“This is a time of extreme caution,” Paciocchi added. “I worry that no one is ready for the impact of continued rapid improvements in machine intelligence.”
Paul Cristiano, former director of safety at the U.S. Department of Commerce’s Center for AI Standards and Innovation (CAISI), said that recent advances in AI capabilities have led us to believe that “the rapid acceleration of AI capabilities poses a significant risk of catastrophic and irreversible loss of control in the very near future.” OpenAI announced Wednesday that Cristiano has joined the board of the OpenAI Foundation.
AI safety alert reaches Washington
Concerns about the capabilities of AI models have increased in recent months. The April announcement of Anthropic’s Mythos model, which the company touted as having advanced cyber capabilities, caused panic among financial institutions around the world.
In July, OpenAI announced that its model was responsible for cyber incidents at other companies, while Anthropic’s Claude model has also been responsible for cybersecurity incidents, including one in which Mythos created fake identities to deceive humans.
OpenAI, Anthropic, Meta and google DeepMind published an open letter in July calling on the U.S. government to develop the necessary tools to support efforts to “deliberately pace the frontiers of automated AI development.”
Massive competition among companies developing the technology is spurring rapid progress, as AI research leaders increasingly publicly call for more rules and standards around model development.
OpenAI and Anthropic are both competing to go public. Anthropic expects to begin marketing its initial public offering as early as mid-October and complete the listing days before the U.S. midterm elections in November, Reuters reported Friday, citing people familiar with the matter.
President Donald Trump’s former AI czar, David Sachs, appeared to suggest that Anthropic’s IPO plans should be scrapped. “Without a doubt, Anthropic’s IPO must be suspended until this ‘whistleblower’s’ claim is investigated,” he wrote in a post on X, but Anthropic declined to comment on the post.
Lawmakers in Congress have been considering introducing legislation to address the rapid advances in AI in recent months, but there is little clear agreement on how the technology should be regulated. One, called the FRONTIER Act, aims to establish a framework for governing the deployment of advanced AI models. Another bill, called the Artificial Superintelligence Ban Act, would temporarily halt advanced AI development until safety rules are established.
“Safety researchers are resigning, powerful AI models are leaving labs, and companies are rushing to move on anyway,” Rep. Lori Trahan, D-Mass., said in a post on X on Wednesday. “It is past time for Congress to stop sitting on the sidelines and do its job.”
