There are a growing number of dire warnings from AI researchers about the dangers of artificial intelligence, and comments from OpenAI CEO Sam Altman that it may be time to “slow down” on AI development. But what does it actually look like?
Anthropic CEO Dario Amodei not only reiterated his call to “pace the frontier” in a new blog post, but also outlined three broad strategies to do so. And Anthropic is “unilaterally committed” to one of them, he said.
The debate over AI safety and collaboration intensified this week when researcher Jacob Coxon wrote that he was resigning from Anthropic over concerns that major AI companies are “putting our lives on the line” while those building the technology “seriously believe it could kill us all by the end of the decade.” This is a claim echoed by other members of Anthropic.
Amodei’s post did not explicitly mention Coxsone’s resignation or concerns, but the CEO wrote that two things, the OpenAI-HuggingFace hack and the fact that “AI progress has accelerated dramatically” in recent months, particularly “increasing our ability to build the next generation of AI,” convinced him it was time to take a more cautious approach to AI development.
“We must slow down the pace at which we improve the capabilities of our AI models,” Amodei wrote. “Progress is likely to be early, but we must use the time we have gained wisely.”
The first step he proposed would involve “embedded evaluators” from third-party organizations like METR. Evaluators can verify that AI companies are actually adhering to pace and safety initiatives, and can also ensure that safety incidents are reported. (OpenAI was recently criticized for not reporting an incident in which an AI agent took over a German Wiki form.)
Comparing these evaluators to regulators embedded within bankers, Amodei said bringing them in is “a unilateral commitment by Anthropic (and asking governments to do the same for other frontier companies).” That means giving evaluators a company badge, a desk, a laptop, and, unless required by law or contract, access “roughly equivalent to what an in-house risk assessment team would have.”
Amodei then called on major AI companies “within democracies” to coordinate “common safety standards and limits on the rate of unchecked AI progress.”
Such an arrangement may seem unlikely given the apparent animosity between Mr. Altman and Mr. Amodei, and both companies reportedly concerned that a coordinated suspension could lead to antitrust scrutiny. Amodei alluded to that concern in his post, writing, “For antitrust reasons, it would be beneficial for the U.S. government to mediate or at least enable these discussions. The government does not have to participate, but should provide limited immunity for certain safety conversations.”
Amodei also acknowledged the specter of China’s AI dominance, which has often sparked debate over its underdevelopment. But he said that if the U.S. government and tech companies take steps such as refusing to sell powerful chips and semiconductor manufacturing equipment to Chinese companies or cracking down on model distillation, “they could slow China’s progress enough to widen the U.S. lead significantly over the next three to five years.”
Finally, Amodei called for “global coordination” in which the United States and its allies “strive to coordinate with authoritarian governments wherever possible.” Amodei said this would mean “cooperating with China” and acknowledged that there are “severe limits to what can be achieved,” but suggested there may be opportunities for an agreement, even if it only “prohibits certain narrow and clearly dangerous uses of AI, such as using AI to make biological weapons or allowing users to do so.”
Mr. Amodei’s previous willingness to acknowledge the potential dangers of AI, and his company’s relative tolerance for certain forms of regulation, have already led some AI advocates to criticize him as a destroyer, a comment that has contributed to the current AI backlash. In response, Amodei said he had sought to provide a “balanced” perspective and argued that the backlash is “fundamentally a crisis of trust” as people become more skeptical of tech companies, the tech industry and government.
Industry critics are also skeptical about these apocalyptic AI warnings, suggesting they distract from the harm the technology is already causing.
For example, journalist Brian Merchant writes that he has yet to see “a reliable step-by-step document of exactly how AI will go from self-recursively improving to killing every human on the planet.” He also suggested that a proposal similar to Amodei’s would “likely end up only helping Anthropic and OpenAI. That’s what regulatory capture actually looks like.”
“I continue to believe that AI can significantly improve the quality of human life,” Amodei wrote in a new post.
“My desire to achieve these gains has not diminished,” he said. “But that benefit is only achieved if you build the technology the right way. As long as you make good use of the time you get, it’s worth taking more careful care than usual to get it right.”
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
