Concerns about AI self-improvement raise ‘existential’ concerns at Anthropic, OpenAI

This report is from this week’s The Tech Download newsletter. Is it what you see? You can subscribe here.

Concerns about the safety of AI systems and their potential to wipe out humanity gained new viral momentum this week.

Evan Hubinger, head of alignment at Anthropic, told X that he believes there is a more than 10% chance that AI will kill all humans within the next 10 years, after a colleague resigned over safety concerns.

Further warnings have since been issued by researchers at both Anthropic and OpenAI. It sparks a social media frenzy.

But in a reply to Mr. Hubinger’s own post, it became clear where exactly his concerns lay.

“What I’m concerned about is superintelligence arising from recursive self-improvement. As we say, it’s happening sooner than we think,” he said.

Recursive self-improvement (RSI) is when AI itself helps improve the process of building new models, potentially leading to a spiral of capabilities such as better systems building better systems.

The concern is that as AI gains control over how new models are trained, the very humans who built those systems in the first place could lose control.

Both OpenAI and Anthropic have said in recent months that the autonomous model is improving faster than expected.

“Our internal data indicates that Claude is accelerating AI development, a path that could lead to recursive self-improvement, or the AI ​​autonomously building more capable successors,” Anthropic posted on X in June. “It’s happening faster than we thought, and its impact deserves further attention.”

Although AI has not yet reached the RSI stage, the development of AI systems is already accelerating. Anthropic said in an August blog post about RSI that its engineers are shipping an average of eight times more code per quarter between 2021 and 2025.

“AI is already at a level where it can introduce some new ideas,” Vincent Konitzer, a computer science professor at Carnegie Mellon University, told me. “Therefore, it is very difficult to predict at what point this process will begin to significantly accelerate AI capabilities.”

Jakub Paciocki, chief scientist at OpenAI, said on Saturday that he was concerned that “no one is prepared for the impact of continued rapid advances in machine intelligence.”

“If AI development continues on its current trajectory, the systems we will see over the next few years will exhibit similar or even greater leaps in capability, and will increasingly drive their own development,” he said in a company blog post.

This week, following the sudden resignation of Jacob Coxon, social media was flooded with warnings about RSI from researchers at both leading institutes.

“I can’t overstate how dangerous speeding towards RSI is,” Jasmine Wang, an OpenAI researcher who works on integrity research, said Wednesday night.

“We don’t yet have a viable scientific plan to solve the risks posed by recursively self-improving AI. Look into it,” said Anna Wang, who works on AGI safety and regulation at Anthropic.

Anthropic concluded the RSI blog post with three possible scenarios.

In one scenario, progress at the frontier stagnates and AI capabilities diffuse widely. Antropic said it believes that is unlikely.

The second possibility is that AI labs will continue to make profits under human control and change the way the world works. Mr Anthropic said this was “likely”.

However, in other scenarios, AI systems could be capable of fully recursive self-improvement, with the role of humans “significantly reduced in their development.”

In this future, “the thing we are least certain about is how the coordination problem (the challenge of ensuring that AI pursues goals consistent with humans) will or will not be resolved.”

Before Aidan Gomez took the top job at AI startup Cohere, he was one of the co-authors of the 2017 research paper “Attending Is All You Need,” also known as the Transformer paper.

This breakthrough became the basis for technologies such as ChatGPT, Claude, Gemini, and nearly every major large-scale language model in use today.

Cohere, which develops specialized AI models and applications for business, aims to stand out in the industry by positioning itself as a non-US and non-Chinese player capable of delivering “sovereign” AI.

As companies become increasingly concerned about who has access to their data, where that data is processed, and what that ultimately means for their business, Cohere offers a different perspective.

Throughout the conversation, Gomez discussed some of the biggest topics in AI, from cybersecurity challenges to China.

Some of the AI ​​models are “the most powerful cyberweapons ever created,” Gomez said. And when it comes to AI models from China, Gomez said, U.S. lab leadership is “evaporating very quickly.”

I hope you enjoy the episode.

— Arjun Kharpal, Senior Technical Correspondent