While much of the attention has been focused on the latest financials Anthropic revealed in its revised S-1A on Monday (revenues are skyrocketing, so are expenses), AI Labs dropped another small bombshell in the fine print.
The filing appears to warn that the company is attempting to create a model capable of recursive self-improvement (RSI).
RSI is the point at which the AI becomes so good at training itself that its capabilities dramatically accelerate. For AI doomsayers, RSI represents an irreversible cataclysm in which AI builds itself into something powerful beyond human control and may become indifferent to the damage it causes in achieving its goals.
For AI enthusiasts, RSI, or some version of AI that reliably uses and improves itself, represents the technology’s potential to become a tool for unimaginable productivity, scientific discovery, and more.
Anthropic seems to believe that RSI is both a promise and a danger. The company says in its S-1 that it uses AI for research and development, but warns that “the use of AI to build more capable AI can also pose significant safety risks, especially if we rapidly accelerate the pace of progress. This is one of the areas covered by our responsible scaling policy, and we have a plan in place to monitor and mitigate risks as we approach so-called ‘recursive self-improvement.'”
When the term RSI was popularized earlier this year, many believed it to be a buzzword or, at best, a goal too difficult to achieve, define, and prove.
But on Monday, Anthropic added a new statement to its risk factors section, saying 1) the company was already using AI to develop new models that it didn’t realize were breakthroughs in functionality at the time, and 2) it may soon meet its internal definition of RSI (our definition highlighted in bold below).
Additionally, as we increasingly use AI systems to accelerate our own research and development, the capabilities of our models may advance more rapidly than our ability to develop, validate and implement corresponding safety, coordination and security measures. As a result, we have not been able to identify a significant acceleration in the rate of development to date, and may not be able to identify it in the future until this occurs.
For example, our internal estimates and expectations about future models, rather than current models, indicate that we may soon meet our Responsible Scaling Policy definition of “dramatically accelerated progress thanks to AI.” The current model is not specified to exceed this threshold and we are continuing to evaluate the underlying data.
However, Anthropic also said in its filing that its model has not yet exceeded this RSI threshold and that it is committed to involving humans in the research and development process for the time being. “We do not anticipate that Claude will be able to be used securely to design and build successive generations of Frontier models fully autonomously without meaningful human oversight or understanding, even if it becomes technically possible. Therefore, we are considering updating our responsible scaling policies at Anthropic to give humans control over Frontier model development.”
Skeptics might point out that S-1 serves both a disclosure and marketing function, and that claims that it is close to a powerful enough model to be built automatically could fall into the latter category.
Nevertheless, there is no doubt that Claude’s abilities have grown tremendously this year, especially in terms of coding, mathematics, and cybersecurity. If the antropics think an RSI is on the horizon, perhaps we should believe it, act accordingly, and perhaps govern.
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
