Leading AI firm Anthropic has warned that humans risk losing control over AI systems in the very near future if a concerning trend with its Claude model continues.
The $1 trillion startup revealed that Claude now writes more than 80 per cent of its own code – up from less than 10 per cent in February last year.
Anthropic claims that the Claude-written code is currently roughly at parity to human-written code, and is expected to be much better within the next year.
“That trend points to an AI system capable of fully autonomously designing and developing its own successor,” Anthropic’s Marina Favaro and Jack Clark wrote in a blog post on Thursday.
“AI that can build itself would be a major development in the history of technology – one that could bring enormous good for the world in science, healthcare, and beyond.
“But full recursive self-improvement also might increase the risks of humans losing control over AI systems.”
Anthropic noted that, until recently, artificial intelligence advances have relied entirely on human engineers and developers. But the company is now increasingly delegating the development of its AI to the AI systems themselves in an effort to speed up progress.
These advances are putting Claude on a path towards “recursive self-improvement”, which could see it improve itself without human intervention.
Anthropic added that it would “likely be a good thing” to effectively slow the development of the technology in order to give more time to deal with the potential implications.
A slowdown or pause of AI development would require broad industry consensus, as well as political cooperation amid intense geopolitical pressures and rivalries.
“A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions,” Anthropic’s post stated.
“It would also require that each can verify that the others have actually stopped… None of this is necessarily impossible in principle – the world has built verification regimes for other complex technologies – but those regimes took decades to build both the infrastructure and the trust. We don’t have that long.”
Anthropic said it plans to work with policymakers, researchers and other AI companies in the coming months in an effort to prepare for the arrival of self-improving AI systems.
It is not the first time there has been such a call to pause AI development, with thousands of technologists calling for a six month moratorium on AI training in 2023 following the release of ChatGPT.
The open letter, organised by the non-profit Future of Life Institutes, cited the danger of society-wide loss of control of AI, though no pause was achieved.