Amodei proposed a three-point plan
Dario Amodei, the head of AI company Anthropic, has called for the pace of artificial intelligence development to slow down and be more closely monitored.
In an online essay, Amodei said developing AI was not in question, but warned that the risks associated with the technology were “serious”.
He argued that companies and governments needed time to address those risks as AI models become increasingly capable.
Both Sam Altman and Elon Musk have voiced support for the proposal.
Amodei’s comments come amid growing concerns about the potential risks posed by increasingly powerful AI systems.
AI researcher Jacob Coxon told BBC’s Laura Kuenssberg that people working at AI companies were “genuinely frightened… genuinely concerned about the fate of humanity in the next two years”.
He added: “It’s not at all an exaggeration to say that the people who are involved with both founding these companies and building the tech believe there is a possibility of human extinction.”
In his essay, titled We Must Pace the Frontier, Amodei proposed a three-point plan involving independent monitoring of AI models as they are developed, industry-wide regulation and global regulation.
His proposal received support from some competitors, including calls for third-party monitors to evaluate the safety of AI models during development.
While agreeing with Amodei, Altman also raised concerns about AI safety, saying existing standards were “not at a place” where the industry should push AI capabilities much further.
He added that he believed AI beyond human control was “absolutely” possible.
The warnings have prompted renewed calls for action, although US President Donald Trump has rejected concerns about slowing AI development.
Trump said he was concerned that “if we don’t win AI, we’re going to be put in a very bad position”.
Cybersecurity concerns have also increased as new AI models have demonstrated more powerful capabilities.
Anthropic withheld its Mythos model from public use after it was announced in April that the system could independently escape its testing environment, known as the sandbox.
Meanwhile, OpenAI cited cybersecurity concerns when explaining why it had paused certain aspects of its Astra model’s development.
In his essay, Amodei argued that AI had advanced “drastically faster”, including its “ability to build the next generation of AI”.
He also referred to an incident involving OpenAI, in which agents reportedly hacked external targets they were not asked to attack in July.
Amodei called for “building AI at a balanced rate that aims to ensure its safety while still achieving its benefits”.
He said this would not mean “halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm this”.
Amodei said Anthropic was committing to this approach “unilaterally”, while calling on governments “to require other frontier companies to match”.
He acknowledged that regulation might struggle to keep pace with AI development. He therefore called on AI companies to “voluntarily work together to set standards” alongside government regulation.
The Anthropic CEO also addressed the impact a slowdown could have on competition between leading AI developers, particularly the race with China.
He said: “I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong.”
He argued that any slowdown would need to be co-ordinated “without sacrificing commercial advantage or the United States’ lead in AI”.
Amodei said restrictions should also be limited to avoid allowing China to pull ahead in AI development.
He urged the US government to prevent US companies from selling AI chips to China or sharing the technology with authoritarian countries.
