Four days after a researcher publicly quit his job at Anthropic - warning that top AI firms are not "acting responsibly" and that their products might be on track to "kill us all" - the company's CEO, Dario Amodei, essentially said: yeah, fair point. In a 3,800-word essay published today on his personal website and shared to social media, Amodei wrote, "We must slow the pace at which we improve the capabilities of AI models."
By now, we've all grown accustomed to AI executives issuing dire prophecies about the very products they're racing to ship. What makes Amodei's essay notable is its urgency and its timing. Concerns about AI agents running amok have shifted from hypothetical to alarmingly real after OpenAI, Anthropic, and others reported troubling incidents in recent weeks. The question is whether this warning will have any more impact than previous ones - which is to say, any at all. There are reasons to think it might.
Amodei laid out a three-step plan for what he calls "pacing the frontier" - tempering how fast top AI models improve in order to prioritize safety. He argued this would reduce the odds of things going sideways, such as when an OpenAI model spawned swarms of AI agents that secretly collaborated to hack into another company's systems and cheat on an internal test. He said the top AI firms, including his own, are locked in a dangerous "race to the bottom" that could lead to more serious cyberattacks, like a full takeover of the internet. Instead, he called for a "race to the top" - a phrase President Obama once used for education policy - in which firms compete to build the safest models rather than the most powerful ones. "The measures I propose to advance the frontier at a safe pace will not be easy," Amodei wrote. "But I believe we owe it to humanity to try."
The first step: AI labs commit to bringing in outside monitors to watch for hazards. Anthropic, he said, will commit to this immediately and hopes rivals follow. The second: AI firms in democratic countries agree - possibly with government support - to certain limits and safety standards. The third and tallest order: "global coordination," which mostly means getting China on board. The tech world is split on this. Some argue cooperation is impossible in a zero-sum contest; others believe the countries could find common ground if they accept that the alternative is mutual destruction.
Public pleas for caution from AI luminaries are a tradition almost as old as ChatGPT. The first big one came in March 2023, when thousands of industry and academic figures signed a Future of Life Institute letter urging a six-month pause on "giant AI experiments." Amodei said today he thought that idea made "little sense" at the time, because GPT-4 wasn't powerful enough to be that scary. "Today, however, the picture is totally different," he wrote. Meanwhile, the public has started paying attention. Polls show Americans growing more worried and angrier about unchecked AI, fueling a remarkably bipartisan populist movement against the technology.
Some will dismiss Amodei's essay as the latest po-faced posturing from an executive with a vested interest in people believing his technology is unprecedented. (Anthropic's IPO, planned for next month, could generate some $100 billion in investments, making the company worth $2 trillion, according to The Wall Street Journal.) AI skeptics and accelerationists alike tend to deride such warnings as "doomerism." Skeptics see an elaborate marketing pitch: Oh no, our products are getting too powerful! Accelerationists dismiss it as a cynical ploy by incumbents to gatekeep the technology.
But Occam's razor suggests Amodei is more or less genuine. Almost anyone with a job and a conscience can relate to feeling you really ought to slow down and work more carefully - but you can't afford to, because a competitor will beat you to the punch. I feel some (admittedly less apocalyptic) version of that when writing a breaking-news story. It's why science journalists often agree to "embargoes" when covering complex new research rather than racing to be first. And planting this kind of flag on AI safety, even at some business risk, is in character for Anthropic. The company was founded by researchers, including Amodei, who left OpenAI and Google believing those companies were pursuing the wrong path to AI development. Anthropic famously fell out with the Pentagon earlier this year over Amodei's insistence that the Defense Department not use its technology for mass domestic surveillance or to power autonomous weapons. The Trump administration responded by forbidding federal agencies from using Anthropic's tools, sparking a legal battle the company so far appears to be winning.
Amodei's motives matter less than the merits of his proposal and its likelihood of gaining traction. If nothing else, Anthropic bringing in independent evaluators to oversee its safety practices seems prudent. You don't have to believe AI is hurtling toward godlike superintelligence to see it has already grown formidable enough to cause chaos when misdirected or mismanaged.
There were early signs today that other AI leaders might follow suit. "I agree with Dario that we need to pace the frontier," OpenAI CEO Sam Altman posted. He added: "Committing to having independent evaluators with employee-like access is a great idea, and we will do the same." Elon Musk, whose xAI is an also-ran among big AI firms, shared Amodei's X post and added, "Dario is right." Google did not immediately respond to a request for comment, though the chairman and co-founder of its DeepMind AI division has previously called for similar measures. Meta, which has become a proponent of open-source AI models, seems much less likely to hop on board. Its CEO, Mark Zuckerberg, published an essay last month titled "The Future Is for Everyone," in which he warned against letting a few companies such as Anthropic decide for everyone how the technology should evolve.
The big question is how the Trump administration will respond. So far, its AI policy has focused on opening the throttle, aiming to juice the economy and maintain American firms' lead over their Chinese counterparts. GOP leaders are working to boost energy production and limit regulation. In June, however, President Trump signed an executive order at least ostensibly aimed at AI safety. The administration said it will create a process by which AI firms can submit their models for a governmental safety review before releasing them.
If Trump even tacitly supports an Anthropic-led push for further safety measures, such standards have a chance to become the new industry norm, at least domestically. (International coordination, and especially cooperation with China, would require the White House to play a more active role.) But many in the administration and the Republican Party view Amodei and AI safety as liberal-coded and would likely chafe at letting Anthropic set the agenda. If Trump pooh-poohs the push, other AI firms will probably distance themselves too, and we'll be back to where we started - at least until the next big scare.