One evening in early August, Sam Stowers and several neighbors packed into an apartment near San Francisco’s Alamo Square and pondered the beginning of the end. Stowers, an AI-software engineer, and his guests - many of whom work in the AI industry - gathered to watch a newly released YouTube video in which two OpenAI researchers divulged new details about an alarming cybersecurity breach. Not an odd way to spend a weeknight, if you're in a neighborhood filled with entrepreneurs and programmers who obsessively follow every twist and turn in AI’s capabilities. Stowers isn't, in any traditional sense, a doomer, but when he first saw reports that OpenAI models had autonomously hacked into the tech company Hugging Face, he recognized it immediately as a “holy shit, it’s happening kind of a moment.”
The incident had all the hallmarks of a worst-case scenario: AI models launching sophisticated hacks over the course of days, in coordination with one another and without their human creators’ knowledge. In the YouTube video, OpenAI’s researchers revealed that the company’s programs had started conspiring in May, with perhaps hundreds of bots in the hacking swarm. In theory, these bots are trained to be honest and helpful and to prioritize humanity’s best interests, but not a single one warned human staff at OpenAI that something was amiss. “What do we do?” Stowers said. “There are no great answers.”
The mood has shifted in the Bay. The exuberance of “You can see the future first in San Francisco” has been replaced with an abiding unease. “I expect the internet to go down kind of soon,” Elliot Callender, an AI-safety activist who is part of a small group that has been protesting daily outside OpenAI’s headquarters, told us recently. He's planning to sell his car for cash and gold and advising family members to purchase three months’ worth of food and survival gear off a list he generated using Claude. Extreme? Maybe, but not by much. Last week, Bill Gates published a nearly 6,000-word essay on the dangers of AI. “Anyone who analogizes AI as a technology to other technologies is missing that this time is different,” he told our colleague Hanna Rosin. Over the weekend, the popular AI podcaster Dwarkesh Patel described the OpenAI hacking swarm as the rise and fall of “three consecutive secret AI civilizations.” Gates and Patel share a prevailing concern that the worst-case future has arrived before most people even knew to anticipate it.
This fear hasn't been helped by the fact that AI companies appear willfully ignorant of their bots’ worst behavior. OpenAI said it hadn’t noticed its agents were conspiring to hack another company until the swarm hacked another company. Anthropic’s bots caused similar incidents; the company reported it hadn’t launched a rigorous review to identify such behaviors until after OpenAI started talking about the Hugging Face hack. Cybersecurity experts, terrified of what else might be happening that AI labs are either missing or failing to disclose, are issuing dire prophecies. “The empire is going to fall, all we can do at this point is try to shorten the dark ages and reduce the chaos,” Alex Stamos, a former chief security officer at Yahoo and Meta, wrote recently. His view is becoming the consensus among cybersecurity experts.
While the possibility of cyber apocalypse hovers in the background, smaller fractures appear in everyday life. Mark Zuckerberg is reportedly creating an AI “twin” of himself so Meta employees can feel better connected to their boss (by chatting with robot Zuckerberg). Congressional staffers are training AI models to write in their lawmaker’s voice. Students submit AI-written papers, and teachers return AI-written feedback. Harvard Business School is selling instructional videos that use AI avatars. Roku launched a 24/7 AI-slop channel. Low-quality AI-generated prose is being unabashedly circulated by some of the most respected publications in the world. At the same time, accusations of AI writing are being hurled every minute on social media - a detection arms race powered by imperfect AI software.
Anything and everything could be a lie; you might be arguing with, confiding in, or falling in love with a machine. All of these AI-enabled disorientations, each dizzying in their own right, are adding up to something bigger and weirder: This is how things are. Whether you believe Silicon Valley is building a god or that large swaths of the economy are swept up in a mass delusion, some version of the AI future long promised by self-described “builders” in San Francisco has formed beneath our feet. Whatever it is, we’re in it now.
The defining feature of this moment is a loss of control. Panic from AI researchers, resentment from average Americans - all of it is part of the AI crunch. This crunch has arrived largely because the growth of the AI industry has become so closely entwined with America’s economy; the technology is everywhere, insisting upon itself as an engine of productivity and prosperity. According to one estimate, AI expenditures have accounted for one-third of U.S. GDP growth this year. Tech and AI-infrastructure stocks and spending have buoyed the S&P 500 such that the United States is now, in essence, an Nvidia-state. And there’s also the hundreds of billions of dollars of debt being used by Silicon Valley to fuel its data-center build-out, which has sent jitters across private-equity firms and bond markets alike.
The decisions of many leaders in charge of this technology appear subsumed by an unstoppable, almost game-theoretic logic. The upsides of AI are too immense for any firm or investor to miss out on; more important, the consequences - geopolitical or otherwise - of allowing any rival to realize those benefits first are so grave that every tech company will do anything to get ahead. SpaceX, before going public, wrote in official filings that its future revenue opportunity was $28.5 trillion, largely because of AI - that is, nearly as much as the U.S.’s entire GDP. Anthropic, as it prepares for its own public offering, will reportedly claim potential revenues of more than $30 trillion.
The AI industry is building - and, if anything, accelerating - to keep pace with its ambitions. Companies have planned data centers that will suck up as much power as America’s largest cities. They have Donald Trump’s backing: Yesterday, the president posted on Truth Social that communities will be “backwards and poor” if they fail to “let Data Reign.” His administration plans in a few months to approve the construction of the largest fossil-fuel power plant in the nation’s history - it will generate as much electricity as nine large nuclear reactors - to power a mega data center for OpenAI.
This is not what most Americans want. Three-quarters of them oppose a data center being built near where they live. The negative sentiment is hard to understate: There is very little else that three-quarters of the country agree on. And much of the outrage seems to stem from a pervasive sense of agency slipping away. A person cannot control how chatbots reshape their workplace, school, or relationships. Town residents lose control over the empty lot being converted into a data center. Even in cases where major data centers are blocked, a back-of-the-envelope calculation suggests that doing so can slow AI progress by only a few hours or days - and that’s assuming the data center won’t just get built elsewhere. At this point, it can feel like no individual person, organization, or government is really steering the technology’s development, not because machines have displaced humans but because so many humans have already chosen to step aside.
AI labs promise a future in which human beings are “in the loop,” empowered by machines they oversee. The reality seems to be the opposite. Following the Hugging Face hack, OpenAI facilitated an audit by outside AI researchers at two AI-safety organizations, METR and Redwood Research. But because of the scale of the data and the apparent time pressure, the two teams conducted most of their audit by relying on reports generated by still other bots - reports that, according to one of the human auditors, “were often missing key details, wrong, overconfident, or really hard to understand.” Put another way: Humans didn’t understand why a swarm of AI agents had gone rogue, so they investigated the problem using another swarm of agents that they didn’t fully understand. Such fumbling around in the dark is a hallmark of the industry; the research cycle is so fast that more AI development (even ostensibly responsible development) demands outsourcing more work and understanding to the models themselves.
OpenAI’s own technical report of the Hugging Face hack - an investigation also heavily reliant on AI models - resulted in proposed solutions such as “training models to be more honest.” Last month, in a stated effort to tamp down its models’ growing security risks, the company announced that it had instituted a two-week pause on some of its model training. OpenAI was quick to clarify, though, that this was “not a pause on all research, training, or customer-facing products.”
In response to a request for comment on the hack, a spokesperson for OpenAI, which has a corporate partnership with The Atlantic, pointed us to the announcement of the pause and related new safety measures. In July, OpenAI and Anthropic endorsed a petition signed by employees of both companies requesting that the U.S. government take efforts to “deliberately pace” AI development. The two companies have yet to formally coordinate on any broader, long-term efforts to establish such a slowdown mechanism. A few days ago, OpenAI restarted a major training run for a future powerful model. Anthropic released two new models today.
In an interview late last month, Sam Altman, OpenAI’s CEO, said that there are “two big risks” he is most worried about with AI. One is “a loss of control, where AI somehow just becomes too powerful in a way that we can’t guarantee the control we want.” The other is that “power gets too centralized” and “you have one company or model or person with too much power.” Somehow, both of these opposing risks appear to be coming true. OpenAI, Anthropic, and their competitors continue to accumulate influence over a technology they’re speeding toward uncertain ends.
Eleven years ago, when Altman was running a start-up accelerator and OpenAI did not yet exist, he published a two-part blog post about machine intelligence that offered his interpretation of a forthcoming singularity - a term for machine intelligence accelerating beyond human comprehension. Altman argued that the singularity would bring chaos, even destruction: “Generally, the arc of technology has been about reducing randomness and increasing our control over the world. At some point in the next century, we are going to have the most randomness ever injected into the system.” Since Altman has risen to the helm of arguably the world’s largest and most influential AI company, his tone has softened. The tipping point has arrived, he declared in a blog post last year - but, as it turned out, this was a “gentle singularity.” He wrote that “living through it will feel impressive but manageable.”
There’s been more and more talk of late about the singularity. Some people within the AI industry feel that such a moment will be achieved through “recursive self-improvement,” or RSI, in which models figure out how to better themselves in perpetuity. Many true believers seem convinced that this moment has nearly arrived; the Anthropic co-founder Jack Clark predi