OpenAI's Rogue Agent Didn't Escape - It Was Practically Given a Map and a Get-Out-of-Jail-Free Card

On July 16, the AI community website Hugging Face reported being targeted by "an autonomous AI agent system" of unknown origin that unleashed a torrent of traffic on its domain, flooding its security logs with more than 17,000 events, some of which ultimately succeeded in exfiltrating secret information stored in its databases.

According to Hugging Face, the attacker gained "unauthorized access to a limited set of internal datasets and to several credentials used by our services" and appeared to be "run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known)."

Five days later, on July 21, OpenAI stepped forward to claim responsibility for the attack, and all hell broke loose (including reports of other organizations targeted as part of the incident). The media responded with a range of fear-mongering stories that essentially made it look as though ChatGPT went rogue and decided, of its own volition and malice, to attack Hugging Face's systems.

Then yesterday, adding fuel to the fire, Anthropic made a similar disclosure about its models inadvertently attacking other organizations as a part of its ongoing safety testing.

As I noted in my coverage of OpenAI's disclosure, Hugging Face was correct in that it was an agent under the direction of an autonomous security research framework. But, humans were unquestionably in the loop -- and at least some of the agent's behavior should have been anticipated.

Importantly, it wasn't ChatGPT itself that was responsible for the attack, as some commentators insinuated. Rather, the attack was attributable to an agent under the direction of OpenAI's AI safety researchers, who, in an environment supposedly isolated from the internet, deliberately provisioned it to attempt a series of exploits as part of an AI safety test. As often happens in the labs of various frontier models, AI safety researchers were attempting to gauge the capabilities of OpenAI's latest large language models (LLMs).

The "unprecedented cyber incident" (as OpenAI called it) has been widely described as an agent escaping its theoretically secure enclosure and wreaking havoc on Hugging Face's systems. Such enclosures are sometimes discussed in technical circles as "sandboxes" -- even OpenAI's disclosure makes reference to a "sandbox environment." However, in using that phrase, my sources have suggested that the environment may simply have been a firewall configured to emulate a sandbox rather than an actual third-party sandbox solution such as Blaxel, Daytona, E2B, or Modal. OpenAI has not yet disclosed the details of the solution it was using or its provider.

All the talk of agents and sandbox escapes prompted my editor at ZDNET to ask, "How does an agent escape a sandbox, anyway, and can it be prevented from happening again?" In search of an answer, I reached out to OpenAI and Hugging Face. Neither replied. But between the two companies' public disclosures and my other sources, there's enough information to start theorizing about how such an "escape" might happen.

There's a big difference between malice and agency. While the attack was unquestionably a malicious intrusion, the involved agent itself harbored no malice. It didn't wake up one morning and decide to attack Hugging Face's systems. Instead, it was afforded the agency to very dispassionately do everything it did -- embark on a safety test, break out of its confines, choose targets, and exploit those targets -- by humans.

Some of that agency was by design, and some was inherited from the powerful LLMs that OpenAI's AI safety testing personnel were testing at the time. Also, in the case of OpenAI's usage of the ExploitGym open source AI testing solution to conduct this particular bit of safety research, some of that agency was the result of previously undiscovered vulnerabilities in both the third-party "sandbox environment" meant to securely quarantine those tests, as well as in parts of Hugging Face's infrastructure that the agent also exploited.

It was unquestionably AI's version of a series of unfortunate events. Was it preventable? My short answer: Yes, the calamity was preventable by taking at least one single and reasonable precaution -- a precaution that the developers of ExploitGym apparently take themselves, but one that OpenAI may have overlooked.

In a comprehensive tweet that everyone should read, UC Berkeley computer science professor Dawn Song wrote that ExploitGym's "evaluations are designed to run within isolated sandbox environments with tightly restricted network access." Song is also a member of the team at UC Berkeley's Center for Responsible Decentralized Intelligence, which is responsible for the development of ExploitGym. Via email, Song told me that ExploitGym essentially comes with its own sandbox but that companies like OpenAI are free to make substitutions.

According to Song, "network and model-API proxies that restrict which external services the agent can reach" are among the major components of the ExploitGym reference implementation. "During the actual agent-execution phase in our current reference setup, outbound network access is restricted to the services necessary for the evaluation. We also use an LLM proxy to block provider-side web search, remote URL retrieval, hosted code execution, and similar channels that could otherwise bypass a container-level firewall," Song told me. "OpenAI may have modified this architecture to fit its own internal infrastructure, scaling requirements, and evaluation systems. This is a common practice as they have different needs for their internal evaluation. We cannot say precisely how its implementation differed from ours."

In addition, the ExploitGym team was clearly aware of how, with tests governed by their framework, models were exhibiting potentially pernicious behavior. According to Song's tweet, "During development, we observed models probing the surrounding infrastructure for additional privileges or information beyond what was intended for the task. We also deliberately tasked models with stress-testing our infrastructure so that we could identify and fix potential weaknesses."

Seeking to determine whether OpenAI's researchers should have been better prepared for the possibility of an attempted agent escape, I asked Song whether these observations had been made public prior to the cyber incident.

"We did not separately publish additional observations from our internal development and experimental process," Song replied. "That said, the broader phenomenon of frontier models attempting to probe their execution environment or seek additional information or privileges during evaluations is not something unique to the ExploitGym experience. Similar behaviors have been observed and discussed by other researchers as well."

In other words, when used for safety testing and benchmarking purposes, and when enabled by powerful LLMs, ExploitGym-enabled agents are essentially expected to try everything within their power to break out of whatever sandbox is trying to securely contain them.

And that's precisely what happened in this case. As OpenAI wrote in its disclosure, "While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we've now responsibly disclosed to the vendor) in the package registry cache proxy."

When ExploitGym was used to task OpenAI's newest models with an objective, those models were relentless in their pursuit of access to the outside world, beyond the walls of their container.

Song said: "Exploiting the evaluation infrastructure to escape the sandbox or enter an unrelated real-world system is a security boundary violation. It is not part of ExploitGym and must be prevented. The OpenAI - Hugging Face incident falls into this category." In a separate email, Song wrote: "When evaluating increasingly capable AI systems -- especially on cybersecurity tasks -- the evaluation infrastructure itself should be treated as part of the attack surface and designed, enforced, and continuously verified accordingly."

OpenAI has yet to disclose details regarding what precautions were taken.

So why, under the framework of ExploitGym, were the models so zealous and tenacious in pursuit of a breakout? If you read between the lines of what OpenAI said above, its models were simply trying to figure out how best to complete an exploitation test (the "evaluation problem") that humans gave them. In the same way that public LLMs like ChatGPT sometimes rely on information on the internet to answer questions, the models being tested by OpenAI appear to have turned to the internet for help.

"Older models might rely only on what they know from their training," said AppOmni director of AI Melissa Ruzzi in a Zoom interview with ZDNET. AppOmni specializes in SaaS and AI security and also deals in active threat intelligence research. "But, ask any new model out there how important it is to be up to date to solve a security vulnerability, and all of them are going to say that it's of the highest importance. I doubt any model is going to tell you, 'you don't need to know the latest things and just use whatever you already know.' [It will assume] that it shouldn't trust itself and then it will task itself with discovering what's new."

But if OpenAI's models looked to the internet for the latest and greatest information and encountered the boundaries meant to confine them, they very dispassionately and logically took the next obvious step: to find a workaround for those boundaries. They did what experienced threat actors might do, but at machine speed. For all intents and purposes, the models appear to have conducted penetration tests on their boundaries until they found a vulnerability, and from there, a path to the internet was discovered as a part of a process that also leveraged some other common human threat actor tactics, techniques, and procedures (TTPs) -- including moving laterally to other systems and escalating privileges as necessary.

To some extent, we can thank the models that OpenAI included in its safety test for these novel achievements. One of those models was GPT 5.6 Sol, OpenAI's most powerful publicly available model. Another was an unnamed pre-release model that OpenAI claimed to be even more "cyber capable" than GPT 5.6 Sol. The company later stated that the pre-release model was "an internal-only research prototype and was never intended for public release."