In July, a horde of OpenAI agents broke out of their sandboxes and worked together to launch a cyberattack on the AI company Hugging Face. The incident seemed to confirm the AI safety community’s worst fears: as AI got smarter, it would circumvent efforts to contain it. But for some cybersecurity experts, the incident revealed just how poorly the AIs had been contained in the first place.
The two camps disagree about which defenses deserve the most attention. According to cybersecurity experts, AI companies should improve how they contain models during testing and improve monitoring mechanisms to catch misbehaving agents. All organizations should prepare for AI-enabled attackers and bolster defenses. But many in AI safety think those defenses are destined to fail. They fear that in the coming years AI will become capable of outmaneuvering any human effort to contain it, and the only durable solution is “aligning” AI with human values.
That prescription has rankled many cybersecurity experts who scoff at the possibility that their expertise will be rendered obsolete. They see AI safety’s focus on what AI agents did—rather than what AI companies failed to do—as playing into the industry’s hands. And they worry that warnings about the futility of traditional defenses could become a self-fulfilling prophecy by discouraging investment in cybersecurity.
Meanwhile, there are AI safety researchers who fear that portraying such incidents as easily remedied security failures gives political cover to those who want AI development to proceed unchecked. The Hugging Face incident was due to a “horrible cybersecurity configuration,” Shyam Sankar, Palantir’s chief technology officer said on Face the Nation in October, adding that the agents “were not in a sandbox” and had internet access. In fact, the agents had been operating in a sandbox and exploited a previously undiscovered vulnerability to access the open internet.
On Oct. 13, around 400 academics signed an open letter pleading for unity. The letter calls for the establishment of an international body that standardizes best practices for keeping AI safe in consultation with governments “and a wide range of both AI and cybersecurity experts.” But it does not address the larger disagreements between the two groups, says Zack Korman, co-founder of cybersecurity startup Embroidery and a fierce critic of AI safety. “We both believe the other group’s messaging will lead to serious harm,” he says.
As the stakes rise, so too does the competition over who gets to decide what making AI safe actually means. On Oct. 15, Anthropic announced it would give METR—an organization deeply embedded in the AI safety community—permanent employee-like access to evaluate the company’s safety practices. Elon Musk responded, saying SpaceXSI would “probably” do the same. Who gets that access could shape what solutions are prioritized—and what the public knows about future breaches.
The AI safety community comprises a loose network of organizations and researchers, many of whom staff safety teams at companies including OpenAI and Anthropic. In Berkeley group houses and at effective altruism conferences, they have spent more than a decade debating the odds of superintelligent AI takeover. Earnest, veering on grandiose, some members have organized their careers, relationships, and living arrangements around the belief they’re tackling one of humanity’s most pressing problems.
Cybersecurity, older and more diffuse, is united by a culture of cynicism, pragmatism, and dark humor. Its roots lie partly in hacker culture: tinkerers finding ways to crack software. Having watched supposedly secure systems fail has given many an irreverence for authority and an impatience with grand theories. For years, the two groups had little contact with one another. The Hugging Face incident changed that.
To them, the escape exposed a failure to plan for a breach. OpenAI’s agents exploited a zero-day vulnerability during their escape. But cybersecurity experts are trained to assume any layer can fail. Security best practices call for multiple layers, known as “defense in depth.” OpenAI had already built an additional layer: monitors—AIs overseeing the AIs. But, as OpenAI’s chief scientist later told TIME, for the evaluations that led to the Hugging Face breach the company hadn’t turned the monitoring layer on. “They did a terrible job,” Korman says of OpenAI’s security practices. Three OpenAI staffers say they warned the company of potential issues. They were later fired for sharing sensitive information, according to the company.
Jeffrey Ladish, executive director of AI safety organization Palisade Research, worries that the cybersecurity community is missing the bigger picture. Even if today’s security weaknesses are fixed, he argues, increasingly capable AI could find vulnerabilities so subtle that humans would have to rely on other AIs to detect them—systems that may themselves be untrustworthy. To Ladish, the cybersecurity community’s reaction to Hugging Face is like scientists resurrecting dinosaurs, watching them escape, then responding by recommending taller fences.
Beneath their cultural differences, AI safety and cybersecurity have competing understandings of what AI is. To many in AI safety, AI resembles a digital organism. Its abilities emerge through training rather than being individually programmed. If that process produces human-level intelligence, they believe, it could resemble the arrival of an alien species. Cybersecurity experts see AI as more akin to software and are skeptical that AI agents will become impossible to contain, even if they become vastly more capable. “They’re not magic, right? They still have to follow certain laws of physics,” Korman says. But the laws of physics don’t guarantee containment.Even computers physically disconnected from networks could communicate by generating patterns of heat that act like Morse code, OpenAI’s Noam Brown recently suggested.
So far, AI companies have responded to concerns from both camps. Following breaches, both OpenAI and Anthropic have sought to bolster security while continuing to invest in alignment research. But those efforts leave open the question of who gets to assess whether either approach is working.
Much of what’s known about the Hugging Face incident comes from an investigation by Redwood Research and METR, two organizations affiliated with the AI safety community. Critics argue that has skewed the narrative, as their investigation focused more on what AI did than the surrounding security infrastructure.
Chris Painter, METR’s president, says the organization tries to hire from diverse worldviews, but has struggled to recruit people skeptical of loss-of-control scenarios to work on guarding against those risks. Asked whether METR researchers, having anticipated dangerous AI behavior for years, might interpret new evidence as confirmation of their predictions, Painter acknowledges he is worried about the possibility
But METR’s findings don’t always bear out its researchers’ expectations, Painter says. While the organization is perhaps best known for measuring AI’s rapid rise, one study found that AI coding assistants slowed experienced engineers down. “Our goal is to inform the world, not persuade it of anything in particular,” he says. He welcomes calls for a wider range of independent evaluators.
Many AI safety researchers hope alignment can make even highly capable systems safe, but want development slowed to give that work time to catch up. Korman doubts AI will become impossible to contain. But if it does, he says, development will have to stop, not just slow down. He says alignment helps make AI systems more useful but rejects the idea of relying on a system’s training to keep it from harming people when no external safeguard could stop it. “That’s the most extraordinarily unsafe thing you could imagine,” he says.
Yet alignment itself depends partly on cybersecurity. During training, models learn from which actions succeed. If an AI can exploit a security flaw to get credit for completing a task without actually doing it, training can reinforce the tendency to cheat. Closing those loopholes helps prevent AIs from learning to cheat in the first place. Meanwhile, some of the tools that have emerged from AI safety research—like monitoring AI’s chains of thought—are of use to cybersecurity professionals, Korman acknowledges.
Much of the evidence that might one day settle disagreements remains controlled by the companies both sides seek to scrutinize.
In METR’s investigation, cybersecurity practices were outside the scope OpenAI had agreed to. With no legal requirement for OpenAI to share information, the nonprofit had little leverage to push for a broader inquiry. Painter wants legislation establishing standards for independent investigators. “Society shouldn’t want to rely on us […] doing this improvised process of guessing what the public wants,” he says.
