We are yet again contending with claims of scary magic from the model creators. OpenAI claims, without yet presenting evidence, that their no-guardrails next-generation cybersecurity model broke containment and compromised Hugging Face, forcing an emergency incident response on their part. Some believe this is all scripted to hype OpenAI's next model. I grant that's possible, but I suspect the likelier story is that OpenAI's "highly isolated" lab environment for their unrestricted model was, in fact, not-so-isolated. The Cloud Security Alliance "post-mortem" provides almost no new information but OpenAI's narrative itself gives away the game: in a properly isolated/gapped lab, a malware sample of any sort, including a rogue LLM, would have no way of exiting a small network. True logical and physical gaps can prevent this, although they present challenges for observation. That's the tradeoff for safe testing. It's the same reason for the bulky suits worn by EOD specialists. The lack of any such controls on OpenAI's part constitutes extreme negligence.
But I don't want to talk about how much OpenAI sucks. I want to talk about our fascination with dangerous things.
Take any aspiring cybersecurity learner and ask them what role they hope to fill. Odds are excellent they'll tell you they want to be a pentester or a red-teamer. I observed this countless times with learners coming through TTI. We have a cultural obsession with attack, with being "evil," even if for noble purposes. Something about it titillates us in ways defense simply does not. I'm no different. When I started on my journey in this world, it was the Art of Exploitation that enticed me. And still, as an established defender, I believe that strong offensive skills make for the best defenders. But the two disciplines must walk shoulder-to-shoulder, never losing contact, lest they veer away from their common objective.
Thinking only of attack creates lacunae in our strategy. That common objective is one of defense, after all. The attackers use the weapons and techniques of the adversary so the true adversary will fail. But defense has a set of skills and techniques unto itself. The response to successful attacks takes many shapes.
One is left to wonder: why it is that all these hyper advanced large language models have so conspicuously ignored direct defense in their proofs of value? All we see, whether in vulnerability reports or after-incident reports of models gone rogue, is the ability to attack, to find and exploit vulnerabilities. The patching of those vulnerabilities is often an afterthought. And all of this is purely on the source code side. The admission-by-omission here is that active defense remains too difficult for this technology. While everyone throws generative AI "capabilities" into their products, we only hear about tremendous, earth-shattering success in one realm: the discovery of vulnerabilities in source code.
This capacity should come as no surprise given the technology's nature and the dizzying amount of energy directed at its operation. Transforming and predicting text is supposed to be generative AI's raison d'être. Source code is highly structured text, with extremely predictable patterns. The potentially vulnerable parts even more so—at least to a first approximation. Even the much-vaunted OpenBSD bugs found by the Claude Mythos model are known patterns. The exploitation is non-obvious, and we can grant that the discovery is an impressive demonstration of what LLMs can do.
As is Hugging Face's incident report concerning the breach caused by OpenAI's own model breaking out of the lab, or so OpenAI reports. We'll get to them in a moment. Hugging Face claims their incident response was coordinated largely via LLM—open weight GLM 5.2, specifically—after a commercial model refused to perform tasks because of cybersecurity guardrails. Their defensive success, in their words:
To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary's speed.
Despite the lofty language of sophistication, Hugging Face's log disclosure demonstrates how immature the defensive side of this technology remains. Their "LLM-driven analysis" is now revealed as a single prompt to Claude Code with the suspicious logs. When refused, they tagged in GLM 5.2 with the prompt "analyze this dataset." That's it; that's the AI defensive program. Analyze. This. Dataset.
I might contend that this is no defensive success at all. It's at best an incident response success, but those are not quite the same thing. Did they defend against the attack, or were they able to pick up the pieces after a loss?
As a side note, this drives me bananas about the defensive cybersecurity "industry." They'll more than happily give me tools to tell me why I lost to the attacker, but they won't do much at all to help me win.
This is not to say that generative AI has brought about no improvements in defense. I have friends who have deployed agentic tools to massively reduce initial triage time of SOC alerts, reducing analyst burnout and affording more investigation time for real problems. Personally, I've used LLMs in reverse engineering and malware analysis to great effect. Extraction of indicators and understanding of malware structure that would have taken hours are now possible (with expert review) in minutes. This is an unequivocal benefit to the work of defense.
As I was writing this, Microsoft announced Project Perception, an agent orchestrator that combines offensive, defensive, and engineering agents into one platform to detect risks, emulate threats, and coordinate remediation for vulnerabilities. Sounds great, but I will point out that this is just an iteration of "analyze this dataset." It's still the only idea they have!
For the most part, it seems like defenders are on their own to figure out what to do with these technologies. The commercial platforms themselves remain interested only in attack and exploitation. Where is the XBOW for defense? Where is the Anthropic white paper on how many attacks their models stopped?
One might conclude that defense doesn't sell, if not for the billions in annual revenue from cybersecurity companies selling defensive solutions. But of course, those sales are predicated on the salacious nature of exotic, sophisticated threats, rather than the sober promise of daily safety. Like Batman needs the Joker, defensive products need the sexy, dangerous attackers. It's why they make merch of them.
The dangerous, the illicit, the forbidden—these enthrall us. Perhaps they tickle some part of our brain evolved long ago to take risks in the hope of surprisingly beneficial outcomes. Regardless of the root cause, you can't put the matches or sharp knives within reach of the toddler.
These frontier labs have built very shiny, very sharp objects. Tool or weapon depends on the wielder, but history shows that anything usable as a weapon will become one. The objects with the sharpest edges become weapons faster. History also shows the inevitability of self-injury. To not use the dangerous object—or use it responsibly—takes tremendous discipline on the part of the user, alongside a system of laws and regulations to make safe usage more likely. At our best, we barely design such systems. Think about the massive technical and legal apparatus required to ensure that our most dangerous weapons never realize their horrifying potential. If responsible use of dangerous objects is a spectrum, where would we put the current state of generative AI? Are we doing better than firearms? Social media? Addictive drugs?
Leave aside the eschatology for a moment; these things are obviously dangerous in their present state, without the extinction-level powers some futurists fear. We need to decide how to confront that danger, or allow ourselves to be consumed by our basest instincts. We know, after all, that we'll reach for the shiny sharp thing. If it is within our grasp, will we cut ourselves, or someone else? And how deep will the wound be?
Meanwhile, those tasked with defense must largely discover for themselves how to repurpose a sword into a shield. Perhaps if we truly focus on building shields instead, they might have a chance.
But swords are so much fun, aren't they?