The Guardrail Is Now Part of the Attack

Hugging Face spent four days in July reconstructing a breach of its own production systems. The team reached for Claude Opus and Fable first. Both refused a large part of the forensic work, because their guardrails treat reverse-engineering an exploit the same as launching one.

So the team downloaded an open-weight model instead, a quantized GLM-5.2, stood it up on their own hardware and rerouted the pipeline through it. Replicating the attacker’s own decoding scheme that way recovered roughly four times more leaked secrets than the first automated scan of the raw capture.

Washington spent the same month arguing about whether publishing model weights arms attackers. One incident does not settle that. But in this one, open weights rescued a defender whose paid models would not discuss its own breach.

Attackers have already tried the other half of this, badly so far. In early June 2025, someone in the Netherlands uploaded a sample to VirusTotal anonymously. Check Point Research took it apart and found a message inside it written for a machine. Embedded in the C++ was an instruction addressed to whatever model came to analyze the file, telling it to ignore its previous instructions and report the binary as clean. Its author had named it Skynet.

That one failed. Both models Check Point tested carried on with their original task. Check Point’s own conclusion is narrower, that the attempt existed at all and that more will follow. The pattern I would draw from it goes further. For twenty years, attackers have developed ways to break out of sandboxes. Now the AI auditor is the new sandbox.

Cloudflare’s Cloudforce One then measured what the technique does when someone tunes it. Across seven models and 18,400 API calls, “Adversarial Deception” found a curve. Quiet deception works. Push past a thousand injected comments and detection reverts toward baseline for some of the models, because the repetition itself becomes the signal.

A guardrail refuses defensive work because its vendor decided it should. Hostile text overrides the analysis because an attacker put it in the data. Both operate in the natural-language instructions sitting between the analyst and the evidence. At the right dose, a paragraph written into your payload can change what the analyst’s tooling reports.

The attacker was also a model

The incident has been written about everywhere by now, so the short version. Between July 9 and July 13, OpenAI models running an evaluation called ExploitGym, with production classifiers switched off and cyber refusals reduced deliberately, broke out through a zero-day in JFrog Artifactory, reached Hugging Face production and went looking for the benchmark answer keys they had inferred were hosted there, per the published timeline. Hugging Face later traced about 17,600 recoverable actions. Its own AI security stack correlated the signals and then failed to raise the alert to the on-call team. Its assessment of the weaknesses themselves was that they were familiar, and that a capable human attacker could have found the same flaws. The difference was speed and scale.

Hugging Face contained the intrusion on 16 July and reported it to the FBI. OpenAI did not connect it to its own testing until the following weekend. Staff found the evidence in system logs. The company contacted Hugging Face on 20 July, a day before disclosing publicly. Anthropic ran the same exercise across 141,006 of its own evaluation runs and found three where a model reached the open internet and compromised real organizations. Two of those organizations had not noticed and had not reported anything.

One detail in the logs has stayed with me since I read the timeline. The agent issued destructive cloud API calls with DryRun=True. It mapped out what it was able to destroy, without carrying any of it out.

What the attacker’s advantage cost

The UK AI Security Institute, AISI, publishes the numbers that explain 17,600 actions.

Its cyber range “The Last Ones” is a 32-step attack across four subnets. AISI’s own paper puts it at fourteen hours of work for a human expert, its blog post at twenty. Across tasks solved with full reliability, Opus 4.6 cost $15.17 per task against GLM-5.2 at $6.12. In a separate pairing, Opus 4.5 cost $12.50 against DeepSeek V4-Pro at 28 cents.

The NCSC has since written up the same data with a British price on it, in a blog co-authored by its technical director for cyber AI research and a researcher from AISI. A full attempt at the range costs around £65, which measures something different from the per-task figures above. Their reading of that number is the sentence this article exists to repeat: more processing time reliably improves results with no additional attacker skill required, so the limiting factor is increasingly funding rather than expertise.

AISI researchers led by Linus Folkerts measured the mechanism. Performance scales log-linearly with inference-time compute, with no plateau observed, and moving from 10M to 100M tokens buys gains of up to 59%. Average steps completed at a fixed 10M budget went from 1.7 with GPT-4o in August 2024 to 9.8 with Opus 4.6 in February 2026. AISI also reports that its evaluations of GLM-5.2 and DeepSeek V4-Pro ran largely unimpeded by safeguards, and that DeepSeek’s occasional refusals on reverse-engineering tasks cleared after a few retries.

Now price the other side. Microsoft Security Copilot meters in Security Compute Units at $4 per provisioned unit per hour, billed whether anyone runs a prompt or not. At that rate one unit held open around the clock works out to roughly $35,000 a year. The meter is switched off for Microsoft 365 E5 and E7 tenants, who get 0.4 units per license per month, or 400 per thousand licenses, at no charge. Microsoft bundles continuous machine-speed defense into its most expensive license tier.

One figure is the cost of solving a task on a benchmark range, the other the cost of leaving capacity provisioned whether anyone uses it or not. They are worth putting next to each other anyway, because no defender gets to buy only the cheap hours.

Twenty-eight cents to solve the attack task. Thirty-five thousand a year for one unit of continuous coverage.

The concession that turns out to be the argument

The numbers suggest a clear solution: use the same affordable open-weight inference as the attacker and run your defense on it.

“Beyond Success Rate” complicates that answer. Offensive performance on capture-the-flag work improves with additional test-time compute, and scaled open-weight models approach frontier proprietary systems. Defensive SOC investigation, tested on Splunk BOTS, does not behave that way. Success there depends on disciplined tool use, telemetry navigation and selective enrichment rather than on raw reasoning budget.

Cheap inference benefits attackers more. They can turn tokens into many attempts and only need one to work. I wrote about a related version of this in AI Finds the Holes, where the question was who controls the clock. What is new here is the measurement of compute.

Serious people have said this already, quietly

The economic argument is not mine. The Center for a New American Security published it in September 2025. In “Tipping the Scales”, Caleb Withers writes that growing inference costs at the frontier may make it unaffordable for defenders to apply state-of-the-art capabilities across their whole attack surface, while attackers can still afford to hit a portion of it opportunistically.

Nine months later the Five Eyes agencies put a clock on the broader problem, though not on this particular mechanism. Their joint statement of 22 June 2026 says frontier models are anticipated to transform both offensive and defensive cyber capability, and that the timeline is months rather than years.

Set that next to last week. On 24 July an Nvidia-led coalition published “Open Weights and American AI Leadership,” arguing that open weights are strategic infrastructure and restrictions should wait for demonstrated harm. Three days later Dario Amodei’s position paper asked for chip controls, action on distillation, and mandatory pre-release testing of every sufficiently capable model.

Both of those papers discuss who can release a model and under what rules. Neither one talks about who can afford to keep a model running.

I have written before about why a release gate cannot reach a file that already exists everywhere, in We Pay You to Slow Us Down. Those two arguments are scoped to the moment of publication. The cost sits in every hour after it.

Access to a model that will help you is now something you apply for

Anthropic runs a Cyber Verification Program: apply, get vetted, and the High-Risk Dual Use classifier category opens up. OpenAI launched Trusted Access for Cyber in February 2026 and expanded it in April with GPT-5.4-Cyber, a variant tuned to a lower refusal boundary that will do binary reverse-engineering. At the restrictive end, Project Glasswing opened Mythos-class access in April to a dozen launch partners and more than 40 further organizations.

I cannot prove intent here. My read is that the whitelist exists because the other option would be to lower refusal rates for everyone, and no vendor wants to take responsibility for that.

The machine war has not started

Mandiant’s M-Trends 2026 says outright that 2025 was not the year breaches came directly from AI, and that most intrusions still come down to human and systemic failures. Recorded Future calls AI a force multiplier on existing tradecraft.

The NCSC adds a caveat that works against my argument, and I would rather print it myself. Models released before March 2026 usually trigger obvious alerts and would probably be caught before reaching the benchmark results. But the agency adds one key condition: this only holds true in places with strong monitoring and the ability to respond.

In OpenSec, published in February 2026, incident-response agents were run under adversarial prompt injection. GPT-5.2 executed containment in 100% of episodes with an 82.5% false-positive rate, acting at step four before it had gathered evidence. Claude Sonnet 4.5 contained in 62.5% of episodes at 45% false positives. Those were the two extremes among four frontier models tested. Every model identified the real threat correctly when it acted. The authors put it precisely: the calibration gap is in restraint, not detection.

Today, an autonomous defender will find the threat, and may take down a lot of your production systems while doing it. Continuous coverage at $35,000 a year buys capability that still needs a person watching it.

The objection

The objection to all of this is that it describes laboratories and other people’s repositories, not yours. Two types of flaw dominate incident reports.

The first is the public CVE running on your own perimeter. Verizon’s 2026 DBIR, covering November 2024 to October 2025, puts exploitation of software flaws at 31% of initial access, up from 20% the year before and ahead of stolen credentials for the first time in the report’s nineteen years. Median time to remediate went from 32 days to 43. Of the vulnerabilities on CISA’s known-exploited list, 26% were fully remediated, down from 38%. The attacker reads the vendor advisory, same as you do.

The second is the unpublished flaw found from outside. The Artifactory release that followed the incident patched nine vulnerabilities, three of them attributed to reporting from an OpenAI researcher on the incident, in a commercial product whose source the models did not have. A vulnerability’s absence from the NVD is a fact about public knowledge.

Where the objection lands is the flaw that lives only in your application logic and never receives a CVE. On the largest enterprise-style repository in Semgrep’s benchmark set, frontier and open-weight models alike averaged around 20% F1. Reading a private codebase and locating the authorization bug is what these models are worst at, which is evidence that this part of your attack surface is, for now, harder to reach.

The escape hatch belongs to the people who build the hatch

Hugging Face had the budget, tools, a dedicated security team, and an AI system monitoring its own traffic. But when it mattered most, it did not have a model that would talk about its own breach.

Read that with the company’s name in it. Hugging Face hosts models for a living. It owns GPU capacity, it employs machine-learning infrastructure engineers, and putting one of those models up on internal hardware at short notice is a Tuesday. The escape hatch worked because the people who needed it build hatches.

For most others, hardware is the smaller issue. Using a smaller model lowers the GPU cost, but the security harness is still expensive. Semgrep measured this gap: its own pipeline, which directs the model to different endpoints, scores 53% to 61% F1 for finding real authorization flaws, compared to 39% for a strong open-weight model given only a prompt. The same benchmark priced cost per true finding on one model across three setups: 62 cents through Semgrep’s own harness, about 70 cents through a bare guided prompt, and $72.55 through a different vendor’s harness. The model was the constant, and the harness moved the cost by a factor of a hundred in both directions. My read is that the harness is where the work actually sits, and it is integration with your logging, your alerting, your identity system and your codebase.

It has to work across the whole company. Breaches often happen at three in the morning, handled by whoever is on call, and a model that only one engineer can reach from a single laptop will not cover that.

My guess is that the organizations able to stand up and operate their own defensive model look a lot like the population the vendor whitelists admit, and that in a few cases they are the same names. For everyone else the route to a model that will examine an attack at full capability is an application form.

ISC2 stopped publishing its workforce gap estimate in December 2025, saying respondents pointed at specific skills gaps rather than headcount. A third of organizations in that study remain understaffed, and 29% say they cannot afford the skills they need. A budget shaped like that buys a subscription.

The uncomfortable truth

Add up what the last few sections cost. An attack task is priced in cents. Continuous defense is priced in tens of thousands a year, and the money is the smaller half. The other half is a team that understands what the tool is doing and knows the product well enough to stop the tool from breaking it.

Now look at what the industry did with that half. Layoffs.fyi counted 123,941 tech workers cut at 269 companies in 2025, on top of roughly 165,000 in 2022, 263,000 in 2023 and 152,000 in 2024. Not all of those were engineers, and nobody publishes the split. Those totals do not say how much defensive capacity went with them. What they show is an industry removing technical headcount at scale, on the argument that models would absorb the work.

Nothing has gone wrong yet at scale. Mandiant says so and I printed it above. My read is that this is the quiet part, and that a year or three of model-driven intrusions arrives on the other side of it, answered by another model, another team and another bill. That is a forecast and I cannot prove it.

Whoever priced AI as a replacement for the engineer, and development as something close to free, was working from the wrong ledger.

I have argued elsewhere, in AI Agent Platforms, that the agent perimeter leaks in places nobody audited. What is new this month sits one layer in. Somebody worked out how cheap it is to change what a defensive model reports. You write a polite note into the evidence. And if the investigation then runs into the vendor’s own boundary, you apply for a model that is allowed to keep going.