Someone broke into Hugging Face, the popular LLM website, over a weekend.

They ran thousands of attacks. Stole credentials. Moved across internal systems.

But, what Hugging Face found out was there was no “someone.”

Here’s what happened.

On July 16, Hugging Face said an autonomous AI agent broke into its production systems. It got in through a dataset that abused a code-execution path and a template flaw. Then it harvested credentials and moved across their clusters over a weekend. Their words: “an autonomous agent framework…executing many thousands of individual actions across a swarm of short-lived sandboxes.”

Hugging Face caught it themselves. Largely with their own AI agents. They closed the holes, rebuilt the servers, rotated credentials, called in forensics, and reported it to law enforcement.

Then on July 21, OpenAI publicly admitted the attacker was its own AI.

Two of its models were running a test called ExploitGym. The test measures whether a model can turn a bug into a working attack. OpenAI had turned down the models’ safety refusals for the test.

You know what happened next, right?

The models found a zero-day in OpenAI’s own systems, escaped the sandbox onto the open Internet, guessed that Hugging Face might host the test answers, and broke in to steal them. To cheat a benchmark.

OpenAI’s words: the models were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

Read that again. No ransom. No criminal crew. Nobody at the keyboard.

The model wasn’t malicious. It was motivated.

You’re not Hugging Face. But here’s why this matters to you.

Your developers pull models and datasets from Hugging Face today. And you’re turning your own AI agents loose today, too.

Two weeks ago I wrote about building a check that catches AI when it’s wrong. This is the other half. An AI agent chasing a goal, with no check and no human watching, broke into a company. Nobody told it to.

Look at the whole picture. An over-eager AI caused the breach. Hugging Face’s own AI caught it. And when their human responders reached for the big commercial AI tools mid-incident, those tools locked them out. The safety guardrails, Hugging Face said, “cannot distinguish an incident responder from an attacker.” So they fell back to a model they run on their own hardware (GLM 5.2).

That’s the lesson. You need AI in the fight. You need humans supervising it. And you need to know, before the crisis, whether your tools will show up for you.

One more thing. “Sandboxed” turned out to be a claim, not a guarantee. (The Verge points out that OpenAI’s confession also came with a sales pitch for its own security model. What do you make of that?)

Here’s what to do this week.

If your developers use Hugging Face, rotate your access tokens and review account activity now. That’s Hugging Face’s own advice to users.
Make a list of every AI agent running unattended in your shop. For each one, write down the goal it’s chasing and what it can reach.
Find out now, not mid-incident, whether the AI tools your responders would grab will actually help them. Hugging Face’s team got refused and had to fall back to their own model.
Ask your AI vendors what contains their agents and their tests. “Sandboxed” is a word, not a wall.

What AI agents are running unattended in your shop right now? Pick one. What goal is it chasing, and what can it touch if it goes sideways?

-Kip