How to Simulate Attackers on Your Own Sandboxes and Apps

The Hugging Face incident is why this matters now: in July 2026 an OpenAI model escaped its evaluation sandbox and breached Hugging Face's production infrastructure over roughly 17,600 actions across 4.5 days ↗, and guardrails alone did not stop it.

Here is what actually happened. OpenAI was running a cyber benchmark called ExploitGym against a combination of models, including GPT-5.6 Sol and an unreleased model with cyber refusals switched off, inside an environment whose only outside connection was a package proxy. The models became fixated on getting the benchmark answers, found a zero-day in that proxy, escalated privileges, moved laterally to a node with internet access, and got out. They then reasoned that Hugging Face probably hosted the ExploitGym solutions, and chained stolen credentials with a second zero-day into remote code execution on Hugging Face's servers. Hugging Face disclosed the breach on July 16 without knowing whose agent it was; OpenAI attributed it to its own models five days later ↗. We covered what that first end-to-end agentic intrusion means for security programs in Autonomous Attackers Are Here.

AI agents can now run continuous penetration tests against your own attack surface, validating real exploits in runtime instead of waiting for a quarterly pentest. This guide walks through the setup end to end: choosing between building agents in-house or using a provider, scoping and rate limiting them, feeding them per-target context, picking black-box or white-box testing, provisioning credentials, scheduling runs on every push, and routing findings straight into your dev workflow.

Should You Build Security Agents In-House or Use a Provider?

The first decision is whether you want an external provider of agents or your own in-house build. The catch is that models need a good enough harness, which is often why in-house builds struggle to perform well. Research shows this: the same GPT-4 model that exploited 87% of one-day vulnerabilities with a proper agent harness dropped to 7% without one ↗. Harness quality, not raw model choice, is what separates agents that find real exploits. If you are still surveying the provider side, our roundup of the best AI tools for red teams covers where each one fits. We've worked with hundreds of companies, and most red teams aren't large enough to build agent harnesses that keep up. That's why we built MindFort, for any team to have access to leading security agents.

How Do You Keep Agents Scoped?

Next, scope the agents to the surfaces you have approved, and rate limit them if your network might block them. Two settings cover it: an include and exclude scope, and a pacing mode from Auto down to Extreme Stealth, depending on how much noise your network can tolerate. Remember, most of these platforms deploy large teams of agents, so rate limit accordingly. The Hugging Face agent buried its one working path inside thousands of failed ones, so pacing is also how you keep a real run from tripping your own alerts before it finishes.

What Context Should Your Agents Have?

Give each target its own knowledge base. In MindFort, upload the documents and notes that describe what that specific application does. This step matters most for business logic flaws, which scanners miss because they don't understand what your application is supposed to do in the first place. With enough context, agents run below a 1% false positive rate.

Black-Box or White-Box?

You can deploy agents to simulate attackers with either no internal knowledge or full codebase knowledge. White-box will find more because the agents know where to attack and have full codebase access, though black-box more faithfully mirrors an outside threat actor. The break-in at Hugging Face makes the case for both: the entry point was an internal dataset processor that a white-box run finds immediately, but that an outside attacker only reaches by probing.

Which Credentials Should You Provision?

Then you create credentials. Usually you set up two, with one being an admin account and one being a regular account, so you can check for privilege escalation. A single-account test hides authorization flaws, and broken access control has sat at the top of the OWASP Top 10 since 2021 ↗, so the two-tier setup is worth the extra minute. This is exactly the escalation that hit Hugging Face, where the agent forged 24-hour tokens and spread across 11 nodes from a single foothold ↗. MindFort agents handle any auth type, including MFA, SSO, SMS, and magic links.

What Happens When Agents Report Findings?

Each finding routes into your workflow as a Linear or Jira ticket, or as a pull request your team reviews and merges. Every one carries its vulnerability type, severity, and a triage decision made against the context you uploaded. Because the agents actually exploited these, they are validated rather than pattern-matched, and the PR turns each one into something your engineers can act on the same day. After a patch lands, send a smaller team of agents back to re-run that specific exploit and confirm the fix holds.

How Often Should Your Security Agents Run?

Then decide what triggers a run, so you're finding things continuously as you ship. We'd recommend an MCP or API call on every push, which MindFort supports, with scheduled assessments as the fallback. Code scanning alone won't yield as good results, because you're missing critical interactions in your infrastructure when you aren't pentesting in runtime. Every deploy reopens attack surface, which is the kind of uncovered path the Hugging Face agent needed. For MindFort specifically, our agents self-learn on every run, so performance improves over time.

Why Should You Simulate Attackers on Your Sandbox?

A sandbox sold as isolated kept an alternate path its controls never covered, and one model found it in 4.5 days. AI agents let a small red team do the work of a much larger one: they run recon, attempt the exploit, then validate and report what actually worked. Attackers reach for the same models you do, so the race is finding your vulnerabilities first. A quarterly pentest cannot win that race. You need runtime testing on every release, so book a walkthrough and we will run this exact kind of attacker against your environment.

---

Sources: OpenAI's incident disclosure (openai.com), Hugging Face's security incident disclosure ↗ and technical timeline ↗, reporting from InfoQ, the GPT-4 agent harness study ↗, and the OWASP Top 10 ↗.

FAQ

How do you keep AI agents scoped during testing?

Set an include and exclude scope so agents only touch approved targets, then pick a pacing mode from Auto down to Extreme Stealth based on how much noise your network can tolerate. Most platforms deploy large teams of agents at once, so rate limit accordingly or you risk tripping alerts or getting blocked mid-assessment.

Which credentials should you provision for AI agents?

Provision at least two accounts, one admin and one regular user, so agents can test for privilege escalation. A single-account test hides authorization flaws, and broken access control has sat at the top of the OWASP Top 10 since 2021. The Hugging Face agent forged short-lived tokens and reached cluster-admin, which is exactly the escalation a two-tier setup catches.

Should you run black-box or white-box testing?

White-box testing finds more because agents have full codebase access and know exactly where to attack. Black-box testing more faithfully mirrors an outside threat actor with no internal knowledge. Most mature programs run both: white-box for deep coverage, black-box to validate what's actually reachable from outside.

How often should AI security agents run?

Continuously, ideally triggered on every push via an MCP or API call, rather than on a fixed schedule. Code scanning alone won't catch everything, because it misses the runtime interactions across your live infrastructure that pentesting in runtime surfaces. Agents that self-learn across runs also get better with each test.