Introducing NexBench: MindFort's Internal Model Evaluation

Attackers now find and exploit 0-days faster than defenders can keep up, which is why continuous pen-testing against attack surfaces that change often has become table stakes. Better models have made that testing faster, but running the same agents has become token- and cost-inefficient: they run for 8-16 hours at a time, sometimes every day. That leaves a gap between how fast attackers find 0-days and how fast teams can fix them. We built MindFort to make security efficient, performant, and abundant.

This led us to become interested in how efficient agents were when executing offensive security tasks. For us to be able to create these offensive security agents, we needed them to be capable of running for multiple hours at a time, continuously through the year, which led us to create an internal benchmark measuring models' ability to balance token efficiency and performance.

Current cybersecurity evals share one flaw: they are boxed in. Real environments are messy and ambiguous, with layered vulnerabilities that can be chained together in different ways. They come in different deployment and authentication shapes and different sizes, which makes it hard for agents to even work their way in, and that is half the battle. So we built our own, modeled on the thousands of real environments we have watched our agents work through.

Introducing, NexBench. NexBench demonstrates the real world environments that agents find themselves in. By running our agents on NexBench, we aim to understand which models can find the most validated vulnerabilities, have the best price to performance ratios, and exhibit long running coherence and ability.

Our eval provides signals of a model's ability to write, reason and execute real exploits against a target, while measuring their price to performance ratio and how well the harness handles the complex environment. In our research, we have found that evaluations of agents in existing benchmarks far outperform the same models/agents in real world environments.

On raw performance, GPT-5.6 performed best with 87 validated findings on its best run. Opus 4.8 came in second with 68 total findings (Figs. 1 and 2).

Note: The leaderboards below rank all twelve models by the highlighted column, each shown at its best run across every reasoning effort. No model or effort is dropped from the evaluation. Click any column header to re-sort.

Where it starts getting interesting is when we start to implement scoring, assigned based on severity of the finding. We assigned a score of 1 to low findings, 2 for medium findings, and 3 for high/critical findings. In general, critical and high findings were harder to find, since they required chaining multiple vulnerabilities together, which makes this metric a good read on overall performance in offensive security workflows. Each score was calculated as shown below and the greatest score out of 3 runs per model was used.

When weighting severity against each finding, Grok 4.5 did impressively well, scoring just below GPT-5.6 Sol and beating out Opus 4.8. Grok did take the full runtime to get there, more than double that of Opus, Luna, Sol, and others, but within the time a pen test should reasonably take it outperformed most models at a very efficient token rate. Another fast model was Kimi K3, which scored a 42 in 49 minutes (Fig. 3).

GPT-5.6 Sol also scored as the most expensive model when looking at cost to accepted finding in our evals, with Sonnet coming in second (Fig. 4). Surprisingly, Kimi K2.7, K3, and GLM-5.2 all scored the best in terms of cost per finding, which we will explore later in regards to overall efficiency.

Looking at token and cost efficiency, we can then deduce which models were able to make the most of token and dollar cost. Our x-axis measures validated points per USD, cost efficiency as a ratio, with our y-axis measuring performance per token, token efficiency. Kimi K2.7 ended up having the best dollar to finding ratio, but token use inefficiency ended up weighing down overall results (Fig. 5). Among the hosted models, Grok 4.5 performed in the inverse, with strong token efficiency, but was more expensive than Kimi K2.7, along with GLM-5.2. The standout on token efficiency, though, is Qwen3.8 27B, run locally on our own hardware: it validated 45 findings on just 6.85M total tokens. That works out to roughly 6.6 findings per million tokens, several times the rate of any hosted model, at about $1.14 per validated finding.

Kimi K3 ended up scoring similarly to Grok 4.5 in terms of token efficiency (1.03 findings per 1M tokens vs 1.02 findings per 1M tokens) but scored significantly better in terms of dollar efficiency, at more than double the findings per USD, making it a really cost efficient choice.

Modeling score to cost in dollars, we can model the pareto frontier of the above models, based on available budgets. Grok on high ended up scoring best, with an additional $300 in spend needed by Sol on high effort to surpass Grok's performance on total points discovered (Fig. 6).

A similar pattern can be observed when modeling the pareto frontier of token cost to findings, with each Grok model appearing along the frontier, no matter the reasoning effort. Additionally, K3 scored on the pareto frontier for both cost and token efficiency (Fig. 7).