Things to Build

security companies that already handle adversarial code should sell that infrastructure to teams running cyber evals. my bet is that their experience operating it could save customers the work of building and securing their own test environments, and they're prob way better at it too

you'd bring an eval harness and vulnerable targets, define which systems the agent can reach, and get an isolated environment with the dependencies installed. the provider would manage network access and credentials outside the agent's control, preserve traces, and reset the environment between runs.

the oai hugging face incident shows why the connected services matter too. agents abused a permitted package service to reach the internet before compromising hugging face. a service aimed at preventing a repeat would need to contain that whole path, including what the package service could access.

there's alr some semi-players in the space, but not yet fully fleshed out as a business model. CrowdStrike sells sandbox analysis of malware and URLs. E2B provides isolated microVMs, and Modal offers sandboxes with network restrictions. Cloudflare supports keeping credentials outside the sandbox and restricting internet access. NVIDIA's OpenShell also enforces policies outside the agent's reach. the strat here i think is to sell all of these things full-suite

i'd look for customers who need someone to operate the full cyber testing environment. the first buyers i'd try are AI teams running offensive evals without a dedicated containment team. they'd pay for recurring capacity and for keeping the environment secure as their harness changes. they then continuously come to us each time they run evals OR we can sell them the infra, reusable target setups and evidence that our containment mechanisms get better all the time (since it's a core part of our actual security product as well). this works especially well since these customers can then run tests asap that they'd otherwise delay due develop the proper safeguards.

this feels like a plausible expansion for a security company with relevant infra and operators, already driving agent first security ops. a standalone business would need enough reusable work to avoid becoming bespoke consulting. the advantage would have to come from continuously testing and operating the boundary; being a security company alone doesn't establish that yours works better, esp if you're not doing these in your daily ops, i don't see anyone paying deloitte cyber for example and trusting them to know how to effectively sandbox adversarial agents.

i'd first test whether customers will pay for that operating responsibility, and whether the restrictions still let them run useful evals. obviously there's some fde / integration frontloading that needs to be done here, if every labs specific eval needs custom infra, adapting your containment controls gets harder.

if the same infra can support both products, security companies get another revenue stream from work they already have to do, and customers get to run adversarial evals sooner with safeguards they can actually test.