Running Thousands of Agent Eval Episodes Without a Single Live API Call
Stateful simulators give each episode its own resettable world to grade.

To use an API for agent evals without hitting production, run the agent against a stateful simulator that reproduces the API’s behavior and gives each episode its own resettable world. Running thousands of episodes without live API calls requires more than realistic responses. The environment must remember actions, expose their cumulative effects, and support grading against the final world state.
Why live API calls break at the scale agent evaluation actually demands
A team launches thousands of parallel eval episodes against a live API and hits rate limits within seconds. Each agent may need several calls to complete its task, so throttling interrupts whole trajectories. A suite that takes seconds locally can take hours or days against a live sandbox. Throughput is only the first constraint, however.
Live service state also changes between runs, so the same scenario can produce different outcomes for reasons unrelated to agent behavior. Writes create a more serious obstacle to repetition. An agent that creates invoices, closes tickets, or initiates payments cannot safely rewind those effects between episodes. Adding episodes compounds these problems because more concurrent calls encounter both changing state and the accumulated effects of earlier actions.
What does agent eval at scale require from a test environment?
An action-taking agent needs an environment with four properties, beginning with isolation per episode. An invoice created in episode 47 must not appear in the invoice list for episode 48. Determinism requires the environment to respond consistently to the same actions from the same starting state. Environmental determinism removes harness noise so that differences in outcomes reflect agent behavior.
Clean reset makes that starting state available again without manual cleanup, allowing automated pipelines to run unattended. World-state observability supplies the evidence needed to grade the result. The Intuit StuntDouble RFC asks graders to determine “did the agent leave the world in the right condition? An agent’s final message cannot answer that question on its own. A fifth requirement follows from running these episodes concurrently, which is horizontal scalability using the same execution mechanism for a handful of episodes and tens of thousands.
Why stateless mocks satisfy none of those requirements for multi-turn, action-taking agents
Stateless mocks remain appropriate for unit-testing tool bindings and agent logic. Stateless mocks are limited when an evaluation depends on earlier actions. A static mock returns a fixture, while a conditional mock selects a response from the current request. Neither keeps track of the world that earlier requests changed. A stateful simulator differs because its response depends on both the request and the accumulated state.
Consider an agent that creates an invoice, lists invoices, and then attempts an operation that should conflict with an earlier modification. The list must include the new invoice. The conflict must arise from the resource’s current condition, rather than from a fixture chosen independently of the earlier write. The RFC states that “the created invoice never shows up in the list unless the scenario author hand-writes both outputs to be consistent — and re-writes them for every trajectory variant”.
A canned 409 response can test error handling, but it cannot establish that the agent caused the conflict through its preceding actions. A status fixture that stays the same also cannot support a create, poll, and cancel lifecycle. Branching trajectories make manual consistency increasingly difficult to maintain. Even read-dominant agents run into this limit when they check whether an order exists before creating it. Stateless fixtures can be repeatable and isolated as responses, but they do not provide an observable, resettable world in which those decisions have consequences.
What stateful simulation adds and why episode-scoped state is the right unit
A stateful simulator begins each episode with a reproducible seed, such as a small document store of plain JSON customers, orders, and invoices. Creates, updates, and deletes modify that episode’s store. Later reads show those writes, so the scenario author does not have to predict every trajectory. The episode owns its state throughout the run. At completion, its final world can be compared with the expected state and then discarded or reset.
The τ-bench / τ²-bench pattern implements tools as read/write functions over a JSON database and grades by comparing final state with expected state. This permits a valid but unexpected route to pass. It also exposes an agent that calls the expected tools but leaves corrupted data behind. Tool order alone cannot reliably tell these outcomes apart.
A shared process-level store allows one episode’s writes to alter another episode’s reads, forcing serialization if interference is to be avoided. Episode-scoped state removes that dependency while preserving reproducible seeds and observable results. The RFC describes the goal as preserving “the library's existing guarantee that concurrent runs never interfere”. That guarantee lets the same evaluation design move from local execution into a parallel pipeline.
How this architecture fits into a CI/CD pipeline for agent development
The practical pipeline begins with deterministic unit tests on every PR, using stateless mocks for tool bindings and output parsing. Those tests catch schema and parsing regressions without needing a simulated world. A targeted agent-layer suite then exercises critical multi-turn paths against stateful simulators, resetting between episodes. This suite can complete in seconds locally. With their own simulator copies, hundreds of parallel episodes can also run within a pipeline job.
The full suite runs nightly or when the agent or prompt changes, distributing thousands of episodes across containers with isolated state. Its aggregate scores can gate releases. That coverage requires upfront scenario authoring, however, including seed data, expected outcomes, and failure conditions. Automation executes those scenarios repeatedly. Local and CI copies do not have to share service capacity as they would in a shared sandbox.
Grading can separate probabilistic language output from concrete actions. An LLM-as-Judge scores output against a rubric and fails the job below a threshold, while world-state comparison grades the action layer without a judge. Version control must cover the environment as well. Model version, prompt version, and simulator version all affect results because a change in LIST behavior can alter an agent’s outcome even when its model and prompt remain unchanged.

Why simulator fidelity to the real API is not optional in this workflow
Once simulated episodes become release gates, those gates can be trusted only if the simulator is faithful to the API. A renamed field, new required parameter, or changed status code can leave a hand-maintained simulator accepting behavior that production rejects. Passing scores can then make the agent seem ready when it is not. The simulator may return plausible data without signaling any discrepancy.
Silent errors are especially difficult to separate from agent behavior. Unrecognized query syntax might return an empty set rather than an error, or a misspelled field might be ignored. Figures that look right can hide the fact that different references return the same results. A harness must check that the simulated API both responds and behaves correctly.
The operational requirement is continuous automated verification against the real API on every release, with published results that make discrepancies visible. This is separate from the high-volume agent episodes. Those episodes need no live calls, while the simulator’s verification process does. For RL training pipelines, environments should also be versioned as containerized artifacts with reproducible image hashes, and fidelity should be tested continuously against current production data after launch.
Fault injection as a first-class concern in stateful episode design
A happy-path suite leaves resilience largely unmeasured. Stateful episodes can introduce latency spikes, timeouts, 429 responses, partial responses, and missing fields while preserving the effects of actions already completed. A timeout after a non-idempotent create is particularly revealing because a retry may produce duplicate records. The resulting state shows whether recovery preserved the intended outcome.
Faults therefore belong in the scenario definition, with the failing call, error, and invocation specified precisely. Random failures at random times prevent reliable comparison between runs. Stateless mocks can return errors, but they cannot expose the cumulative consequences of recovery without maintaining the world those actions change. Final-state inspection distinguishes an agent that merely reports a failure from one that leaves a partially completed task or duplicate records behind.
Practical simulator coverage across the APIs agents most commonly call
With an empty simulator, authors have to create every customer, charge, and invoice before the task can begin. Pre-seeded, interrelated records allow episodes to start mid-task instead. Cross-service identity is equally important for agents whose work spans Slack, Stripe, and Lob. Otherwise, individually plausible API responses can describe incompatible versions of the same customer.
The Stripe simulator covers the full billing lifecycle with pre-seeded customers and charges in multiple states. These include a charge declined and paid on a replacement card, one refunded in full, one disputed, and one bank debit pending. Those states support more than create-and-succeed scenarios. The same clients, invoice numbers, and dates appear across the Slack, Resend, PayPal, and Lob simulators, providing coherent data for multi-service episodes.
The Kalshi simulator illustrates a different state boundary. It provides a local exchange with Trade API v2, price-time-priority matching, and a WebSocket feed for trading bot evaluation. One process holds one exchange, and every client of that port shares its markets and order books. A process therefore defines the shared-world boundary that episode isolation must account for. Teams designing concurrent runs must account for that boundary rather than assuming separate clients receive independent state.
For RL training, deployment must support thousands of simultaneous container instances, each with independent state and no shared infrastructure coordination. Container packaging makes that execution pattern possible. It does not by itself establish fidelity or isolation, which still depend on the simulator’s behavior and state ownership. Coverage therefore includes both the API features represented and the conditions under which episodes can safely run together.
What a scalable agent eval workflow looks like end to end
To implement this workflow, scenario authors define a seed state, the faults to inject at specified invocations, and the expected final world. Each execution container instantiates its own simulator copy, loads that seed, runs the agent to completion, and captures the resulting state. The scenario is written once and executed repeatedly. Because no episode depends on another’s mutable world, execution can expand horizontally without coordinating their writes.
Grading then compares actual and expected state, with trajectory assertions serving as a secondary diagnostic layer. An unexpected tool sequence need not fail if it reaches the correct outcome. An agent that follows the expected sequence still fails if it leaves the wrong final state. Scores from the episodes are combined for the CI gate, while output rubrics grade parts of the response that state comparison cannot assess. A regression can therefore block release before the agent acts against real user data.
The release process must pin and track the model, prompt or agent, and simulator versions together. A simulator update is a release event because it changes the environment in which scores are produced. Separate fidelity monitoring checks that environment against the live API and exposes drift as a simulator regression. This division keeps production calls out of the thousands of agent episodes without treating the simulator as an unquestioned substitute for reality.
Under this design, a prompt-induced behavior regression can be caught in CI within minutes. The evaluation does not wait for shared sandbox capacity or require cleanup of irreversible production actions. Its capacity grows through independent worlds, while its credibility depends on reproducible scenarios and verified API behavior.


