Sandboxed High-Fidelity API Simulators for Reproducible Agent Integration Tests

Stateful, seeded, observable simulators catch agent failures that stateless mocks miss.

· 8 min read
Cover illustration for “Sandboxed High-Fidelity API Simulators for Reproducible Agent Integration Tests”
AI Agent Testing · November 18, 2026 · 8 min read · 1,798 words

Reproducible agent integration tests require sandboxed API simulators that preserve and evolve state across sequential calls. To evaluate an agent against realistic API responses without touching production, the simulator must combine isolation, statefulness, determinism, and observability. Tests then need a known starting state, deliberate failures, and continuous checks against live API behavior.

Why stateless mocks cannot reproduce the failure modes that matter most for agents

An agent creates an order, receives a 201 with an ID, polls that ID until its status changes, and then cancels it. A stateless mock returns its fixture regardless of what happened before. It can report “captured” before any capture occurs, leaving the polling loop to run indefinitely or terminate on the wrong state. The conflict path may never become reachable. The mock passes a sequence the real API would reject.

A successful response to one request does not establish that the next request sees the resource in the correct state. Agents depend on that continuity because their next action follows from the response they just received. Static fixtures can exercise individual responses and error handlers, but they cannot establish whether the agent followed a valid sequence of transitions.

In multi-agent systems, this problem also affects handoffs and shared state. Each agent can pass its own unit tests while the system as a whole deadlocks, loops, or hands off the same task again and again. Tests that check only individual tool calls also miss a poisoned document that hijacks a planner or state that drifts after many handoffs. LLM non-determinism compounds the gap. The same prompt can produce a different plan on the next run, so one green result does not establish reliable behavior.

The four architectural properties a simulator must have to make agent tests both isolated and reproducible

Isolation prevents actions inside the sandbox from reaching the real world. Without it, a runaway loop can escape test containment and become a customer transaction or a sent Slack message. Location alone is not a sufficient boundary. Where the agent runs and what it can reach are separate questions, requiring a private filesystem, blocked or allowlisted outbound calls, protection against privilege escalation, and hard limits on CPU and memory.

Statefulness preserves the consequences of actions within a run. A created resource must be available to later reads, lists, updates, and deletes, with each response reflecting prior changes. Removing persistence severs that causal chain. In a payment flow, creating a charge, capturing it, and refunding it must each change what a subsequent GET returns. Static and conditional mocks work for many error-path tests, but create-then-read flows need stored state.

Determinism keeps the conditions in which a failure happens the same. A fixed seed and recorded state make the environment replayable instead of leaving developers to chase changing test data. The starting conditions must remain the same. This removes environmental variation that would otherwise obscure whether a changed result came from the agent or its surroundings.

Observability makes the sequence available for inspection. Without a full trace, isolation produces a black box in which a failure is contained but unexplained. The trace needs every tool call, argument, response, handoff, and state diff, together with recorded planning reasoning and the final action. A correct action reached for the wrong reason can otherwise pass unnoticed. Together, these four properties define the criteria for evaluating the simulator’s remaining capabilities.

Why state must be reset to a known baseline between runs, and what "known baseline" actually means

State must persist within a run and stop persisting across runs. Otherwise, an earlier test can change the conditions of a later one, making execution order determine the result. Implicit scenario state needs explicit clearing too. Parallel execution makes this requirement stricter because concurrent agents must not observe each other’s writes. Stateless mocks handle requests independently, while stateful simulators need deliberate state management to achieve comparable independence.

A known baseline is a realistic, pre-seeded world whose contents are determined by the seed. Starting empty forces every test to construct its own setup, which is slow and brittle and tends to produce minimal datasets. Realistic data exposes different faults. Nullable nested fields, references to deleted records, heterogeneous array members, and money represented as integer cents all challenge assumptions that clean fixtures leave intact.

ReqRes Agent Sandbox illustrates this kind of seeding with cursor-paginated users whose requested limits can be silently clamped and orders that reference deleted products whose name and price fields are null. Those conditions are deterministic per seed. The broader requirement is rollback to that pre-seeded baseline after each run. For parallel CI workloads, each run also needs an isolated state namespace. Reset and separation together keep the test’s history from becoming an undeclared input.

How fault injection fits into the simulator's job, and why it cannot be bolted on afterward

An agent’s response to degraded APIs must be tested deliberately before production. The simulator therefore needs to inject rate limits, server errors, gateway failures, timeouts, malformed responses, partial content, conflicts, and empty responses. Each exercises a different recovery condition. A test that only proves the agent can finish when every request succeeds leaves those conditions unexamined.

The point of injection must follow the workflow’s state. A 409 conflict that becomes valid only after a particular sequence cannot be represented faithfully by attaching an arbitrary error response to a standalone request. The simulator needs to know what preceded it.

ReqRes has fifteen failure modes that tests can trigger on purpose through a dedicated scenarios endpoint and marks them with an intentional header so error reporters can keep them off alert dashboards. Returning immediately with a simulated-delay header preserves the testing signal without exhausting connection pools by holding connections open. The test must distinguish an intentional simulated failure from a failure of the test infrastructure.

Asynchronous workflows add another set of state-dependent failures. Out-of-order webhooks, token timeouts, and late retries require a simulator that understands asynchronous state. Cancellation tests, for example, must show that cancellation does not fire while approval is pending and that the subscription stays the same if approval is rejected. Restart behavior belongs in the same sequence. Testing whether the workflow survives a restart and cancels exactly once requires more than an isolated error response.

What continuous verification against the live API means and why a simulator that skips it will silently drift

A simulator can satisfy these architectural requirements and still become inaccurate. APIs change the shapes of their responses, add fields, change what errors mean, and change how rate limits work. Without repeated comparison against the live API, the simulator accumulates divergence while its tests continue to pass. The resulting confidence applies to the simulation.

The source of the simulator does not remove this risk. Spec-derived mocks inherit the accuracy of the OpenAPI document, while handwritten mocks depend on the author’s understanding of the API. Neither is verified by construction. For reinforcement learning, divergence also changes the training signal because the agent can optimize for behavior that exists only in the simulator. A verifier that has drifted can miss an integration defect and also reward the wrong behavior.

Before every release, the simulator should be compared with the live API, with the results published as a fidelity score. A falling score should block the build.

What a production-grade stateful simulator looks like for the APIs agents commonly call — Slack, GitHub, Stripe, PayPal, and others

Slack, GitHub, Stripe, and PayPal cannot be treated as interchangeable collections of endpoints because their stateful behaviors and failure paths differ. Endpoint coverage alone is insufficient. A production-grade simulator needs realistic pre-seeded data, controllable state, and verification against the real API before release. Published fidelity results then give teams a basis for judging the behavior their tests depend on.

Stripe’s first-party test mode has test card numbers, simulated webhook events, and a dashboard that shows test transactions in a parallel API environment. Teams use it to test the whole payment flow in CI without moving real money. The sequence is the test’s core. Creating a charge, capturing it, and refunding it must each put the resource in a different state that later reads show. A fixture returning a successful payment response does not establish that those transitions work.

PayPal illustrates why missing events belong in integration tests. A documented pattern creates a sandbox order, approves it through a sandbox buyer account, and deliberately drops the merchant webhook. The reconciliation agent then uses PayPal APIs to confirm capture, check for disputes, and issue a refund. Each step depends on the earlier state. A stateless mock cannot represent the successive states of an order being created, a webhook being dropped, and reconciliation being confirmed.

Slack offers a different comparison through the self-hostable agent-slack fixture server, invoked with go run ./cmd/mockslack. Its maintenance and verification model distinguishes this integration-testing fixture from a continuously verified simulator. A manually maintained fixture can support integration tests, but its existence alone does not establish fidelity to current Slack behavior. Teams need the same evidence when they assess GitHub or another API simulator.

For agent evaluation and reinforcement learning, production readiness also includes deterministic, controllable state and horizontal scaling across containers. Scale must preserve isolation between simultaneous runs. More containers do not improve an evaluation if their agents can read or overwrite one another’s state.

How can you structure a CI pipeline for agent integration tests that use a stateful simulator?

Contract tests and smoke tests should run first and finish quickly without having to start a full stateful simulator. Stateful integration tests follow for multi-step flows, asynchronous webhooks, and injected failures. Lightweight checks should remain lightweight. A test that needs only an in-memory check gains nothing from the additional setup of a full sandbox.

Each stateful run should begin in its own disposable environment, restored to the pre-seeded baseline before the suite executes. After the run, the environment should be discarded so that no state reaches the next test. Parallel containers need independent state as well. Short-lived, narrowly scoped credentials should be injected only after restoration, never baked into an image or snapshot. This keeps the restored environment separate from the credentials used for that execution.

Rate-limit handling, timeout behavior, and recovery from 5xx responses belong in the mandatory gate. The simulator’s published fidelity score should also be a build input, with a decline since the last verified release triggering an alert before results are trusted. A green happy path is not enough. Both the agent’s recovery behavior and the simulator’s accuracy need to support the release decision.

With a fixed seed, a known baseline, and a continuously verified simulator, CI produces a failure that can be replayed against the same environment. The recorded trace supplies the calls, handoffs, and state changes needed to examine that failure. Tuesday’s failure can be investigated on Wednesday without reconstructing an unknown starting world.

Filed underAI Agent Testing

More in AI Agent Testing