Modeling Operation Order in Integration Tests With a Stateful API Simulator
Stateful simulators preserve resources across API calls to test complete workflows.

Integration testing against stateful APIs requires a stateful simulator that preserves resources across calls, enforces valid transitions, and resets to a known baseline before each test. Tests should execute complete workflows and assert on individual responses and the state after each operation. The integration requires the correct order of operations, and checks against the real API keep the simulator trustworthy without live calls in every test run.
Why individual call correctness is not enough to verify an integration
A POST returns 201, but the service does not persist the resource. An isolated test accepts the response, while a separate GET test receives a configured success payload. Both tests pass even though the data is missing. A sequenced test would expose the failure because its GET must retrieve what the preceding POST created.
A fixed response for GET /orders/42 cannot reflect a preceding POST /orders/42/cancel, and a configured "captured" status can appear before capture ever happens. Individual calls look correct while the workflow contradicts the service’s rules. IcePick points to the semantic gap in interface specifications, which describe endpoints and data types but do not fully describe the service’s logic or how its behavior changes. Endpoint correctness is necessary, but it cannot establish that one operation leaves the service ready for the next.
What “operation order” means as a testing primitive
With operation order, the sequence of state transitions is what the test checks. Each operation has preconditions that must hold for it to run and postconditions that describe what must hold afterward. A charge ID produced during creation becomes input to capture, while capture establishes the state required for a refund. Reordering those calls changes which operations are valid. Even an unchanged GET can require different responses before and after fulfillment.
IcePick formalizes state-mutating operations as TLA+ actions with preconditions and postconditions. Its postPlayer example requires that the player does not exist before insertion and does exist afterward. For a test, the practical consequence is straightforward. Success must leave a condition that later operations can use as well as return an acceptable status code.
Spec2SeqFuzz (2026) builds sequences through next-category prediction, using the executed prefix to help choose the next API category. Its payload reuse mechanism lets later requests reuse parameters extracted from earlier responses. That distinction is important because the sequence must preserve both behavioral dependencies and the runtime values that connect its steps.
Why stateless mocks cannot model operation order
A stateless mock selects a configured response without remembering earlier calls. It has no internal representation of whether a resource was created, captured, updated, or deleted. Consequently, a mock configured to return a captured charge cannot distinguish creation followed by capture and refund from creation followed directly by refund. Both paths can receive the same successful response. There is no current state against which to reject the invalid transition.
Stateless mocks suit isolated endpoint contract checks, where the question concerns a request or response. They cannot establish that a resource referenced by a later call was actually produced earlier. A successful GET may describe a resource that never existed.
Recorded cassettes and record-and-replay playback retain the responses from a particular execution path, but replay does not supply a resource model. A deviation from the recorded sequence can produce the wrong response or no response. Recording a valid workflow therefore does not establish how the service behaves when the caller chooses another path.
What a stateful simulator actually tracks across a call sequence
A stateful simulator keeps a world model up to date. Creation adds a resource that later calls can address by ID, updates mutate that resource, and successful deletion removes it. Reads consult the current model. Transition guards use the same model to reject operations when their preconditions are not met, such as a refund before capture.
IcePick’s state emulator illustrates how execution results affect that model. A successful creation adds the resource, while a successful DELETE removes it. If no matching data exists, typically because a prior POST failed, dependent DELETE or PUT operations are marked NOT_TESTED and skipped. A missing prerequisite is not treated as successful coverage.
Pre-seeded realistic resources broaden the available starting states. Tests can check reads, updates, deletions, and transitions without first creating every resource they need. Reset provides the corresponding boundary between tests. Resetting the simulator to a known baseline before each sequence keeps changes made by one test from affecting where another test starts.
Modeling a realistic multi-step workflow through the Stripe charge lifecycle
In the modeled Stripe charge lifecycle, POST /v1/charges creates a charge in "pending" state and returns its ID. Every subsequent operation references that same resource. The simulator must retain it, so a read at this point reflects the pending state.
Next, POST /v1/charges/:id/capture transitions the charge to "captured". A subsequent GET /v1/charges/:id must expose that change. Only then is POST /v1/charges/:id/refunds valid in this workflow. Skipping capture must cause the rejection described in the documentation, and each read in the sequence must show the charge’s state when the request is made.
The same requirement extends to webhook-driven payment workflows. PayPal’s webhook simulator produces demonstration events that are not associated with any App and do not appear in the dashboard event viewer. Those events alone cannot establish the relationship between an order, its capture, its refund, and webhook delivery. A stateful test double must model those connected steps to make the complete workflow reproducible.
Extending the model to producer-consumer sequences across GitHub and messaging APIs
Payment workflows are one instance of a producer-consumer dependency. An earlier operation produces state or a value that a later operation requires. For GitHub, repository creation, commit push, and PR opening form such a chain. The branch SHA produced by the push becomes input to the PR-open step. A simulator that accepts a PR without valid branch state allows a path that the real API would reject.
For Resend, POST /emails returning 200 and the email.sent webhook firing do not establish delivery. Delivery is a separate downstream fact. A simulator must preserve that distinction so application state driven by webhooks reflects the event that actually occurred, rather than treating sending and delivery as interchangeable.
Spec2SeqFuzz uses producer-consumer dependencies to link later requests to artifacts made by earlier ones. The same structural relationship is central to RESTler. Without a valid prerequisite resource, a dependent request cannot meaningfully test the intended transition, even if its payload has the right shape.
How do you write integration tests where operation order matters?
A sequence-aware test should establish its starting state, execute the workflow, and assert on each transition. The starting point can be a reset baseline, a precisely constructed fixture, or suitable pre-seeded resources. The beforeEach and afterEach pattern provides boundaries for that setup and cleanup.
During execution, response assertions remain necessary. They establish that each call returned the expected status code and response shape. State assertions then establish that the operation had its required effect, such as leaving a newly created charge in "pending" state. A transition-guard assertion checks the negative case by trying a refund before capture and checking for the rejection described in the documentation.
IcePick’s Glacier contract language expresses behavioral requirements through executable semantic contracts based on first-order logic and REST semantics. That approach makes success depend on more than HTTP status codes. It can work without a lot of setup for each test because pre-seeded data provides the surrounding resource graph. The test can concentrate on the operations and state changes it is intended to verify.
Running sequence-aware tests in CI without touching live APIs
Live API dependencies introduce constraints unrelated to the code under test. Rate limits cap how many runs can take place, and third-party latency and downtime can cause a build to fail even when the code change is correct. Live sandbox state also persists unless teardown removes it. A charge left in an intermediate state can therefore affect the next run.
A local stateful simulator removes those external dependencies while retaining the workflow’s state requirements. Its baseline can be restored before each sequence, and its transition guards check calls throughout the run. Running locally also works in offline and air-gapped environments without outbound calls or live credentials. Network access is not required for behavioral depth.
The important distinction is between isolation and statelessness. CI needs isolated tests, but a test of a multi-step workflow needs shared state among the calls within that workflow. Deterministic reset supplies both properties by preserving state during a sequence and discarding its effects before the next one. This eliminates shared-state flakiness caused by dependence on earlier test runs.
Fault injection as a first-class part of sequence testing
A fault injected midway through a workflow tests more than error handling at one endpoint. By then, preceding operations may already have changed the resource. The client must decide whether to retry, roll back, or treat the partial state as terminal. Correct recovery depends on what happened before the fault.
Faults should therefore follow a predefined, repeatable schedule governing timing, frequency, and intensity. Latency at a selected step tests whether the client times out and retries while the resource remains unresolved. A mid-sequence 429 tests backoff and preservation of state across the retry. A later 503 tests how the application handles a partially completed workflow.
Successful status codes also need scrutiny. An HTTP 200 with an incorrect payload can lead the next operation to consume corrupt state without triggering error handling. IcePick identifies this oracle problem as a reason to verify behavior beyond status codes. Without a stateful model, fault injection becomes guesswork because the test cannot establish the resource’s condition before failure or after recovery.
Applying the same sequence model to AI agents that call external APIs
An AI agent must follow the same transition rules as any other API client, but it can choose a different call sequence from one run to the next. A fixed set of responses cannot cover every path that results. The simulator must check each action against the state at that point. That allows evaluation to distinguish an invalid sequence from a valid sequence that produces the wrong outcome.
Multi-turn workflows make this dependence explicit. A write in turn two must be visible to a read in turn four, or the agent proceeds with a false world model. AWS’s ToolSimulator, within Strands Evals, addresses shared state across tool calls through share_state_id.
Reinforcement learning training also needs deterministic behavior, a reset between episodes, and horizontal scaling across thousands of training containers that run in parallel. The same state and action must produce the same result. Repeated, varied evaluations are also needed because single-run success rates can overestimate reliability.
Keeping the simulator honest through fidelity verification against the real API
A simulator can enforce transitions consistently and still enforce the wrong transitions. If the real API changes, tests may continue passing against behavior that production rejects. Sequence tests are especially at risk because a change to a field or status value can break the link between operations even when each response shape is still acceptable.
Continuous verification must therefore check the simulator against the real API before each release for the operations and transitions it models. The comparison must include state outcomes and endpoint shapes. A published fidelity score expresses the fraction of operations and transitions where behavior matches. It makes the fidelity claim auditable and gives teams a concrete basis for judging the simulator.
IcePick identifies outdated specifications that are not synchronized with implementation as a related testing risk. Building a simulator from a specification does not remove that risk unless behavior is rechecked against the service. The practical choice is a simulator with published fidelity scores and a continuous verification process. Live API comparison then establishes the model’s credibility, while routine CI tests exercise complete workflows without live-service rate limits, latency, or credential exposure.


