API Simulator Fidelity Scoring and Drift Detection

Integration testing against stateful APIs requires a simulator that preserves state across calls, so tests can exercise ordered workflows without hitting production. “API Simulator Fidelity Scoring and Drift Detection” explains why that simulator also needs continuous comparison against the real API. A fidelity score is trustworthy only while a verification process keeps its evidence current.

Why a fidelity score without a maintenance process is a false promise

A simulator-backed test suite passes, but the integration fails in production because the real API now requires a field the simulator still treats as optional. The simulator continues to answer every request. Its responses look legitimate, and the tests accept them as correct. The defect escapes because the test environment no longer represents the system it stands in for.

A published fidelity score can deepen that false confidence when it describes an earlier version of reality. The real API may deprecate a response key or tighten an error code without disrupting the simulator’s surface. A score records a comparison at a particular moment. Without maintenance, it offers no evidence that the same comparison would pass after those changes.

For AI agents, the consequence also affects attribution. Unrecognised query syntax may produce empty result sets rather than errors, while a misspelled field may be ignored so that a filter returns the full dataset. Either response can change how an agent behaves while still looking valid. An independent fidelity measure is therefore necessary to distinguish an agent failure from an environment that supplied silently incorrect results.

What a fidelity score actually measures and how one is constructed

A fidelity score compares how the simulator and the real API behave across a defined test surface. The comparison has several distinct dimensions. Response shape fidelity covers fields, types, and nesting, while state transition fidelity covers how the simulated world changes across create, read, update, and delete calls. Error and edge-case fidelity covers the status codes and error bodies returned when inputs are invalid, resources are missing, or constraints are violated. Behavioral fidelity covers business rules, including a 409 when an already-completed order is cancelled or a 429 when a rate limit is crossed.

The deepest structural gap separates stateless fixtures from stateful behavior. A mock that returns the same fixture for every status request cannot show the changes a polling loop expects. The loop may never terminate or may terminate immediately, and a returned ID may not match the resource previously created. A conflict path may be unreachable altogether. Internal consistency does not establish fidelity when the test double omits the transition under test.

Scoring shows those differences by replaying a canonical test suite against both the real API and the simulator. Response body diffs, status code distributions, and field presence rates are the evidence used to score each dimension and work out an aggregate. The component scores preserve distinctions the aggregate can conceal. Coverage defines the boundary of the claim. A simulator can match every covered endpoint perfectly while behaving incorrectly on endpoints or workflows the suite never exercises.

How real APIs change and why those changes break simulator fidelity silently

Once a score is published, even an additive API change can erode its relevance. A new optional field becomes consequential when an agent begins depending on it, and a new enum value creates a branch an integration must handle. Behavioral tightening changes existing expectations more directly. A previously nullable field may become non-null, a deleted resource may begin returning 404, or an error may move from one status-code class to another.

Undocumented changes create a different problem because the specification and the simulator can become wrong together. The spec may say a field is always there while the live API starts returning null under load. A simulator built from that spec continues returning the field. Spec agreement then conceals runtime disagreement. Similarly, changed webhook payloads or retry semantics can escape comparisons limited to immediate request and response behavior.

These changes share a quiet failure mechanism. The simulator still responds, its own rules still agree with each other, and its tests can keep passing. The divergence becomes visible only when external evidence challenges those rules. Spec-to-spec tools such as oasdiff find changes between documented contracts, but they cannot show whether a running API follows either contract. That limitation makes direct comparison with live behavior an essential part of maintaining a score, rather than an optional supplement to specification checks.

The three layers where drift is detectable and what each one catches

Spec-to-spec diffing in CI forms the first layer. It compares the current and proposed OpenAPI specifications to catch deliberate breaking changes before they merge, and oasdiff has 350 breaking-change detection rules within 773 total API change checks and can run as a GitHub Action. Optic’s GitHub repository was archived on January 12, 2026 following Atlassian’s acquisition, and the tool is no longer maintained. Whichever tool is used, this layer needs two valid spec files and cannot tell whether the live implementation matches them.

Runtime contract validation forms the second layer. Schemathesis generates property-based tests from OpenAPI or GraphQL schemas and reports response-schema violations. Pact tests the inverse relationship by asserting that provider changes preserve expectations registered by consumers. PactFlow Drift, launched March 2026, generates test suites from OpenAPI specs using AI and runs them in CI to check implementation conformance. These checks expose code that changed without a corresponding contract update, but their interpretation still depends on a contract that reflects intent.

Live response monitoring forms the third layer. Scheduled polling compares actual responses with an OpenAPI spec or a learned baseline, with no spec required for baseline mode. FlareCanary classifies changes as breaking, warning, or informational and uses multi-sample baseline learning to reduce false positives from conditional fields. It also monitors MCP server tool schemas for AI agents. Its free tier checks 5 endpoints daily, and it also has paid plans.

The layers form a stack. Specification checks cover declared changes, contract tests exercise conformance, and live monitoring observes runtime divergence beyond those test runs. A field that becomes null under load or an undocumented status-code change needs evidence from running endpoints.

Measuring drift severity and deciding when a fidelity score must be rebuilt

Detection establishes that something changed. An additive field and a broken state transition need to be judged differently. Severity therefore depends on both the scale of the difference and the fidelity dimension it affects. Statistical measures help quantify distribution changes, while semantic checks establish whether the simulator still represents the API’s rules.

Population Stability Index, or PSI, offers a conventional ladder originating in credit scoring. Below 0.1 means drift is negligible, 0.1–0.25 means it is moderate, and above 0.25 means it is significant. For simulator verification, that ladder can be applied to tracked response field distributions. Kolmogorov-Smirnov tests and Jensen-Shannon divergence provide complementary distribution measures. None establishes whether a field’s meaning changed while its type remained the same.

AI agent testing also needs two-stage measurement. One stage examines whether the simulator’s candidate response set has widened or narrowed, and the other examines what it actually returns in a run. Even with a wider candidate set, individual outputs may look unchanged. Measuring only those outputs can therefore hide drift that becomes visible in aggregate, which is distinct from detecting a shift in a numerical field distribution.

A practical invalidation policy combines these signals. It uses operational triggers for re-verification: a live monitor reports a breaking change, PSI crosses 0.25 on a tracked distribution, or a previously passing canonical end-to-end test fails on a state transition. A separate simulator-fidelity signal, analogous to the proposed result.simulator_fidelity property for LLM simulators, allows CI to fail or warn on simulator drift without misclassifying it as an agent regression.

What a continuous verification loop looks like end-to-end

Before every simulator release, the verification loop replays the canonical suite against the live API and the simulator at the same time. It compares outputs across every scored dimension and fails the release if any dimension falls below its threshold. Runtime contract validation supplies evidence about implementation conformance within that process. On every PR, spec-to-spec diffing catches deliberate changes to the contract before they merge. Scheduled live response polling in staging and production monitoring then checks for changes between release checks.

A drift alert starts the loop’s decision process of classifying severity, identifying affected fidelity dimensions, and triggering re-verification under the invalidation policy. A changed response key may be a shape discrepancy, while a new post-deletion response may also affect an ordered workflow. Re-running the suite tests those consequences. Fixing only the observed field would leave the score’s broader behavioral claims unexamined.

Stable reference data makes those comparisons meaningful. An empty environment can produce false negatives because account limits, currency validation, and resource ownership checks need realistic inputs to activate. So the loop needs pre-seeded data that is realistic and changes little enough to give useful diffs.

Fault injection also needs to be part of the same release cycle. Latency injection, 429 simulation, and 503 injection need to be checked because real APIs can change their rate-limit and error behavior without notice. Resilience tests need those paths to exercise representative failures. After each successful verification run, the loop republishes the fidelity score with a timestamp so consumers can see when the evidence was refreshed.

How can you do integration testing against stateful APIs with continuous verification

For integration tests where order of operations changes the outcome, a stateful simulator provides the necessary foundation for this loop. A write must affect the next read. Resource creation, updates, and deletion must change the responses that follow, beyond just choosing unrelated fixtures. Continuous verification then establishes whether those transitions still resemble the real API. The relevant comparison is not simply whether a simulator returns plausible JSON, but whether an ordered workflow reaches the same outcomes.

Stripe’s native sandbox mirrors production behavior through sandbox objects that simulate transactions, supported webhooks, simulated readers, and directly testable failure scenarios. A simulator compared against Stripe thus needs evidence that covers more than response shape. Its score must cover webhook delivery behavior and command-line-tool-triggered event fidelity for the paths it includes. The sandbox also has an explicit boundary. IC+ pricing cannot be tested there, so the fidelity score must disclose that gap.

PayPal’s sandbox uses virtual accounts to simulate production behavior and retries failed webhook deliveries. Its simulated webhook events have a separate limitation. PayPal’s official simulator page states that those events are for demonstration purposes, are not associated with any app, and do not appear in the dashboard event viewer. A fidelity score must account for that documented divergence. Otherwise, a match with a demonstration event could be taken as evidence of a match in an application-linked webhook workflow.

For AI agent and RL environment testing, statefulness also shapes whether the environment can give valid evaluation results. Later reads must still see earlier writes, state must be easy to control and reset, and parallel runs need horizontal scale so that shared state does not let them interfere with each other. These properties make ordered tests possible without depending on production state. A current fidelity score then supplies the evidence needed to interpret their results. When that evidence is stale or weak, an apparent agent failure remains entangled with the possibility that the simulator was wrong.