Skip to content

Enforce the published OpenAPI contract with property-based CI #56

Description

@yhay81

Problem

The OpenAPI document is linted statically, but no independent black-box gate verifies it against the running server. A deterministic Schemathesis v4.24.3 discovery run selected all 46 operations and generated 403 cases. It found 19 unique contract failures:

  • 18 responses used valid runtime error statuses such as 404 or 422 that the operation did not document;
  • DELETE /suppressions/{email} returned { object, email, deleted } while the shared deletion schema required { object, id, deleted };
  • eight operations repeatedly rejected inputs that the OpenAPI schema described as valid because runtime string constraints were stricter.

No generated case caused a 5xx response, but generated clients and direct HTTP users cannot currently rely on the published error and validation contract.

Scope

Align the checked-in OpenAPI document with the existing runtime contract and add a deterministic, loopback-only property-based CI job. This is contract hardening, not a runtime behavior change.

Acceptance criteria

  • Every protected operation documents the shared JSON error shape for non-success responses without masking server errors.
  • Suppression deletion documents the actual email-keyed response.
  • OpenAPI request constraints match the corresponding Zod constraints for the cases identified by the discovery run.
  • Pin the official Schemathesis v4.24.3 container by OCI digest rather than a floating tag.
  • Refuse any non-loopback contract-test target before starting the test tool.
  • Exercise all 46 operations against local in-memory HayaSend with deterministic generation.
  • Require not_a_server_error, status-code, content-type, and response-schema conformance.
  • Do not persist generated request or response bodies as artifacts.
  • Keep Node, Python, and Go official SDK gates green.
  • Document the new direct HTTP compatibility evidence and tool/version pin.

Non-goals

This issue does not replace scenario-based SDK tests, connect to AWS or SES, or require generated tests to create every stateful resource graph.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcompatibilityOfficial SDK and API behavioral compatibilityroadmapAccepted public roadmap work

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions