Comprehensive test suite as a mandatory quality gate before merging to master

[lucaosti]

Goal

Build a single, comprehensive test suite that is run and passed before anything merges into master — the mandatory quality gate for the default branch.

AGENTS.md already requires green CI before merge. This issue is about making that requirement mean something: today CI runs mypy, pytest, tsc, lint and Jest, which catches broken code but does not prove the product works.

Why this is needed now

Two findings from the project review make this urgent:

1. CI was not running on master at all. The workflow triggered on branches: [main, rag] — neither of which is the default branch — so pushes and pull requests to master ran no checks. Fixed in chore(ci): document contribution workflow and fix CI branch triggers #124, but it went unnoticed for weeks, which is exactly what an unreliable gate looks like.
2. A large body of work (Integrate the mvp-testing development line into master #130) is about to land from mvp-testing: 117 files, +12,428 lines, including migrations 0003–0006. Merging that without a trustworthy gate is how regressions enter the default branch invisibly.

Scope

The suite must cover every layer, and must fail loudly rather than skip silently.

1. Static and unit

• Type checking across backend and frontend, no ignored errors
• Lint with zero warnings tolerated
• Backend unit tests (~388 functions today) — audit for tests that assert nothing or are skipped
• Frontend unit tests
• QVAC service tests

2. Integration

• Every API endpoint: success path, auth failure, validation failure, not-found
• Authorization: a user cannot read or modify another user's courses, documents, sessions or certificates — asserted explicitly, not assumed
• Course isolation: an evidence pack never contains chunks from another course
• Database migrations: alembic upgrade head from empty and from each prior revision; downgrade where supported

3. Pipeline end-to-end

• Upload → parse → normalize → chunk → index → retrieve → answer, on real fixture documents (PDF, PPTX, DOCX)
• Failure paths: corrupt file, encrypted PDF, empty document, unsupported type
• Behaviour with the QVAC service down, Redis down, worker not running — degradation must be graceful and honest

4. RAG quality regression

• One authoritative evaluation suite with a committed baseline (depends on Consolidate the four overlapping RAG evaluation artefacts onto one baseline #141)
• A merge must not silently degrade retrieval quality — define the threshold that fails the build
• Report before/after in the PR, as AGENTS.md requires

5. Contract stability

• The evidence pack shape, the study action registry and the public API responses are contracts; a change to them must be a deliberate, visible test update rather than a silent break

6. Build and deploy

• Docker images build
• docker compose up reaches a healthy state for every service
• Health endpoints report accurately when a dependency is down

Requirements for the gate itself

• Deterministic. A flaky gate is worse than no gate — it teaches people to re-run until green. Any flaky test is quarantined and fixed, not retried.
• Fast enough to be respected. If the full suite is slow, split it: fast checks on every push, full suite required before merge.
• No silent skips. A skipped test must be reported as skipped and justified.
• Enforced, not advisory. Configure branch protection on master so the suite is a required status check and cannot be bypassed.
• Honest about what it does not cover. Document the gaps rather than implying total coverage.

Tasks

• Audit existing coverage and list what is genuinely untested
• Decide the split between per-push and pre-merge stages
• Implement the missing layers above
• Add fixture documents small enough to commit but real enough to be meaningful
• Configure branch protection on master with the suite as a required check
• Document in AGENTS.md exactly what must be green and how to run it locally
• Verify the gate actually blocks: open a deliberately broken PR and confirm it cannot merge

Acceptance criteria

• A pull request that breaks any layer above cannot merge into master
• Branch protection configured and verified with a deliberately failing PR
• The suite runs locally with one documented command per layer
• Zero flaky tests; any quarantined test tracked with its own issue
• Coverage gaps documented honestly

Note on sequencing

Worth building the structure before #130 lands, so the integration is the first thing the gate protects rather than the first thing that slips past it.