Test gate 4/6 · RAG quality regression gate

[lucaosti]

Sub-issue of #146 — layer 4 of 6. Depends on #141 (single authoritative evaluation suite).

Goal

Make retrieval quality a merge criterion, so a change cannot silently degrade answers.

The problem this solves

Retrieval quality degrades invisibly. Every test can pass, the application works, and answers are quietly worse — a changed chunk size, a reordered pipeline stage, an upgraded embedding or parsing dependency. Nothing in the current gate would catch it.

This is not hypothetical: #144 upgrades ML and PDF parsing dependencies, and #140 restructures the retrieval pipeline. Both can degrade quality while every unit test passes.

Prerequisite

#141 must land first. Four overlapping evaluation artefacts currently exist with no authoritative baseline; a regression gate on an ambiguous baseline is worse than none, because it manufactures false confidence.

Tasks

Baseline

• Commit a baseline result from the authoritative suite, generated on a known corpus
• Pin the corpus — the baseline is meaningless if the documents behind it change
• Record environment: models, versions, relevant config

Threshold

• Define what constitutes a regression. Retrieval scores are noisy; a single query moving is not a regression, an aggregate drop is
• Establish the noise floor by running the suite several times unchanged and measuring variance
• Set a threshold above the noise floor, and document the reasoning

Gate

• Run the suite when retrieval, chunking, parsing, prompts or ML dependencies change
• Fail the build on an aggregate drop beyond the threshold
• Report before/after in the PR, satisfying the AGENTS.md requirement mechanically rather than by convention
• Make an intentional change reviewable: a genuine improvement that shifts the baseline should update it deliberately, with justification

Cost

• The suite requires models and is slow. Decide whether it runs on every PR or only on PRs touching relevant paths, and document the choice

Acceptance criteria

• A deliberate quality regression is caught by the gate — verified by introducing one
• The threshold sits above measured noise, with the measurement recorded
• Baseline updates are explicit and justified, never silent
• Runtime cost documented and acceptable

Note

Verify by breaking it: shrink the retrieval pool or disable reranking, confirm the gate fails, then revert. An untested gate is an assumption.