Skip to content

test: coverage audit — real fixtures, dead tests removed, streaming and MMR covered - #171

Merged
lucaosti merged 1 commit into
mvp-testingfrom
test/coverage-audit-and-gaps
Aug 3, 2026
Merged

test: coverage audit — real fixtures, dead tests removed, streaming and MMR covered#171
lucaosti merged 1 commit into
mvp-testingfrom
test/coverage-audit-and-gaps

Conversation

@lucaosti

@lucaosti lucaosti commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Closes #154.

What this does

The coverage-audit layer of the quality gate (#146, layer 1/6): measure what's actually tested, find what isn't, and close the highest-value gaps rather than chase a percentage.

Skipped tests: 19 → 0

17 of the 19 skips had nothing to do with hardware or models — they were guarded on two fixture files that were never committed: docs/src/bitcoin_technical_document.pdf and bitcoin_creative_commons_en.pptx. *.pdf/*.pptx are globally gitignored (local upload cruft), and these two fixtures got caught by that same rule instead of being carved out as an exception.

Generated real, structured fixtures rather than placeholders — a 4-page PDF (headings, paragraphs, a formula, a table) and a 5-slide PPTX — using pymupdf and python-pptx, both already backend dependencies, so nothing new was added. All 17 previously-skipped tests pass against them unmodified.

The remaining 2 (test_chunker.py, test_ingester_parser.py) guarded on module_1_ingestor / module_2_parser / module_3_micro_chunker — never declared as a dependency anywhere, permanently skipped since introduction. They describe a three-level chunking hierarchy that predates the current two-level parent/child design in pipeline.py. Deleted, not fixed: a test for an architecture the code no longer has is misleading, not just inert.

Assertion-free scan

4 flagged by a static scan of every test_* function for a missing assert/pytest.raises. All four are legitimate does-not-raise checks — pytest fails a test on an uncaught exception, so their absence of a raise is the assertion. Strengthened one (test_module_aliases_are_registered_without_error) with an explicit check of the registered alias, since that was cheap and is the actual point of the test.

New coverage: streaming and MMR

The two real gaps the audit surfaced: the SSE streaming paths (chat_service.stream_answer, study_service.stream_dispatch) at ~53-54% despite being the primary interaction mode today, and reranker.mmr_select's actual diversity algorithm — previously only its no-model fallback was exercised.

  • 11 streaming tests: cache short-circuit (no retrieval on a hit), QUIZ/ORAL/RETRIEVE always buffered never streamed, progressive token yield, empty-stream → buffered-generate fallback, a QVAC [ERROR] token mid-stream treated as failure rather than shown to the user, full degradation to a raw source snippet when everything is down.
  • 4 MMR tests with a fake embedding model returning controlled vectors: a diverse lower-relevance chunk beats a redundant higher-relevance one for the second slot, lambda_=1.0 matches pure relevance order, inference failure degrades to a plain top-k slice.

One test documents a finding rather than an assumption I got right the first time: when both streaming and buffered generation fail, the fallback answer is raw_answer — QVAC's own bare string from the initial retrieval call — not pack.context_block(). So retrieved evidence sitting right there in the pack can be silently discarded if raw_answer happens to be empty. The identical branch exists in the non-streaming dispatch() path, so this isn't new. Filed as #172 rather than changed here — this issue is about testing and reporting, not product behaviour.

Numbers

Before After
Backend tests 915 946
Skipped 19 0
Coverage (branch) 78.02% 82.04%
chat_service.py 53% 80%
study_service.py 54% 73%
reranker.py 54% 77%
pipeline.py 61% 73%

mypy clean across 84 files.

Not done here (by design)

Per-endpoint success/validation/not-found matrix and per-service error-path mapping — #154 already flagged auth coverage as complete (test_authorization_matrix.py) and asked for a report, not a rewrite of every layer. The remaining line-level gaps (ChromaDB fallback path in chat_service, two-hop compare/derive merge logic, the low-level _stream_generate SSE parser) are listed in the closing comment on #154 as scoped candidates for whichever of layers 2-6 fits each.

…nd MMR covered

Closes #154.

## Skipped tests: 19 -> 0

17 of 19 skips were not model- or hardware-dependent — they were guarded on
two fixture files, docs/src/bitcoin_technical_document.pdf and
bitcoin_creative_commons_en.pptx, that were never committed. *.pdf/*.pptx
are globally gitignored (local upload cruft), and these two fixtures got
caught by that same rule instead of being carved out as an exception.

Generated real, structured fixtures instead of placeholders: a 4-page PDF
(headings, paragraphs, a formula line, a table) and a 5-slide PPTX, built
with pymupdf and python-pptx — both already backend dependencies, so no
new dependency was added. All 17 previously-skipped tests pass against
them unmodified.

The remaining 2 skips (test_chunker.py, test_ingester_parser.py) guarded
on module_1_ingestor / module_2_parser / module_3_micro_chunker, which
were never declared as a dependency anywhere and have been permanently
skipped since their introduction. They describe a three-level chunking
hierarchy (section -> paragraph -> micro) that predates the current
two-level parent/child design in app/workers/pipeline.py (see the
58b468b refactor that unified them). Deleted rather than fixed: keeping
tests for an architecture the code no longer has is actively misleading,
not just inert.

## Assertion-free test audit

4 flagged by static scan; all four are legitimate does-not-raise checks
(pytest fails on an uncaught exception, so absence of a raise is a real
assertion). Strengthened test_module_aliases_are_registered_without_error
with an explicit check of the registered alias, since that state was
cheap to assert on and is the actual point of the test.

## New coverage: streaming and MMR

Coverage audit surfaced two real gaps: the SSE streaming paths
(chat_service.stream_answer, study_service.stream_dispatch) at ~53-54%
despite being the primary interaction mode, and reranker.mmr_select's
actual diversity algorithm — only its no-model fallback was tested.

- 11 new tests for both streaming paths: cache short-circuit, QUIZ/ORAL/
  RETRIEVE always buffered, progressive token yield, empty-stream fallback
  to buffered generate, [ERROR] token mid-stream treated as failure,
  full degradation to a raw source snippet.
- 4 new MMR tests using a fake embedding model with controlled vectors:
  a diverse lower-relevance chunk beats a redundant higher-relevance one,
  lambda_=1.0 matches pure relevance order, inference failure degrades to
  a top-k slice.

One test documents a found-but-not-fixed behaviour rather than asserting
what I assumed: when both stream and buffered generation fail, the
fallback answer is raw_answer (QVAC's own bare string), not
pack.context_block() — so retrieved evidence can be silently discarded
if raw_answer happens to be empty. Identical branch exists in the
non-streaming dispatch() path. Filed as a follow-up rather than changed
here, since this issue is about testing and reporting, not product
behaviour.

## Numbers

| | Before | After |
|---|---|---|
| Backend tests | 915 | 946 |
| Skipped | 19 | 0 |
| Coverage (branch) | 78.02% | 82.04% |
| chat_service.py | 53% | 80% |
| study_service.py | 54% | 73% |
| reranker.py | 54% | 77% |
| pipeline.py | 61% | 73% |

mypy clean across 84 files.
@lucaosti
lucaosti merged commit 6ea62ea into mvp-testing Aug 3, 2026
5 checks passed
@lucaosti
lucaosti deleted the test/coverage-audit-and-gaps branch August 3, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant