Test gate 3/6 · End-to-end pipeline and graceful degradation tests

[lucaosti]

Sub-issue of #146 — layer 3 of 6.

Goal

Prove the ingestion-to-answer pipeline works end to end on real documents, and degrades honestly when it cannot.

Why this layer matters most

Unit tests cover the pieces; this is the only layer that proves they compose. The pipeline spans a Python service, a Node service, a queue, a vector store and a database — every integration point between them is untested by unit tests, and the pipeline is the product.

Happy path

• Upload → parse → normalize → chunk → index → retrieve → answer, for a real PDF, a real PPTX and a real DOCX
• Assert at each stage, not only at the end: parsed output faithful to source, chunks carry correct metadata, index queryable, retrieved chunks relevant, citations resolve to the correct page/slide
• Content actually retrievable after ingestion — the single most important assertion in the suite

Fixtures

• Commit small but genuine documents: a slide deck with real structure, a PDF with headings/tables/formulas, a DOCX
• Keep them small enough for CI, real enough to be meaningful — a one-page "hello world" PDF proves nothing about parsing lecture material
• Include one document with known-difficult content (formula-heavy, or a table spanning pages)

Failure paths

• Corrupt PDF
• Password-protected PDF
• Empty document, or one that parses to no text
• Unsupported file type
• File exceeding the size limit
• A document that parses but yields zero chunks

Each must produce an actionable error and leave the document in a coherent state — never stuck in processing forever.

Degradation

• QVAC service unavailable → study actions fail clearly, not with an opaque 500
• QVAC_LLM_ENABLED=false → retrieval-only mode returns passages for every action
• Redis unavailable → cache misses, system still functions
• ARQ worker not running → documents queue rather than silently vanishing
• Database unavailable → health endpoint reports it accurately

Acceptance criteria

• Full pipeline verified on all three formats with stage-by-stage assertions
• Every failure path leaves a coherent, inspectable state
• No degradation scenario produces an opaque error or an infinite processing state
• Fixtures committed and documented

Note

The "stuck in processing forever" case is already known — the README troubleshooting table lists it. It should be a test, not a documented workaround.

[lucaosti]

Partial progress in #166.

Landed — the degradation half:

• QVAC unreachable → empty evidence pack, not a propagated transport error
• Generation fails after successful retrieval → falls back to retrieved passages rather than erroring
• rag_only mode → asserts /generate is never called, for every one of the eight study actions
• Redis unavailable → cache misses instead of failures; token blacklist degrades to in-memory
• Database unreachable → /health reports degraded honestly
• Reranker backends missing or failing → vector order preserved, rerank_score left at 0.0

Not landed — the end-to-end half, which is what this issue is really about:

• Upload → parse → normalize → chunk → index → retrieve → answer on real documents, with assertions at each stage rather than only at the end
• Committed fixtures: a genuine slide deck, a PDF with headings/tables/formulas, a DOCX
• Failure paths: corrupt PDF, password-protected PDF, empty document, unsupported type, oversized file, document that parses to zero chunks
• The "document stuck in processing forever" case — currently a documented workaround in the README troubleshooting table, which is exactly the kind of thing that should be a test

The degradation tests are in tests/integration/test_course_isolation.py.

[lucaosti]

Partial progress in #173 (merged into mvp-testing).

Closed:

• Pipeline-level failure handling verified for real (not assumed): unsupported extension, corrupt PDF, empty PDF, and the "parses but yields zero chunks" case — all reach a terminal, inspectable state (ERROR with a message, or READY with an honest zero count), never stuck.
• API-level validation: unsupported MIME → 415, oversized file → 413, both with no orphaned CourseDocument row; a file at exactly the 50 MB boundary → accepted.
• The upload-queueing half of the README's documented "stuck in processing forever" symptom: no arq_pool → falls back to BackgroundTasks; arq_pool present → enqueues through it rather than silently running in-process.

Still open (this issue stays open):

• Password-protected PDF — not done, because I hadn't verified what pymupdf4llm actually raises for one and didn't want to assert against a guess
• PPTX/DOCX-specific failure paths (only PDF covered; the happy path already covers all three formats via the fixtures from test: coverage audit — real fixtures, dead tests removed, streaming and MMR covered #171)
• docker compose up reaching a healthy state / health endpoints under degradation — that's layer 6 (Test gate 6/6 · Enforce the gate — branch protection, flakiness, speed #159) territory, tracked there instead

955 backend tests now (was 915 at the start of the coverage-audit pass), 82.26% branch coverage, mypy clean across 84 files.