Found during a validity audit of #5.
src/export/analysis.ts hard-codes pair_complete: true, and analysis/paired-bootstrap.ts filters on that field plus two others that are numbers by construction. The filter can therefore never exclude a row.
The consequence is that an attempt with no quality verdict enters a paired difference estimand as a hard 0 rather than being excluded as missing. quality_outcome_available_a / _b are recorded and consulted by nothing.
This sits in the analysis path and should land with its own tests, including a case where one arm of a pair has no evaluation result.
Found during a validity audit of #5.
src/export/analysis.tshard-codespair_complete: true, andanalysis/paired-bootstrap.tsfilters on that field plus two others that are numbers by construction. The filter can therefore never exclude a row.The consequence is that an attempt with no quality verdict enters a paired difference estimand as a hard
0rather than being excluded as missing.quality_outcome_available_a/_bare recorded and consulted by nothing.This sits in the analysis path and should land with its own tests, including a case where one arm of a pair has no evaluation result.