23_parity_and_reproducibility.md

Parity and Reproducibility

Parity Layers

• check_snapshot_raw_parity.py: raw NDJSON vs snapshots
• check_data_pipeline_parity.py: pipeline-focused parity report output

Strictness

• Value and dtype equality: strict
• Row and column order: not required where datasets are logically unordered

Reproducibility Protocol

1. Import/refresh canonical raw sources.
2. Run pipeline.
3. Regenerate manifests.
4. Validate parity.
5. Compare checksums in manifests.

Drift Triage

If parity fails:

1. Inspect data/manifests/parity_report.json
2. Generate QA report: scripts/data_qa_report.py
3. Locate failing dataset + transform stage
4. Re-run pipeline for targeted dataset if needed