Skip to content

Add analysis doc for HurumNet vs notebook output discrepancies#17

Open
mtauraso wants to merge 1 commit into
mtauraso-debugfrom
codex/investigate-output-differences-between-models
Open

Add analysis doc for HurumNet vs notebook output discrepancies#17
mtauraso wants to merge 1 commit into
mtauraso-debugfrom
codex/investigate-output-differences-between-models

Conversation

@mtauraso

@mtauraso mtauraso commented May 7, 2026

Copy link
Copy Markdown

Motivation

  • Document likely causes for differing outputs between the self-contained notebook model and the Hyrax-converted model to aid debugging and reproduction efforts.
  • Summarize architectural, preprocessing, and training/interpretation mismatches that can change logits or probabilities and prioritize fixes.
  • Provide a clear, minimal reproduction procedure and actionable recommendations for aligning Hyrax behavior with the notebook reference.

Description

  • Add a new analysis document at docs/hurum_net_discrepancy_analysis.md that enumerates observed mismatches and their consequences.
  • Identify key discrepancies including an input-channel mismatch (KBModNet instantiated with in_channels=3 in the notebook vs hard-coded in_channels=1 in hurum_net.py), differing preprocessing pipelines (normalize_stamps() vs Hyrax percentile clipping/min-subtraction/sum normalization), and output-interpretation differences (notebook uses F.softmax(... )[:, 1] while Hyrax returns raw logits).
  • Cover additional differences in training behavior such as target encoding, lack of gradient clipping, optimizer/scheduler setup, and class-balanced sampling, and recommend concrete checks and fixes.
  • Provide a suggested minimal reproduction procedure and a prioritized fix order to resolve the most likely root causes first.

Testing

  • No automated tests were added or run for this change because it is documentation-only.
  • This change does not modify runtime code and therefore requires no code-based test updates to validate the analysis document.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant