Skip to content

Probabilistic metrics & ensemble plumbing - consolidated track for v0.8.0 #649

Description

@sadamov

I have accumulated 7 open PRs that each add a piece of probabilistic / ensemble groundwork independently. I propose to consolidate them into a single curated PR that lands alongside or just before the Graph-EFM port (#648).

Why consolidate rather than merge piecemeal:

  1. Each contributor guessed at input shape conventions for ensemble tensors ((M, B, T, N, F), (B, S, ...), (M, ..., N, d_state) are all in flight). Sequential merges lock in inconsistency.
  2. There is no probabilistic model on main yet (feat: add latent encoder/decoder infrastructure for Graph-EFM port #648 is draft), so the metrics have no downstream consumer to validate the API against.
  3. Several PRs touch the removed ARModel (post-Refactor model class hierarchy into composable Forecaster/StepPredictor layers #208) and need rewrites against ForecasterModule regardless.

PRs being folded in (closing with credit):

PR Author What gets absorbed
#226 @shauryam2807 sample-based CRPS (crps_ens) + ensemble spread (spread_squared)
#402 @hritikkumarpradhan energy-score CRPS + --ensemble_members CLI wiring
#489 @Debadri-das expand_ensemble_batch / fold_ensemble_batch utilities
#521 @Debadri-das 4D ensemble tensor support + zarr export bridge
#523 @Debadri-das pred_std post-softplus scaling
#541 @aryannzzz rank histogram (Talagrand)
#296 @Shivampal157 metrics module test coverage (CRPS / NLL / reductions)

API contract to design here before implementation:

  1. Ensemble tensor shape: pick one convention and stick with it (proposal: (B, S, T, N, F) everywhere, S = ensemble member dim, immediately after batch).
  2. Where ensemble metrics get registered (extend DEFINED_METRICS? sibling ENSEMBLE_METRICS dict? metric receives ensemble-aware flag?).
  3. What ForecasterModule.validation_step / test_step logs when an ensemble forecaster is active.
  4. Whether pred_std lives on the forecaster, the step predictor, or both.

Ownership: TBD, @Sir-Sloth-The-Lazy ?

Authors of the absorbed PRs: thanks for the work. Your changes will be cited as the source for the corresponding pieces of the consolidated PR. If you want to be involved in the design discussion above, please subscribe to this issue.

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions