Skip to content

Add MeanReciprocalRank metric for recommender systems (#2631)#3795

Open
sheetalsattiraju wants to merge 1 commit into
pytorch:masterfrom
sheetalsattiraju:feature/mean-reciprocal-rank
Open

Add MeanReciprocalRank metric for recommender systems (#2631)#3795
sheetalsattiraju wants to merge 1 commit into
pytorch:masterfrom
sheetalsattiraju:feature/mean-reciprocal-rank

Conversation

@sheetalsattiraju

@sheetalsattiraju sheetalsattiraju commented Jun 28, 2026

Copy link
Copy Markdown

What does this PR do?

This PR adds MeanReciprocalRank (MRR) as a new metric under ignite/metrics/rec_sys/, alongside the existing HitRate and NDCG metrics. The implementation supports a single or list of top_k values, an ignore_zero_hits flag to optionally exclude users with no relevant items, distributed training via sync_all_reduce, and returns a list of MRR@k values ordered by sorted top_k. A corresponding test file is included, mirroring the structure of test_hitrate_metric.py.

Why was this PR needed?
In issue #2631, there was discussion around implementing other Recommender System metrics (besides HitRate and NDCG@k). I worked on MRR, a similar metric to HitRate. Mean Reciprocal Rank (MRR) measures the average of the reciprocal ranks of the first relevant item in the predicted ranking for each user. If no relevant item is found in the top-k predictions, the reciprocal rank for that user is 0. The math is as follows: MRR@K = (1 / N) * sum(1 / rank_i) for i = 1 to N.

While HitRate measures whether a relevant item appears anywhere in the top-k, MRR additionally captures where the first relevant item ranks. Without it, users evaluating recommender systems with ignite had no built-in way to compute MRR and would need to implement it manually outside the framework.

What is the relevant issue number?
Issue #2631

Does this PR meet the acceptance criteria?
[x] Tests added for new/changed behavior -- tests/ignite/metrics/rec_sys/test_mrr.py mirrors test_hitrate_metric.py, covering unit tests, edge cases, and distributed tests
[x] Follows project style guide -- implementation mirrors the structure, decorators, type hints, and docstring format of the existing HitRate metric
[x] No breaking changes introduced -- add-on, no existing metrics or APIs were modified

@github-actions github-actions Bot added the module: metrics Metrics module label Jun 28, 2026
@aaishwarymishra

Copy link
Copy Markdown
Collaborator

can you also add corresponding tests for the metric, you may also have to add it in docs/source/metrics.rst

@aaishwarymishra

Copy link
Copy Markdown
Collaborator

hi I just noticed there is a pr already opened for this , thanks for the pr but I will encourage you to maybe look for some other issue :)
pr #3573

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: metrics Metrics module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants