Add MeanReciprocalRank metric for recommender systems (#2631)#3795
Open
sheetalsattiraju wants to merge 1 commit into
Open
Add MeanReciprocalRank metric for recommender systems (#2631)#3795sheetalsattiraju wants to merge 1 commit into
sheetalsattiraju wants to merge 1 commit into
Conversation
Collaborator
|
can you also add corresponding tests for the metric, you may also have to add it in |
Collaborator
|
hi I just noticed there is a pr already opened for this , thanks for the pr but I will encourage you to maybe look for some other issue :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds MeanReciprocalRank (MRR) as a new metric under ignite/metrics/rec_sys/, alongside the existing HitRate and NDCG metrics. The implementation supports a single or list of top_k values, an ignore_zero_hits flag to optionally exclude users with no relevant items, distributed training via sync_all_reduce, and returns a list of MRR@k values ordered by sorted top_k. A corresponding test file is included, mirroring the structure of test_hitrate_metric.py.
Why was this PR needed?
In issue #2631, there was discussion around implementing other Recommender System metrics (besides HitRate and NDCG@k). I worked on MRR, a similar metric to HitRate. Mean Reciprocal Rank (MRR) measures the average of the reciprocal ranks of the first relevant item in the predicted ranking for each user. If no relevant item is found in the top-k predictions, the reciprocal rank for that user is 0. The math is as follows: MRR@K = (1 / N) * sum(1 / rank_i) for i = 1 to N.
While HitRate measures whether a relevant item appears anywhere in the top-k, MRR additionally captures where the first relevant item ranks. Without it, users evaluating recommender systems with ignite had no built-in way to compute MRR and would need to implement it manually outside the framework.
What is the relevant issue number?
Issue #2631
Does this PR meet the acceptance criteria?
[x] Tests added for new/changed behavior -- tests/ignite/metrics/rec_sys/test_mrr.py mirrors test_hitrate_metric.py, covering unit tests, edge cases, and distributed tests
[x] Follows project style guide -- implementation mirrors the structure, decorators, type hints, and docstring format of the existing HitRate metric
[x] No breaking changes introduced -- add-on, no existing metrics or APIs were modified