[recipe] feat: add routing-aware replay utilities for MoE RL

[kaining-never-stop]

This is a small, focused PR following the RFC here:

verl-project/verl#6805

The idea is to keep the first version deliberately lightweight: a self-contained routing_aware_replay recipe for comparing replay masks in MoE RL, without touching verl core.

What is included:

• Fisher-weighted replay mask construction
• budget-matched uniform/random controls
• compact replay diagnostics
• a CPU-only synthetic example
• unit tests for the mask and diagnostics behavior

This is not meant to be a full training recipe yet. I kept it CPU-testable so it is easier to review first; if the direction looks useful, I can add a small MoE training config or align the schema with an upstream router replay output format in a follow-up.

I ran:

python examples/synthetic_router_replay_demo.py
python -m unittest discover -s tests
pytest tests

and the ruff hooks on the new Python files:

pre-commit run ruff --files <new python files>
pre-commit run ruff-format --files <new python files>

[gemini-code-assist]

Code Review

This pull request introduces the routing_aware_replay utility, which provides tools for studying routing-aware replay policies in MoE RL post-training, including Fisher-weighted replay masks, budget-matched baselines, and diagnostics. The review feedback highlights a few improvement opportunities: handling identical scores during min-max normalization to avoid discarding high identical scores, validating that tau lies between theta_low and theta_high in the configuration schema, and adding corresponding unit tests to verify the behavior of identical scores.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[kaining-never-stop]

Thanks, these edge cases make sense. I pushed 7eb7c50 to handle them:

• identical positive scores now keep the replay mask on instead of being normalized away;
• identical zero scores still produce a zero mask;
• tau now has to stay between theta_low and theta_high;
• added tests for those cases.

I re-ran the demo, unittest, pytest, ruff, and the ruff pre-commit hooks on the new Python files.

[kaining-never-stop]

Just checking in. The bot review comments have been addressed, and the PR is ready for maintainer review. Happy to rename the recipe or trim the scope if that would make the first version easier to review.

[kaining-never-stop]

Hi @wuxibin89, the workflow has been approved and the pre-commit check is green. All current review threads are resolved.

When convenient, could you help confirm whether this lightweight, CPU-testable mask and diagnostics utility fits the scope of verl-recipe, or suggest a suitable reviewer? Since verl now has router-replay support in core, I’m happy to align the interface with the existing schema or adjust the initial scope based on maintainer guidance. Thanks!

[kaining-never-stop]

Closing this draft because its Fisher-weighted replay and expert-adaptation scope does not align with the existing R3 semantics. I have revised verl Discussion #6805 to a narrower proposal: a default-off Megatron R3 diagnostics hook for native-vs-replay route comparison. I will continue only with that core observability scope after validation and maintainer feedback.