[algo] fix: remove dead code in GPG advantage estimator

[EazyReal]

What

Removes dead code in compute_gpg_outcome_advantage (verl/trainer/ppo/core_algos.py), addressing Item B of #6478, plus an adjacent device-mismatch fix.

1. id2std is never read. The function builds an id2std dict and populates it per group, but normalization divides centered scores by f_norm, not by the group std. Pure dead code.

2. The alpha parameter is a no-op. alpha (default 1.0) is unconditionally overwritten by alpha = bsz / m.clamp(min=1) before its first use, so any caller-supplied value is silently discarded. No caller passes it (ray_trainer.py builds adv_kwargs without alpha, and the function keeps **kwargs), so removing the parameter is non-breaking.

3. Singleton-group device mismatch. The singleton branch set id2mean[idx] = torch.tensor(0.0) (always CPU), while the multi-response branch uses torch.mean(...) on the scores' device. On GPU this mismatches in the advantage loop. Use a plain 0.0 (device-agnostic; identical on CPU).

Change

• Delete the id2std dict and the two lines that populate it.
• Remove the dead alpha parameter (and its docstring entry).
• Set the singleton mean to 0.0 instead of a CPU tensor.

The live local alpha = bsz / count_nonzero(scores) and its use in the scaling line are kept — that is the actual GPG scaling. Behavior is unchanged on CPU.

Scope

Only Item B of #6478. Item A (the negative-approx-kl clamp) is handled separately in #6538 and is not touched here.

Test

Adds tests/trainer/ppo/test_gpg_advantage_on_cpu.py (CPU-only, a new file to avoid conflicts with #6538/#4677 which also edit test_core_algos_on_cpu.py):

• test_gpg_singleton_group_returns_raw_score — a single non-zero-scored response gets advantage == raw masked score.
• test_gpg_applies_n_over_nonzero_scaling — a 2-response group (scores 4, 0) yields alpha * (score - mean) = 2 * (score - 2).

[CLAassistant]

All committers have signed the CLA.

[gemini-code-assist]

Code Review

This pull request removes the unused alpha parameter and the unused id2std dictionary from the compute_gpg_outcome_advantage function, and introduces unit tests to verify the GPG outcome advantage calculations. The review feedback correctly identifies a potential device mismatch issue where id2mean[idx] is initialized with a CPU tensor (torch.tensor(0.0)), which can lead to runtime errors if the input scores are on a GPU. It is recommended to use a device-agnostic Python float 0.0 or explicitly match the device and dtype of the input scores.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[EazyReal]

Good catch — fixed. The singleton-group mean is now a plain 0.0 instead of torch.tensor(0.0), so it no longer mismatches GPU scores in the advantage loop (identical on CPU). Folded it into this cleanup since it's the same function. (The same idiom appears in a few sibling estimators; happy to do those in a separate focused PR.)