[recipe] feat: add nemo_gym/browser — step-wise GRPO on the NeMo Gym interactive_browser environment

[waple0820]

What this adds

nemo_gym/browser/ — step-wise GRPO training on the NeMo Gym interactive_browser environment. The policy drives a live browser through one tool (observe / navigate / click / type / finish) and is scored per episode.

Nothing under nemo_gym/ is modified. The only change outside the new folder is one row in the root README.

Why a second path next to the existing nemo_gym recipe

The existing recipe hands a whole rollout to NeMo Gym through RolloutCollectionHelper and receives a finished result. That fits short, deterministic environments.

A browser rollout runs for tens of steps against live sites, and every step can fail for reasons unrelated to the policy: the environment session is lost, a page never loads, the judge endpoint is unreachable. Those failures arrive as reward=0, indistinguishable from a policy that simply did not solve the task. We measured what that costs: on a 64-way-concurrency run, 1172 of 1280 rollouts across 20 steps failed in session reset, and because each one entered GRPO as a legitimate zero it depressed the group baseline and inflated the advantage of the survivors — the batch trained away from correct behaviour.

So this recipe keeps the loop on the verl side, using verl's own ToolAgentLoop: the trainer bounds each step, classifies an infrastructure failure while it happens, and resamples instead of learning from it.

The two paths attach to different verl extension points — agent_loop_manager_class versus default_agent_loop — share no symbols, and can be used side by side.

Layout

File	Purpose
browser_agent_loop.py	BrowserTool speaking the NeMo Gym HTTP contract (/seed_session, browser_*, /verify), one aiohttp cookie jar per rollout so a rollout maps to exactly one environment session; BrowserToolAgentLoop adds an episode deadline and per-sample env_invalid flags
judge.py	Binary LLM judge for open-ended tasks; an unusable verdict is reported as unusable rather than as a zero
group_stats.py	grpo_env_aware advantage estimator keeping flagged samples out of the group baseline
dataset.py	NeMo Gym rollout rows to tools_kwargs (subclasses the existing NeMoGymJSONLDataset)
prepare_webvoyager_data.py	Task list (Hugging Face dataset or local JSONL) to rollout inputs
configs/, submit_webvoyager.sh, config.env.example	Environment config paths, tool config, Slurm entry point
tests/	Offline: no GPU, no browser, no NeMo Gym server

Reward

Two paths, chosen per task. A task carrying verifier_metadata (final_url, url_contains, dom_contains, answer_equals) is scored by the environment's own /verify. Everything else is scored by a binary LLM judge in the recipe, because the environment's verifier is deterministic-only today. If NeMo Gym grows a judge-backed verifier, this should move there and the recipe should just read /verify.

Environment failures, and the one upstream gap

Classification, resampling and loss masking work as-is. Keeping flagged samples out of the GRPO baseline needs the flags to reach the estimator, and registered estimators receive non_tensor_batch only for gdpo (verl/trainer/ppo/ray_trainer.py, the else branch of compute_advantage). The recipe ships two ways around that, both opt-in:

• NEMO_GYM_BROWSER_DROP_INVALID=1 — a rollout still invalid after its retries returns nothing, and the V1 trainer skips a sample whose agent loop returned nothing, so the failure never enters the batch. No patching; this is what the submit script uses.
• patches.py + sitecustomize.py — wrap compute_advantage and dispatch to grpo_env_aware only for batches that actually carry the flags, forwarding everything else untouched, so other recipes and stock GRPO runs are unaffected even with the module loaded. Ray workers are separate processes, hence sitecustomize rather than a driver-side import — the same reasoning as nemo_gym/server_patch.py, which notes it is "only instantiated when [the recipe] is active, so non-recipe users are unaffected".

Both disappear if verl passes non_tensor_batch to registered estimators, which I would be happy to send as a separate two-line PR:

adv_kwargs["non_tensor_batch"] = data.non_tensor_batch
adv_kwargs["batch"] = data.batch

An all-zero response mask is deliberately not used to infer invalidity: a policy that stops on its first token also produces zero tokens, and that zero is a legitimate outcome that belongs in the baseline. There is a test for exactly that case.

Dependencies

• Requires feat(resources_servers): add interactive browser RL environment (local Playwright + remote CDP backends) NVIDIA-NeMo/Gym#1865, which adds resources_servers/interactive_browser. That PR is open, not merged. Until it lands, NEMO_GYM_ROOT must point at that branch — happy to hold this PR until then.
• Pins a newer verl than the sibling recipe (30119a25 versus 695ac0eb): this path needs the V1 trainer and vLLM 0.18, so it carries its own REQUIRED_VERL.txt and uses verlai/verl:vllm018.latest.
• No task data is committed; prepare_webvoyager_data.py pulls it.

Validation

Internal run, 2026-07-21, Qwen3-8B, 16×H100, 60 steps, 8 tasks × 8 trajectories per step: critic/rewards/mean rose from ~0.105 over the first ten steps to ~0.289 over the last ten. That run used our own environment service; this PR is the port onto the upstream interactive_browser contract, so the numbers are context for the recipe's shape rather than a claim about this exact code path.

Offline tests pass and pre-commit (ruff) is clean:

pytest recipe/nemo_gym/browser/tests -q     # 8 passed

smoke_environment.py walks one episode over the environment's HTTP contract (seed, act, observe, verify) with no GPU and no policy, so an environment problem is separable from a training problem before a node is spent:

gym env start --resources-server interactive_browser --no-agent --no-model
python recipe/nemo_gym/browser/smoke_environment.py --url http://127.0.0.1:8000