[CI] Shard multimodal standard 4 tests

[khluu]

Summary

• run three pytest commands in multi-modal-models-standard-4-other-whisper as four deterministic shards
• isolate the Gemma4 image and video CUDA-graph nodes as two one-node pytest processes on shard 0, preserving them without retries, skips, or xfails
• run the singleton memory-leak test exactly once on shard 0, avoiding empty-selection exit 5 and duplicate coverage
• keep the AMD mirror at one job

Why

Buildkite build #83851 took 50.857 minutes for this job, including 49.171 minutes across its four pytest commands and 1.687 minutes of outer fixed cost. Four shards target less than 20 minutes with headroom for model imbalance.

The AMD parallelism override depends on ci-infra #473. This PR must not merge first.

Current-main blocker

Targeted Buildkite #83935 is rejected for acceptance. The isolated test_vit_cudagraph_image[gemma4] process loaded the model and compiled its graph, then repeatedly failed 249,561,088-byte allocations with only 160/94/28 MiB free on the 33,280-MiB device. It ended in the same PyTorch NVML_SUCCESS == r allocator assertion seen in the earlier sharded attempts. Because the image and video commands are intentionally joined with &&, the image failure prevented the independently isolated video node from executing in this run.

The build is also a false green under this draft command shape. All four jobs finished with exit status 0 and 13.461/7.940/10.397/8.088-minute walls even though shard 0 contains the failed pytest. The failing image command is the left side of an AND-list, so shell -e does not stop there; the generated command continued through the remaining test commands and returned the last successful status. The green Buildkite state therefore is not valid acceptance evidence.

This is a current-main resource regression rather than a sharding defect or smaller worker allocation:

• baseline #83851 and #83935 both used H200 MIG workers with driver 580.159.03, CUDA 13.0, 15 MiB / 33,280 MiB visible at startup, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, and the same 18.92-GiB post-load KV-cache figure for Gemma4
• the failure reproduces cross-host: baseline used h200-ci-4-12 GPU 2 slot 3, while #83935 used h200-ci-4-15 GPU 3 slot 2
• baseline #83851 ran v0.27.2rc1.dev78+g8e6d8e4f6 and passed its 20-node vit command; current #83935 ran v0.27.2rc1.dev92+g27f27c530
• the relevant behavioral boundary between those revisions is upstream [MRV2][Multimodal] Enable encoder cuda graph for model runner v2 #49852 / 66728fe, “Enable encoder cuda graph for model runner v2,” which changes the encoder-cudagraph path this exact test exercises

Keep this PR draft and do not skip, xfail, or change the resource semantics. It is not ready for the accepted set until current main can run the underlying Gemma4 case on the existing worker allocation.

Validation

• parsed .buildkite/test_areas/models_multimodal.yaml with PyYAML
• uvx pre-commit run --files .buildkite/test_areas/models_multimodal.yaml
• git diff --check
• generated a full pipeline locally with ci-infra indexSelectLargeIndex: block: [308,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed. #473 head e006ba64b39ffd4f179824f088ff10e88af5635f; verified NVIDIA parallelism 4, AMD parallelism 1, three deterministic sharded commands, Gemma4 coverage 2/2 as separate one-node processes, singleton memory-leak coverage 1/1 exactly once, fail-closed unexpected-index handling, and dependency preservation
• #83935 runtime manifests cover the complete complementary non-Gemma vit union 18/18 across shards as 5/2/6/5 with zero duplicates
• #83935 additionally emitted the isolated image-Gemma manifest, for 19 observed unique vit IDs; the video-Gemma manifest was not emitted because the preceding isolated image process failed and the two commands are intentionally joined with &&
• all four #83935 jobs finished with exit status 0, but shard 0's log contains 1 failed for the isolated image-Gemma command; this terminal false-green result is explicitly rejected
• source selection and the full ci-infra indexSelectLargeIndex: block: [308,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed. #473 render preserve the intended exact 2 Gemma4 + 18 complementary vit IDs without skips, xfails, retries, or duplication

Duplicate-work check

Searched open PRs by the exact step key and sharding keywords. No open PR shards this job; source-dependency and ROCm-gating changes are unrelated.

AI assistance was used to prepare and validate this CI-only change.