[CI] Shard Hybrid SSM NixlConnector PD accuracy tests into 4 config groups - #52354
Draft
khluu wants to merge 2 commits into
Draft
[CI] Shard Hybrid SSM NixlConnector PD accuracy tests into 4 config groups#52354khluu wants to merge 2 commits into
khluu wants to merge 2 commits into
Conversation
…roups Split hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus (~45.7m wall in build 83851) into parallelism: 3 via a new opt-in CONFIG_INDICES env in config_sweep_accuracy_test.sh that selects configs from the chosen array by zero-based index. Unset means run everything, so the other jobs sharing the sweep script (tp/dp_ep/sw_attn variants and the AMD mirrors) are unchanged. CONFIG_EXPECTED_COUNT=7 makes each shard fail loudly if the hybrid config array changes without rebalancing the index lists. Buckets are balanced on measured per-config runtime: shard 0 = configs 0,4 (~14.5m), shard 1 = configs 5,6 (~13.2m), shard 2 = configs 1,2,3 (~16.3m); expected wall ~18m incl. ~1.7m install/canary per shard. Timeout 60 -> 30. The AMD mirror keeps its own unchanged command and is pinned to parallelism: 1 (stays a single unsharded job; requires the pipeline generator change in vllm-project/ci-infra#473). Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Build 83924 passed 3/3 but the worst shard landed at 19.42m with only
0.58m under the 20m target while measured config-time variance is
~1.4m. Rebalance to parallelism: 4 with index buckets {0} / {4,5} /
{6,1} / {2,3} (per-config walls from 83924: 8.22 / 6.17+6.80 /
6.63+6.30 / 5.58+5.80m -> projected max ~14.7m incl. setup). Also
validate BUILDKITE_PARALLEL_JOB explicitly so an unexpected index fails
instead of passing empty. Union stays {0..6} = 7/7 exactly once with
CONFIG_EXPECTED_COUNT=7 on every shard; AMD mirror still pinned to one
unsharded copy; the shared sweep script is unchanged in this commit.
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
khluu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Phase 2 of the CI job-splitting effort (45–60m jobs → shards <20m).
hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpusran 45.7m in build 83851.Validation (terminal, build 83937,
VLLM_CI_BRANCH=amd-mirror-parallelism-override@ e006ba64)Targeted build 83937 on this exact head: 4/4 passed, walls 10.18 / 14.30 / 14.80 / 13.22m — max 14.80m with real margin (3.09× vs baseline).
CONFIG_INDICES=0(1 config, 8.52m), shard 14,5(6.13m + 6.50m), shard 26,1(6.67m + 6.38m), shard 32,3(5.87m + 5.65m) — {0} ⊎ {4,5} ⊎ {6,1} ⊎ {2,3} = all 7 hybrid configs exactly once,CONFIG_EXPECTED_COUNT=7guard active and each shard's sweep printed its all-tests-passed line.e006ba64): NVIDIAparallelism: 4/ AMDparallelism: 1, AMD ROCm command byte-identical and unsharded.CONFIG_INDICESunset → all 7 configs run in original order (tp/dp_ep/sw_attn/AMD/mooncake callers unchanged); count-guard mismatch exits 1 before any test.History
The first head (3-way,
79ebca72…) validated green in build 83924 (walls 16.18/15.20/19.42m) but was rejected on margin: 19.42m left 0.58m under ~1.4m observed config-time variance. Rebalanced toparallelism: 4with buckets {0}/{4,5}/{6,1}/{2,3} and an explicit 0–3 index validator. 83924 remains union/performance evidence.Approach
Opt-in
CONFIG_INDICESenv in the sharedconfig_sweep_accuracy_test.sh(comma-separated zero-based indices; unset = run everything) plus aCONFIG_EXPECTED_COUNTfail-closed guard so a newly added config can never be silently skipped, and a fail-closedBUILDKITE_PARALLEL_JOBvalidator. Timeout 60 → 30. AMD mirror unchanged, pinned to one copy viamirror.amd.parallelism: 1(vllm-project/ci-infra#473).Notes
gh pr listfor the step key and file).