[CI] Shard LoRA TP (Distributed) into 4 timing-balanced groups - #52350
Draft
khluu wants to merge 3 commits into
Draft
[CI] Shard LoRA TP (Distributed) into 4 timing-balanced groups#52350khluu wants to merge 3 commits into
khluu wants to merge 3 commits into
Conversation
Split the 7 sequential per-file pytest commands of lora-tp-distributed (~45.4m wall in build 83851) across parallelism: 3 using per-shard command guards, balanced on measured per-file runtime: shard 0 ~15.2m (gptoss 12.7m + chatglm3 2.5m), shard 1 ~15.3m (llama 7.4m + gemma4 7.9m), shard 2 ~14.3m (qwen3_with_multi_loras 6.0m + qwen35_densemodel 4.2m + olmoe 4.1m). Both env exports still run on every shard; file flags (-x) unchanged; every file runs on exactly one shard. Timeout 60 -> 25. num_devices: 4 is per shard, so peak accelerator use goes 4 -> 12 while wall drops ~45m -> ~16m. Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
An index outside 0-2 would skip every per-shard guard and let the job pass empty; validate BUILDKITE_PARALLEL_JOB explicitly before any test command and exit 1 with a diagnostic otherwise. Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Two validation runs (83921, 83936) show per-file test time is stable
(max delta 0.23m) but fixed setup varies 1.3-4.9m on identical work.
The N=3 buckets already sit at the 14.5m balance ideal, so no 3-way
arrangement stays safely under 20m when the setup tail hits; move to
parallelism: 4 with gptoss (12.2m, single-file floor) isolated:
{gptoss} / {llama, qwen35} / {gemma4, olmoe} / {qwen3_multi, chatglm3}
= 12.2/11.4/11.2/8.6m test, worst-observed-setup projection ~17.1m.
Index validator now accepts 0-3; exports, flags, per-file guards and
exact once-each coverage unchanged.
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
khluu
force-pushed
the
ci-shard-lora-tp-distributed
branch
from
August 14, 2026 15:16
b28aa58 to
afa3315
Compare
khluu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Phase 2 of the CI job-splitting effort (45–60m jobs → shards <20m).
lora-tp-distributedran 45.4m in build 83851.Validation (terminal, build 83941,
VLLM_CI_BRANCH=amd-mirror-parallelism-override@ e006ba64)Targeted build 83941 on this exact head: 4/4 passed, walls 13.58 / 14.22 / 12.93 / 9.35m — max 14.22m with real margin (3.19× vs baseline).
-v -s -x) and both env exports unchanged; the 0–3 index validator ran first on every shard.Approach
parallelism: 4with per-shard command guards plus a fail-closedBUILDKITE_PARALLEL_JOB0–3 validator. Buckets balanced on per-file walls that were stable across three runs (83921/83936/83941): gptoss (12.1–12.2m) is the single-file floor and gets its own shard, so N=4 is also the ceiling of useful N.Why 4-way
The N=3 layout validated green twice but was rejected on margin: per-file test time was stable across runs, but fixed setup varied 1.3–4.9m on identical work (83936 shard 1 hit a 4.93m outlier → 19.52m wall, 0.48m margin). The N=3 buckets already sat at the ~14.5m balance ideal, so no 3-way arrangement stays safely under 20m when the setup tail hits. N=4 projects ~17.1m even with the observed worst-case setup. Builds 83921/83936 remain rejected-margin evidence with valid 7/7 union proofs.
Notes
gh pr listfor the step key and file; the sharding merged to main for the single-GPUlorastep is a different key, rebased over cleanly).