[CI] Expand Basic Models (Extra Initialization) to 14 shards (<20 min) - #52351
Draft
khluu wants to merge 1 commit into
Draft
[CI] Expand Basic Models (Extra Initialization) to 14 shards (<20 min)#52351khluu wants to merge 1 commit into
khluu wants to merge 1 commit into
Conversation
basic-models-tests-extra-initialization currently runs 4 shards at ~53-60m each (~224 GPU-min of large-model-init tests). Per-model init timing from the #83851 shard-3 log is a broad flat distribution (slowest single model ~2m, mean ~35s) with no single-test floor and no peelable heavy family, so the clean split is a higher pytest-shard count. Raise parallelism 4 -> 14 (already uses --num-shards/--shard-id) to land each shard ~16m (<20m target) with headroom for the observed ~1.13x hash imbalance. Per-shard timeout 100 -> 30. No mirror.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
basic-models-tests-extra-initializationcurrently runs 4 shards at ~53–60 min each (~224 GPU-min of large-model-init tests) — well over the Phase-2 <20 min target. Expand it to 14 shards.Structural analysis (why higher N, not a peel)
Parsed per-model init durations from the #83851 shard-3 log (105 models, 58.8 min, matching the wall):
Qwen3_5ForConditionalGeneration~124 s (~2 min); then a 50–83 s tail; mean ~35 s/model.*ForConditionalGeneration) inits landing there by nodeid hash.So peeling a family can't cut the 224 GPU-min; the clean split is a higher pytest-shard count (the step already uses
--num-shards/--shard-id).Sizing
Floor = 224 min ÷ 20 = 12 shards; with the observed ~1.13× hash imbalance, N=14 lands the worst shard ~18 min (N=12 would peak ~21 min).
parallelism: 4 → 14, per-shardtimeout_in_minutes100 → 30, no mirror. This adds 10 h200 copies — the repeated per-shard setup / total GPU-min will be recorded from the run.Validation — build #83919 (targeted
basic-models-tests-extra-initialization, 14/14 green)Running N items in this shardmanifests on both sides, set-difference 0 in both directions (cross-checked by @cuong). Resharding changes no coverage. The collected-386 / selected-385 line is runtime deselection that varies per machine (1–2 shards here; 1/6/2/1 in the baseline) and is present in both configurations — not a gap introduced here.Part of the Phase-2 CI-overhaul sharding pass (kernels/basic-models workstream), one PR per key.