[Bug]: Data-parallel startup ignores DP in startup_omp_num_threads, causing CPU thread oversubscription and 15x slower weight loading (engine-ready timeout)

[janbernloehr]

Your current environment

The output of python collect_env.py

collect_env.py could not be run interactively in the environment where this
failure was captured (a batch scheduler job whose container is torn down at
exit). The following is the environment as recorded from the job's own package
manifest and device inventory:

vLLM version         : 0.26.1rc1.dev602+g65b7662d3
PyTorch              : 2.13.0+cu130
CUDA runtime         : 13.0.2
Python               : 3.12.3
OS                   : Ubuntu 24.04 (x86_64)
GPU driver           : 580.167.08
GPUs                 : 8x NVIDIA B200 (183359 MiB each, compute capability 10.0,
                       148 SMs); 4 GPUs used by this run
CPU                  : Intel(R) Xeon(R) Platinum 8570, 224 logical CPUs
                       (nproc inside the container reports 224)
System memory        : ~2.0 TiB

Relevant packages:
  transformers            5.15.0
  triton                  3.7.1
  nvidia-nccl-cu13        2.30.7
  flashinfer-python       0.6.16.post3
  flashinfer-cubin        0.6.16.post3
  flashinfer-jit-cache    0.6.16.post3+cu130
  compressed-tensors      0.17.0
  runai-model-streamer    0.16.1
  nvidia-nvshmem-cu13     3.4.5

🐛 Describe the bug

Summary

startup_omp_num_threads() divides the machine's CPU count by the number of
local worker processes within one engine, with no term for data parallelism.
With --data-parallel-size N on a single node, N engine cores each independently
claim the entire CPU count for their torch thread pool. On a 224-CPU node at
DP=4 that is 4 x 224 = 896 torch threads competing for 224 CPUs.

The visible consequence is a ~15x slowdown in model weight loading, which in
turn blows the engine-ready deadline and crashes startup:

(ApiServer_0 pid=3383311) TimeoutError: Timed out waiting for engine core
processes to start. This is often caused by slow weight loading for large
models. Waited 600s (configured by VLLM_ENGINE_READY_TIMEOUT_S). To increase
the timeout, set the environment variable: VLLM_ENGINE_READY_TIMEOUT_S=<seconds>

All api_server_count frontends raise it, followed by:

RuntimeError: Frontend process failed during engine core initialization.
See root cause above. Failed frontend proc(s): {'ApiServer_0': None}

and BrokenPipeError: [Errno 32] Broken pipe from each
Worker_DP{0..3}_EP{0..3} as the executor is torn down mid-load.

Note the 600s figure is vLLM's own default (envs.py:
VLLM_ENGINE_READY_TIMEOUT_S: int = 600). No timeout override was set for this
run.

Mechanism

vllm/utils/torch_utils.py:

def startup_omp_num_threads(num_local_procs: int) -> int:
    return max(1, available_cpu_count() // max(1, num_local_procs))

vllm/v1/executor/multiproc_executor.py passes self.local_world_size:

set_multiprocessing_worker_envs(self.local_world_size)

and local_world_size is derived only from tensor, pipeline, and prefill-context
parallelism — the assertion just above it reads
world_size == tensor_parallel_size * pipeline_parallel_size * prefill_context_parallel_size, with no data-parallel factor.

So each engine core computes its divisor as if it were the only process on the
node. This is directly observable in the failing run's log: with
--tensor-parallel-size 1 --data-parallel-size 4, all four engine cores report

[multiproc_executor.py:149] DP group leader: node_rank=0, node_rank_within_dp=0,
  ... world_size=1, local_world_size=1

local_world_size=1 on a 224-CPU node yields 224 // 1 = 224 threads per
engine, 896 node-wide.

The function correctly early-returns when the user has already set
OMP_NUM_THREADS, so the bug only bites the default path.

Reproduction

# upstream nightly image
docker run --gpus all --ipc=host --shm-size=32g \
  -v /path/to/models:/models \
  vllm/vllm-openai:nightly \
  vllm serve /models/Kimi-K2-Thinking-NVFP4 \
    --host 0.0.0.0 --port 8000 \
    --trust-remote-code \
    --no-enable-prefix-caching \
    --served-model-name kimi-k2-thinking \
    --load-format runai_streamer \
    --attention-backend FLASHINFER_MLA \
    --kv-cache-dtype fp8 \
    --tensor-parallel-size 1 \
    --pipeline-parallel-size 1 \
    --data-parallel-size 4 \
    --max-model-len 10240 \
    --gpu-memory-utilization 0.95 \
    --enable-expert-parallel \
    --async-scheduling \
    --compilation_config.cudagraph_mode FULL_DECODE_ONLY \
    --compilation_config.max_cudagraph_capture_size 2048 \
    --moe-backend flashinfer_trtllm

Checkpoint: nvidia/Kimi-K2-Thinking-NVFP4 (public on Hugging Face).
Hardware: single node, 4x B200, 224 logical CPUs. No OMP_NUM_THREADS and no
VLLM_ENGINE_READY_TIMEOUT_S set.

The reproducer needs a high core count and a checkpoint made of many small
tensors; see "Why block-quantized checkpoints amplify this" below.

Expected behavior

Startup completes. Each engine's torch thread pool is sized so that the
aggregate across all local engines does not exceed the machine's CPU count.

Actual behavior

Weight loading takes 1822s and startup fails at the 600s engine-ready deadline.
From the failing run:

[serve.py:123] Defaulting api_server_count to data_parallel_size (4).
...
(Worker_DP0_EP0) [gpu_model_runner.py:5483] Model loading took 154.87 GiB
  memory and 1822.406850 seconds

The frontends abandon startup at 600s while the workers are still loading; the
load would have needed 1822s, i.e. ~1222s past the deadline it had to meet.

Single-variable A/B isolating the thread count

Two runs, identical image, identical checkpoint, identical flags, same
DP=4 topology, same 224-CPU node class. The only difference is whether
OMP_NUM_THREADS is set, which is exactly the variable the early-return in
set_multiprocessing_worker_envs controls:

OMP_NUM_THREADS	threads/engine	node-wide	weight load	server startup	outcome
unset (default)	224	896	1822.4 s	never completes	TimeoutError at 600 s
56 (= 224/4)	56	224	118.3 s	360 s	serves normally, 1921 tok/s

15.4x on weight loading from thread count alone. The load time is highly
reproducible: three independent runs on the default path measured 1822.4s,
1816.7s, and 1815.5s (0.38% spread).

A dose-response across topologies on the same 224-CPU node class, all
block-quantized checkpoints:

topology	threads/worker	node-wide threads	weight load
DP=1	56	224	56 s
DP=4, OMP_NUM_THREADS=56	56	224	118 s
DP=8, OMP_NUM_THREADS=56	56	448	671 s
DP=4, default	224	896	1822 s
DP=8, default	224	1792	2059 s

On a 144-CPU node the same DP=4 configuration loads in 230s — fewer cores means
a smaller divisor numerator, so less oversubscription. Higher core count makes
this worse, which is the signature of the divisor being wrong rather than of a
resource shortage.

Why block-quantized checkpoints amplify this

The slowdown scales with tensor count, not bytes. The NVFP4 checkpoint above
deserializes into ~277k small tensors, most just above torch's internal
GRAIN_SIZE threshold for parallelizing an elementwise op — so nearly every
tensor pays full thread-pool dispatch overhead against an oversubscribed pool.
Equivalent bf16 checkpoints load as ~51 large tensors and are essentially
unaffected; FP8 variants of the same models also pass. So this reproduces most
sharply on NVFP4/MXFP4/block-quantized weights.

Why this surfaces as a crash only with vllm serve

Worth noting for anyone trying to reproduce via the module entrypoint: the
slowdown occurs either way, but only vllm serve turns it into a hard failure.

vllm serve defaults api_server_count to data_parallel_size (in
vllm/entrypoints/cli/serve.py), which routes through run_multi_api_server
and the externally-managed client path, where MPClient polls with a
VLLM_ENGINE_READY_TIMEOUT_S budget. python -m vllm.entrypoints.openai.api_server never references api_server_count — its
__main__ calls uvloop.run(run_server(args)) unconditionally — and takes the
self-managed path, where wait_for_engine_startup polls at
STARTUP_POLL_PERIOD_MS and continues indefinitely with only logger.debug
output. On that path the same configuration silently absorbed a 2010s startup
with no error at all.

A related detail on the deadline itself: the poll sits inside
while identities:, so each received ready-message restarts the budget. It is a
per-message idle timeout rather than a cumulative startup deadline. Because this
load emits no intermediate ready messages, the very first poll exceeds 600s
anyway.

Regression range

Bisected with per-commit wheels from https://wheels.vllm.ai/<sha>, holding the
base image, checkpoint, flags, and all other dependency versions fixed:

• last known good: 52c0e3cb08b8178829e1f2db4ac90e9bb98c8a5f
  (0.26.1rc1.dev327+g52c0e3cb0) — weight load 114.8 s, serves normally
• first bad: 7635a9002baecca64909dbcf8b1d461d26fb879d
  (0.26.1rc1.dev328+g7635a9002) — weight load 1961.7 s

That boundary is PR #49919 ("[Core] Explicitly manage torch CPU threads in
workers"), which introduced startup_omp_num_threads.

Caveat, stated explicitly: this bisect pair was measured at
--tensor-parallel-size 4 with no data parallelism. It brackets the
introduction of the explicit thread-management code, and shows that code made
this configuration much slower; it does not by itself isolate the missing DP
term. The DP-specific evidence is the A/B and dose-response tables above.

Suggested fix

Include the local data-parallel replica count in the divisor, so the aggregate
thread count across co-located engines respects the CPU budget. Both fields are
already available on ParallelConfig (data_parallel_size_local and
local_world_size), so the executor can pass
local_world_size * max(1, data_parallel_size_local) instead of
local_world_size alone.

A node-wide accounting would be more robust still, since headless/multi-node
layouts can place a differing number of engines per node.

Prior art on this knob

• PR [Bugfix] set OMP_NUM_THREADS to 1 by default when using the multiproc_gpu_executor #6109 (merged 2024) originally set a blanket OMP_NUM_THREADS=1 for the
  multiprocessing GPU executor; PR [Core] Explicitly manage torch CPU threads in workers #49919 superseded that with explicit
  per-worker thread management.
• PR [Misc] Set correct OMP_NUM_THREADS for executors #22773 (closed, not merged) addressed OMP thread selection for the CPU
  backend's VLLM_CPU_OMP_THREADS_BIND path — a different executor, but the
  same class of problem.

Workaround for users hitting this

Set OMP_NUM_THREADS explicitly to available_cpus / (local engines x workers per engine) before launching. The early-return in
set_multiprocessing_worker_envs then leaves it alone. Raising
VLLM_ENGINE_READY_TIMEOUT_S is not a good workaround — it converts the crash
into a silent multi-fold startup regression.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the
  documentation and FAQ for help before opening a new issue.

---

This issue was drafted with assistance from the opus AI model.

[cr-zhao]

I'd like to work on this.