Model download workers contend on Hugging Face local-dir lock

[danehans]

Component

Setup/Standup

Describe the bug

The model download Job runs hf download --local-dir with Hugging Face's default eight workers. On a shared GKE Premium Filestore PVC, workers contend on the same local-directory metadata lock:

Fetching 27 files: 0%|          | 0/27 [00:00<?, ?it/s]
Still waiting to acquire lock on /cache/models/Qwen/Qwen3-32B/.cache/huggingface/.gitignore.lock (elapsed: 30.7 seconds)

Seven identical wait messages appeared at the same elapsed time, consistent with one worker holding the lock and the other seven waiting. The download eventually completed in this case, but the contention delays startup and can look like a hung download.

The current template also executes pip install -U --user huggingface_hub at Job runtime and then runs hf auth login even though HF_TOKEN is already provided in the environment:

https://github.com/llm-d/llm-d-benchmark/blob/05d82d5aa8f925d253d0aab49b708f3103623f7f/config/templates/jinja/04_download_job.yaml.j2

Installing the newest package at runtime makes the Job non-reproducible. In this run it upgraded huggingface_hub from 1.21.0 to 1.27.0 and emitted dependency-conflict warnings for packages already in the benchmark image.

Suggested changes:

• Install a tested, pinned huggingface_hub version in the benchmark image instead of upgrading it at Job runtime.
• Let the CLI consume HF_TOKEN directly rather than persisting it with hf auth login.
• Make the download worker count configurable and support --max-workers 1 for network-backed/shared PVCs. A low configurable default may also be appropriate.

The hostPath download DaemonSet has similar runtime installation and login behavior.

Steps to reproduce

1. Create a GKE cluster with a shared Premium Filestore RWX model PVC.
2. Configure a GPU scenario to download Qwen/Qwen3-32B.
3. Provide a valid HF_TOKEN and run llmdbenchmark standup.
4. Inspect the download-model Job pod logs.
5. Observe concurrent waits for .cache/huggingface/.gitignore.lock.

Only one download Job pod and one hf download process were running when this was observed.

Additional context or screenshots

Hugging Face documents eight as the default hf download --max-workers value:

https://huggingface.co/docs/huggingface_hub/main/en/package_reference/cli

Using --local-dir creates the .cache/huggingface metadata directory within the destination:

https://huggingface.co/docs/huggingface_hub/en/guides/download