Skip to content

test(coordinator): parallelize the e2e suite - #2302

Draft
revit13 wants to merge 2 commits into
llm-d:mainfrom
revit13:parallel-coord
Draft

test(coordinator): parallelize the e2e suite#2302
revit13 wants to merge 2 commits into
llm-d:mainfrom
revit13:parallel-coord

Conversation

@revit13

@revit13 revit13 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?
/kind test

What this PR does / why we need it:

Which issue(s) this PR fixes:
Parallelizes the coordinator e2e suite, applying the same mechanism the router
e2e already uses (#1931 / PR #2033): ginkgo run --procs=N over one Kind
cluster, with per-process namespaces and NodePorts.

Measured on a local dev machine (22 cores, 31 GB, WSL2), not CI:
make -f Makefile.coord.mk test-e2e-coordinator-run drops from 8m41s to 5m05s
at the default E2E_NUM_PROCS=5, a 1.7x speedup. CI runners are smaller, so
expect a different absolute number there

  • Suite lifecycle. SynchronizedBeforeSuite provisions the cluster, CRDs,
    and a single vllm-render once on process 1; every other process only builds
    its own client. ReportAfterSuite tears down what process 1 created.

  • Per-group infra. A testWrapper mirroring the router's creates the
    namespace, Envoy, and the Services/ServiceAccounts/RBAC each group binds to in
    BeforeAll, and removes them in AfterAll. Each of the 9 scenarios becomes
    its own When(..., ginkgo.Ordered, testWrapper(...)) group so Ginkgo can
    distribute them; Ordered keeps a group on the process that built its Envoy.

  • Shared renderer. One vllm-render in the base namespace serves every
    process. The pipeline config gains ${RENDER_NAMESPACE} for the render step
    so it resolves cross-namespace while the gateway stays on ${NAMESPACE}.

  • Per-process NodePort. shared-envoy-resources.yaml takes
    ${ENVOY_NODE_PORT} instead of a hardcoded 30080, matching the router's
    envoy.yaml; without it processes 2..N collide on one port.

  • Shared helpers. SetupNamespace/DeleteNamespace move to test/utils,
    replacing near-identical copies in both suites. DeleteNamespace waits
    ReadyTimeout rather than ExistsTimeout: finalizing a namespace means
    reaping every pod in it, and the EPP's drain alone exceeds 30s.

  • Runner. run_ginkgo_suite takes an optional timeout; the coordinator
    passes 90m. E2E_NUM_PROCS (default 5) is forwarded through
    Makefile.coord.mk so ginkgo --procs and the suite's numProcesses agree.

Fixes #

Release note (write NONE if no user-facing change):

NONE

@revit13
revit13 requested review from a team, roytman and shmuelk as code owners August 6, 2026 11:10
@revit13
revit13 requested review from liu-cong and vMaroon August 6, 2026 11:10
@revit13
revit13 marked this pull request as draft August 6, 2026 11:10
@github-actions github-actions Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 6, 2026
@revit13
revit13 force-pushed the parallel-coord branch 8 times, most recently from 2aed284 to 646aae7 Compare August 11, 2026 03:39
Signed-off-by: Revital Sur <eres@il.ibm.com>
Signed-off-by: Revital Sur <eres@il.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant