EPP Sending Requests to not ready vLLM instances inside a Launcher Pod

[dumb0002]

In the FMA path, the dual-pods controller decides when the launcher pod is bound and applies the variant and inferencePool labels, and the launcher pod's readiness reflects the launcher control-plane API, not the just-spawned vLLM instance behind it. So the pod becomes a routing target before vLLM's port is up, and EPP forwards into a closed socket → 503.

The gap it's that the launcher pod's readiness signal doesn't reflect whether the specific vLLM instance just spawned on it is serving.

As result of this issue with see significant failed requests in the WVA+FMA benchmarking experiments.

	FMA+WVA run	WVA-only run
Backend container	FMA launcher	vLLM container directly
Startup probe on the inference port (:8000)?	No — launcher's readiness was its own control-plane API	Yes — httpGet /v1/models :8000
Pod admitted to Service/InferencePool while vLLM port not ready?	Yes (immediately after dual-pods controller bound the requester ↔ launcher and applied the variant and inferencepool labels)	No — pod stays NotReady until /v1/models returns 200
Result	EPP routed traffic to launcher → vLLM TCP refused → HTTP 503	EPP never saw the new pod as an endpoint until vLLM was serving → zero scale-up failures

See example below:

Second requester pod is not ready yet and any requests sent to the EPP envoy fails.

kubectl -n braulio-wva-fma  get pods  -o wide
NAME                                                              READY   STATUS      RESTARTS   AGE     IP             NODE               NOMINATED NODE   READINESS GATES
access-to-harness-data-workload-pvc                               1/1     Running     0          161m    10.130.4.34    pokprod-b93r43s3   <none>           <none>
download-model-fmtz2                                              0/1     Completed   0          163m    10.130.4.33    pokprod-b93r43s3   <none>           <none>
fma-requester-qwen-qwe-df5cf70e-wen3-32b-86c777fc-j8lrv           0/1     Running     0          99s     10.128.5.56    pokprod-b93r44s2   <none>           <none>
fma-requester-qwen-qwe-df5cf70e-wen3-32b-86c777fc-ntc7l           1/1     Running     0          150m    10.129.7.53    pokprod-b93r39s0   <none>           <none>
inference-perf-89yk1spi                                           1/1     Running     0          4m35s   10.129.4.81    pokprod-b93r39s1   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-2z6wx                     2/2     Running     0          161m    10.129.7.49    pokprod-b93r39s0   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-b9lgk                     2/2     Running     0          161m    10.130.4.35    pokprod-b93r43s3   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-ddtlf                     2/2     Running     0          161m    10.128.11.39   pokprod-b93r38s0   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-fgnhc                     2/2     Running     0          161m    10.131.6.244   pokprod-b93r43s1   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-fsqtc                     2/2     Running     0          161m    10.130.2.169   pokprod-b93r44s0   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-hdmmk                     2/2     Running     0          161m    10.128.5.2     pokprod-b93r44s2   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-jj2mx                     2/2     Running     0          161m    10.131.4.234   pokprod-b93r44s3   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-k2zgx                     2/2     Running     0          161m    10.129.4.69    pokprod-b93r39s1   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-mxs5t                     2/2     Running     0          161m    10.130.8.30    pokprod-b93r43s2   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-rv9vg                     2/2     Running     0          161m    10.129.8.182   pokprod-b93r39s2   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-szrsz                     2/2     Running     0          161m    10.130.7.182   pokprod-b93r43s0   <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-w5zp9                     0/2     Pending     0          161m    <none>         <none>             <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-xmxjq                     0/2     Pending     0          161m    <none>         <none>             <none>           <none>
launcher-fma-qwen-qwe-df5cf70e-wen3-32b-xxrc7                     2/2     Running     0          161m    10.131.3.248   pokprod-b93r44s1   <none>           <none>
qwen-qwe-df5cf70e-wen3-32b-fma-dp-dual-pods-controller-78bm7fz8   1/1     Running     0          161m    10.128.11.37   pokprod-b93r38s0   <none>           <none>
qwen-qwe-df5cf70e-wen3-32b-fma-dp-launcher-populator-85786qpldt   1/1     Running     0          161m    10.128.11.38   pokprod-b93r38s0   <none>           <none>
qwen-qwe-df5cf70e-wen3-32b-router-epp-8c5cb8ffc-prszr             2/2     Running     0          13m     10.128.5.43    pokprod-b93r44s2   <none>           <none>
wva-controller-manager-6dc99f6df5-mgm99                           1/1     Running     0          163m    10.128.11.35   pokprod-b93r38s0   <none>           <none>

Snapshot from logs of EPP envoy container:

{"time":"2026-07-01T15:17:00.911Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"4986e87e-e508-4ab6-b12a-622abd19d297","duration_ms":2722}
{"time":"2026-07-01T15:17:02.623Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"d04e30e9-69d7-4ac5-91e3-54a97cef53b6","duration_ms":1011}
{"time":"2026-07-01T15:17:03.094Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"89b8c490-4ba2-42b9-868d-5853d9957452","duration_ms":541}
{"time":"2026-07-01T15:17:04.050Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"2a81156f-c13a-4029-b33e-44c9317f72e9","duration_ms":2}
{"time":"2026-07-01T15:17:04.257Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"a5503b03-3047-47f8-8dfb-d182811c7c5c","duration_ms":2}
{"time":"2026-07-01T15:17:04.282Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"d3c3e736-0777-425d-875a-7fcb6e0339d2","duration_ms":2}
{"time":"2026-07-01T15:17:05.989Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"af6974c1-97cc-4caa-b364-1189823fa5fb","duration_ms":2}
{"time":"2026-07-01T15:17:07.356Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"f35914ef-9871-48e9-a010-bea00b69e94b","duration_ms":1188}
{"time":"2026-07-01T15:17:07.360Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"fb1f5fe3-da09-40d3-b538-167a3f75367a","duration_ms":1185}
{"time":"2026-07-01T15:17:07.625Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"21746df4-058c-4968-998d-e1e31a0cd006","duration_ms":921}
{"time":"2026-07-01T15:17:07.631Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"2f9eb918-296b-49f9-a798-4cf2d48b2a17","duration_ms":916}
{"time":"2026-07-01T15:17:07.704Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"e38ef6f9-27ae-43df-aa1c-314b52cd844a","duration_ms":845}
{"time":"2026-07-01T15:17:08.029Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"39960c6a-69f2-4ab9-8aef-4c8fe94991f7","duration_ms":521}
{"time":"2026-07-01T15:17:08.120Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"a8dc127e-360f-4b71-9727-02b255768ad7","duration_ms":430}
{"time":"2026-07-01T15:17:08.346Z","status":503,"flags":"UF","upstream_host":"10.128.5.2:8000","request_id":"542594a3-2a12-497a-b824-d7b598bc6bb5","duration_ms":206}

[MikeSpreitzer]

While we are still patching minor release 0.6, the dual-pods controller should be changed to apply the ISC's labels only after the vllm instance start replying positively to its readiness test.

Using the TCP proxy, which is planned for minor release 0.7, will entail also making the dual-pods controller cease applying InferencePool-matching labels to launchers.