Skip to content

Commit 9f9085a

Browse files
jtechappsjtechapps
authored
Add performance load tests to record EPP resource usage (#1740)
* Add performance load tests to record EPP resource usage Creates a new nightly test that runs inference-perf job and records llm-d-router resource usage to a markdown file in the repo. Create python helper script that deploys llm-d-router in current kubectl context with specified configuration, llm-d-simulator pods, and runs specified inference-perf job against the endpoint then records CPU, memory, and e2e_scheduler duration. Signed-off-by: Jacob Murry <jacobmurry@google.com> * Fix perf test github action fix the github workspace Signed-off-by: Jacob Murry <jacobmurry@google.com> * Refactor perf tests into test/perf directory - move configs and results into test/perf directory - hyperlink router config and inference-perf config in results table - move optimize baseline results into own directory Signed-off-by: Jacob Murry <jacobmurry@google.com> --------- Signed-off-by: Jacob Murry <jacobmurry@google.com>
1 parent a2edf6c commit 9f9085a

9 files changed

Lines changed: 1212 additions & 0 deletions

‎.github/workflows/nightly-router-perf-test-optimized-baseline-10k-1k.yaml‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
name: Nightly - Router Perf Test (Optimized Baseline 10k-1k)
2+
3+
on:
4+
schedule:
5+
- cron: '0 9 * * *' # Run daily at 09:00 UTC
6+
workflow_dispatch: # Enable manual triggering from UI
7+
inputs:
8+
gcp_project_id:
9+
description: 'GCP Project ID'
10+
default: 'llm-d-scale'
11+
required: true
12+
type: string
13+
gke_cluster_name:
14+
description: 'GKE Cluster Name'
15+
default: 'llm-d-ap-usc1-router-perf'
16+
required: true
17+
type: string
18+
gke_cluster_zone:
19+
description: 'GKE Cluster Zone/Region'
20+
default: 'us-central1'
21+
required: true
22+
type: string
23+
sim_replicas:
24+
description: 'Number of simulator replicas'
25+
default: '10'
26+
required: true
27+
type: string
28+
router_config:
29+
description: 'Path to consolidated router Helm values configuration'
30+
default: 'test/perf/config/router-configs/optimized-baseline.yaml'
31+
required: true
32+
type: string
33+
test_name:
34+
description: 'Test identifier name'
35+
default: 'optimized-baseline-job1'
36+
required: true
37+
type: string
38+
perf_job:
39+
description: 'Performance job configuration path'
40+
default: 'test/perf/config/shared_prefix_job1.yaml'
41+
required: true
42+
type: string
43+
44+
permissions:
45+
contents: write
46+
actions: read
47+
48+
jobs:
49+
run-perf-benchmark:
50+
runs-on: ubuntu-latest
51+
env:
52+
# This cluster must have the Gateway API Inference Extension CRDs
53+
# already installed in it.
54+
GCP_PROJECT_ID: ${{ inputs.gcp_project_id || 'llm-d-scale' }}
55+
GKE_CLUSTER_NAME: ${{ inputs.gke_cluster_name || 'llm-d-ap-usc1-router-perf' }}
56+
GKE_CLUSTER_ZONE: ${{ inputs.gke_cluster_zone || 'us-central1' }}
57+
SIM_REPLICAS: ${{ inputs.sim_replicas || '10' }}
58+
ROUTER_CONFIG: ${{ inputs.router_config || 'test/perf/config/router-configs/optimized-baseline.yaml' }}
59+
TEST_NAME: ${{ inputs.test_name || 'optimized-baseline-job1' }}
60+
PERF_JOB: ${{ inputs.perf_job || 'test/perf/config/shared_prefix_job1.yaml' }}
61+
HF_TOKEN: ${{ secrets.HF_TOKEN }}
62+
steps:
63+
- name: Checkout llm-d/llm-d-router
64+
uses: actions/checkout@v6
65+
66+
- name: Checkout kubernetes-sigs/inference-perf
67+
uses: actions/checkout@v6
68+
with:
69+
repository: kubernetes-sigs/inference-perf
70+
path: inference-perf
71+
72+
- name: Authenticate to Google Cloud
73+
uses: google-github-actions/auth@v2
74+
with:
75+
credentials_json: ${{ secrets.GKE_SA_KEY }}
76+
77+
- name: Set up gcloud CLI and kubectl
78+
uses: google-github-actions/setup-gcloud@v2
79+
with:
80+
project_id: ${{ env.GCP_PROJECT_ID }}
81+
install_components: 'kubectl,gke-gcloud-auth-plugin'
82+
83+
- name: Get GKE credentials
84+
run: |
85+
gcloud container clusters get-credentials "${{ env.GKE_CLUSTER_NAME }}" \
86+
--zone "${{ env.GKE_CLUSTER_ZONE }}" \
87+
--project "${{ env.GCP_PROJECT_ID }}"
88+
89+
- name: Install helm
90+
run: |
91+
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
92+
93+
- name: Set up Python
94+
uses: actions/setup-python@v5
95+
with:
96+
python-version: "3.10"
97+
98+
- name: Install Python dependencies
99+
run: |
100+
python3 -m pip install --upgrade pip
101+
python3 -m pip install pyyaml
102+
103+
- name: Run Performance Test Script
104+
run: |
105+
python3 test/perf/run_nightly_perf.py \
106+
--router-config ${{ env.ROUTER_CONFIG }} \
107+
--test-name ${{ env.TEST_NAME }} \
108+
--perf-chart ${{ github.workspace }}/inference-perf/deploy/inference-perf \
109+
--perf-job ${{ env.PERF_JOB }} \
110+
--sim-replicas ${{ env.SIM_REPLICAS }} \
111+
--results-dir test/perf/results/optimized-baseline \
112+
--router-machine-family e2 \
113+
--gcp-project ${{ env.GCP_PROJECT_ID }}
114+
115+
- name: Upload Performance Results
116+
uses: actions/upload-artifact@v4
117+
if: always()
118+
with:
119+
name: epp-perf-results
120+
path: test/perf/results/optimized-baseline/
121+
122+
- name: Commit and push updated results
123+
if: success() && github.ref == 'refs/heads/main'
124+
run: |
125+
git config --global user.name "github-actions[bot]"
126+
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
127+
git add test/perf/results/optimized-baseline/
128+
git commit -m "Auto-update EPP nightly performance results [skip ci]" || exit 0
129+
git push

‎test/perf/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# EPP Performance Benchmarking Suite
2+
3+
This directory contains the performance testing pipeline for the Endpoint Picker (EPP) of the `llm-d-router`. The pipeline is used to deploy the router in standalone mode, run stress tests using `inference-perf`, and capture metrics such as CPU/Memory utilization and routing latency.
4+
5+
## Directory Layout
6+
7+
- **`config/`**: Configuration manifests for test runs.
8+
- **`router-configs/`**: Router Helm configuration values recipes (e.g. `optimized-baseline.yaml`).
9+
- `llm-d-sim-deployment.yaml` / `llm-d-sim-service.yaml`: Kubernetes manifests for deploying the vLLM simulator.
10+
- `shared_prefix_job1.yaml`: Performance test workload specification defining load stages, API target, and request distributions.
11+
- **`results/`**: Execution results logged as Markdown tables, grouped by test recipe name (e.g., `results/optimized-baseline/`).
12+
- **`run_nightly_perf.py`**: The Python orchestrator script responsible for test namespace setup, application deployment (EPP & simulator), metrics scraping, and markdown generation. It assumes that `kubectl` is already configured to target an active Kubernetes cluster.
13+
14+
---
15+
16+
## Running Locally
17+
18+
To execute the performance suite locally, you must target an active GKE cluster. Ensure your `kubectl` context is configured correctly.
19+
20+
### Prerequisites
21+
- Python 3.10+
22+
- `pyyaml`
23+
- `helm`
24+
- A GKE cluster with Gateway API and Inference Extension CRDs installed.
25+
26+
### Execution Command
27+
28+
Run the orchestrator script from the repository root:
29+
30+
```bash
31+
python3 test/perf/run_nightly_perf.py \
32+
--router-config test/perf/config/router-configs/optimized-baseline.yaml \
33+
--test-name optimized-baseline-job1 \
34+
--perf-job test/perf/config/shared_prefix_job1.yaml \
35+
--sim-replicas 10 \
36+
--gcp-project <your-gcp-project-id> \
37+
--results-dir test/perf/results/optimized-baseline \
38+
--router-machine-family e2
39+
```
40+
41+
### Parameters
42+
- `--router-config`: Path to the consolidated Helm values file for the EPP.
43+
- `--test-name`: Unique name for this test run (defines the markdown filename).
44+
- `--perf-job`: Path to the `inference-perf` job file config.
45+
- `--sim-replicas`: Number of simulator pods to scale.
46+
- `--gcp-project`: GCP Project ID hosting your GKE cluster.
47+
- `--results-dir`: Output path for appending markdown result metrics.
48+
- `--router-machine-family`: Optional node affinity mapping (e.g. `e2`, `c3`).
49+
- `--no-cleanup`: (Optional) Skip namespace deletion on test completion (useful for debugging).
50+
51+
---
52+
53+
## Nightly GitHub Actions Run
54+
55+
The benchmarking pipeline runs daily via GHA:
56+
- **Workflow Path**: `.github/workflows/nightly-router-perf-test-optimized-baseline-10k-1k.yaml`
57+
- **Schedule**: Daily at 09:00 UTC.
58+
- **Actions**:
59+
1. Authenticates to GCP and configures `kubectl` to point to the GKE development cluster.
60+
2. Runs `run_nightly_perf.py` (specifying `--router-machine-family e2` for node affinity).
61+
3. Appends the metrics results to `test/perf/results/optimized-baseline/optimized-baseline-job1.md`.
62+
4. Automatically commits and pushes the updated results file back to the repository.

‎test/perf/config/llm-d-sim-deployment.yaml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: llm-d-sim
6+
name: llm-d-sim
7+
namespace: llm-d-sim
8+
spec:
9+
replicas: 10
10+
selector:
11+
matchLabels:
12+
app: llm-d-sim
13+
template:
14+
metadata:
15+
labels:
16+
app: llm-d-sim
17+
spec:
18+
topologySpreadConstraints:
19+
- maxSkew: 1
20+
topologyKey: topology.kubernetes.io/zone
21+
whenUnsatisfiable: DoNotSchedule
22+
labelSelector:
23+
matchLabels:
24+
app: llm-d-sim
25+
containers:
26+
- args:
27+
- --model
28+
- Qwen/Qwen3-8B
29+
- --port
30+
- '8000'
31+
- --max-model-len
32+
- '131072'
33+
- --max-num-seqs
34+
- '256'
35+
- --max-waiting-queue-length
36+
- '2048'
37+
- --inter-token-latency
38+
- 1ms
39+
- --prefill-time-per-token
40+
- 4us
41+
- --prefill-overhead
42+
- 5ms
43+
- --enable-kvcache
44+
- --kv-cache-size
45+
- '10000000'
46+
- --block-size
47+
- '16'
48+
- --render-timeout
49+
- 120s
50+
- --mm-render-timeout
51+
- 120s
52+
env:
53+
- name: POD_NAME
54+
valueFrom:
55+
fieldRef:
56+
fieldPath: metadata.name
57+
- name: POD_NAMESPACE
58+
valueFrom:
59+
fieldRef:
60+
fieldPath: metadata.namespace
61+
- name: POD_IP
62+
valueFrom:
63+
fieldRef:
64+
fieldPath: status.podIP
65+
image: ghcr.io/llm-d/llm-d-inference-sim:v0.9.0
66+
imagePullPolicy: IfNotPresent
67+
livenessProbe:
68+
httpGet:
69+
path: /health/ready
70+
port: 8000
71+
initialDelaySeconds: 15
72+
periodSeconds: 10
73+
name: vllm-sim
74+
ports:
75+
- containerPort: 8000
76+
name: http
77+
protocol: TCP
78+
- containerPort: 5557
79+
name: zmq-kv-events
80+
protocol: TCP
81+
readinessProbe:
82+
httpGet:
83+
path: /health/ready
84+
port: 8000
85+
initialDelaySeconds: 10
86+
periodSeconds: 5
87+
resources:
88+
requests:
89+
cpu: '1'
90+
memory: 8Gi
91+
initContainers:
92+
- args:
93+
- launch
94+
- render
95+
- Qwen/Qwen3-8B
96+
- --port=8082
97+
- --max-model-len=131072
98+
command:
99+
- vllm
100+
env:
101+
- name: VLLM_ALLOW_LONG_MAX_MODEL_LEN
102+
value: '1'
103+
- name: TOKENIZERS_PARALLELISM
104+
value: 'true'
105+
- name: OMP_NUM_THREADS
106+
value: '4'
107+
- name: HF_TOKEN
108+
valueFrom:
109+
secretKeyRef:
110+
key: token
111+
name: hf-secret
112+
image: vllm/vllm-openai-cpu:v0.19.1
113+
imagePullPolicy: IfNotPresent
114+
livenessProbe:
115+
failureThreshold: 10
116+
httpGet:
117+
path: /health
118+
port: 8082
119+
initialDelaySeconds: 15
120+
periodSeconds: 10
121+
name: vllm-render
122+
ports:
123+
- containerPort: 8082
124+
name: http-render
125+
protocol: TCP
126+
readinessProbe:
127+
failureThreshold: 10
128+
httpGet:
129+
path: /health
130+
port: 8082
131+
initialDelaySeconds: 10
132+
periodSeconds: 5
133+
resources:
134+
requests:
135+
cpu: '1'
136+
memory: 8Gi
137+
restartPolicy: Always

‎test/perf/config/llm-d-sim-service.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: llm-d-sim-svc
5+
namespace: llm-d-sim
6+
labels:
7+
app: llm-d-sim
8+
spec:
9+
selector:
10+
app: llm-d-sim
11+
ports:
12+
- protocol: TCP
13+
port: 8000
14+
targetPort: 8000
15+
name: http
16+
- protocol: TCP
17+
port: 5557
18+
targetPort: 5557
19+
name: zmq-kv-events
20+
type: ClusterIP

0 commit comments

Comments
 (0)