Setup and Teardown Intermittent Errors in the FMA + WVA & FMA + KEDA workflows

[dumb0002]

Component

Setup/Teardown

Describe the bug

CI nightly (ocp-wva-fma-warmstart) failing in two phases: (a) GCP auth Invalid JWT Signature in standup, (b) transient git fetch timeout during WVA-controller kustomize uninstall in teardown

Summary

The WVA+FMA and KEDA+FMA workflows are failing in two independent, non-code-related places. Both are infrastructure/tooling issues (expired credentials + a transient remote fetch), not benchmark logic — but they abort the pipeline and should be hardened/fixed.

---

a) Standup — GCP service-account auth failure (invalid_grant: Invalid JWT Signature)

Where: google-github-actions/setup-gcloud step (GKE-targeted job / project llm-d-scale).

Error:
Successfully authenticated
Error: google-github-actions/setup-gcloud failed with: failed to execute command
`gcloud --quiet config set project llm-d-scale`:
ERROR: (gcloud.config.set) There was a problem refreshing auth tokens for account
llm-d-e2e@llm-d-scale.iam.gserviceaccount.com:
('invalid_grant: Invalid JWT Signature.', {'error': 'invalid_grant',
 'error_description': 'Invalid JWT Signature.'})

---

b) Teardown — WVA controller uninstall fails on transient git fetch timeout

Where: teardown step [01] uninstall_helm, WVA controller uninstall (OpenShift / pokprod). llm-d-benchmark v0.7.0.

Error:
Failed to uninstall WVA controller in llmdbenchcicdfwns-ocp: error: accumulating resources:
accumulation err='accumulating resources from
'github.com/llm-d/llm-d/guides/workload-autoscaling/wva-config/platform/ocp?ref=main':
evalsymlink failure on '/tmp/tmpvto1fvfc/github.com/.../ocp?ref=main' :
lstat /tmp/tmpvto1fvfc/github.com: no such file or directory':
hit 27s timeout running '/usr/bin/git fetch --depth=1 https://github.com/llm-d/llm-d main'

Severity

Both are transient/infra-level (no code defect), but each fully fails its phase. (b) is a good candidate for teardown hardening so intermittent GitHub slowness doesn't fail the workflows

Steps to reproduce

Run manually any of the following workflows:

• WVA + FMA: https://github.com/llm-d/llm-d-benchmark/actions/workflows/ci-benchmark-ocp-fma-wva.yaml
• KEDA + FMA: https://github.com/llm-d/llm-d-benchmark/actions/workflows/ci-benchmark-ocp-fma-keda.yaml

Additional context or screenshots

See following previous run workflows here run:
- https://github.com/llm-d/llm-d-benchmark/actions/runs/30470472819/job/90645445264
- https://github.com/llm-d/llm-d-benchmark/actions/runs/30416439168/job/90463910134

[dumb0002]

cc: @aavarghese @Vezio @maugustosilva

[Vezio]

@maugustosilva I'm assuming maybe a token was rotated on the GKE cluster? I know we don't have much control over that but do you know of anything @maugustosilva - I'd assume this issue would have propagated to other CI jobs.

Error:
Successfully authenticated
Error: google-github-actions/setup-gcloud failed with: failed to execute command
gcloud --quiet config set project llm-d-scale:
ERROR: (gcloud.config.set) There was a problem refreshing auth tokens for account
llm-d-e2e@llm-d-scale.iam.gserviceaccount.com:
('invalid_grant: Invalid JWT Signature.', {'error': 'invalid_grant',
'error_description': 'Invalid JWT Signature.'})

[Vezio]

@dumb0002 have you seen this occur locally?

Error:
Failed to uninstall WVA controller in llmdbenchcicdfwns-ocp: error: accumulating resources:
accumulation err='accumulating resources from
'github.com/llm-d/llm-d/guides/workload-autoscaling/wva-config/platform/ocp?ref=main':
evalsymlink failure on '/tmp/tmpvto1fvfc/github.com/.../ocp?ref=main' :
lstat /tmp/tmpvto1fvfc/github.com: no such file or directory':
hit 27s timeout running '/usr/bin/git fetch --depth=1 https://github.com/llm-d/llm-d main'

Or is this always during CICD? I'm wondering if there is a transient network problem since you're pulling down resources from a remote. I wonder if you can implement either a retry or something equivalent so that it doesn't immediately error out if there is a hung fetch?

The library wouldn't be able to fix the network problems, but we could sort've dance around transient issues by having an implicit retry policy for these types of things.

Thoughts from you @dumb0002 and @aavarghese ?

[dumb0002]

@Vezio, I only hit this issue in the CICD not locally. It seems an transient error indeed but I'm constantly hitting it in every run sometimes even after multiple manual retries since Yersterday afternoon.

@aavarghese, any thoughts on adding the suggested retry or something equivalent?

[aavarghese]

I did two runs of the EPP+KEDA+FMA workflow with a space of 30 mins, and both failed on the gcloud CLI step..so i'm blocked. Retry not helping:

18s
Run google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /runner/_work/_temp/2ab1e01c-2e44-4f3f-90a7-2e660eb58d16 -f /runner/_work/_temp/65089216-a8c1-4621-bac6-f43d923d1645
Successfully authenticated
Error: google-github-actions/setup-gcloud failed with: failed to execute command `gcloud --quiet config set project llm-d-scale`: ERROR: (gcloud.config.set) There was a problem refreshing auth tokens for account llm-d-e2e@llm-d-scale.iam.gserviceaccount.com: ('invalid_grant: Invalid JWT Signature.', ***'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'***)
Please run:

  $ gcloud auth activate-service-account --key-file=SERVICE_ACCOUNT_FILE_PATH

Or:

  $ gcloud auth login --cred-file=SERVICE_ACCOUNT_FILE_PATH

to obtain new credentials.

If you have already logged in with a different account, run:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.

[Vezio]

@aavarghese and @dumb0002 let's isolate the two issues.

The retry as I mentioned, is for your git fetch that pulls WVA as highlighted in the issue reported:

Error:
Failed to uninstall WVA controller in llmdbenchcicdfwns-ocp: error: accumulating resources:
accumulation err='accumulating resources from
'github.com/llm-d/llm-d/guides/workload-autoscaling/wva-config/platform/ocp?ref=main':
evalsymlink failure on '/tmp/tmpvto1fvfc/github.com/.../ocp?ref=main' :
lstat /tmp/tmpvto1fvfc/github.com: no such file or directory':
hit 27s timeout running '/usr/bin/git fetch --depth=1 https://github.com/llm-d/llm-d main'

In this situation, you could add a retry to fetch again x number of times until exhausted.

The gcloud issue is not related to the retry solution I proposed - it is seperate. That seems to be related to a token as part of the overall CICD. This is something I don't have access to, I believe @maugustosilva might have access, but I believe we'd need to have some help from the Google folks on GKE.