Component
Setup/Teardown
Desired use case or feature
llmdbenchmark run tears down the harness launcher pod when a run finishes, but leaves the rest of what it created in the namespace:
pod/access-to-harness-data-workload-pvcservice/llm-d-benchmark-harnesspvc/workload-pvcconfigmap/llm-d-benchmark-preprocessesconfigmap/llm-d-benchmark-run-parameters
Keeping the PVC between runs is useful (workload data survives), but there is no CLI path to remove any of this when the user is done. The only way out is a hand-written kubectl delete naming each resource. That causes three problems:
- The PVC bills until deleted.
workload-pvc is backed by shared storage (a Filestore instance on GKE). A user who follows a guide, runs one benchmark, and deletes their stack the way the guide shows still pays for the instance indefinitely, and nothing tells them it exists
- The resource names are CLI internals. llm-d's
helpers/benchmark.md will document the five-resource kubectl delete above so guides can link to it as a stopgap. That list silently breaks the next time the CLI renames anything.
- A stale PVC wedges retries. A PVC's StorageClass is immutable and the CLI reuses an existing claim, so a run that failed against the wrong StorageClass fails every retry until the user discovers the claim and deletes it by hand (
ProvisioningFailed ... multi writer with mount access type on RWO-only default classes is the common case).
Proposed solution
llmdbenchmark cleanup --namespace <ns> [--keep-pvc]
- Deletes everything the CLI created in the namespace. Idempotent; missing resources are not an error.
--keep-pvc preserves workload-pvc for users who want workload data to survive between runs (today's default behavior, made explicit).- Implementation suggestion: label every resource the CLI creates (e.g.
app.kubernetes.io/managed-by=llmdbench) and have cleanup delete by selector. That removes the name-list coupling for the CLI itself and lets external docs replace their hardcoded lists with one command.
Acceptance:
- After
run followed by cleanup, kubectl get all,pvc,configmap -n <ns> shows none of the resources listed above.
cleanup on a namespace with no benchmark resources exits 0.- A run that failed on a wrong StorageClass can be retried after
cleanup without manual PVC surgery.
Alternatives
- Keep documenting the
kubectl delete list in consuming repos (status quo). Rots whenever the CLI renames a resource, and every consumer maintains its own copy.
- Delete the PVC automatically at the end of each run. Removes the billing hazard but loses cross-run workload-data reuse, which is worth keeping as the default.
Additional context or screenshots
Motivated by llm-d/llm-d#2213, which moves the hand-maintained deletion list into helpers/benchmark.md ("Cleaning up harness resources") as a stopgap.
Component
Setup/Teardown
Desired use case or feature
llmdbenchmark runtears down the harness launcher pod when a run finishes, but leaves the rest of what it created in the namespace:pod/access-to-harness-data-workload-pvcservice/llm-d-benchmark-harnesspvc/workload-pvcconfigmap/llm-d-benchmark-preprocessesconfigmap/llm-d-benchmark-run-parametersKeeping the PVC between runs is useful (workload data survives), but there is no CLI path to remove any of this when the user is done. The only way out is a hand-written
kubectl deletenaming each resource. That causes three problems:workload-pvcis backed by shared storage (a Filestore instance on GKE). A user who follows a guide, runs one benchmark, and deletes their stack the way the guide shows still pays for the instance indefinitely, and nothing tells them it existshelpers/benchmark.mdwill document the five-resourcekubectl deleteabove so guides can link to it as a stopgap. That list silently breaks the next time the CLI renames anything.ProvisioningFailed ... multi writer with mount access typeon RWO-only default classes is the common case).Proposed solution
--keep-pvcpreservesworkload-pvcfor users who want workload data to survive between runs (today's default behavior, made explicit).app.kubernetes.io/managed-by=llmdbench) and havecleanupdelete by selector. That removes the name-list coupling for the CLI itself and lets external docs replace their hardcoded lists with one command.Acceptance:
runfollowed bycleanup,kubectl get all,pvc,configmap -n <ns>shows none of the resources listed above.cleanupon a namespace with no benchmark resources exits 0.cleanupwithout manual PVC surgery.Alternatives
kubectl deletelist in consuming repos (status quo). Rots whenever the CLI renames a resource, and every consumer maintains its own copy.Additional context or screenshots
Motivated by llm-d/llm-d#2213, which moves the hand-maintained deletion list into
helpers/benchmark.md("Cleaning up harness resources") as a stopgap.