e2e: Check for GPU Driver/ML framework compatibilty

[mikkeloscar]

#11567 introduced an AMI with updated nvidia drivers

This adds GPU related e2e tests that verifies various GPU and ML framework support against the AMI/drivers. This way we can ensure to not break support over time or if we do we can do it consciously.

[mikkeloscar]

👍

[mikkeloscar]

👍

[demonCoder95]

Does it make much sense to assert the version here? We control the version of the image ourselves in the function call. Also this increases maintenance overhead for the test-case, where we'll need to update this script every time we update the CUDA version.

I think we can drop the assert, and just keep the rest of the code. It should compile and run fine for us to be able to test properly for any version.

[zalando-robot]

🤖 LLM Suggestions

[High] Non-Zalando container registries used in production-facing tests

Two of the three new tests pull images from public Docker Hub registries:

• nvidia/cuda:13.2.1-devel-ubuntu24.04
• pytorch/pytorch:2.12.1-cuda13.2-cudnn9-runtime

Zalando policy requires images to come from container-registry.zalando.net (ZCR) in production contexts. Even for e2e tests running in the cluster, pulling from Docker Hub introduces rate-limiting risks, supply-chain risks, and may violate compliance requirements. The first test was already changed to use nvcr.io/nvidia/... which is NVIDIA's own registry but still not ZCR. Consider either mirroring these images to ZCR or confirming that test clusters are explicitly exempted from this policy.

Applied

[High] Infinite loop with no termination guard if pod never reports Terminated

A 2-minute deadline has been added to the loop — this is correctly fixed in the current diff.

[High] Log-pattern mismatch causes silent test pass even when log check fails

The log-pattern check now uses framework.Failf(...) when the pattern is not found — this is correctly fixed in the current diff.

Other non-critical findings

[Medium] runGPUTest By(...) message is misleading for non-vector tests

The helper still uses By("Creating a vector pod which runs on a GPU node") for all three tests, including the CUDA compile and PyTorch tests. This makes test output logs confusing. Pass a description string or derive it from the image name:

By(fmt.Sprintf("Creating GPU test pod with image %s", image))

[Medium] Inline CUDA source in a shell heredoc is fragile and hard to maintain

The CUDA kernel test embeds a 3-line cat > /tmp/t.cu <<EOF ... EOF heredoc inside a Go string literal inside a bash -c command. This is difficult to read, hard to debug when it fails, and Go string escaping rules interact poorly with shell quoting. Consider storing the CUDA source as a Go raw string literal and using a ConfigMap to inject it, or at minimum move the entire command to a named Go constant with a comment.

[Low] container.Command vs container.Args semantics

For the vector-add test, command is nil, so the container's default entrypoint runs. For the CUDA and PyTorch tests, the full command (binary + args) is passed as Command. In Kubernetes, Command overrides ENTRYPOINT and Args overrides CMD. Passing ["python", "-c", "..."] as Command is correct, but ["bash", "-c", "...script..."] is also placed in Command. This is functionally fine but it might be cleaner to split entrypoint from arguments using container.Args for the script portion.

[Low] application label value is hardcoded as "vector-add" for all tests

All three tests share labels := map[string]string{"application": "vector-add"}. The CUDA compile and PyTorch tests are not vector-add workloads. While this is cosmetically minor and doesn't affect test correctness, it could confuse observability tooling. Pass labels as a parameter or derive from the test context.

Tokens Used: Input: 8,488 | Output: 1,094 | Cache Write: 8,486 | Cache Read: 0 | Total cost: $0.0482