Prevent operand pods from starting before nvidia driver is loaded

When an operand pod (mig-manager, device-plugin, GFD, etc.) is created
while the nvidia driver module is not loaded, the nvidia container runtime
wrapper falls back to plain runc/crun. This means no NVIDIA libraries are
injected into the container — the container is permanently broken for its
lifetime.

The init containers in these pods gate on the toolkit-ready sentinel file,
but this file can be validly present from a running operator-validator
even while the driver is temporarily unloaded (e.g., during a driver
container restart). The sentinel file system validates that components
were installed correctly but does not validate that the nvidia kernel
module is currently loaded — which is the actual condition required for
the container runtime to inject NVIDIA libraries.

Add a /proc/modules check for the nvidia kernel module alongside the
existing toolkit-ready file check in all operand init containers. This
ensures the main container is not created until the nvidia module is
loaded, guaranteeing the runtime wrapper invokes the real
nvidia-container-runtime (which injects libraries) instead of falling
back to runc.

Additionally, add preStop lifecycle hooks to the toolkit and driver
containers to clean up sentinel files (toolkit-ready, driver-ready) on
graceful termination. This prevents downstream init containers from
passing on stale files when pods are properly terminated during upgrades
or drains.

Co-authored-by: Shiva Kumar <shivaku@nvidia.com>
Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>

Author: rajathagasthya

assets/gpu-feature-discovery/0500_daemonset.yaml

@@ -29,7 +29,7 @@ spec:
         - name: toolkit-validation
           image: "FILLED BY THE OPERATOR"
           command: ['sh', '-c']
-          args: ["until [ -f /run/nvidia/validations/toolkit-ready ]; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
+          args: ["until [ -f /run/nvidia/validations/toolkit-ready ] && grep -q '^nvidia ' /proc/modules; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
           securityContext:
             privileged: true
           volumeMounts:

assets/state-container-toolkit/0500_daemonset.yaml

@@ -77,6 +77,10 @@ spec:
               fieldPath: metadata.namespace
         imagePullPolicy: IfNotPresent
         name: nvidia-container-toolkit-ctr
+        lifecycle:
+          preStop:
+            exec:
+              command: ["/bin/sh", "-c", "rm -f /run/nvidia/validations/toolkit-ready"]
         securityContext:
           privileged: true
           seLinuxOptions:

assets/state-dcgm-exporter/0800_daemonset.yaml

@@ -28,7 +28,7 @@ spec:
       - name: toolkit-validation
         image: "FILLED BY THE OPERATOR"
         command: ['sh', '-c']
-        args: ["until [ -f /run/nvidia/validations/toolkit-ready ]; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
+        args: ["until [ -f /run/nvidia/validations/toolkit-ready ] && grep -q '^nvidia ' /proc/modules; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
         securityContext:
           privileged: true
         volumeMounts:

assets/state-dcgm/0400_dcgm.yml

@@ -28,7 +28,7 @@ spec:
       - name: toolkit-validation
         image: "FILLED BY THE OPERATOR"
         command: ['sh', '-c']
-        args: ["until [ -f /run/nvidia/validations/toolkit-ready ]; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
+        args: ["until [ -f /run/nvidia/validations/toolkit-ready ] && grep -q '^nvidia ' /proc/modules; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
         securityContext:
           privileged: true
         volumeMounts:

assets/state-device-plugin/0500_daemonset.yaml

@@ -28,7 +28,7 @@ spec:
       - image: "FILLED BY THE OPERATOR"
         name: toolkit-validation
         command: ['sh', '-c']
-        args: ["until [ -f /run/nvidia/validations/toolkit-ready ]; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
+        args: ["until [ -f /run/nvidia/validations/toolkit-ready ] && grep -q '^nvidia ' /proc/modules; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
         securityContext:
           privileged: true
         volumeMounts:

assets/state-driver/0500_daemonset.yaml

@@ -150,7 +150,7 @@ spec:
         lifecycle:
           preStop:
             exec:
-              command: ["/bin/sh", "-c", "rm -f /run/nvidia/validations/.driver-ctr-ready"]
+              command: ["/bin/sh", "-c", "rm -f /run/nvidia/validations/.driver-ctr-ready /run/nvidia/validations/driver-ready"]
       - image: "FILLED BY THE OPERATOR"
         imagePullPolicy: IfNotPresent
         name: nvidia-peermem-ctr

assets/state-mig-manager/0600_daemonset.yaml

@@ -28,7 +28,7 @@ spec:
         - name: toolkit-validation
           image: "FILLED BY THE OPERATOR"
           command: ['sh', '-c']
-          args: ["until [ -f /run/nvidia/validations/toolkit-ready ]; do echo waiting for nvidia container toolkit to be setup; sleep 5; done"]
+          args: ["until [ -f /run/nvidia/validations/toolkit-ready ] && grep -q '^nvidia ' /proc/modules; do echo waiting for nvidia container toolkit to be setup; sleep 5; done"]
           securityContext:
             privileged: true
           volumeMounts:

assets/state-mps-control-daemon/0400_daemonset.yaml

@@ -30,7 +30,7 @@ spec:
         - image: "FILLED BY THE OPERATOR"
           name: toolkit-validation
           command: ['sh', '-c']
-          args: ["until [ -f /run/nvidia/validations/toolkit-ready ]; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
+          args: ["until [ -f /run/nvidia/validations/toolkit-ready ] && grep -q '^nvidia ' /proc/modules; do echo waiting for nvidia container stack to be setup; sleep 5; done"]
           securityContext:
             privileged: true
           volumeMounts: