docs: multi-SG Prometheus prereq for ai-service-metrics check (#939)

Author: njhensley

docs/integrator/eks-dynamo-networking.md

@@ -8,11 +8,42 @@ If system components and GPU workloads are on different node groups/security gro
 - `Unable to create lease` (etcd unreachable)
 - `JetStream not available` (NATS unreachable)
 
+The conformance validator's `ai-service-metrics` check adds a third requirement:
+it dials Prometheus over the cluster Service (typically
+`kube-prometheus-prometheus.monitoring.svc:9090`). The orchestrator Job that
+runs the check tolerates every taint and has no node-affinity toward
+Prometheus, so the kube-scheduler may place it on any worker node — including
+one whose ENI is in a security group that cannot reach the Prometheus pod.
+
+When that happens, the dial times out at 5 s and the check is marked `failed`:
+
+```
+[SERVICE_UNAVAILABLE] Prometheus unreachable at http://kube-prometheus-prometheus.monitoring.svc:9090 — verify network connectivity
+```
+
+The outcome is **non-deterministic from run to run** on the same cluster: the
+first scheduling decision is a tie-break, and subsequent runs are dominated by
+image-locality scoring (whichever node pulled the validator image first keeps
+winning). A re-run on a "freshly working" cluster is therefore not a reliable
+signal that the SG topology is correct.
+
+This is tracked in [issue #933](https://github.com/NVIDIA/aicr/issues/933);
+this page documents the cluster-side prerequisite until the validator gains
+`podAffinity` toward Prometheus.
+
 ## Required Security Group Rules
 
 Allow ingress from the GPU node security group to the system node security group on:
-- TCP `2379`
-- TCP `4222`
+- TCP `2379` — etcd (dynamo-platform)
+- TCP `4222` — NATS / JetStream (dynamo-platform)
+- TCP `9090` — Prometheus (required for the `ai-service-metrics` conformance check)
+
+The `9090` rule is symmetrically required: the validator orchestrator pod may
+land on **any** worker node, so every node group whose pods can host the
+orchestrator must be able to reach the Prometheus pod's IP on `9090`. On
+clusters with separate customer/system ENI subnets (e.g. DGXC EKS), this means
+the system SG must accept ingress from the customer SG (and any other worker
+SG), not only from itself.
 
 Example:
 
@@ -28,10 +59,17 @@ aws ec2 describe-instances \
   --query "Reservations[0].Instances[0].SecurityGroups[*].GroupId" \
   --output text
 
-# 2) Allow etcd + NATS from GPU SG -> system SG
+# 2) Allow etcd + NATS + Prometheus from GPU SG -> system SG
 aws ec2 authorize-security-group-ingress --group-id <system-sg-id> \
   --protocol tcp --port 2379 --source-group <gpu-sg-id>
 
 aws ec2 authorize-security-group-ingress --group-id <system-sg-id> \
   --protocol tcp --port 4222 --source-group <gpu-sg-id>
+
+aws ec2 authorize-security-group-ingress --group-id <system-sg-id> \
+  --protocol tcp --port 9090 --source-group <gpu-sg-id>
 ```
+
+If the cluster has more than two worker security groups, repeat the `9090`
+rule for each non-system SG that can host pods — the validator orchestrator
+has no scheduling preference and may land on any of them.

docs/user/validation.md

@@ -359,6 +359,35 @@ Common reasons and their cause:
 | `skipped - no-cluster mode` | `message` | `--no-cluster` was passed — the runner short-circuits every phase before dispatching any Job | Remove the flag to run behavioral checks |
 | `skipped due to previous phase failure` | `message` | An earlier phase failed and subsequent phases are skipped | Fix the earlier phase first, then re-run |
 
+### `ai-service-metrics` fails with "Prometheus unreachable"
+
+On EKS clusters that split worker and system pods across separate security
+groups (e.g. DGXC EKS with distinct customer/system ENI subnets), the
+conformance check `ai-service-metrics` can fail non-deterministically with:
+
+```
+[SERVICE_UNAVAILABLE] Prometheus unreachable at http://kube-prometheus-prometheus.monitoring.svc:9090 — verify network connectivity
+```
+
+The validator orchestrator Job tolerates every taint and has no node-affinity
+toward Prometheus, so the kube-scheduler may place it on any worker node —
+including one whose ENI is in a security group whose ingress to the
+Prometheus-hosting SG is missing or asymmetric. The outcome is **not stable
+across re-runs**: image-locality scoring tends to keep the pod on whatever
+node won the first scheduling decision, so a passing run on a fresh cluster
+does not prove the SG topology is correct.
+
+This is a cluster-side prerequisite, not an AICR bug per se — see
+[EKS Dynamo Networking Prerequisites](../integrator/eks-dynamo-networking.md#required-security-group-rules)
+for the SG ingress rules required for Prometheus (`tcp/9090`). The underlying
+issue is tracked at [#933](https://github.com/NVIDIA/aicr/issues/933).
+
+Workaround when SG changes are not available: re-run the check until the
+orchestrator lands on a node whose SG can reach Prometheus, then leave the
+image cached there so image-locality keeps subsequent runs on the same node.
+This is unreliable and should not be used as the steady-state validation
+strategy.
+
 ### Benchmark Job stuck or timed out
 
 Each performance check has a Job-level `activeDeadlineSeconds` set by the catalog's `timeout:`. For `inference-perf`, the full pipeline (workload ready → endpoint health → benchmark) can take up to 30 min on cold-start clusters. If it still times out: