Improve integration test robustness

[aishukamal]

Three tasks to improve integration test reliability, discovered during the channel and orchestrator suite development:

1. Robustness soak: run integration suite 100 times on fresh nodes

Run the full suite (--phase all) 100 times, each on a freshly provisioned node (cold image pull, cold DaemonSet startup). Vary GPU types (L4, H100) across runs. Goal: identify flakes from cold-start timing, spot preemption recovery, and non-deterministic engine behavior. Report the failure rate and classify each failure.

2. Fix standalone agent pod lifecycle

The standalone agent pod is cleaned up (via t.Cleanup) before the channel workload test can exec into it for PID discovery. The channel test calls h.findPID(t, "channel_workload") which execs into snapshot-agent-test, but that pod was already deleted by a prior test group's cleanup.

Repro: run --phase standalone on any GPU node with the channel tests enabled (#105). The channel standalone test fails with exec failed: pods "snapshot-agent-test" not found.

Fix: ensure the standalone agent pod outlives all test groups that need it — either restructure cleanup ordering or scope the agent lifecycle to the entire standalone phase rather than individual engine groups.

3. Fix SGLang non-determinism on L4

TestStandalone/SGLangCompound produces different generations before and after compound C/R on L4 GPUs. The same test passes consistently on H100.

Repro: run --phase standalone on an L4 node. SGLangCompound fails with inference changed after restore.

Either the assertion is too strict for smaller GPUs (floating-point differences in quantization/scheduling change outputs) or SGLang's restore path has a real non-determinism bug on L4. Investigate and fix — either relax the assertion (e.g. compare logprobs instead of exact text) or file upstream.

4. Add retry logic to Kubernetes node update operations in orchestrator harness

The orchestrator harness (tests/integration/orchestrator/harness.go) uses bare Nodes().Update() calls for label mutations (exclusiveLabel, labelNode, and their t.Cleanup restore paths). Kubernetes node updates are sensitive to timing — concurrent modifications cause conflict errors. Add retry-on-conflict logic (e.g. retry.RetryOnConflict) to all node update calls in the harness.