|
1 | 1 | # Snapshot Agent Integration Tests |
2 | 2 |
|
3 | | -End-to-end tests for snapshot-agent backends on real GPU hardware, in both standalone and K8s deployment modes. |
| 3 | +End-to-end tests for snapshot-agent backends on real GPU hardware. Every |
| 4 | +phase exercises a real install path — the way a user actually deploys the |
| 5 | +component, not a test-only build: |
4 | 6 |
|
5 | | -The test suite is written in Go and runs inside the cluster: `run.sh` deploys a test-runner pod, copies the repo source into it, and executes `go test` there. The Go harness deploys the snapshot-agent and inference engine pods itself — one engine at a time, so a single free GPU is enough. |
| 7 | +| Phase | Test | Fixture | |
| 8 | +|---|---|---| |
| 9 | +| `standalone` | `TestStandalone` | `make standalone` artifacts (`bin/snapshot-agent` + `bin/cuda-checkpoint`), built in-cluster and run from a plain Debian base image | |
| 10 | +| `k8s` | `TestK8s` | **official snapshot-agent Helm chart** (DaemonSet pinned to `TEST_NODE`) | |
| 11 | + |
| 12 | +The k8s phase runs against the chart the way a user deploys it, so every |
| 13 | +backend (CUDA checkpoint via watcher-discovered PIDs, vLLM sleep/wake, SGLang |
| 14 | +release/resume) is exercised through the official chart exactly once. |
| 15 | +`run.sh` installs the chart fixture before starting `go test` and fails fast |
| 16 | +with a chart-attributed error if the chart pod does not become Ready — a |
| 17 | +broken chart never shows up as a test failure. |
| 18 | + |
| 19 | +The test suite is written in Go and runs inside the cluster: `run.sh` deploys a test-runner pod, copies the repo source into it, and executes `go test` there. The Go harness deploys the agent (standalone phase) and inference engine pods itself — one engine at a time, so a single free GPU is enough. |
6 | 20 |
|
7 | 21 | All snapshot/restore calls go through the **Python client** (`timeslice.snapshot_agent`, invoked via `agentctl.py`), so the entire client layer is covered. |
8 | 22 |
|
9 | | -- `run.sh` — launcher (deploy runner, copy source, install the Python client, `go test`, cleanup) |
| 23 | +- `run.sh` — launcher (build image, install chart fixture, deploy runner, copy source, build `make standalone`, install the Python client, `go test`, cleanup) |
10 | 24 | - `runner.yaml` — test-runner pod + RBAC |
11 | 25 | - `harness.go` / `engines.go` — harness: pod lifecycle, exec/HTTP helpers, pod specs |
12 | 26 | - `agentctl.py` — thin CLI over the Python client; builds `BackendConfig` protos in Python from primitive flags |
13 | 27 | - `standalone_test.go` / `k8s_test.go` — the test cases |
14 | 28 |
|
15 | | -## Adding a test |
| 29 | +## Testing your changes |
16 | 30 |
|
17 | | -Add a `t.Run(...)` inside the engine group that provides the pods it needs, using the harness helpers: |
| 31 | +Everything runs from your working directory — uncommitted changes included — |
| 32 | +so no commit or merge is needed at any layer: |
18 | 33 |
|
19 | | -```go |
20 | | -h.WithEngine(t, VLLM, func(t *testing.T, e *Engine) { |
21 | | - t.Run("MyNewTest", func(t *testing.T) { |
22 | | - before := h.Inference(t, e) // deterministic completion |
23 | | - h.SnapshotOK(t, "my-job", vllmSleepConfig(e.Endpoint(), 1)) |
24 | | - vram := h.VRAMMiB(t, e) // GPU memory in use |
25 | | - h.RestoreOK(t, "my-job", vllmWakeConfig(e.Endpoint())) |
26 | | - RequireFreedAndCorrect(t, vram, before, h.Inference(t, e)) |
27 | | - }) |
28 | | -}) |
| 34 | +```bash |
| 35 | +TEST_NODE=<gpu-node> ./tests/integration/snapshot-agent/run.sh \ |
| 36 | + --build --project <gcp-project> |
29 | 37 | ``` |
30 | 38 |
|
31 | | -A new engine is an `EngineSpec` in `engines.go`. |
| 39 | +`--build` has Cloud Build produce the agent image from the working directory |
| 40 | +(tagged `integ-<commit>` so repeated runs don't collide with node image |
| 41 | +caches) and runs both phases against it. `run.sh` then copies the local |
| 42 | +workspace into the cluster, so the standalone phase's `make standalone` |
| 43 | +build, the Helm chart (installed from local `deploy/`), the Python client, |
| 44 | +and the test code all come from the workspace too. |
32 | 45 |
|
33 | | -## Prerequisites |
34 | | - |
35 | | -- A GKE cluster with at least 1 free GPU |
36 | | -- `gcloud` and `kubectl` on the machine running the tests |
37 | | - (Go and everything else run inside the cluster) |
38 | | -- A snapshot-agent image. Build one from the repo root with: |
| 46 | +Alternative (pre-built image — any registry the cluster can pull from): |
39 | 47 |
|
40 | 48 | ```bash |
41 | 49 | gcloud builds submit --config=cloudbuild-image.yaml \ |
42 | | - --substitutions=_IMAGE=gcr.io/<your-project>/snapshot-agent:dev . |
43 | | -``` |
| 50 | + --substitutions=_IMAGE=gcr.io/<project>/snapshot-agent:dev . |
44 | 51 |
|
45 | | -This builds from your working directory, so local modifications are included — no commit needed. Requires the Cloud Build API (`gcloud services enable cloudbuild.googleapis.com`) and permission to push to the project's registry; GKE nodes in the same project can pull from `gcr.io/<project>` by default. |
| 52 | +TEST_NODE=<gpu-node> ./tests/integration/snapshot-agent/run.sh \ |
| 53 | + --agent-image gcr.io/<project>/snapshot-agent:dev |
| 54 | +``` |
46 | 55 |
|
47 | | -## Running |
| 56 | +## Prerequisites |
48 | 57 |
|
49 | | -```bash |
50 | | -./tests/integration/snapshot-agent/run.sh \ |
51 | | - --image gcr.io/<your-project>/snapshot-agent:dev \ |
52 | | - --project <your-project> \ |
53 | | - --cluster <your-cluster> \ |
54 | | - --zone <your-zone> |
55 | | -``` |
| 58 | +- A GKE cluster with at least 1 free GPU |
| 59 | +- `gcloud` and `kubectl` on the machine running the tests |
| 60 | + (Go and everything else run inside the cluster) |
| 61 | +- For the k8s phase: a snapshot-agent image built from the official |
| 62 | + Dockerfile (`--build` does this for you; `cloudbuild-image.yaml` defaults |
| 63 | + to `docker/snapshot-agent/Dockerfile`). The standalone phase needs no |
| 64 | + image — it builds the agent from source in the test runner. |
| 65 | +- Cloud Build API enabled (`gcloud services enable cloudbuild.googleapis.com`) |
| 66 | + and permission to push to the project's registry; GKE nodes in the same |
| 67 | + project can pull from `gcr.io/<project>` by default. |
56 | 68 |
|
57 | 69 | ## Options |
58 | 70 |
|
59 | 71 | ```text |
60 | | ---image IMAGE Snapshot-agent container image (required) |
61 | | ---project PROJECT GCP project (runs gcloud get-credentials) |
| 72 | +--agent-image IMAGE Snapshot-agent image the k8s phase installs via the |
| 73 | + official chart (required for k8s/both unless --build) |
| 74 | +--build Build the agent image from the working directory via |
| 75 | + Cloud Build (requires --project); explicit |
| 76 | + --agent-image overrides |
| 77 | +--project PROJECT GCP project (image pushes with --build; also used by |
| 78 | + gcloud get-credentials with --cluster) |
62 | 79 | --cluster CLUSTER GKE cluster name |
63 | 80 | --zone ZONE GKE cluster zone |
64 | 81 | --model MODEL Model to load (default: Qwen/Qwen2.5-0.5B) |
65 | | ---phase PHASE "standalone", "k8s", or "both" (default: both) |
66 | | ---skip-cleanup Leave the test-runner pod running for debugging |
| 82 | +--phase PHASE "standalone", "k8s", or "both" (default) |
| 83 | +--skip-cleanup Leave the test-runner pod and chart fixture running |
| 84 | + for debugging |
67 | 85 | ``` |
68 | 86 |
|
69 | | -Environment: `TEST_NODE=<node-name>` pins the suite to a specific node instead |
70 | | -of the default pick (first node with a free GPU by requests). Use it when the |
71 | | -cluster runs workloads that occupy GPUs without requesting them (time-slicing |
72 | | -experiments), which the default pick cannot see. |
| 87 | +Environment: |
| 88 | + |
| 89 | +- `TEST_NODE=<node-name>` — required for the k8s phase (the chart is pinned |
| 90 | + to this node); for the standalone phase it pins the suite instead of the |
| 91 | + default pick (first node with a free GPU by requests). Use the pin when |
| 92 | + the cluster runs workloads that occupy GPUs without requesting them |
| 93 | + (time-slicing experiments), which the default pick cannot see. |
| 94 | +- `CHART_AGENT_PORT=<port>` — port for the chart-deployed agent (default |
| 95 | + 9001), so the suite can coexist with an unrelated agent on the default |
| 96 | + port (the chart runs on hostNetwork). |
73 | 97 |
|
74 | 98 | ## Exit code |
75 | 99 |
|
|
0 commit comments