Skip to content

Commit 3a53fa1

Browse files
oguzkilcanoguzkilcan
committed
test: verify upgrade drain honors grace period and aborts on failure
Before Talos 1.13 the drain happened inside machined as part of the upgrade sequence. It evicted every pod with a hardcoded 60 second grace period, and it logged eviction failures instead of returning them, so the node rebooted whether or not the workload had actually finished shutting down. Omni now runs the drain itself for cluster on 1.13 and up, passing each pod's own terminationGracePeriodSeconds through and refusing to reboot when the drain does not complete. Nothing covered that end to end. The new suite parks a two-replica workload behind a PodDisruptionBudget, has it ignore SIGTERM for six minutes, then upgrades the cluster and rebuilds the drain timeline by sampling pod, node and ClusterMachineConfigStatus state. It checks that the eviction carried the pod's full grace period rather than a fixed one, that the drain failure surfaced instead of being swallowed, and that the node's boot ID only changed once the pod was gone. Signed-off-by: Oguz Kilcan <oguz.kilcan@siderolabs.com>
1 parent 09a97cd commit 3a53fa1

7 files changed

Lines changed: 693 additions & 2 deletions

File tree

‎.github/workflows/ci.yaml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -416,6 +416,79 @@ jobs:
416416
path: ${{ github.workspace }}/integration-test
417417
retention-days: "5"
418418
continue-on-error: true
419+
e2e-drain-grace-period:
420+
runs-on:
421+
group: large
422+
if: contains(fromJSON(needs.default.outputs.labels || '[]'), 'integration/e2e')
423+
|| contains(fromJSON(needs.default.outputs.labels || '[]'), 'integration/e2e-drain')
424+
needs:
425+
- integration-talemu
426+
- integration-qemu
427+
- e2e-qemu
428+
- e2e-talemu
429+
- e2e-auth0
430+
- e2e-saml
431+
- e2e-helm
432+
steps:
433+
- name: gather-system-info
434+
id: system-info
435+
uses: kenchan0130/actions-system-info@59699597e84e80085a750998045983daa49274c4 # version: v1.4.0
436+
continue-on-error: true
437+
- name: print-system-info
438+
run: |
439+
MEMORY_GB=$((${{ steps.system-info.outputs.totalmem }}/1024/1024/1024))
440+
441+
OUTPUTS=(
442+
"CPU Core: ${{ steps.system-info.outputs.cpu-core }}"
443+
"CPU Model: ${{ steps.system-info.outputs.cpu-model }}"
444+
"Hostname: ${{ steps.system-info.outputs.hostname }}"
445+
"NodeName: ${NODE_NAME}"
446+
"Kernel release: ${{ steps.system-info.outputs.kernel-release }}"
447+
"Kernel version: ${{ steps.system-info.outputs.kernel-version }}"
448+
"Name: ${{ steps.system-info.outputs.name }}"
449+
"Platform: ${{ steps.system-info.outputs.platform }}"
450+
"Release: ${{ steps.system-info.outputs.release }}"
451+
"Total memory: ${MEMORY_GB} GB"
452+
)
453+
454+
for OUTPUT in "${OUTPUTS[@]}";do
455+
echo "${OUTPUT}"
456+
done
457+
continue-on-error: true
458+
- name: checkout
459+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # version: v7.0.0
460+
- name: Unshallow
461+
run: |
462+
git fetch --prune --unshallow
463+
- name: Set up Docker Buildx
464+
id: setup-buildx
465+
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # version: v4.2.0
466+
with:
467+
driver: remote
468+
endpoint: tcp://buildkit-amd64.ci.svc.cluster.local:1234
469+
timeout-minutes: 10
470+
- name: Mask secrets
471+
run: |
472+
echo "$(sops -d .secrets.yaml | yq -e '.secrets | to_entries[] | "::add-mask::" + .value')"
473+
- name: Set secrets for job
474+
run: |
475+
sops -d .secrets.yaml | yq -e '.secrets | to_entries[] | .key + "=" + .value' >> "$GITHUB_ENV"
476+
- name: run-integration-qemu
477+
env:
478+
INTEGRATION_TEST_ARGS: --test.run TestIntegration/Suites/(CleanState|DrainTerminationGracePeriod)$
479+
TEST_CLASS: integration-qemu
480+
WITH_DEBUG: "true"
481+
WITH_RACE: "true"
482+
run: |
483+
sudo -E make run-integration-test
484+
- name: save-integration-qemu-artifacts
485+
if: always()
486+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # version: v7.0.1
487+
with:
488+
name: integration-qemu-e2e-drain-grace-period
489+
path: ${{ github.workspace }}/integration-test
490+
retention-days: "5"
491+
continue-on-error: true
419492
e2e-enterprise:
420493
runs-on:
421494
group: large

‎.github/workflows/e2e-drain-grace-period-cron.yaml‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# THIS FILE WAS AUTOMATICALLY GENERATED BY KRES, PLEASE DO NOT EDIT.
2+
#
3+
# Generated on 2026-08-05T11:13:11Z by kres 93d28ca.
4+
5+
concurrency:
6+
group: ${{ github.head_ref || github.run_id }}
7+
cancel-in-progress: true
8+
'on':
9+
schedule:
10+
- cron: 30 1 * * *
11+
name: e2e-drain-grace-period-cron
12+
jobs:
13+
default:
14+
runs-on:
15+
group: large
16+
steps:
17+
- name: gather-system-info
18+
id: system-info
19+
uses: kenchan0130/actions-system-info@59699597e84e80085a750998045983daa49274c4 # version: v1.4.0
20+
continue-on-error: true
21+
- name: print-system-info
22+
run: |
23+
MEMORY_GB=$((${{ steps.system-info.outputs.totalmem }}/1024/1024/1024))
24+
25+
OUTPUTS=(
26+
"CPU Core: ${{ steps.system-info.outputs.cpu-core }}"
27+
"CPU Model: ${{ steps.system-info.outputs.cpu-model }}"
28+
"Hostname: ${{ steps.system-info.outputs.hostname }}"
29+
"NodeName: ${NODE_NAME}"
30+
"Kernel release: ${{ steps.system-info.outputs.kernel-release }}"
31+
"Kernel version: ${{ steps.system-info.outputs.kernel-version }}"
32+
"Name: ${{ steps.system-info.outputs.name }}"
33+
"Platform: ${{ steps.system-info.outputs.platform }}"
34+
"Release: ${{ steps.system-info.outputs.release }}"
35+
"Total memory: ${MEMORY_GB} GB"
36+
)
37+
38+
for OUTPUT in "${OUTPUTS[@]}";do
39+
echo "${OUTPUT}"
40+
done
41+
continue-on-error: true
42+
- name: checkout
43+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # version: v7.0.0
44+
- name: Unshallow
45+
run: |
46+
git fetch --prune --unshallow
47+
- name: Set up Docker Buildx
48+
id: setup-buildx
49+
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # version: v4.2.0
50+
with:
51+
driver: remote
52+
endpoint: tcp://buildkit-amd64.ci.svc.cluster.local:1234
53+
timeout-minutes: 10
54+
- name: Mask secrets
55+
run: |
56+
echo "$(sops -d .secrets.yaml | yq -e '.secrets | to_entries[] | "::add-mask::" + .value')"
57+
- name: Set secrets for job
58+
run: |
59+
sops -d .secrets.yaml | yq -e '.secrets | to_entries[] | .key + "=" + .value' >> "$GITHUB_ENV"
60+
- name: run-integration-qemu
61+
env:
62+
INTEGRATION_TEST_ARGS: --test.run TestIntegration/Suites/(CleanState|DrainTerminationGracePeriod)$
63+
TEST_CLASS: integration-qemu
64+
WITH_DEBUG: "true"
65+
WITH_RACE: "true"
66+
run: |
67+
sudo -E make run-integration-test
68+
- name: save-integration-qemu-artifacts
69+
if: always()
70+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # version: v7.0.1
71+
with:
72+
name: integration-qemu
73+
path: ${{ github.workspace }}/integration-test
74+
retention-days: "5"

‎.github/workflows/slack-notify-ci-failure.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# THIS FILE WAS AUTOMATICALLY GENERATED BY KRES, PLEASE DO NOT EDIT.
22
#
3-
# Generated on 2026-07-09T13:52:50Z by kres 14a38a3.
3+
# Generated on 2026-08-05T11:13:11Z by kres 93d28ca.
44

55
'on':
66
workflow_run:
@@ -19,6 +19,7 @@
1919
- e2e-cluster-import-cron
2020
- e2e-minor-talos-upgrade-cron
2121
- e2e-rotate-ca-cron
22+
- e2e-drain-grace-period-cron
2223
- e2e-enterprise-cron
2324
types:
2425
- completed

‎.github/workflows/slack-notify.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# THIS FILE WAS AUTOMATICALLY GENERATED BY KRES, PLEASE DO NOT EDIT.
22
#
3-
# Generated on 2026-07-09T13:52:50Z by kres 14a38a3.
3+
# Generated on 2026-08-05T11:13:11Z by kres 93d28ca.
44

55
'on':
66
workflow_run:
@@ -19,6 +19,7 @@
1919
- e2e-cluster-import-cron
2020
- e2e-minor-talos-upgrade-cron
2121
- e2e-rotate-ca-cron
22+
- e2e-drain-grace-period-cron
2223
- e2e-enterprise-cron
2324
types:
2425
- completed

‎.kres.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -602,6 +602,19 @@ spec:
602602
skipArtifactDownload: true
603603
environmentOverride:
604604
INTEGRATION_TEST_ARGS: "--test.run TestIntegration/Suites/(CleanState|RotateCA)$"
605+
- name: e2e-drain-grace-period
606+
needsOverride:
607+
[integration-talemu, integration-qemu, e2e-qemu, e2e-talemu, e2e-auth0, e2e-saml, e2e-helm]
608+
crons:
609+
- "30 1 * * *"
610+
runnerGroup: large
611+
triggerLabels:
612+
- integration/e2e
613+
- integration/e2e-drain
614+
artifacts:
615+
skipArtifactDownload: true
616+
environmentOverride:
617+
INTEGRATION_TEST_ARGS: "--test.run TestIntegration/Suites/(CleanState|DrainTerminationGracePeriod)$"
605618
- name: e2e-enterprise
606619
needsOverride:
607620
[integration-talemu, integration-qemu, e2e-qemu, e2e-talemu, e2e-auth0, e2e-saml, e2e-helm]

0 commit comments

Comments
 (0)