WIP: tests: (scale and servicing) use framework context for update-loop goroutine lifecycle

[bfjelds]

Three fixes for scale test reliability:

1. Storm framework integration — use tc.Context() instead of context.Background() for proper goroutine lifecycle management
2. Per-iteration SSH proxy lifecycle — cancel tunnels before finalize to prevent goroutine accumulation
3. Improved initramfs diagnostics — distinguish genuine initramfs failures from transient warnings where VM booted to systemd

Addresses goroutine leak warnings and false-positive initramfs attribution in scale test runs.

[bfjelds]

/azp run [GITHUB]-trident-pr-e2e

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[bfjelds]

/azp run [GITHUB]-trident-pr-e2e

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Improves Storm scale test reliability by tying long-running goroutines to the framework test-case context, tightening per-iteration SSH tunnel cleanup, and making initramfs diagnostics less prone to false positives.

Changes:

• Switch servicing test execution to pass storm.TestCase through so tests can use tc.Context() / tc.BackgroundWaitGroup().
• Add per-iteration SSH proxy cancellation + wait to avoid accumulating proxy goroutines/processes across update iterations.
• Refine serial log parsing to separate transient dracut warnings from true initramfs boot failures.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
tools/storm/utils/ssh/ssh.go	Makes SSH reverse port forward startup more strict and adjusts behavior on context-cancelled shutdown.
tools/storm/servicing/trident.go	Adds a context-aware runner that passes storm.TestCase into servicing test functions.
tools/storm/servicing/tests/update.go	Uses framework context/WaitGroup for background servers, adds per-iteration SSH proxy lifecycle, and improves dracut/initramfs diagnostics.

Comments suppressed due to low confidence (1)

tools/storm/servicing/tests/update.go:279

• The temp file handle (stageLogLocalTmpFile) is never closed, and the temp path is not removed on several early-return paths after creation (e.g., SCP download failure, later stage/finalize error returns before the explicit os.Remove). Close the file promptly after getting the name, and ensure the temp path is removed on all exits from the iteration (a common approach is to scope per-iteration cleanup via an inner function so defers don’t accumulate across iterations).

		stageLogLocalTmpFile, err := os.CreateTemp("", "staged-trident-full")
		if err != nil {
			cleanupProxies()
			return fmt.Errorf("failed to create temp staging log file: %w", err)
		}
		stageLogLocalTmpPath := stageLogLocalTmpFile.Name()

		err = stormssh.ScpDownloadFile(vmConfig.VMConfig, vmIP, "/var/log/trident-full.log", stageLogLocalTmpPath)
		if err != nil {
			return fmt.Errorf("failed to download staged trident log: %w", err)
		}

[bfjelds]

/azp run [GITHUB]-trident-pr-e2e

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

tools/storm/servicing/tests/update.go:279

• The temp file handle is never closed (stageLogLocalTmpFile.Close()), and the temp path is only removed on the “success” path. Any early return after creation (e.g., SCP failure, copy failure, stage errors) will leak temp files; at scale this can exhaust disk/inodes and file descriptors. A concrete fix is to close the temp file immediately after getting its name, and ensure the temp path is removed on all paths (e.g., via a per-iteration-scoped defer in an inner function, or explicit cleanup before each return after creation).

		stageLogLocalTmpFile, err := os.CreateTemp("", "staged-trident-full")
		if err != nil {
			cleanupProxies()
			return fmt.Errorf("failed to create temp staging log file: %w", err)
		}
		stageLogLocalTmpPath := stageLogLocalTmpFile.Name()

		err = stormssh.ScpDownloadFile(vmConfig.VMConfig, vmIP, "/var/log/trident-full.log", stageLogLocalTmpPath)
		if err != nil {
			return fmt.Errorf("failed to download staged trident log: %w", err)
		}

[bfjelds]

/azp run [GITHUB]-trident-pr-e2e

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.