integration-testing.md

Integration testing — Hulumi M3 weekly workflow

Hulumi's per-PR test suite runs entirely on Pulumi mocks. Real-AWS integration is reserved for the dedicated weekly/manual workflow, not PRs, and runs in a sandbox AWS account via GitHub Actions OIDC. This document spells out the workflow, the auth path, the cost contract, and the open-source threat model.

Workflow

.github/workflows/weekly-integration.yml

• Trigger: cron 0 4 * * 0 (Sundays 04:00 UTC) + workflow_dispatch for ad-hoc runs.
• Matrix: tier ∈ {sandbox, startup-hardened} — serialized with max-parallel: 1 because AccountFoundation touches account-wide AWS services.
• Job timeout: 30 minutes total (15-min Pulumi up + 10 min destroy + 5 min slop, per the M3 contract's eventual-consistency window).
• Auth: OIDC AssumeRoleWithWebIdentity against AWS_SANDBOX_OIDC_ROLE_ARN. No long-lived AWS credentials.
• State backend: prefer a self-managed S3 backend via the PULUMI_BACKEND_URL repository secret. Pulumi Cloud via PULUMI_ACCESS_TOKEN remains supported for maintainers who opt in, but the workflow refuses to run with both configured. When neither is set, the workflow runs CONTRACT-ONLY mode (mock unit + integration tests, no actual pulumi up).

Required repo configuration

Type	Name	Source	Required?
Secret	AWS_SANDBOX_ACCOUNT_ID	account ID of the sandbox AWS account	yes
Secret	AWS_SANDBOX_OIDC_ROLE_ARN	ARN of the IaC role with hulumi:iac-role=true tag	yes
Variable	AWS_SANDBOX_REGION	AWS region (default us-east-1)	yes
Secret	PULUMI_BACKEND_URL	private S3 backend URL	preferred real-AWS path
Secret	PULUMI_ACCESS_TOKEN	Pulumi Cloud personal access token	optional alternative

The first three are set during sandbox bootstrap (see deployment/sandbox-account.md). PULUMI_BACKEND_URL should point at a private, versioned, SSE-encrypted S3 bucket in the same sandbox account. The workflow creates or hardens that bucket idempotently, blocks public access, and refuses non-S3 backends in CI.

The S3 backend bucket is deliberately out-of-band from the Pulumi stacks under test. Routine pulumi destroy calls delete resources recorded in the stack being destroyed; they do not delete the backend bucket or its versioned state objects unless a Pulumi program explicitly manages that bucket as a resource. The weekly workflow may create or harden the backend bucket, but it never deletes it.

Open-Source Threat Model

The repository is public, so CI must assume hostile pull requests and curious readers:

• Real AWS credentials are OIDC-only. No static AWS keys, Pulumi Cloud tokens, passphrases, kubeconfigs, app private keys, sandbox account IDs, role ARNs, or backend bucket URLs are committed or stored as public Actions variables.
• The OIDC trust policy is branch-scoped to repo:kerberosmansour/hulumi:ref:refs/heads/main; forked PRs and feature branches cannot assume the sandbox role.
• The Pulumi state backend is not public. S3 state must use Block Public Access, versioning, and server-side encryption. The workflow refuses file:// or ambient local state in CI.
• The sandbox account is isolated from production and shared workloads. Use the constrained role policy in weekly-integration-iam-policy.json instead of attaching account-wide administrator access.
• The integration path does not create internet-facing compute, load balancers, public security groups, public repositories, or public S3 buckets. AccountFoundation exercises account-level controls only: CloudTrail, Config, GuardDuty, Security Hub, IAM password policy, KMS, and private log/state buckets.
• Logs must not print secrets, sandbox account IDs, backend bucket URLs, or full state exports. Failure artifacts are limited to Pulumi working metadata and should be deleted once the failure is understood.

Failed-run cleanup

If a real-AWS run fails during teardown, use the e2e-cleanup workflow instead of ad hoc console deletion. It is destructive (drains S3 + pulumi destroy + removeStack) so it is gated behind the protected e2e-cleanup GitHub Environment (required reviewers) and only runs on refs/heads/main; it also verifies the state bucket is owned by the sandbox account before destroying anything. Pass the 10-character suffix from the failed stack name (sandbox-<suffix>). The cleanup script selects that Pulumi stack from the private backend, drains only S3 buckets whose physical name starts with af-e2e-<suffix>-, then runs pulumi destroy and removeStack.

The cleanup path is intentionally Pulumi-state driven. @hulumi/drift is useful for classifying drift, but it is not a deletion engine and its real-AWS cleanup scenarios are still roadmap work; the cleanup workflow builds the drift package only as a dependency check and keeps the actual destructive action scoped to Pulumi-owned e2e state.

The first reconciler-backed S3 proof lives in packages/drift/tests/integration/reconciler-s3.integration.test.ts. It is double-gated by HULUMI_INTEGRATION=1 and HULUMI_RECONCILER_AWS_INTEGRATION=1; without both flags it only emits a visible skip notice. When enabled in the sandbox account, it creates one scoped versioned S3 bucket, proves plan mode is non-mutating, executes the S3 sweeper, and verifies no in-scope bucket remains. The test intentionally uses the AWS SDK directly for this first proof; the workflow and Pulumi-stack fixture follow in #97 / later integration work.

The first non-S3 reconciler proof lives in packages/drift/tests/integration/reconciler-cloudwatch-log-group.integration.test.ts and uses the same double gate. When enabled in the sandbox account, it creates one scoped CloudWatch Logs log group, proves plan mode does not mutate it, proves weak ownership evidence blocks execution, reports shared singleton resources as retained without mutation, injects a pre-delete executor failure to prove the result is resumable, retries with the real CloudWatch Logs executor, and verifies that no log group with the generated test prefix remains. afterAll also lists the generated prefix and deletes leftovers, so success and failure paths share the same cleanup contract.

The maintainer workflow drift-reconciler-cleanup keeps plan and execute permissions separate. Plan mode assumes AWS_RECONCILER_PLAN_ROLE_ARN, writes only a redacted plan-intent artifact, and does not set the live execute flag. Execute mode assumes AWS_RECONCILER_EXECUTE_ROLE_ARN, requires the protected aws-reconciler-execute GitHub environment, and runs only the gated S3 proof. Use separate IAM policies: reconciler-plan-iam-policy.json for read-only planning and reconciler-s3-execute-iam-policy.json for the narrow S3 execute proof.

The weekly integration workflow can run the S3 and CloudWatch Logs reconciler proofs after assuming the sandbox role through GitHub OIDC, but only when maintainers explicitly set the repository variable HULUMI_RECONCILER_AWS_INTEGRATION=1. PR CI never sets that flag.

The guarded state-transition model for future broad execute-mode work lives in HulumiReconciler.tla with the checked invariant summary in HulumiReconciler-verified.md. Any broader execute-mode feature must update or link to that model before it is enabled.

The first drift-classifier real-AWS proof lives in packages/drift/tests/integration/drift-classify.integration.test.ts. It keeps the repo-wide HULUMI_INTEGRATION=1 gate and also requires a configured Pulumi backend (PULUMI_BACKEND_URL or PULUMI_ACCESS_TOKEN). When enabled in the sandbox account, it creates one short-lived Pulumi-managed S3 bucket, mutates its tags through the AWS SDK, waits for the CloudTrail PutBucketTagging event, asserts DriftClassifier.classify() returns ConsoleBreakGlass / high, then calls classify() again to prove the cache prevents a second adapter poll. afterAll always runs pulumi destroy, removes the stack, and deletes the local Pulumi work directory.

The EksClusterFoundation EKS contract lives in packages/k8s-baseline/tests/integration/eks/eks-cluster-foundation.eks.test.ts. It is gated by HULUMI_INTEGRATION_EKS=1 and HULUMI_EKS_SANDBOX_CLUSTER. Without both it records a visible skip. When enabled, the intended live checks are endpoint mode, audit log types, Pod Identity/add-on metadata, and managed node launch-template IMDSv2 posture. The test must not persist kubeconfigs, service-account tokens, or cloud credentials.

Cost contract

Resource	Per-run cost	Notes
CloudTrail management events	$0	First trail per region is free.
AWS Config	~$0.01–0.05	Recorder + items enumerated, torn down each run.
GuardDuty (basic)	<$0.10	Enable + scan + disable in the test window.
GuardDuty (extended features × 5)	<$0.20	Each feature billed per scan.
Security Hub + 2 standards	<$0.05	Per-check pricing; 2 standards = double cost.
KMS CMK ring (4 keys)	$0 if torn down each run	$1/month per key if orphaned.
S3 (log bucket)	<$0.01	Object count negligible.
S3 (Startup-Hardened log target)	<$0.01	Short-lived server-access-log target bucket.
S3 (drift-classify fixture bucket)	<$0.01	One tagged bucket, destroyed after the test.
CloudWatch Logs (Startup-Hardened)	<$0.01	365-day retention but tiny ingest.

Expected weekly cost with clean teardown: under $1/run, typically $0.20–$0.50. First-ever run might spike to $3–$5 as Config does its initial full-account enumeration.

If teardown fails: GuardDuty + KMS + CloudWatch Logs can drift to $30–60/month cumulative. The $20/month sandbox-account budget alarm is the safety net (see deployment/sandbox-account.md).

Local run

# Mocks-only — no AWS credentials needed.
pnpm --filter @hulumi/baseline test

# Real-AWS integration — opt-in. Requires:
#   - HULUMI_INTEGRATION=1 to flip the integration gate
#   - a self-managed S3 Pulumi backend OR Pulumi Cloud token
#   - AWS credentials (SSO into the sandbox account or aws-vault)
HULUMI_INTEGRATION=1 \
PULUMI_BACKEND_URL='s3://hulumi-pulumi-state-<sandbox-account-id>?region=us-east-1' \
AWS_REGION=us-east-1 \
pnpm --filter @hulumi/baseline test -- tests/integration/

Current status: the weekly workflow is wired for a real backend and the AccountFoundation sandbox and Startup-Hardened lanes have real Pulumi Automation API up/assert/destroy tests. Both lanes assert the returned CloudTrail, Config recorder, GuardDuty, Security Hub, and KMS outputs through AWS API calls before teardown. The Startup-Hardened lane also creates a scoped S3 server-access-log target bucket for the internal log bucket and deletes it during cleanup. The drift-classifier lane has one real-AWS S3 console-drift proof with cache-hit verification. Failure-injection cleanup tests and the remaining drift real-AWS scenarios remain explicit it.todo() / skipped roadmap work tracked in integration-testing-roadmap.md. That is intentional: the project must not pretend that a success-path pass is full e2e coverage.

Eventual-consistency contract

AWS service enablement is asynchronous. AccountFoundation orders its sub-resources via Pulumi dependsOn:

• aws.securityhub.Account depends on aws.guardduty.Detector + every aws.guardduty.DetectorFeature.
• aws.securityhub.StandardsSubscription depends on the Hub.
• aws.cfg.DeliveryChannel depends on the Recorder.

The original M3 design used a pulumi.dynamic.Resource polling probe that waited for aws.guardduty.getDetector().status === "ENABLED" up to 10 minutes. That approach conflicts with vitest's worker pool — Pulumi's closure-serialization step requires Node's trace_events module which isn't available in the test workers. Direct dependsOn provides equivalent ordering for the real-AWS path because AWS's CreateDetector call resolves only after the detector is ENABLED.

The escape hatch (packages/baseline/src/aws/probes/poll.ts) is kept for v1.1+ probe additions where a separate dependsOn isn't sufficient. The no-sleep-in-source AST test asserts every use of setTimeout / sleep / await new Promise lives inside probes/.

What an "integration green" looks like

For the currently implemented AccountFoundation lanes, a green weekly integration run means:

• pulumi up for AccountFoundation(tier: "sandbox") and AccountFoundation(tier: "startup-hardened") completes via Pulumi Automation API using OIDC and the configured backend.
• The stack returns real provider outputs for CloudTrail, Config, GuardDuty, Security Hub, and the four KMS keys, and each output is checked through the matching AWS API.
• pulumi destroy and removeStack run in afterAll, and the local Pulumi work directory is removed.
• The four test-created KMS keys are not left in Enabled state after destroy. AWS's normal pending-deletion state counts as cleaned up.
• A manual dispatch with tier=sandbox only runs the sandbox matrix lane. A manual dispatch with tier=startup-hardened only runs the Startup-Hardened lane; tier=both runs both serially.
• If HULUMI_DRIFT_INTEGRATION=1 is enabled for the workflow, the drift classifier creates one S3 fixture bucket, observes an out-of-band tag mutation through CloudTrail, returns ConsoleBreakGlass / high, proves the second classification is served from cache, and destroys the fixture stack.

The remaining AccountFoundation roadmap target is the deliberate failure-injection cleanup test. Broader drift real-AWS scenarios remain explicitly gated until their account-wide assertions are implemented.

Edge platform integration lanes

Hulumi edge-platform integration is opt-in and split by provider surface. Without the listed env vars, each package test:integration command emits a skipped test whose title names the missing variables. The lanes are designed for a separate battle-test project first; this repo's release gate may remain Battle-test pending when mock/unit/policy evidence is green and the limitation is documented.

Cloudflare edge lane

Command:

pnpm --filter @hulumi/cloudflare-baseline test:integration

Required env vars:

Env var	Purpose
HULUMI_CLOUDFLARE_INTEGRATION=1	Explicit opt-in.
CLOUDFLARE_API_TOKEN	Sandbox token for Cloudflare provider operations.
HULUMI_CLOUDFLARE_ACCOUNT_ID	Account for tunnel and account-level edge fixtures.
HULUMI_CLOUDFLARE_ZONE_ID	Existing sandbox zone for DNSSEC, DNS record, WAF, and bot fixtures.

Required assertions for the real lane:

• ZoneFoundation DNSSEC output shape includes status/material outputs.
• PublicHostname defaults proxy-eligible public-app records to proxied mode.
• EdgeWafBaseline deploys managed rulesets only where the plan permits and records unsupported controls otherwise.
• Cleanup records the zone id, test hostname, ruleset ids, tunnel id, and exact manual deletion steps if teardown fails.

GitHub deployment lane

Command:

pnpm --filter @hulumi/platform-patterns test:integration

Required env vars:

Env var	Purpose
HULUMI_GITHUB_EDGE_INTEGRATION=1	Explicit GitHub opt-in.
HULUMI_GITHUB_SANDBOX_OWNER	Sandbox owner/org.
HULUMI_GITHUB_SANDBOX_REPOSITORY	Disposable or dedicated sandbox repo.

Required assertions for the real lane:

• DeploymentRepositoryFoundation creates protected environments with reviewer and branch-policy evidence visible through provider/API state.
• Reusable workflow allowlist and full-length SHA pinning are exercised by the workflow-governance linter fixture.

AWS origin lane

Command:

pnpm --filter @hulumi/platform-patterns test:integration

Required env vars:

Env var	Purpose
HULUMI_AWS_EDGE_INTEGRATION=1	Explicit AWS opt-in.
HULUMI_AWS_OIDC_PROVIDER_ARN	GitHub Actions OIDC provider ARN in the sandbox account.
HULUMI_AWS_DEPLOY_POLICY_ARN	Least-privilege deploy policy ARN attached to the test role.
HULUMI_AWS_TEST_REGION	Region for ALB/NLB security-group fixtures.

Required assertions for the real lane:

• GitHubAwsOidcDeploymentRole renders a narrow trust policy and usage block.
• CloudflareOriginIngress tunnel mode serializes hostname binding.
• Allowlist+AOP mode restricts load-balancer ingress to Cloudflare source CIDRs and target ingress to the load-balancer security group.

Scenario cookbooks:

• EKS service exposed through Cloudflare Tunnel
• ALB origin restricted to Cloudflare plus AOP
• GitHub OIDC deployment pipeline
• Origin IP rotation after Cloudflare onboarding
• Build provenance helper usage
• External battle-test checklist (docs/cookbooks/hulumi-edge-platform-battle-test.md)

Failure modes + diagnostics

Symptom	Likely cause	Fix
OIDC AssumeRoleWithWebIdentity denied	Trust-policy sub filter mismatch	Confirm repo:kerberosmansour/hulumi:ref:refs/heads/main and that the workflow ran on main
Security Hub subscription fails	GuardDuty Detector still in CREATING state	The dependsOn chain handles this — re-run; if it persists, AWS region anomaly
pulumi destroy leaves orphans	Race between Config recorder and DeliveryChannel teardown	Re-run pulumi destroy manually; M5 will add an orphan-resource sweeper
Integration job times out at 30 min	pulumi up exceeded 15-min eventual-consistency window	Check stack export artifact uploaded on failure; investigate which sub-resource didn't reach READY

Out of scope

• Multi-region runs: M3 ships single-region per matrix tier. Multi-region testing arrives with M5's SLSA release.
• AccountFoundation running across an AWS Organization: orgAccountIds arg wires the Config aggregator + KMS deny-without-tag policy, but Hulumi v1 does not deploy across multiple accounts in one pulumi up. M5's SCP template documents the Org-wide story.
• Drift detection: M4 ships @hulumi/drift; the weekly integration does not classify drift in M3.