Harden AWS stack lifecycle protections (#175)

## What changed

- enforce a CloudFormation stack policy after every applied deploy or
upgrade so retained DynamoDB, S3, and inbound KMS resources cannot be
replaced or deleted by an ordinary update
- enable and verify CloudFormation termination protection after
deployment, including no-template-change reconciliation
- add `status aws --detect-drift` with bounded polling and metadata-only
drift output
- require `operational` status to have verified termination protection,
the retained-resource policy, and an `IN_SYNC` drift result
- make protected cleanup plan-able but require the exact stack name plus
`--disable-termination-protection`; restore protection when delete
submission fails
- update the dedicated AWS integration workflow and operator
documentation for the protected lifecycle

## Why

The AWS path is the active Certified candidate, but a successful
CloudFormation deployment alone did not protect the stack from
accidental deletion, did not prevent updates from replacing retained
customer-data resources, and only exposed the last reported drift state.
These controls are required before the path can be treated as a credible
Resend migration target.

This is the first implementation tranche of #174. It does not claim SES
production readiness or close the terminal-delivery and dogfood evidence
gates.

## User and operator impact

Applied deploys and upgrades now finish only after both stack
protections are verified. A fresh drift check is explicit because it
performs an AWS operation. Deletion remains possible, but the normal
protected stack needs a separate, visible acknowledgement before the CLI
disables termination protection.

## Validation

- `npm ci` — 0 vulnerabilities
- `npm outdated --json` — no outdated npm dependencies
- `npm run check`
- `npm test` — 607 passed, 6 skipped
- `npm run build`
- `npm run check:conformance`
- `npm run lint:openapi`
- `npm run check:workers` — 38 passed and Wrangler dry-run passed
- `npm run site:build && npm run site:verify`
- `actionlint .github/workflows/aws-integration.yml`
- `git diff --check`

## Safety

No AWS resources are changed by this PR itself. All AWS mutation remains
exact-account gated and explicit. Drift output excludes property
differences and values. Cleanup never purges retained customer data.

Author: yhay81

.github/workflows/aws-integration.yml

@@ -228,6 +228,16 @@ jobs:
             'require("node:fs").readFileSync(process.argv[1], "utf8").trim().split("\n").forEach(JSON.parse)' \
             "$RUNNER_TEMP/hayasend-retention-update.jsonl"
 
+          npm run --silent cli -- status aws \
+            --account "$AWS_TEST_ACCOUNT_ID" \
+            --stack "$STACK_NAME" \
+            --region "$AWS_REGION" \
+            --detect-drift \
+            > "$RUNNER_TEMP/hayasend-status.json"
+          node -e \
+            'const value=JSON.parse(require("node:fs").readFileSync(process.argv[1],"utf8")); if (value.stack.termination_protection !== true || value.stack.stack_policy.retained_resources_protected !== true || value.stack.drift.status !== "IN_SYNC" || value.stack.drift.detected_now !== true) process.exit(1)' \
+            "$RUNNER_TEMP/hayasend-status.json"
+
           updated_retention_resource="$(
             aws cloudformation list-stack-resources \
               --stack-name "$STACK_NAME" \
@@ -462,10 +472,17 @@ jobs:
           fi
 
           if [[ "$stack_exists" = true ]]; then
-            sam delete \
-              --stack-name "$STACK_NAME" \
+            npm run --silent cli -- cleanup aws \
+              --account "$AWS_TEST_ACCOUNT_ID" \
+              --stack "$STACK_NAME" \
               --region "$AWS_REGION" \
-              --no-prompts || {
+              --apply \
+              --confirm-stack "$STACK_NAME" \
+              --disable-termination-protection || {
+                aws cloudformation update-termination-protection \
+                  --disable-termination-protection \
+                  --stack-name "$STACK_NAME" \
+                  --region "$AWS_REGION" >/dev/null 2>&1 || true
                 aws cloudformation delete-stack \
                   --stack-name "$STACK_NAME" \
                   --region "$AWS_REGION"

README.md

@@ -420,27 +420,31 @@ npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws \
 Apply creates but does not immediately execute a CloudFormation change set.
 HayaSend retrieves the exact new change-set ARN, prints its resource changes,
 and refuses removals, indeterminate actions, or possible replacements unless
-`--allow-destructive-changes` is also present. It never changes DNS. See the
+`--allow-destructive-changes` is also present. It then verifies a
+retained-resource stack policy and termination protection. It never changes
+DNS. See the
 [CLI guide](docs/cli.md#plan-and-deploy-to-aws) for inbound options, parameter
 preservation, finite Lambda log retention, failure recovery, and output
 privacy.
 
 The same CLI covers the complete stack lifecycle:
 
 ```bash
-npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
+npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws --detect-drift
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws --apply
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws \
-  --apply --confirm-stack hayasend
+  --apply --confirm-stack hayasend --disable-termination-protection
 ```
 
-`status aws` combines CloudFormation state and drift, SES sending readiness,
-stack-resource failures, CloudWatch alarms, public API health, and the
-dashboard link. `cleanup aws` is plan-first and deliberately retains the
-DynamoDB table, payload bucket, and enabled inbound data resources; it prints
-their physical IDs for a separate retention or destruction decision. See the
+`status aws --detect-drift` runs a fresh bounded CloudFormation drift check and
+combines its metadata-only result with protection state, SES sending
+readiness, stack-resource failures, CloudWatch alarms, public API health, and
+the dashboard link. `cleanup aws` is plan-first, requires a separate explicit
+protection-disable acknowledgement, and deliberately retains the DynamoDB
+table, payload bucket, and enabled inbound data resources; it prints their
+physical IDs for a separate retention or destruction decision. See the
 copy-paste [AWS quickstart](docs/aws-quickstart.md) and the
 [operations runbook](docs/operations.md).
 

docs/aws-integration-testing.md

@@ -10,7 +10,9 @@ non-mutating plan mode. It then repeats the exact account, Region, stack, log
 retention, and tags with explicit `--apply`. This proves account pinning, SES
 and stack preflight, clean SAM validation/build, creation of an unexecuted
 change set, change-set inspection, and execution by the retrieved change-set
-ARN.
+ARN. It also verifies that the CLI enabled termination protection, installed
+the retained-resource stack policy, and completed a fresh `IN_SYNC` drift
+check.
 
 After the clean install, the workflow creates empty groups at the four legacy
 Lambda-generated names with no retention policy, changes `LogRetentionDays`
@@ -39,8 +41,9 @@ AWS Budget and root-account alerts before the first run. The workflow:
 - sends no email to SES;
 - cancels every test schedule and deletes its temporary SES identity;
 - deletes the synthetic legacy and stack-owned CloudWatch log groups;
-- deletes the stack, then explicitly deletes the S3 bucket and DynamoDB table
-  retained by HayaSend's production-safe deletion policies.
+- explicitly acknowledges termination-protection disable, deletes the stack
+  through `cleanup aws`, then deletes the S3 bucket and DynamoDB table retained
+  by HayaSend's production-safe deletion policies.
 
 Cleanup allows up to 60 seconds for CloudWatch's post-stack deletion view to
 converge. A stack-owned group still visible after that bound is reported,
@@ -54,11 +57,11 @@ debugging, and delete the retained resources immediately afterward.
 Create an environment named `aws-integration`. Restrict deployments to the
 default branch and add a required reviewer. Set these environment variables:
 
-| Variable | Example | Purpose |
-|---|---|---|
-| `AWS_TEST_ACCOUNT_ID` | `123456789012` | hard account allowlist |
-| `AWS_TEST_ROLE_ARN` | `arn:aws:iam::123456789012:role/HayaSendGitHubIntegration` | OIDC role |
-| `AWS_TEST_REGION` | `ap-northeast-1` | isolated test Region |
+| Variable              | Example                                                    | Purpose                |
+| --------------------- | ---------------------------------------------------------- | ---------------------- |
+| `AWS_TEST_ACCOUNT_ID` | `123456789012`                                             | hard account allowlist |
+| `AWS_TEST_ROLE_ARN`   | `arn:aws:iam::123456789012:role/HayaSendGitHubIntegration` | OIDC role              |
+| `AWS_TEST_REGION`     | `ap-northeast-1`                                           | isolated test Region   |
 
 No AWS access-key secret is required.
 

docs/aws-quickstart.md

@@ -43,19 +43,29 @@ npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws --apply
 Apply builds the packaged SAM application, creates an unexecuted
 CloudFormation change set, prints every resource action, and executes it only
 when no unacknowledged removal or possible replacement exists. HayaSend never
-changes DNS.
+changes DNS. After a successful create or update, the CLI enforces a stack
+policy that denies replacement or deletion of retained data resources and
+enables CloudFormation termination protection. Apply fails loudly if either
+protection cannot be verified.
 
 ## 3. Check whether it is ready
 
 ```bash
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
+npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws --detect-drift
 ```
 
+The first command is a read-only snapshot of the last reported drift state.
+The second explicitly starts a new CloudFormation drift check and waits up to
+10 minutes for it to finish. Fresh drift output contains logical IDs, resource
+types, and statuses only; it does not print property values.
+
 The result keeps two decisions separate:
 
-- `operational` requires a stable stack, no drift reported, no problematic
-  stack resources, all discovered HayaSend alarms in `OK`, and a successful
-  public `/healthz` request;
+- `operational` requires a stable stack, verified termination protection, the
+  HayaSend retained-resource stack policy, an `IN_SYNC` drift result, no
+  problematic stack resources, all discovered HayaSend alarms in `OK`, and a
+  successful public `/healthz` request;
 - `send_ready` additionally requires SES production access and account
   sending to be enabled.
 
@@ -105,16 +115,20 @@ First print the deletion plan:
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws
 ```
 
-Cleanup refuses an unmanaged stack, a non-terminal stack, or a stack with
-termination protection enabled. To delete the stack, repeat the exact name:
+Cleanup refuses an unmanaged or non-terminal stack. A protected stack remains
+plan-able, but deletion requires both its exact name and a separate
+acknowledgement that termination protection will be disabled:
 
 ```bash
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws \
   --apply \
-  --confirm-stack hayasend
+  --confirm-stack hayasend \
+  --disable-termination-protection
 ```
 
-The CLI waits for `stack-delete-complete` and verifies that the stack no longer
+The CLI verifies the protection change before submitting deletion. If the
+delete request itself fails, it attempts to re-enable termination protection.
+It then waits for `stack-delete-complete` and verifies that the stack no longer
 exists. It does not purge retained customer data. The DynamoDB table, payload
 bucket, and enabled inbound bucket and KMS key have `DeletionPolicy: Retain`;
 their physical IDs are printed before and after deletion. Decide their
@@ -123,8 +137,9 @@ backup, audit, and privacy policy.
 
 ## Routine operating loop
 
-Run `status aws` after deployment, after updates, after AWS incidents, and
-before production canaries. Subscribe a real on-call destination to the
-`AlarmTopicArn`, confirm that subscription, configure an AWS Budget, and use
-the generated dashboard as the first operational view. The complete response
-procedures are in the [operations runbook](operations.md).
+Run `status aws --detect-drift` after deployment, after updates, after AWS
+incidents, and before production canaries. Subscribe a real on-call
+destination to the `AlarmTopicArn`, confirm that subscription, configure an
+AWS Budget, and use the generated dashboard as the first operational view.
+The complete response procedures are in the
+[operations runbook](operations.md).

docs/cli.md

@@ -678,21 +678,32 @@ secret and never creates DNS records. If CloudFormation fails, the error
 includes redacted recent stack events for recovery. Follow the
 [operations runbook](operations.md) before retrying.
 
+A successful apply also installs a CloudFormation stack policy that rejects
+replacement or deletion of the retained DynamoDB, S3, and inbound KMS
+resources and enables termination protection. The result is not successful
+unless both controls are read back and verified. This protection is also
+reconciled when an apply has no template changes.
+
 ## Inspect AWS status
 
 Use the same expected-account gate for an infrastructure and sending-readiness
 snapshot:
 
 ```bash
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
-```
-
-This is read-only and does not require SAM or build the application. It checks
-the AWS CLI and caller identity, SES production/sending state and quota,
-CloudFormation stack state and last reported drift, individual stack
-resources, stack-owned CloudWatch alarms, and the public `/healthz` endpoint.
-Only problematic resources and alarms are expanded in the result. The output
-also links the generated dashboard and prints exact `upgrade aws`, `cleanup
+npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws --detect-drift
+```
+
+Without `--detect-drift`, this is read-only and does not require SAM or build
+the application. It checks the AWS CLI and caller identity, SES
+production/sending state and quota, CloudFormation stack state, termination
+protection, the retained-resource stack policy, last reported drift,
+individual stack resources, stack-owned CloudWatch alarms, and the public
+`/healthz` endpoint. `--detect-drift` explicitly starts and awaits a fresh
+CloudFormation drift check. It reports only drifted logical IDs, resource
+types, and statuses, never property differences or property values. Only
+problematic resources and alarms are expanded in the result. The output also
+links the generated dashboard and prints exact drift, `upgrade aws`, `cleanup
 aws`, and authenticated `doctor` next steps.
 
 `operational` covers infrastructure, alarms, and public health. `send_ready`
@@ -722,14 +733,16 @@ Cleanup is also plan-first:
 ```bash
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws
 npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws \
-  --apply --confirm-stack hayasend
+  --apply --confirm-stack hayasend --disable-termination-protection
 ```
 
 The CLI refuses stacks without both `Project=HayaSend` and
-`ManagedBy=HayaSendCLI`, non-terminal stacks, and stacks with termination
-protection enabled. Apply requires the exact stack name, starts ordinary
-CloudFormation deletion, waits for `stack-delete-complete`, and verifies that
-the stack is absent.
+`ManagedBy=HayaSendCLI` and non-terminal stacks. Apply requires the exact
+stack name. A protected stack additionally requires
+`--disable-termination-protection`; the CLI disables and verifies that control
+immediately before ordinary CloudFormation deletion. If submission of the
+delete request fails, it attempts to restore the protection. It waits for
+`stack-delete-complete` and verifies that the stack is absent.
 
 Cleanup never purges resources protected by `DeletionPolicy: Retain`. Its
 plan and result identify the retained DynamoDB table, payload bucket, and any