Skip to content

Commit 01d5f36

Browse files
authored
Simplify AWS deployment and operations (#173)
## What changed - add a complete AWS lifecycle: `deploy aws`, `status aws`, `upgrade aws`, and guarded `cleanup aws` - allow one-time non-secret `HAYASEND_AWS_ACCOUNT_ID` / `AWS_REGION` setup while retaining a live STS account gate on every command - make `status aws` combine SES production/sending readiness, CloudFormation state and drift, problematic resources, stack-owned CloudWatch alarms, public HayaSend identity/health, dashboard link, and exact next commands - make updates reuse the existing isolated SAM build and exact change-set inspection, while requiring an existing stable stack - make cleanup plan-first, require both `--apply` and exact `--confirm-stack`, refuse unmanaged/protected/unstable stacks, wait for verified deletion, and preserve/print retained customer-data resources - add a copy-paste AWS quickstart and update the CLI and operations runbooks ## Why The accepted two-speed direction in #81 and #132 makes AWS native + SES the sole active Certified candidate. The AWS path therefore needs to be the easiest path to create, understand, monitor, update, and remove safely before the terminal-delivery proof in #126 and AWS dogfood in #105. A live read-only run against the designated test account correctly reported that no `hayasend` stack exists and SES is healthy but sandbox-only, so it did not claim operational or send readiness. ## Validation - `npm ci` — 0 vulnerabilities - `npm outdated --json` — no outdated npm dependencies - `npm run check` - `npm run check:workers` — 38 tests passed and Wrangler dry-run passed - `npm test` — 603 tests passed, 6 skipped - `npm run build` - `npm run check:conformance` - `npm run lint:openapi` - `npm run site:build && npm run site:verify` - `git diff --check` - AWS CLI 2.36.10 and SAM CLI 1.164.0 ## Safety This does not deploy or delete AWS resources. Real mutation remains plan-first, exact-account gated, change-set inspected, and protected by explicit acknowledgements. Cleanup never purges `DeletionPolicy: Retain` data resources. Closes no evidence gate by itself. #126 must still pass from an exact reviewed main commit before AWS is production-qualified.
1 parent b4e3557 commit 01d5f36

10 files changed

Lines changed: 1663 additions & 106 deletions

File tree

.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -717,7 +717,7 @@ jobs:
717717
jq --exit-status '
718718
.entryCount <= 615 and
719719
.size < 760000 and
720-
.unpackedSize < 3900000 and
720+
.unpackedSize < 4000000 and
721721
([.files[].path] | index("dist/cli.js") != null) and
722722
([.files[].path] | index("NOTICE") != null) and
723723
([.files[].path] |

README.md

Lines changed: 31 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -389,16 +389,21 @@ Requirements:
389389
- an SES-enabled AWS Region
390390
- SES production access before sending to unverified recipients
391391

392-
Start with the non-mutating deployment plan. The explicit account ID prevents
393-
an authenticated shell from silently targeting the wrong account:
392+
Authenticate once, then keep the expected account and Region as non-secret
393+
shell configuration. The CLI checks the live STS identity on every lifecycle
394+
command, so a stale or wrong SSO session cannot silently target another
395+
account:
394396

395397
```bash
396-
aws_account_id="$(aws sts get-caller-identity --query Account --output text)"
397398
HAYASEND_VERSION=X.Y.Z
398-
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws \
399-
--account "$aws_account_id" \
400-
--region ap-northeast-1 \
401-
--stack hayasend
399+
export AWS_PROFILE=your-sso-profile
400+
export AWS_REGION=ap-northeast-1
401+
aws sso login --profile "$AWS_PROFILE"
402+
export HAYASEND_AWS_ACCOUNT_ID="$(
403+
aws sts get-caller-identity --query Account --output text
404+
)"
405+
406+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws
402407
```
403408

404409
The plan validates the tools and template, performs a clean temporary SAM
@@ -408,9 +413,6 @@ AWS. After reviewing it, repeat the command with `--apply`:
408413

409414
```bash
410415
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws \
411-
--account "$aws_account_id" \
412-
--region ap-northeast-1 \
413-
--stack hayasend \
414416
--log-retention-days 30 \
415417
--apply
416418
```
@@ -423,6 +425,25 @@ and refuses removals, indeterminate actions, or possible replacements unless
423425
preservation, finite Lambda log retention, failure recovery, and output
424426
privacy.
425427

428+
The same CLI covers the complete stack lifecycle:
429+
430+
```bash
431+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
432+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws
433+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws --apply
434+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws
435+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws \
436+
--apply --confirm-stack hayasend
437+
```
438+
439+
`status aws` combines CloudFormation state and drift, SES sending readiness,
440+
stack-resource failures, CloudWatch alarms, public API health, and the
441+
dashboard link. `cleanup aws` is plan-first and deliberately retains the
442+
DynamoDB table, payload bucket, and enabled inbound data resources; it prints
443+
their physical IDs for a separate retention or destruction decision. See the
444+
copy-paste [AWS quickstart](docs/aws-quickstart.md) and the
445+
[operations runbook](docs/operations.md).
446+
426447
Before choosing a Region or comparing hosted alternatives, review the
427448
[reproducible AWS cost model](docs/aws-costs.md). It separates SES charges
428449
from HayaSend infrastructure, shows list price and recurring free allowances

docs/aws-quickstart.md

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# AWS quickstart
2+
3+
This is the shortest supported path for creating, checking, updating, and
4+
removing a HayaSend AWS deployment. Use an exact released HayaSend version in
5+
every infrastructure command.
6+
7+
## 1. Sign in and set the target once
8+
9+
Install Node.js 24 or newer, npm 12 or newer, the current AWS CLI v2, and the
10+
current AWS SAM CLI. Sign in with your normal AWS SSO profile:
11+
12+
```bash
13+
export AWS_PROFILE=your-sso-profile
14+
export AWS_REGION=ap-northeast-1
15+
aws sso login --profile "$AWS_PROFILE"
16+
17+
export HAYASEND_AWS_ACCOUNT_ID="$(
18+
aws sts get-caller-identity --query Account --output text
19+
)"
20+
export HAYASEND_VERSION=X.Y.Z
21+
```
22+
23+
`HAYASEND_AWS_ACCOUNT_ID` and `AWS_REGION` are non-secret configuration. Every
24+
HayaSend AWS command calls STS and refuses to continue when the authenticated
25+
account differs. Pass `--account`, `--region`, `--stack`, or `--profile` only
26+
when overriding these defaults. The default stack name is `hayasend`.
27+
28+
## 2. Create the stack
29+
30+
Generate a read-only plan:
31+
32+
```bash
33+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws
34+
```
35+
36+
Review the account, Region, SES state, parameters, tags, and exact apply
37+
command in the JSON result. Then apply:
38+
39+
```bash
40+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws --apply
41+
```
42+
43+
Apply builds the packaged SAM application, creates an unexecuted
44+
CloudFormation change set, prints every resource action, and executes it only
45+
when no unacknowledged removal or possible replacement exists. HayaSend never
46+
changes DNS.
47+
48+
## 3. Check whether it is ready
49+
50+
```bash
51+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
52+
```
53+
54+
The result keeps two decisions separate:
55+
56+
- `operational` requires a stable stack, no drift reported, no problematic
57+
stack resources, all discovered HayaSend alarms in `OK`, and a successful
58+
public `/healthz` request;
59+
- `send_ready` additionally requires SES production access and account
60+
sending to be enabled.
61+
62+
The result includes SES quota, only problematic resource and alarm details,
63+
the CloudWatch dashboard URL, and exact update, cleanup, and deep-diagnostics
64+
commands. It does not read the bootstrap secret or API keys.
65+
66+
For authenticated application and queue diagnostics, set a scoped key locally
67+
and run:
68+
69+
```bash
70+
export HAYASEND_BASE_URL=https://your-api.example
71+
export HAYASEND_API_KEY=your-scoped-diagnostics-key
72+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" doctor
73+
```
74+
75+
Keep the key in an approved secret manager and out of command arguments,
76+
transcripts, and issue reports.
77+
78+
## 4. Update safely
79+
80+
Plan an update to the existing stack:
81+
82+
```bash
83+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws
84+
```
85+
86+
After reviewing the plan:
87+
88+
```bash
89+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws --apply
90+
```
91+
92+
`upgrade aws` refuses a missing or non-terminal stack and uses the same exact
93+
change-set inspection as initial deployment. If CloudFormation proposes any
94+
removal, indeterminate action, or possible replacement, it stops before
95+
execution. Use `--allow-destructive-changes` only after reviewing every
96+
printed destructive action.
97+
98+
Run `status aws` again after every update.
99+
100+
## 5. Remove the running stack
101+
102+
First print the deletion plan:
103+
104+
```bash
105+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws
106+
```
107+
108+
Cleanup refuses an unmanaged stack, a non-terminal stack, or a stack with
109+
termination protection enabled. To delete the stack, repeat the exact name:
110+
111+
```bash
112+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws \
113+
--apply \
114+
--confirm-stack hayasend
115+
```
116+
117+
The CLI waits for `stack-delete-complete` and verifies that the stack no longer
118+
exists. It does not purge retained customer data. The DynamoDB table, payload
119+
bucket, and enabled inbound bucket and KMS key have `DeletionPolicy: Retain`;
120+
their physical IDs are printed before and after deletion. Decide their
121+
retention, export, and eventual destruction separately under the applicable
122+
backup, audit, and privacy policy.
123+
124+
## Routine operating loop
125+
126+
Run `status aws` after deployment, after updates, after AWS incidents, and
127+
before production canaries. Subscribe a real on-call destination to the
128+
`AlarmTopicArn`, confirm that subscription, configure an AWS Budget, and use
129+
the generated dashboard as the first operational view. The complete response
130+
procedures are in the [operations runbook](operations.md).

docs/cli.md

Lines changed: 69 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -571,20 +571,21 @@ are in [Cloudflare cost evidence](cloudflare-costs.md).
571571

572572
## Plan and deploy to AWS
573573

574-
Run the exact released CLI version from any working directory. The first
575-
invocation is a non-mutating plan:
574+
The copy-paste path is in the [AWS quickstart](aws-quickstart.md). Set the
575+
expected account and Region once:
576576

577577
```bash
578-
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws \
579-
--account 123456789012 \
580-
--region ap-northeast-1 \
581-
--stack hayasend
578+
export HAYASEND_AWS_ACCOUNT_ID=123456789012
579+
export AWS_REGION=ap-northeast-1
580+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws
582581
```
583582

584-
`--account` is required even when the AWS CLI already has credentials. The CLI
585-
calls STS and stops before reading SES or CloudFormation if the authenticated
586-
account differs. `--region` can instead come from `AWS_REGION` or
587-
`AWS_DEFAULT_REGION`; `--profile` selects a named AWS profile.
583+
`--account` overrides `HAYASEND_AWS_ACCOUNT_ID`. One of them is always
584+
required even when the AWS CLI already has credentials. The CLI calls STS and
585+
stops before reading SES or CloudFormation if the authenticated account
586+
differs. `--region` can instead come from `AWS_REGION` or
587+
`AWS_DEFAULT_REGION`; `--profile` selects a named AWS profile, while
588+
`AWS_PROFILE` continues to work through the AWS CLI.
588589

589590
The plan:
590591

@@ -610,9 +611,6 @@ Apply only after reviewing the plan:
610611

611612
```bash
612613
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" deploy aws \
613-
--account 123456789012 \
614-
--region ap-northeast-1 \
615-
--stack hayasend \
616614
--tag Environment=production \
617615
--apply
618616
```
@@ -680,6 +678,64 @@ secret and never creates DNS records. If CloudFormation fails, the error
680678
includes redacted recent stack events for recovery. Follow the
681679
[operations runbook](operations.md) before retrying.
682680

681+
## Inspect AWS status
682+
683+
Use the same expected-account gate for an infrastructure and sending-readiness
684+
snapshot:
685+
686+
```bash
687+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
688+
```
689+
690+
This is read-only and does not require SAM or build the application. It checks
691+
the AWS CLI and caller identity, SES production/sending state and quota,
692+
CloudFormation stack state and last reported drift, individual stack
693+
resources, stack-owned CloudWatch alarms, and the public `/healthz` endpoint.
694+
Only problematic resources and alarms are expanded in the result. The output
695+
also links the generated dashboard and prints exact `upgrade aws`, `cleanup
696+
aws`, and authenticated `doctor` next steps.
697+
698+
`operational` covers infrastructure, alarms, and public health. `send_ready`
699+
also requires SES production access and sending to be enabled. A missing stack
700+
is a successful read-only inspection with both values false and an exact
701+
`deploy aws` plan command.
702+
703+
## Upgrade AWS
704+
705+
`upgrade aws` accepts the same parameters and tags as `deploy aws`, but it
706+
requires an existing stable stack:
707+
708+
```bash
709+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws
710+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws --apply
711+
```
712+
713+
Plan mode validates and builds the exact packaged version without changing
714+
AWS. Apply uses the same isolated SAM build, exact new change-set selection,
715+
resource-action output, and destructive-change refusal as deployment. Run
716+
`status aws` after CloudFormation reaches its terminal state.
717+
718+
## Clean up AWS
719+
720+
Cleanup is also plan-first:
721+
722+
```bash
723+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws
724+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws \
725+
--apply --confirm-stack hayasend
726+
```
727+
728+
The CLI refuses stacks without both `Project=HayaSend` and
729+
`ManagedBy=HayaSendCLI`, non-terminal stacks, and stacks with termination
730+
protection enabled. Apply requires the exact stack name, starts ordinary
731+
CloudFormation deletion, waits for `stack-delete-complete`, and verifies that
732+
the stack is absent.
733+
734+
Cleanup never purges resources protected by `DeletionPolicy: Retain`. Its
735+
plan and result identify the retained DynamoDB table, payload bucket, and any
736+
enabled inbound bucket and KMS key. Export, retain, or destroy those resources
737+
through a separate reviewed data-lifecycle procedure.
738+
683739
## Send a hosted template
684740

685741
The CLI supports the same `--template` and repeatable `--var KEY=VALUE` shape

docs/operations.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# Operations runbook
22

33
This runbook is the minimum operating procedure for an AWS deployment.
4+
Use the [AWS quickstart](aws-quickstart.md) for the shortest create, status,
5+
update, and removal path.
46
For disposable end-to-end validation, use the
57
[dedicated-account integration workflow](aws-integration-testing.md).
68
Budget the selected Region and retention policy with the
@@ -15,6 +17,10 @@ The plan-first CLI needs read access for:
1517
- `ses:GetAccount`;
1618
- `cloudformation:DescribeStacks`.
1719

20+
`status aws` additionally uses `cloudformation:ListStackResources` and
21+
`cloudwatch:DescribeAlarms`, then calls the public HayaSend `/healthz`
22+
endpoint. It does not read log events, message data, or secret values.
23+
1824
`--apply` additionally reads CloudFormation change sets and failure events
1925
through `cloudformation:ListChangeSets`,
2026
`cloudformation:DescribeChangeSet`, and
@@ -28,11 +34,53 @@ template, organizational permission boundaries, Region, and enabled inbound
2834
features, then review it through the same infrastructure process as other
2935
production roles.
3036

37+
`cleanup aws --apply` needs `cloudformation:DeleteStack` in addition to the
38+
read operations. CloudFormation still needs its normal service-role or caller
39+
permissions to delete stack-owned resources. Cleanup never empties or deletes
40+
resources protected by `DeletionPolicy: Retain`.
41+
3142
The CLI never reads the bootstrap secret value, edits DNS, or passes
3243
credentials as command-line arguments. Its JSON output contains account IDs,
3344
principal and resource ARNs, domain suffixes, and stack outputs, so retain it
3445
as operational metadata rather than posting it publicly.
3546

47+
## Routine lifecycle
48+
49+
Keep the expected account and Region in non-secret environment variables so
50+
every operation stays short while retaining the live STS safety gate:
51+
52+
```bash
53+
export HAYASEND_AWS_ACCOUNT_ID=123456789012
54+
export AWS_REGION=ap-northeast-1
55+
56+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
57+
```
58+
59+
Run `status aws` after every deploy or upgrade, at the start of an incident,
60+
and before a production canary. Do not treat `operational: true` as proof of
61+
mail delivery: require `send_ready: true`, then complete a controlled send and
62+
confirm the terminal recipient event and mailbox receipt.
63+
64+
For an update, review the plan before apply:
65+
66+
```bash
67+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws
68+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" upgrade aws --apply
69+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" status aws
70+
```
71+
72+
For decommissioning, first record a disposition for retained customer data:
73+
74+
```bash
75+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws
76+
npx --yes "@haya-inc/hayasend@${HAYASEND_VERSION}" cleanup aws \
77+
--apply --confirm-stack hayasend
78+
```
79+
80+
The cleanup result repeats retained resource physical IDs after verified stack
81+
deletion. Do not improvise a bulk purge; handle those resources under the
82+
applicable recovery, compliance, and privacy policy.
83+
3684
## After deployment
3785

3886
1. Subscribe the on-call destination to the `AlarmTopicArn` stack output and

0 commit comments

Comments
 (0)