refactor: shared Neon (drop ACK-RDS); tb-prod tailnet+branch validation overlay

README.md

@@ -6,23 +6,30 @@ clusters via ArgoCD. Migration plan & target architecture:
 [`platform-infrastructure/docs/keycloak-customer-auth-migration.md`](https://github.com/thunderbird/platform-infrastructure/blob/main/docs/keycloak-customer-auth-migration.md).
 Epic: [platform-infrastructure#132](https://github.com/thunderbird/platform-infrastructure/issues/132).
 
+> **Database is shared Neon, not RDS.** Source ECS and the EKS target use the same
+> shared Neon Postgres DB, reached over PrivateLink. There is **no data migration** —
+> the prod cutover is a traffic flip. Validation runs against isolated Neon branches
+> (`mzla-tb-{dev,prod}`). The vestigial ACK-RDS machinery was removed
+> ([platform-infrastructure#579](https://github.com/thunderbird/platform-infrastructure/issues/579)).
+
 ## Layout
 
 ```
 bases/
-  aws/        ACK RDS: DBInstance, DBSubnetGroup, FieldExport
-  keycloak/   namespace, StatefulSet, services, PDB, RDS-endpoint ConfigMap,
+  keycloak/   namespace, StatefulSet, services (incl. metrics), PDB,
               db/admin ExternalSecrets, VMServiceScrape
 overlays/
-  tb-dev/     tailnet-only (Tailscale Ingress); single-AZ dev DB
-  tb-prod/    public Cloudflare tunnel (auth) + tailnet admin Ingress; Multi-AZ DB
+  tb-dev/     tailnet-only (Tailscale Ingress); Neon branch mzla-tb-dev; 2 replicas
+  tb-prod/    tailnet-only validation; Neon branch mzla-tb-prod; 2 replicas
+              (public Cloudflare tunnel + tailnet admin + 3 replicas are kept in the
+              overlay dir but unreferenced until the cutover, #142)
 ```
 
 The ArgoCD app-of-apps for each cluster lives in `platform-infrastructure`
 (`argocd/tb-{dev,prod}/apps/keycloak-customer.yaml`) and points at
-`overlays/<cluster>`. The cluster's Pulumi stack provides the `ack-rds` IRSA role
-and the Keycloak RDS security group; the ACK RDS controller is deployed from
-`platform-infrastructure`.
+`overlays/<cluster>`. The cluster's Pulumi stack (`mzla-tb-{dev,prod}`, Phase 4e)
+provides the Keycloak **Neon PrivateLink endpoint SG + pod SG + IRSA**; the pod SG
+is assigned to the pods via the overlay's `SecurityGroupPolicy`.
 
 ## Build / validate
 
@@ -34,30 +41,36 @@ kustomize build overlays/tb-dev        # or a single overlay
 ## Exposure model
 
 - **Admin is never public.** It is reached over **Tailscale**, like the Staff SSO
-  Keycloak. tb-dev is tailnet-only (whole service). tb-prod exposes only the
-  customer auth host publicly via Cloudflare tunnel; the admin console/REST is on
-  the tailnet (`KC_HOSTNAME_ADMIN`), with public `/admin` denied at the edge.
-- **Realm `tbpro` arrives via the database** (snapshot restore for dev / logical
-  replication for the prod cutover) — there is no `--import-realm`.
+  Keycloak. Both dev and prod are tailnet-only today (whole service). At the prod
+  cutover (#142) tb-prod adds a public Cloudflare tunnel for the auth host, with the
+  admin console/REST kept on the tailnet (`KC_HOSTNAME_ADMIN`) and public `/admin`
+  denied at the edge (Cloudflare Access).
+- **Realm `tbpro` lives in the shared Neon DB** (validated via isolated Neon
+  branches) — there is no `--import-realm` and no data migration.
+
+## DB connectivity (Neon over PrivateLink)
 
-## Placeholders to resolve before deploy
+The overlay's `keycloak/statefulset.yaml` sets `KC_DB_URL_HOST` to the env's Neon
+endpoint (the `mzla-tb-{dev,prod}` branch today) + `KC_DB_URL_PROPERTIES=?sslmode=require`.
+The pods reach Neon over the manually-created Neon interface endpoints, gated by the
+dedicated `mzla-tb-{dev,prod}-keycloak-neondb-privatelink` endpoint SG; the pods carry
+the dedicated pod SG via `keycloak/securitygrouppolicy.yaml`.
 
-Per overlay (`overlays/<cluster>/`), replace:
+## Placeholders / per-env values
 
-| Token | Source |
+| Token / value | Source |
 |-------|--------|
-| `REPLACE_MZLA_ECR/keycloak-customer` + `REPLACE_*_MIRRORED_SHA` (`kustomization.yaml` `images:`) | mirrored image ([platform-infrastructure#531](https://github.com/thunderbird/platform-infrastructure/issues/531)) |
-| `REPLACE_*_RDS_SG_KEYCLOAK_ID` (`aws/db-instance.yaml`) | `pulumi stack output rds_sg_keycloak_id` |
-| `REPLACE_*_PRIVATE_SUBNET_A/B` (`aws/db-subnet-group.yaml`) | `pulumi stack output private_subnet_ids` |
-| tb-dev `KC_HOSTNAME` tailnet host / tb-prod `auth.tb.pro` + Cloudflare zone | confirm tailnet name + that `tb.pro` is a managed Cloudflare zone |
+| `REPLACE_MZLA_ECR/keycloak-customer` (`kustomization.yaml` `images:`) | the per-account mzla ECR mirror (done, [platform-infrastructure#558](https://github.com/thunderbird/platform-infrastructure/pull/558)); each overlay sets `newName` + `newTag` |
+| `keycloak/securitygrouppolicy.yaml` `groupIds` | the cluster's eks-cluster-sg + `pulumi stack output keycloak_customer_pod_sg_id` (`mzla-tb-{dev,prod}`) |
+| `KC_DB_URL_HOST` (overlay `keycloak/statefulset.yaml`) | the env's Neon endpoint (branch today; live shared endpoint at cutover) |
 
 Secrets (AWS Secrets Manager, per account, eu-central-1) referenced by the
 ExternalSecrets: `mzla/<env>/keycloak-customer-db` and
 `mzla/<env>/keycloak-customer-admin` (`{username, password}`).
-`mzla/shared-services/cloudflare-operator` already exists (read cross-account).
+`mzla/shared-services/cloudflare-operator` already exists (read cross-account; used by the cutover Cloudflare resources).
 
-## Not yet included (follow-ups)
+## Not yet included (cutover follow-ups, #142)
 
-- Public `/admin` deny for tb-prod (Cloudflare Access) — gates the prod cutover.
-- DR backup CronJob (port from Staff SSO `argocd/keycloak/dr-backup-*`) before the
-  [#142](https://github.com/thunderbird/platform-infrastructure/issues/142) cutover.
+- Public Cloudflare tunnel (`auth.tb.pro`) + tailnet admin ingress + 3 replicas (files present, unreferenced).
+- Public `/admin` deny for tb-prod (Cloudflare Access).
+- Repoint `KC_DB_URL_HOST` from the validation branch to the live shared Neon endpoint.

bases/keycloak/kustomization.yaml

@@ -6,7 +6,6 @@ resources:
   - statefulset.yaml
   - service.yaml
   - pdb.yaml
-  - rds-endpoint-configmap.yaml
   - db-externalsecret.yaml
   - admin-externalsecret.yaml
   - vmservicescrape.yaml

bases/keycloak/statefulset.yaml

@@ -5,12 +5,14 @@
 # quay.io/keycloak. The `images:` transform in each overlay sets newName + newTag;
 # the string before ":" here is the match key.
 #
-# REALM: tbpro realm data arrives via the database (snapshot restore for dev /
-# logical replication for the prod cutover), so there is no --import-realm. The
-# custom JWT claim mappers, clients, and theme live in the DB + image.
+# REALM: tbpro realm data lives in the SHARED Neon database (source ECS and the
+# EKS target use the same DB), so there is no --import-realm and no data migration
+# -- the cutover is a traffic flip. Validation runs against isolated Neon branches.
+# The custom JWT claim mappers, clients, and theme live in the DB + image.
 #
 # HOSTNAME/EXPOSURE: KC_HOSTNAME is patched per overlay. Admin access is via
-# Tailscale (overlay-managed Ingress); prod additionally sets KC_HOSTNAME_ADMIN.
+# Tailscale (overlay-managed Ingress); dev + prod are tailnet-only today. At the
+# prod cutover (#142) tb-prod adds the public Cloudflare tunnel + KC_HOSTNAME_ADMIN.
 #
 # SCHEDULING: the general arm64 node group (no dedicated keycloak nodes). The
 # custom image must be multi-arch or arm64.
@@ -76,16 +78,12 @@ spec:
             # Database
             - name: KC_DB
               value: postgres
+            # Shared Neon over PrivateLink. DB host is set per-overlay (each env's
+            # Neon endpoint); the overlay also adds KC_DB_URL_PROPERTIES=?sslmode=require.
             - name: KC_DB_URL_HOST
-              valueFrom:
-                configMapKeyRef:
-                  name: keycloak-customer-rds-endpoint
-                  key: host
+              value: REPLACE_OVERLAY_KC_DB_HOST
             - name: KC_DB_URL_PORT
-              valueFrom:
-                configMapKeyRef:
-                  name: keycloak-customer-rds-endpoint
-                  key: port
+              value: "5432"
             - name: KC_DB_URL_DATABASE
               value: keycloak
             - name: KC_DB_USERNAME

bases/kustomization.yaml

@@ -2,7 +2,6 @@
 namespace: keycloak-customer
 
 resources:
-  - aws
   - keycloak
 
 labels:

overlays/tb-dev/kustomization.yaml

@@ -15,10 +15,7 @@ images:
     newTag: keycloak-e7ef91efd43f7619e7fc522fb64737b56efb9ffc
 
 patches:
-  # AWS / RDS env-specific values (SG id, subnet ids, identifier)
-  - path: aws/db-instance.yaml
-  - path: aws/db-subnet-group.yaml
-  # Keycloak workload: tailnet hostname (dev is tailnet-only)
+  # Keycloak workload: tailnet hostname + Neon branch DB host (dev is tailnet-only)
   - path: keycloak/statefulset.yaml
   # ExternalSecret remoteRef keys -> mzla/tb-dev/*
   - target:

overlays/tb-prod/keycloak/securitygrouppolicy.yaml

@@ -0,0 +1,21 @@
+# Assigns the dedicated pod security group to the keycloak-customer pods so they can
+# reach Neon over the dedicated mzla-tb-prod-keycloak-neondb-privatelink endpoint SG.
+# Under POD_SECURITY_GROUP_ENFORCING_MODE=strict the listed SGs REPLACE the node SG,
+# so both are required (and they match the live keycloak pod ENIs):
+#   - sg-08711bcc50c7b1e18 : eks-cluster-sg-mzla-eks-tb-prod01 (cluster comms /
+#       kubelet probes :9000 / CoreDNS)
+#   - sg-08319d6a808f5ad35 : mzla-tb-prod-keycloak-customer-pod (Neon PrivateLink
+#       client; from the mzla-tb-prod pulumi output keycloak_customer_pod_sg_id)
+apiVersion: vpcresources.k8s.aws/v1beta1
+kind: SecurityGroupPolicy
+metadata:
+  name: keycloak-customer
+  namespace: keycloak-customer
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak-customer
+  securityGroups:
+    groupIds:
+      - sg-08711bcc50c7b1e18
+      - sg-08319d6a808f5ad35