Plot a migration to NeonDB

[ryanjjung]

The Kubernetes deployment of Stalwart is built against AWS hosted resources, including an RDS database. We may want to use NeonDB for this instead.

This issue is a stand-in for a substantial amount of work. Break this work out into other tickets as discovery and discussion happens.

• Determine if we can build Neon resources automatically.
  • There exists a NeonDB Kubernetes Operator, but it has some issues:
    • Its readme explicitly says it's not ready for production use.
    • It doesn't seem to support any of the PrivateLink configurations we need.
    • This may be because it says it is only for self-hosted NeonDB instances, which we probably don't want since we've been trying to offload these backend supports to hosting providers.
    • It's not official or endorsed by NeonDB.
  • We have previously tested NeonDB's Pulumi provider, and it has a lot of problems and is also not sufficient for our needs.
• Determine the best way to configure Neon for our needs.
  • Do we do one DB for tb-dev and one for tb-prod?
  • Do we one DB for tb-prod and branch it for tb-dev?
  • There may be PII concerns, and we may need to investigate the data sanitization features of NeonDB.
• Build the database clusters in the manner we determine through the above experimentation.
• Build the PrivateLink resources (the links, the security groups and rules, etc.) to connect this from our EKS cluster.
  • Security groups should be configured with the per-pod-security-group configuration in mind.
• If data migration is needed, migrate that from RDS to Neon.
• Point Stalwart at the Neon DB.