diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md index 0fb7b5839fe..03c82e7ef02 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md @@ -129,6 +129,16 @@ Connect to your Aurora MySQL instance as an admin user and execute the following GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%'; ``` +### Using IAM authentication (optional) {#iam-authentication} + +Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above: + +```sql +CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'; +``` + +See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way. + ## Configure network access {#configure-network-access} ### IP-based access control {#ip-based-access-control} diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md index 3136f1b18cf..f65ec99a976 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md @@ -133,6 +133,16 @@ Connect to your RDS MySQL instance as an admin user and execute the following co GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%'; ``` +### Using IAM authentication (optional) {#iam-authentication} + +Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above: + +```sql +CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'; +``` + +See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way. + ## Configure network access {#configure-network-access} ### IP-based access control {#ip-based-access-control} diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/auth.md b/docs/integrations/data-ingestion/clickpipes/postgres/auth.md index f9f3a9f7919..a9269a98876 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/auth.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/auth.md @@ -15,13 +15,27 @@ import Image from '@theme/IdealImage'; This article demonstrates how ClickPipes customers can leverage role-based access to authenticate with Amazon Aurora and RDS and access their databases securely. -:::warning -For AWS RDS Postgres and Aurora Postgres you can only run `Initial Load Only` ClickPipes due to the limitations of the AWS IAM DB authentication. +## Setup {#setup} -For MySQL and MariaDB, this limitation doesn't apply, and you can run both `Initial Load Only` and `CDC` ClickPipes. -::: +### Before you start {#before-you-start} -## Setup {#setup} +#### Make sure IAM authentication is enabled {#make-sure-iam-authentication-is-enabled} + +1. Login to your AWS Account and navigate to the RDS instance you want to configure. +2. Click on the **Modify** button. +3. Scroll down to the **Database authentication** section. +4. Enable the **Password and IAM database authentication** option. +5. Click on the **Continue** button. +6. Review the changes and click on the **Apply immediately** option. + +#### Setting up the database user {#setting-up-the-database-user} + +Connect to your RDS/Aurora instance and create a database user with IAM authentication enabled, then follow the rest of the steps in your source setup guide to configure your instance for ClickPipes: + +- [AWS Aurora for PostgreSQL source setup guide](./source/aurora#configure-database-user) +- [AWS RDS for PostgreSQL source setup guide](./source/rds#configure-database-user) +- [AWS Aurora for MySQL source setup guide](../mysql/source/aurora#configure-database-user) +- [AWS RDS for MySQL source setup guide](../mysql/source/rds#configure-database-user) ### Obtaining the ClickHouse service IAM role Arn {#obtaining-the-clickhouse-service-iam-role-arn} @@ -39,15 +53,7 @@ For MySQL and MariaDB, this limitation doesn't apply, and you can run both `Init Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be used to access your RDS/Aurora instance. -### Configuring the RDS/Aurora instance {#configuring-the-rds-aurora-instance} - -#### Enabling IAM DB Authentication {#enabling-iam-db-authentication} -1. Login to your AWS Account and navigate to the RDS instance you want to configure. -2. Click on the **Modify** button. -3. Scroll down to the **Database authentication** section. -4. Enable the **Password and IAM database authentication** option. -5. Click on the **Continue** button. -6. Review the changes and click on the **Apply immediately** option. +### Setting up the IAM role {#setting-up-iam-role} #### Obtaining the RDS/Aurora Resource ID {#obtaining-the-rds-resource-id} @@ -55,27 +61,6 @@ Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be 2. Click on the **Configuration** tab. 3. Note the **Resource ID** value. It should look like `db-xxxxxxxxxxxxxx` for RDS or `cluster-xxxxxxxxxxxxxx` for Aurora cluster. Let's call this value `{RDS_RESOURCE_ID}`. This is the resource ID that will be used in the IAM policy to allow access to the RDS instance. -#### Setting up the Database User {#setting-up-the-database-user} - -##### PostgreSQL {#setting-up-the-database-user-postgres} - -1. Connect to your RDS/Aurora instance and create a new database user with the following command: - ```sql - CREATE USER clickpipes_iam_user; - GRANT rds_iam TO clickpipes_iam_user; - ``` -2. Follow the rest of the steps in the [PostgreSQL source setup guide](./source/rds) to configure your RDS instance for ClickPipes. - -##### MySQL / MariaDB {#setting-up-the-database-user-mysql} - -1. Connect to your RDS/Aurora instance and create a new database user with the following command: - ```sql - CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'; - ``` -2. Follow the rest of the steps in the [MySQL source setup guide](../mysql/source/rds) to configure your RDS/Aurora instance for ClickPipes. - -### Setting up the IAM role {#setting-up-iam-role} - #### Manually create IAM role. {#manually-create-iam-role} 1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md index d427532cea2..0ec34f87a30 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md @@ -98,6 +98,17 @@ Connect to your Aurora PostgreSQL writer instance as an admin user and execute t The `clickpipes` publication defines the set of tables whose change events will be streamed to ClickPipes. We recommend against using `FOR ALL TABLES` unless you intend to replicate every table, as including unnecessary tables increases WAL traffic from Postgres to ClickPipes and reduces overall replication efficiency. +### Using IAM authentication (optional) {#iam-authentication} + +Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. This requires the `rds.iam_auth_for_replication` parameter to be set to `1`. Create the user as follows, then grant it the same schema and replication privileges shown above: + +```sql +CREATE USER clickpipes_iam_user; +GRANT rds_iam TO clickpipes_iam_user; +``` + +See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way. + ## Configure network access {#configure-network-access} ### IP-based access control {#ip-based-access-control} diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md index 0aa2c124819..a438b88b991 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md @@ -98,6 +98,21 @@ Connect to your RDS Postgres instance as an admin user and execute the following The `clickpipes` publication defines the set of tables whose change events will be streamed to ClickPipes. We recommend against using `FOR ALL TABLES` unless you intend to replicate every table, as including unnecessary tables increases WAL traffic from Postgres to ClickPipes and reduces overall replication efficiency. +### Using IAM authentication (optional) {#iam-authentication} + +Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above: + +```sql +CREATE USER clickpipes_iam_user; +GRANT rds_iam TO clickpipes_iam_user; +``` + +:::note +IAM authentication for replication requires the `rds.iam_auth_for_replication` parameter to be set to `1`. This is supported from PostgreSQL version 11 onwards; on earlier versions, you can only run `Initial Load Only` ClickPipes. +::: + +See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way. + ## Configure network access {#configure-network-access} ### IP-based access control {#ip-based-access-control}