Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,16 @@ Connect to your Aurora MySQL instance as an admin user and execute the following
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
```

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
```

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
10 changes: 10 additions & 0 deletions docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,16 @@ Connect to your RDS MySQL instance as an admin user and execute the following co
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
```

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
```

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
55 changes: 20 additions & 35 deletions docs/integrations/data-ingestion/clickpipes/postgres/auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,27 @@ import Image from '@theme/IdealImage';

This article demonstrates how ClickPipes customers can leverage role-based access to authenticate with Amazon Aurora and RDS and access their databases securely.

:::warning
For AWS RDS Postgres and Aurora Postgres you can only run `Initial Load Only` ClickPipes due to the limitations of the AWS IAM DB authentication.
## Setup {#setup}

For MySQL and MariaDB, this limitation doesn't apply, and you can run both `Initial Load Only` and `CDC` ClickPipes.
:::
### Before you start {#before-you-start}

## Setup {#setup}
#### Make sure IAM authentication is enabled {#make-sure-iam-authentication-is-enabled}

1. Login to your AWS Account and navigate to the RDS instance you want to configure.
2. Click on the **Modify** button.
3. Scroll down to the **Database authentication** section.
4. Enable the **Password and IAM database authentication** option.
5. Click on the **Continue** button.
6. Review the changes and click on the **Apply immediately** option.

#### Setting up the database user {#setting-up-the-database-user}

Connect to your RDS/Aurora instance and create a database user with IAM authentication enabled, then follow the rest of the steps in your source setup guide to configure your instance for ClickPipes:

- [AWS Aurora for PostgreSQL source setup guide](./source/aurora#configure-database-user)
- [AWS RDS for PostgreSQL source setup guide](./source/rds#configure-database-user)
- [AWS Aurora for MySQL source setup guide](../mysql/source/aurora#configure-database-user)
- [AWS RDS for MySQL source setup guide](../mysql/source/rds#configure-database-user)

### Obtaining the ClickHouse service IAM role Arn {#obtaining-the-clickhouse-service-iam-role-arn}

Expand All @@ -39,43 +53,14 @@ For MySQL and MariaDB, this limitation doesn't apply, and you can run both `Init

Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be used to access your RDS/Aurora instance.

### Configuring the RDS/Aurora instance {#configuring-the-rds-aurora-instance}

#### Enabling IAM DB Authentication {#enabling-iam-db-authentication}
1. Login to your AWS Account and navigate to the RDS instance you want to configure.
2. Click on the **Modify** button.
3. Scroll down to the **Database authentication** section.
4. Enable the **Password and IAM database authentication** option.
5. Click on the **Continue** button.
6. Review the changes and click on the **Apply immediately** option.
### Setting up the IAM role {#setting-up-iam-role}

#### Obtaining the RDS/Aurora Resource ID {#obtaining-the-rds-resource-id}

1. Login to your AWS Account and navigate to the RDS instance/Aurora Cluster you want to configure.
2. Click on the **Configuration** tab.
3. Note the **Resource ID** value. It should look like `db-xxxxxxxxxxxxxx` for RDS or `cluster-xxxxxxxxxxxxxx` for Aurora cluster. Let's call this value `{RDS_RESOURCE_ID}`. This is the resource ID that will be used in the IAM policy to allow access to the RDS instance.

#### Setting up the Database User {#setting-up-the-database-user}

##### PostgreSQL {#setting-up-the-database-user-postgres}

1. Connect to your RDS/Aurora instance and create a new database user with the following command:
```sql
CREATE USER clickpipes_iam_user;
GRANT rds_iam TO clickpipes_iam_user;
```
2. Follow the rest of the steps in the [PostgreSQL source setup guide](./source/rds) to configure your RDS instance for ClickPipes.

##### MySQL / MariaDB {#setting-up-the-database-user-mysql}

1. Connect to your RDS/Aurora instance and create a new database user with the following command:
```sql
CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
```
2. Follow the rest of the steps in the [MySQL source setup guide](../mysql/source/rds) to configure your RDS/Aurora instance for ClickPipes.

### Setting up the IAM role {#setting-up-iam-role}

#### Manually create IAM role. {#manually-create-iam-role}

1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,17 @@ Connect to your Aurora PostgreSQL writer instance as an admin user and execute t

The `clickpipes` publication defines the set of tables whose change events will be streamed to ClickPipes. We recommend against using `FOR ALL TABLES` unless you intend to replicate every table, as including unnecessary tables increases WAL traffic from Postgres to ClickPipes and reduces overall replication efficiency.

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. This requires the `rds.iam_auth_for_replication` parameter to be set to `1`. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER clickpipes_iam_user;
GRANT rds_iam TO clickpipes_iam_user;
```

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,21 @@ Connect to your RDS Postgres instance as an admin user and execute the following

The `clickpipes` publication defines the set of tables whose change events will be streamed to ClickPipes. We recommend against using `FOR ALL TABLES` unless you intend to replicate every table, as including unnecessary tables increases WAL traffic from Postgres to ClickPipes and reduces overall replication efficiency.

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER clickpipes_iam_user;
GRANT rds_iam TO clickpipes_iam_user;
```

:::note
IAM authentication for replication requires the `rds.iam_auth_for_replication` parameter to be set to `1`. This is supported from PostgreSQL version 11 onwards; on earlier versions, you can only run `Initial Load Only` ClickPipes.
:::

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
Loading