Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -129,16 +129,6 @@ Connect to your Aurora MySQL instance as an admin user and execute the following
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
```

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
```

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
10 changes: 0 additions & 10 deletions docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,16 +133,6 @@ Connect to your RDS MySQL instance as an admin user and execute the following co
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
```

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
```

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
55 changes: 35 additions & 20 deletions docs/integrations/data-ingestion/clickpipes/postgres/auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,27 +15,13 @@

This article demonstrates how ClickPipes customers can leverage role-based access to authenticate with Amazon Aurora and RDS and access their databases securely.

## Setup {#setup}

### Before you start {#before-you-start}

#### Make sure IAM authentication is enabled {#make-sure-iam-authentication-is-enabled}

1. Login to your AWS Account and navigate to the RDS instance you want to configure.
2. Click on the **Modify** button.
3. Scroll down to the **Database authentication** section.
4. Enable the **Password and IAM database authentication** option.
5. Click on the **Continue** button.
6. Review the changes and click on the **Apply immediately** option.

#### Setting up the database user {#setting-up-the-database-user}
:::warning
For AWS RDS Postgres and Aurora Postgres you can only run `Initial Load Only` ClickPipes due to the limitations of the AWS IAM DB authentication.

Connect to your RDS/Aurora instance and create a database user with IAM authentication enabled, then follow the rest of the steps in your source setup guide to configure your instance for ClickPipes:
For MySQL and MariaDB, this limitation doesn't apply, and you can run both `Initial Load Only` and `CDC` ClickPipes.
:::

- [AWS Aurora for PostgreSQL source setup guide](./source/aurora#configure-database-user)
- [AWS RDS for PostgreSQL source setup guide](./source/rds#configure-database-user)
- [AWS Aurora for MySQL source setup guide](../mysql/source/aurora#configure-database-user)
- [AWS RDS for MySQL source setup guide](../mysql/source/rds#configure-database-user)
## Setup {#setup}

### Obtaining the ClickHouse service IAM role Arn {#obtaining-the-clickhouse-service-iam-role-arn}

Expand All @@ -53,14 +39,43 @@

Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be used to access your RDS/Aurora instance.

### Setting up the IAM role {#setting-up-iam-role}
### Configuring the RDS/Aurora instance {#configuring-the-rds-aurora-instance}

#### Enabling IAM DB Authentication {#enabling-iam-db-authentication}
1. Login to your AWS Account and navigate to the RDS instance you want to configure.
2. Click on the **Modify** button.
3. Scroll down to the **Database authentication** section.
4. Enable the **Password and IAM database authentication** option.
5. Click on the **Continue** button.
6. Review the changes and click on the **Apply immediately** option.

#### Obtaining the RDS/Aurora Resource ID {#obtaining-the-rds-resource-id}

1. Login to your AWS Account and navigate to the RDS instance/Aurora Cluster you want to configure.
2. Click on the **Configuration** tab.
3. Note the **Resource ID** value. It should look like `db-xxxxxxxxxxxxxx` for RDS or `cluster-xxxxxxxxxxxxxx` for Aurora cluster. Let's call this value `{RDS_RESOURCE_ID}`. This is the resource ID that will be used in the IAM policy to allow access to the RDS instance.

#### Setting up the Database User {#setting-up-the-database-user}

Check failure on line 58 in docs/integrations/data-ingestion/clickpipes/postgres/auth.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Headings

'Setting up the Database User {#setting-up-the-database-user}' should use sentence-style capitalization.

##### PostgreSQL {#setting-up-the-database-user-postgres}

1. Connect to your RDS/Aurora instance and create a new database user with the following command:
```sql
CREATE USER clickpipes_iam_user;

Check warning on line 64 in docs/integrations/data-ingestion/clickpipes/postgres/auth.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.EOLWhitespace

Remove whitespace characters from the end of the line.
GRANT rds_iam TO clickpipes_iam_user;
```
2. Follow the rest of the steps in the [PostgreSQL source setup guide](./source/rds) to configure your RDS instance for ClickPipes.

##### MySQL / MariaDB {#setting-up-the-database-user-mysql}

1. Connect to your RDS/Aurora instance and create a new database user with the following command:
```sql
CREATE USER 'clickpipes_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
```
2. Follow the rest of the steps in the [MySQL source setup guide](../mysql/source/rds) to configure your RDS/Aurora instance for ClickPipes.

### Setting up the IAM role {#setting-up-iam-role}

#### Manually create IAM role. {#manually-create-iam-role}

1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,17 +98,6 @@ Connect to your Aurora PostgreSQL writer instance as an admin user and execute t

The `clickpipes` publication defines the set of tables whose change events will be streamed to ClickPipes. We recommend against using `FOR ALL TABLES` unless you intend to replicate every table, as including unnecessary tables increases WAL traffic from Postgres to ClickPipes and reduces overall replication efficiency.

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. This requires the `rds.iam_auth_for_replication` parameter to be set to `1`. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER clickpipes_iam_user;
GRANT rds_iam TO clickpipes_iam_user;
```

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,21 +98,6 @@ Connect to your RDS Postgres instance as an admin user and execute the following

The `clickpipes` publication defines the set of tables whose change events will be streamed to ClickPipes. We recommend against using `FOR ALL TABLES` unless you intend to replicate every table, as including unnecessary tables increases WAL traffic from Postgres to ClickPipes and reduces overall replication efficiency.

### Using IAM authentication (optional) {#iam-authentication}

Instead of a password, you can authenticate the ClickPipes user via AWS IAM role-based access. Create the user as follows, then grant it the same schema and replication privileges shown above:

```sql
CREATE USER clickpipes_iam_user;
GRANT rds_iam TO clickpipes_iam_user;
```

:::note
IAM authentication for replication requires the `rds.iam_auth_for_replication` parameter to be set to `1`. This is supported from PostgreSQL version 11 onwards; on earlier versions, you can only run `Initial Load Only` ClickPipes.
:::

See [AWS IAM DB authentication (RDS/Aurora)](/integrations/clickpipes/postgres/auth) for the full steps to set up the IAM role and policy required for ClickPipes to authenticate this way.

## Configure network access {#configure-network-access}

### IP-based access control {#ip-based-access-control}
Expand Down
Loading