-
Notifications
You must be signed in to change notification settings - Fork 486
clickpipes: documentation for GCP-native private networking (PSC) #6409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
morsapaes
wants to merge
4
commits into
ClickHouse:main
Choose a base branch
from
morsapaes:docs-psc
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
f9f11ed
Draft for PSC documentation
morsapaes b7da55a
Merge remote-tracking branch 'origin/main' into docs-psc
dhtclk 96773b0
Fix CI: add heading anchors, sidebar entry, and heading casing
dhtclk b808cc0
docs: add ClickPipes Networking section and apply PSC review comments
morsapaes File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
266 changes: 266 additions & 0 deletions
266
docs/integrations/data-ingestion/clickpipes/gcp-psc/postgres.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,266 @@ | ||
| --- | ||
| sidebar_label: 'GCP Private Service Connect for Postgres CDC' | ||
| description: 'Connect a Postgres CDC ClickPipe to a private Cloud SQL for PostgreSQL instance using GCP Private Service Connect.' | ||
| slug: /integrations/clickpipes/gcp-psc/postgres | ||
| title: 'GCP Private Service Connect for Postgres CDC' | ||
| doc_type: 'guide' | ||
| keywords: ['gcp psc', 'private service connect', 'ClickPipes security', 'reverse private endpoint', 'cloud sql', 'postgres cdc', 'private connectivity'] | ||
| integration: | ||
| - support_level: 'core' | ||
| - category: 'clickpipes' | ||
| --- | ||
|
|
||
| # GCP Private Service Connect for Postgres CDC ClickPipes | ||
|
|
||
| You can use GCP [Private Service Connect (PSC)](https://cloud.google.com/vpc/docs/private-service-connect) to give a ClickPipe private, internal-only access to a Postgres source running in your own GCP project — without exposing the database to the public internet or peering VPCs. | ||
|
|
||
| This guide covers the **Postgres CDC** ClickPipe group. Cloud SQL for PostgreSQL is the primary, recommended source. Streaming sources (Kafka, Confluent Cloud) and other source groups have their own guides. | ||
|
|
||
| :::note | ||
| PSC connectivity is only available when your ClickPipe is hosted on GCP. Check the [region matrix](#supported-gcp-regions) before you start. | ||
| ::: | ||
|
|
||
| ## How it works {#how-it-works} | ||
|
|
||
| ClickPipes uses **Reverse Private Endpoints (RPEs)** to reach private sources. For GCP, an RPE is a PSC **endpoint** that the ClickPipes data plane creates inside its own VPC and that consumes a PSC **service attachment** you publish in front of your Postgres instance. | ||
|
|
||
| Two patterns are supported, both using the same RPE endpoint type (`GCP_PSC_SERVICE_ATTACHMENT`): | ||
|
|
||
| | Pattern | When to use | Service attachment | | ||
| | --- | --- | --- | | ||
| | **Cloud SQL native PSC** *(recommended)* | You run Cloud SQL for PostgreSQL and want the simplest setup. | Created automatically by Cloud SQL when you enable PSC on the instance. | | ||
| | **Producer-owned PSC** | Cloud SQL on a private VPC reached through your own internal TCP load balancer, or a self-managed Postgres on Compute Engine. | You create and manage the service attachment yourself. | | ||
|
|
||
| Each RPE provisions a single static internal IP in the ClickPipes VPC. Unlike AWS VPC endpoints, GCP PSC does not propagate DNS, so you tell ClickPipes which private DNS name to map to that IP using `custom_private_dns_mappings`. | ||
|
|
||
| ## Supported Postgres sources {#supported-postgres-sources} | ||
|
|
||
| - Cloud SQL for PostgreSQL (native PSC) | ||
| - Cloud SQL for PostgreSQL on a private VPC, fronted by a producer-owned PSC service attachment | ||
| - Self-managed PostgreSQL on Compute Engine, fronted by a producer-owned PSC service attachment | ||
|
|
||
| ## Prerequisites {#prerequisites} | ||
|
|
||
| - A ClickHouse Cloud service hosted on GCP, in a [supported region](#supported-gcp-regions). | ||
| - A GCP project where Cloud SQL (or your Postgres host) lives. You need IAM rights to: | ||
| - Manage Cloud SQL (`roles/cloudsql.admin`) or your equivalent Postgres infrastructure. | ||
| - Manage PSC service attachments (`roles/compute.networkAdmin`). | ||
| - The ClickHouse Cloud API key/secret for the organization that owns the service (only required if you provision through Terraform or the API). | ||
| - The **ClickPipes consumer project** that you will allow to connect to your service attachment. For ClickPipes production, this is `clickpipes-production`. | ||
|
|
||
| --- | ||
|
|
||
| ## Path A: Cloud SQL native PSC (recommended) {#path-a-cloud-sql-native-psc} | ||
|
|
||
| ### Step 1 — Enable PSC on the Cloud SQL instance {#step-1-enable-psc} | ||
|
|
||
| Either create a new instance with PSC enabled or update an existing one. PSC requires the instance's public IP to be disabled (`ipv4_enabled = false`). | ||
|
|
||
| ```bash | ||
| gcloud sql instances create <INSTANCE_NAME> \ | ||
| --project=<YOUR_PROJECT_ID> \ | ||
| --region=<REGION> \ | ||
| --database-version=POSTGRES_15 \ | ||
| --tier=db-f1-micro \ | ||
| --no-assign-ip \ | ||
| --enable-private-service-connect \ | ||
| --allowed-psc-projects=clickpipes-production | ||
| ``` | ||
|
|
||
| Updating an existing instance: | ||
|
|
||
| ```bash | ||
| gcloud sql instances patch <INSTANCE_NAME> \ | ||
| --project=<YOUR_PROJECT_ID> \ | ||
| --enable-private-service-connect \ | ||
| --allowed-psc-projects=clickpipes-production | ||
| ``` | ||
|
|
||
| The `--allowed-psc-projects` (`psc_config.allowed_consumer_projects` in the API and Terraform) is the auto-accept list. As long as the ClickPipes consumer project is on it, Cloud SQL accepts the endpoint automatically — no manual approval step. | ||
|
|
||
| :::note | ||
| A PSC service attachment can be claimed by only one ClickHouse Cloud service at a time — it cannot be reused across multiple services. If you need to move a service attachment to a different service, contact ClickHouse support to release the existing claim. | ||
| ::: | ||
|
|
||
| ### Step 2 — Read the service attachment URI and DNS name {#step-2-service-attachment-dns} | ||
|
|
||
| These two values are what you hand to ClickPipes. | ||
|
|
||
| ```bash | ||
| gcloud sql instances describe <INSTANCE_NAME> \ | ||
| --project=<YOUR_PROJECT_ID> \ | ||
| --format='value(pscServiceAttachmentLink,dnsName)' | ||
| ``` | ||
|
|
||
| You should get: | ||
|
|
||
| - `pscServiceAttachmentLink` — the service attachment URI in the form | ||
| `projects/<YOUR_PROJECT_ID>/regions/<REGION>/serviceAttachments/<NAME>`. | ||
| - `dnsName` — the synthetic private DNS hostname Cloud SQL assigns to the instance, e.g. | ||
| `<INSTANCE_UID>.<REGION>.sql.goog.`. | ||
|
|
||
| Keep both. ClickPipes needs the service attachment URI to provision the endpoint and the DNS name so the pipe can resolve the host through the RPE. | ||
|
|
||
| Cloud SQL's DNS names end with . (FQDN notation) but the ClickHouse form expects a hostname without it. Drop the trailing dot: | ||
| 3d2deea033cc.swpbkd47gmtq.us-central1.sql.goog | ||
|
|
||
| ### Step 3 — Create the reverse private endpoint in ClickPipes {#step-3-create-rpe} | ||
|
|
||
| #### Option 1: ClickPipes UI {#option-1-clickpipes-ui} | ||
|
|
||
| 1. In ClickHouse Cloud, open your service and go to **Data Sources** > **ClickPipes**. | ||
| 2. Select the data source you want to ingest data from. | ||
| 3. Under **Setup your ClickPipe connection**, toggle on **Use secure connection**, then click **+ Reverse private endpoint**. Click **Create reverse private endpoint** and pick **GCP PSC service attachment**. | ||
| 4. Fill in: | ||
| - **Service attachment URI** — paste the `pscServiceAttachmentLink` from Step 2. | ||
| - **Private DNS name** — paste the `dnsName` from Step 2 (without the trailing dot). | ||
| - **Description** — any human-readable label. | ||
| 5. Click **Create** and wait. The endpoint moves through `Provisioning` → `Ready`. (You will not see `PendingAcceptance` for the native PSC path, because Cloud SQL auto-accepts.) | ||
|
|
||
| #### Option 2: Terraform {#option-2-terraform} | ||
|
|
||
| Use the [`gcp-cloud-sql-native-psc`](https://github.com/ClickHouse/clickpipes-terraform-modules/tree/main/modules/gcp-cloud-sql-native-psc) module from the public [`clickpipes-terraform-modules`](https://github.com/ClickHouse/clickpipes-terraform-modules) repository. It provisions the Cloud SQL instance, enables native PSC, creates the RPE, and optionally creates the ClickPipe. | ||
|
|
||
| ```hcl | ||
| terraform { | ||
| required_providers { | ||
| clickhouse = { source = "ClickHouse/clickhouse", version = ">= 3.14.0" } | ||
| google = { source = "hashicorp/google", version = "~> 6.0" } | ||
| } | ||
| } | ||
|
|
||
| provider "google" { | ||
| project = var.gcp_project_id | ||
| region = var.region | ||
| } | ||
|
|
||
| provider "clickhouse" { | ||
| organization_id = var.clickhouse_organization_id | ||
| token_key = var.clickhouse_cloud_api_key | ||
| token_secret = var.clickhouse_cloud_api_secret | ||
| } | ||
|
|
||
| module "cloud_sql_native_psc" { | ||
| source = "github.com/ClickHouse/clickpipes-terraform-modules//modules/gcp-cloud-sql-native-psc?ref=main" | ||
|
|
||
| project_id = var.gcp_project_id | ||
| region = var.region | ||
| clickhouse_service_id = var.clickhouse_service_id | ||
| psc_allowed_consumer_projects = ["clickpipes-production"] | ||
|
|
||
| # Leave create_clickpipe = false until your source table exists and has data. | ||
| create_clickpipe = false | ||
| } | ||
| ``` | ||
|
|
||
| If you already have a Cloud SQL instance and only want to create the RPE, use the `clickhouse_clickpipes_reverse_private_endpoint` resource directly. Custom private DNS mappings are not an attribute of this resource — they are managed by the separate `clickhouse_clickpipes_reverse_private_endpoint_custom_private_dns` resource, which takes a full replacement list of mappings and references the endpoint by ID: | ||
|
|
||
| ```hcl | ||
| resource "clickhouse_clickpipes_reverse_private_endpoint" "cloud_sql" { | ||
| service_id = var.clickhouse_service_id | ||
| description = "Cloud SQL native PSC endpoint" | ||
| type = "GCP_PSC_SERVICE_ATTACHMENT" | ||
| gcp_service_attachment = "projects/<YOUR_PROJECT_ID>/regions/<REGION>/serviceAttachments/<NAME>" | ||
| } | ||
|
|
||
| resource "clickhouse_clickpipes_reverse_private_endpoint_custom_private_dns" "cloud_sql" { | ||
| service_id = var.clickhouse_service_id | ||
| reverse_private_endpoint_id = clickhouse_clickpipes_reverse_private_endpoint.cloud_sql.id | ||
|
|
||
| mapping = [ | ||
| { private_dns_name = "<INSTANCE_UID>.<REGION>.sql.goog" } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| #### Option 3: API {#option-3-api} | ||
|
|
||
| ```bash | ||
| curl --silent --user $KEY_ID:$KEY_SECRET \ | ||
| -X POST -H "Content-Type: application/json" \ | ||
| https://api.clickhouse.cloud/v1/organizations/<ORG_ID>/services/<SERVICE_ID>/clickpipesReversePrivateEndpoints \ | ||
| -d '{ | ||
| "type": "GCP_PSC_SERVICE_ATTACHMENT", | ||
| "description": "Cloud SQL native PSC endpoint", | ||
| "gcpServiceAttachment": "projects/<YOUR_PROJECT_ID>/regions/<REGION>/serviceAttachments/<NAME>", | ||
| "customPrivateDnsMappings": [ | ||
| { "privateDnsName": "<INSTANCE_UID>.<REGION>.sql.goog" } | ||
| ] | ||
| }' | ||
| ``` | ||
|
|
||
| ### Step 4 — Create the Postgres ClickPipe {#step-4-create-clickpipe} | ||
|
|
||
| In the ClickPipes UI, choose **Postgres CDC** → **Cloud SQL for PostgreSQL** (`cloudsqlpostgres`) and: | ||
|
|
||
| - **Host** — the Cloud SQL `dnsName` you mapped in the RPE. | ||
| - **Port** — `5432`. | ||
| - **Database / User / Password** — the credentials you set on the instance. | ||
| - **Reverse Private Endpoint** — pick the RPE you created in Step 3. | ||
|
|
||
| The pipe routes Postgres traffic through the RPE's internal IP, with DNS resolution coming from the `custom_private_dns_mappings` entry. | ||
|
|
||
| --- | ||
|
|
||
| ## Path B: Producer-owned PSC (private network Cloud SQL or self-managed Postgres) {#path-b-producer-owned-psc} | ||
|
|
||
| Use this path when: | ||
|
|
||
| - Your Cloud SQL instance is on a private VPC reached through an internal TCP load balancer, and not via Cloud SQL's native PSC. | ||
| - You run Postgres on Compute Engine and want to keep it off the public internet. | ||
|
|
||
| The shape is identical to Path A — you publish a PSC service attachment and ClickPipes creates an endpoint for it. The difference is that you own the producer-side stack: internal TCP load balancer, NAT subnet, proxy-only subnet, and the service attachment itself. | ||
|
|
||
| The [`gcp-cloud-sql-private-network-psc`](https://github.com/ClickHouse/clickpipes-terraform-modules/tree/main/modules/gcp-cloud-sql-private-network-psc) module bundles the full producer-side setup. The `psc_consumer_accept_projects` input maps to the service attachment's auto-accept list: | ||
|
|
||
| ```hcl | ||
| module "cloud_sql_private_psc" { | ||
| source = "github.com/ClickHouse/clickpipes-terraform-modules//modules/gcp-cloud-sql-private-network-psc?ref=main" | ||
|
|
||
| project_id = var.gcp_project_id | ||
| region = var.region | ||
| clickhouse_service_id = var.clickhouse_service_id | ||
|
|
||
| psc_consumer_accept_projects = [ | ||
| { project_id = "clickpipes-production", connection_limit = 10 } | ||
| ] | ||
|
|
||
| create_clickpipe = false | ||
| } | ||
| ``` | ||
|
|
||
| If you skip the auto-accept list, the RPE will sit in `PendingAcceptance` until you manually approve the connection on the service attachment: | ||
|
|
||
| ```bash | ||
| gcloud beta compute service-attachments update <SERVICE_ATTACHMENT_NAME> \ | ||
| --region=<REGION> --project=<YOUR_PROJECT_ID> \ | ||
| --consumer-accept-list=projects/clickpipes-production/regions/<REGION>/forwardingRules/<FORWARDING_RULE_ID> \ | ||
| --reconcile-connections | ||
| ``` | ||
|
|
||
| You can find the consumer forwarding rule ID on the RPE detail page in the ClickPipes UI, or as `endpoint_id` in the API/Terraform response. | ||
|
|
||
| --- | ||
|
|
||
| ## Managing existing reverse private endpoints {#managing-reverse-private-endpoints} | ||
|
|
||
| Go to **Data Sources** → **Reverse Private Endpoints** in your service to: | ||
|
|
||
| - See each RPE's status (`Provisioning`, `PendingAcceptance`, `Ready`, `Failed`). | ||
| - Reuse an existing RPE across multiple ClickPipes that target the same Postgres host. | ||
| - Delete RPEs that are no longer in use. Deleting an RPE also tears down the consumer endpoint in the ClickPipes VPC; the service attachment on your side is untouched. | ||
|
|
||
| ## Supported GCP regions {#supported-gcp-regions} | ||
|
|
||
| GCP PSC RPE is available in every region where ClickPipes is hosted on GCP. The PSC endpoint must be created in the **same region** as the service attachment — pick a Cloud SQL region that matches a ClickPipes-on-GCP region. | ||
|
|
||
| For the current list of regions, see [ClickPipes networking → Supported regions](/integrations/clickpipes/networking#supported-regions). | ||
|
|
||
| ## Limitations {#limitations} | ||
|
|
||
| - The RPE endpoint and the service attachment must be in the **same GCP region**. | ||
| - One RPE provisions **one static internal IP**. GCP PSC does not propagate DNS, so you must supply the source's private DNS name via `custom_private_dns_mappings`. | ||
| - Cloud SQL native PSC requires the instance's **public IP to be disabled** (`ipv4_enabled = false`). | ||
| - The Cloud SQL instance and the ClickPipes service must be in projects where the consumer project (`clickpipes-production`) is on the allow list. | ||
| - A PSC service attachment is claimed by a single ClickHouse Cloud service and cannot be reused across multiple services. The claim can be released on request through ClickHouse support. | ||
| - Only `GCP_PSC_SERVICE_ATTACHMENT` is supported as a GCP RPE type. VPC peering is not supported. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should mention here PSC service attachment cannot be reused across multiple ClickHouse Cloud services.