Tiered storage: S3 secret created with empty credentials when volume has no static keys (breaks IAM-role / credential-chain auth)

[MounirBenzi]

Summary

When a tiered-storage S3 volume is configured without static s3_access_key_id / s3_secret_access_key (i.e. relying on an IAM role — EC2 instance profile, ECS task role, or EKS Pod Identity), the tiering service still creates a DuckDB S3 secret with empty KEY_ID/SECRET. DuckDB then signs S3 requests with empty credentials and every tiering move fails with HTTP 403.

Version

11.0.235 (also present on current homer11).

Where

src/storage/ducklake/tiered_storage.go, TieredStorageManager.attachVolume().

For an S3 volume it unconditionally runs (paraphrased):

CREATE SECRET s3_secret_<name> (TYPE S3, KEY_ID '', SECRET '', REGION '');

when the volume config has no keys. There is no branch for "no static credentials → use the AWS credential chain", and the aws extension is not loaded in Start() (only ducklake and sqlite are).

Logs

INF TieredStorageManager: Creating S3 secret volume=cold endpoint="" use_ssl=false
ERR TieringService: Failed to move partition table=... error="failed to insert into destination:
    HTTP Error: Unable to connect to URL s3://<bucket>/... Forbidden (HTTP code 403)"

Reproduction (DuckDB 1.5.3, standalone)

CREATE SECRET s3_secret_cold (TYPE S3, KEY_ID '', SECRET '', REGION '');
SELECT * FROM 's3://<bucket>/test.parquet' LIMIT 1;
-- HTTP 403 Forbidden — "Authentication Failure ... No credentials are provided."

Using the credential chain instead resolves credentials from the environment / instance / container provider:

CREATE SECRET s3_secret_cold (TYPE S3, PROVIDER credential_chain, REGION 'eu-west-2');
-- DuckDB resolves temporary credentials via the AWS SDK default chain

Proposed fix

In attachVolume(), when the volume has no static access key and no custom endpoint (native AWS S3), create the secret with PROVIDER credential_chain rather than empty explicit keys; keep the explicit-key path for static keys and for custom endpoints (MinIO/R2). Also LOAD aws in Start() (best-effort) so the provider is available.

This makes role-based auth (instance profile / ECS task role / EKS Pod Identity) work for tiering without embedding long-lived keys in config.

Happy to open a PR — we have the change running on a fork and can upstream it.

[lmangani]

@MounirBenzi thanks for the report - feel free to open the PR. Our validation focuses on local first options

[adubovikov]

@MounirBenzi any progress here ?