Skip to content

test: add AWS XKS CI test against remote server - #1114

Open
p0wline wants to merge 14 commits into
developfrom
feat/ci-xks-remote
Open

test: add AWS XKS CI test against remote server#1114
p0wline wants to merge 14 commits into
developfrom
feat/ci-xks-remote

Conversation

@p0wline

@p0wline p0wline commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Context

Before this PR, XKS tests only ran locally against a mock server. There was no way to catch regressions in the full AWS integration path.

What this PR does

1. scratchstack 0.10 → 0.11 migration

The SigV4 verification library rejected secret keys longer than 43 characters. Version 0.11 removes that restriction, which was blocking the real AWS infrastructure (key was 52 chars).

2. CI workflow .github/workflows/test_xks_remote.yml

On every PR and push:

  • Builds a non-fips release binary
  • Deploys it to the persistent EC2 test server (aws-xks-kms.cosmian.dev) via SSH
  • Step 1 — Proxy-direct tests: provisions three test keys (aws_xks_kek, encrypt_only_key, decrypt_only_key), enforces usage restrictions (encrypt-only / decrypt-only), then runs the vendored AWS XKS curl test suite directly against the server using SigV4 — no AWS KMS involved
  • Step 2 — End-to-end test: calls aws kms encrypt / aws kms decrypt via the real AWS KMS API, which routes through the configured external key store to our proxy — verifies the full chain from AWS to Cosmian KMS and back

3. Cherry-pick of a14031598 (fix: create a dedicated AWS-XKS user)

Pulled in from your fix/xks_permissions branch. The server now uses the reserved identity [aws-xks-service] for all XKS operations instead of awsPrincipalArn from the request body. This is required for the end-to-end test to work: when AWS KMS calls the proxy, the injected ARN varies per caller, but [aws-xks-service] gets a single permanent grant at startup on every XKS key.

4. test_data submodule update

Points to fix/xks-config-env-override which fixes config.sh to respect env vars (see test_data PR #13). Without this, the vendored test suite ignored the SigV4 credentials injected by the CI workflow.

5. lychee.toml

Added kms\.example\.com and kms\.different\.com to the link-checker exclusions — these are fictional hostnames used in unit tests inside start_kms_server.rs and were causing lychee to fail on every commit.

Required GitHub secrets (repo Cosmian/kms)

Secret Purpose
AWS_XKS_ACCESS_KEY_ID IAM key ID for aws kms encrypt/decrypt (xks-test user)
AWS_XKS_SECRET_ACCESS_KEY Matching IAM secret
AWS_XKS_KEY_ARN ARN of the KMS key backed by the external key store
KMS_XKS_SIGV4_ACCESS_KEY_ID SigV4 access key configured on the proxy server
KMS_XKS_SIGV4_SECRET_ACCESS_KEY Matching SigV4 secret key
KMS_XKS_SSH_PRIVATE_KEY SSH private key to deploy the binary to the EC2 instance
KMS_XKS_SSH_KNOWN_HOSTS SSH known hosts entry for aws-xks-kms.cosmian.dev
KMS_XKS_SSH_USER SSH user on the EC2 instance

p0wline added 11 commits August 14, 2026 16:03
@p0wline
test: add AWS XKS CI test against remote server
c15f2b1 
- New script .mise/scripts/test/test_xks_remote.sh: provisions keys via
  XKS API and runs the vendored curl test suite against the persistent
  server at aws-xks-kms.cosmian.dev (no build, no local server).
- New MISE task test:xks-remote delegating to that script.
- New workflow .github/workflows/test_xks_remote.yml: triggered on push
  to develop/main/xks branches, PRs, and daily schedule; uses concurrency
  group to prevent parallel runs; reads SigV4 creds from secrets.
- nix_helpers.sh: propagate KMS_XKS_* env vars through nix-shell re-entry.
@p0wline
ci: trigger xks-remote on all PRs and pushes
59a392b
@p0wline
fix: use eu-west-1 region for XKS remote test server
991d29b
@p0wline
fix(ci): patch config.sh before test_all to preserve remote credentials
ea6d80b
@p0wline
fix(ci): update test_data submodule + remove runtime config.sh patch
be5b3bd 
test_data now points to fix/xks-config-env-override (d02f52e) which
uses ${VAR:-default} in config.sh — no runtime patching needed.
After Cosmian/test_data#13 is merged, update submodule to develop HEAD.
@p0wline
fix(ci): re-grant XKS permissions on persistent server before test suite
6d1f823 
The remote test server persists keys across CI runs. On the old server
(<=5.26.0), grant_access is skipped when the key already exists, so
Alice and [aws-xks-service] may lose Encrypt/Decrypt/GetAttributes
permissions between runs.

Add grant_xks_access (idempotent re-grant) and update revoke_xks_op to
target both identities so that encrypt-only/decrypt-only restriction
tests work regardless of which server version is deployed.
@p0wline
fix(ci): use correct operation name 'getattributes' in access/grant call
9069ac3 
The /access/grant endpoint deserializes operation_types with serde
rename_all lowercase: the valid value is 'getattributes', not
'get_attributes'. The previous commit silently got HTTP 400 on every
grant call, so Alice's permissions were never applied.

Also capture and log the grant response so future failures are visible.
@p0wline
feat(ci): migrate scratchstack 0.10→0.11, deploy PR binary to XKS ser…
606e010 
…ver before tests

- Upgrade scratchstack-aws-signature 0.10→0.11 (removes ring 0.16.x dep,
  RUSTSEC-2025-0009/0010). API redesign: replace sigv4_verify + SigningKey
  with sigv4_validate_request + SigningKeyService implementing tower_service::Service.
- Remove RUSTSEC-2025-0009/0010 from deny.toml and .cargo/audit.toml.
- Add tower-service = "0.3" direct dep to crate/server.
- test_xks_remote.yml: build non-fips release binary, deploy via SSH before
  tests. Remove daily schedule (CI now owns deployment).
- Add deploy_xks_remote.sh: SSH deploy + systemctl restart + readiness wait.
- Update log-reference.md for renamed SigV4 log variable.
- lychee.toml: exclude ibm.com (503) and sekretz.com (SSL mismatch).
@p0wline
fix(ci): use ubuntu user and sudo for XKS server deploy
5c4e74e
@p0wline
fix(ci): fix SSH remote command quoting (heredoc instead of bash -c)
3a8b771
@p0wline
fix(xks): support SigV4 secret keys >40 chars; update test_data submo…
043c5da 
…dule

KSecretKey::from_str in scratchstack 0.11 only accepts keys of exactly
M-4 chars (default M=44 → 40 chars). XKS spec allows 43-64 chars and
the CI key is longer, causing AuthenticationFailedException: Key too long.

Derive the SigV4 signing key manually via hmac+sha2 to bypass the length
restriction. KSigningKey has no public constructor from raw bytes, so
transmute is used with a documented safety justification.

Also rebase test_data/fix/xks-config-env-override onto develop to bring
in the spire test data directory, fixing the CI spire test failure.
@p0wline
p0wline force-pushed the feat/ci-xks-remote branch from 32abf5e to 043c5da Compare August 14, 2026 14:06
@Manuthor
Manuthor changed the base branch from develop to fix/xks_permissions August 17, 2026 04:47
@Manuthor
Manuthor changed the base branch from fix/xks_permissions to develop August 17, 2026 04:47
@Manuthor
Manuthor changed the base branch from develop to fix/xks_permissions August 17, 2026 04:49
@Manuthor
Manuthor changed the base branch from fix/xks_permissions to develop August 17, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@p0wline @Manuthor