Skip to content

Migrate int and stg e2e jobs to slot-manager #80467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
roivaz wants to merge 3 commits into
base: main
Choose a base branch
from
+93 −122
Draft

Migrate int and stg e2e jobs to slot-manager #80467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
49d69ea
ARO-HCP: Add slots for the new STG e2e subscription
roivaz Jun 12, 2026
f9e1aa3
Add slots to migrate INT e2e sub to the slot catalog
roivaz Jun 12, 2026
343dc44
Migrate INT and STG e2e jobs to slot-manager
roivaz Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 16 additions & 72 deletions ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,15 +236,8 @@ tests:
ARO_HCP_DEPLOY_ENV: int
ARO_HCP_SUITE_NAME: integration/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: int
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-int-quota-slice
- count: 20
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-int
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: int-rh
workflow: aro-hcp-persistent-e2e
Comment on lines +239 to +240

@coderabbitai coderabbitai Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Root cause: VAULT_SECRET_PROFILE values were migrated to *-rh without matching step-level credential profile contract updates across all affected configs.
Affected files: ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yaml, ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__e2e.yaml, ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic.yaml, and ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic-cleanup.yaml.
All these jobs now rely on profile names that the referenced step contracts (and path derivation logic) do not currently show as mounted/declared, which can cause deterministic runtime auth/bootstrap failures.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yaml` around lines 239 -
240, VAULT_SECRET_PROFILE was changed to an int-rh value without updating the
step-level credential profile contracts, so runtime will not find the mounted
credentials; update the step credential declarations used by the aro-hcp jobs to
include the new "-rh" profile names (or revert VAULT_SECRET_PROFILE to the
original profile) so the step contract/path-derivation logic exposes/mounts the
declared profile; specifically, ensure the steps that reference
VAULT_SECRET_PROFILE (the aro-hcp-persistent-e2e workflow and its related
e2e/periodic/periodic-cleanup job step definitions) declare and mount the
"int-rh" (and any other migrated "*-rh") profiles in their credentials/profile
contract blocks to match the env value.

- always_run: false
as: integration-e2e-parallel-ocp-stable
optional: true
Expand All @@ -256,15 +249,8 @@ tests:
ARO_HCP_OPENSHIFT_NODEPOOL_CHANNEL_GROUP: stable
ARO_HCP_SUITE_NAME: integration/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: int
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-int-quota-slice
- count: 20
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-int
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: int-rh
workflow: aro-hcp-persistent-e2e
- always_run: false
as: integration-e2e-parallel-ocp-fast
optional: true
Expand All @@ -276,15 +262,8 @@ tests:
ARO_HCP_OPENSHIFT_NODEPOOL_CHANNEL_GROUP: fast
ARO_HCP_SUITE_NAME: integration/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: int
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-int-quota-slice
- count: 20
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-int
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: int-rh
workflow: aro-hcp-persistent-e2e
- always_run: false
as: integration-e2e-parallel-ocp-nightly
optional: true
Expand All @@ -296,15 +275,8 @@ tests:
ARO_HCP_OPENSHIFT_NODEPOOL_CHANNEL_GROUP: nightly
ARO_HCP_SUITE_NAME: integration/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: int
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-int-quota-slice
- count: 20
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-int
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: int-rh
workflow: aro-hcp-persistent-e2e
- always_run: false
as: stage-e2e-parallel
optional: true
Expand All @@ -321,15 +293,8 @@ tests:
ARO_HCP_DEPLOY_ENV: stg
ARO_HCP_SUITE_NAME: stage/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: stg
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-stg-quota-slice
- count: 30
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-stg
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: stg-rh
workflow: aro-hcp-persistent-e2e
- always_run: false
as: stage-e2e-parallel-ocp-stable
optional: true
Expand All @@ -341,15 +306,8 @@ tests:
ARO_HCP_OPENSHIFT_NODEPOOL_CHANNEL_GROUP: stable
ARO_HCP_SUITE_NAME: stage/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: stg
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-stg-quota-slice
- count: 30
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-stg
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: stg-rh
workflow: aro-hcp-persistent-e2e
- always_run: false
as: stage-e2e-parallel-ocp-fast
optional: true
Expand All @@ -361,15 +319,8 @@ tests:
ARO_HCP_OPENSHIFT_NODEPOOL_CHANNEL_GROUP: fast
ARO_HCP_SUITE_NAME: stage/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: stg
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-stg-quota-slice
- count: 30
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-stg
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: stg-rh
workflow: aro-hcp-persistent-e2e
- always_run: false
as: stage-e2e-parallel-ocp-nightly
optional: true
Expand All @@ -388,15 +339,8 @@ tests:
ARO_HCP_OPENSHIFT_NODEPOOL_CHANNEL_GROUP: nightly
ARO_HCP_SUITE_NAME: stage/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: stg
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-stg-quota-slice
- count: 30
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-stg
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: stg-rh
workflow: aro-hcp-persistent-e2e
- always_run: false
as: prod-e2e-parallel
optional: true
Expand Down
22 changes: 4 additions & 18 deletions ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,8 @@ tests:
ARO_HCP_DEPLOY_ENV: int
ARO_HCP_SUITE_NAME: integration/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: int
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-int-quota-slice
- count: 20
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-int
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: int-rh
workflow: aro-hcp-persistent-e2e
- as: stage-e2e-parallel
max_concurrency: 4
postsubmit: true
Expand All @@ -64,15 +57,8 @@ tests:
ARO_HCP_DEPLOY_ENV: stg
ARO_HCP_SUITE_NAME: stage/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: stg
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-stg-quota-slice
- count: 30
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-stg
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: stg-rh
workflow: aro-hcp-persistent-e2e
- as: prod-e2e-parallel
max_concurrency: 4
postsubmit: true
Expand Down
6 changes: 4 additions & 2 deletions ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic-cleanup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,8 @@ tests:
<{{.Status.URL}}|View logs>'
steps:
env:
VAULT_SECRET_PROFILE: int
CUSTOMER_SUBSCRIPTION: ARO SRE Team - INT (EA Subscription 3)
VAULT_SECRET_PROFILE: int-rh
test:
- ref: aro-hcp-deprovision-expired-resource-groups
- as: delete-expired-stage-resource-groups
Expand All @@ -153,7 +154,8 @@ tests:
<{{.Status.URL}}|View logs>'
steps:
env:
VAULT_SECRET_PROFILE: stg
CUSTOMER_SUBSCRIPTION: ARO HCP E2E Hosted Clusters - Stage - 00
VAULT_SECRET_PROFILE: stg-rh
test:
- ref: aro-hcp-deprovision-expired-resource-groups
- as: delete-expired-prod-resource-groups
Expand Down
33 changes: 6 additions & 27 deletions ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,8 @@ tests:
ARO_HCP_DEPLOY_ENV: int
ARO_HCP_SUITE_NAME: integration/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: int
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-int-quota-slice
- count: 20
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-int
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: int-rh
workflow: aro-hcp-persistent-e2e
timeout: 4h0m0s
- as: stage-e2e-parallel
cron: 0 2 * * *
Expand All @@ -53,15 +46,8 @@ tests:
ARO_HCP_DEPLOY_ENV: stg
ARO_HCP_SUITE_NAME: stage/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: stg
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-stg-quota-slice
- count: 30
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-stg
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: stg-rh
workflow: aro-hcp-persistent-e2e
timeout: 4h0m0s
- as: stage-e2e-parallel-ocp-nightly
cron: 0 23 * * *
Expand All @@ -80,15 +66,8 @@ tests:
ARO_HCP_OPENSHIFT_NODEPOOL_CHANNEL_GROUP: nightly
ARO_HCP_SUITE_NAME: stage/parallel
LOCATION: uksouth
VAULT_SECRET_PROFILE: stg
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-stg-quota-slice
- count: 30
env: LEASED_MSI_CONTAINERS
resource_type: aro-hcp-test-msi-containers-stg
workflow: aro-hcp-e2e
VAULT_SECRET_PROFILE: stg-rh
workflow: aro-hcp-persistent-e2e
- as: prod-e2e-parallel
cron: 0 2 * * *
reporter_config:
Expand Down
8 changes: 8 additions & 0 deletions ci-operator/step-registry/aro-hcp/persistent-e2e/OWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
approvers:
- aro-hcp-sl-approvers
- geoberle
- deads2k
reviewers:
- aro-hcp-sl-reviewers
- geoberle
- deads2k
15 changes: 15 additions & 0 deletions ...erator/step-registry/aro-hcp/persistent-e2e/aro-hcp-persistent-e2e-workflow.metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"path": "aro-hcp/persistent-e2e/aro-hcp-persistent-e2e-workflow.yaml",
"owners": {
"approvers": [
"aro-hcp-sl-approvers",
"geoberle",
"deads2k"
],
"reviewers": [
"aro-hcp-sl-reviewers",
"geoberle",
"deads2k"
]
}
}
16 changes: 16 additions & 0 deletions ci-operator/step-registry/aro-hcp/persistent-e2e/aro-hcp-persistent-e2e-workflow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
workflow:
as: aro-hcp-persistent-e2e
steps:
allow_best_effort_post_steps: true
pre:
- ref: aro-hcp-lease-acquire
- ref: aro-hcp-write-config
test:
- ref: aro-hcp-test-persistent
post:
- ref: aro-hcp-gather-test-visualization
- ref: aro-hcp-gather-custom-link-tools
- ref: aro-hcp-lease-release
documentation: |-
Runs the E2E suite against a persistent (pre-deployed) environment using slot-manager
for subscription and identity-container selection. No provisioning or deprovisioning.
13 changes: 10 additions & 3 deletions ci-operator/step-registry/aro-hcp/test/persistent/aro-hcp-test-persistent-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,19 @@ export CLUSTER_PROFILE_DIR="/var/run/aro-hcp-${VAULT_SECRET_PROFILE}"
export AZURE_CLIENT_ID; AZURE_CLIENT_ID=$(cat "${CLUSTER_PROFILE_DIR}/client-id")
export AZURE_TENANT_ID; AZURE_TENANT_ID=$(cat "${CLUSTER_PROFILE_DIR}/tenant")
export AZURE_CLIENT_SECRET; AZURE_CLIENT_SECRET=$(cat "${CLUSTER_PROFILE_DIR}/client-secret")
export CUSTOMER_SUBSCRIPTION; CUSTOMER_SUBSCRIPTION=$(cat "${CLUSTER_PROFILE_DIR}/subscription-name")
export SUBSCRIPTION_ID; SUBSCRIPTION_ID=$(cat "${CLUSTER_PROFILE_DIR}/subscription-id")
export AZURE_TOKEN_CREDENTIALS=prod

env_file="${SHARED_DIR}/aro-hcp-slot.env"
if [[ -f "${env_file}" ]]; then
# shellcheck disable=SC1090
source "${env_file}"
export LOCATION="${SELECTED_LOCATION:-${LOCATION:-}}"
else
export CUSTOMER_SUBSCRIPTION; CUSTOMER_SUBSCRIPTION=$(cat "${CLUSTER_PROFILE_DIR}/subscription-name")
Comment on lines +14 to +20

@coderabbitai coderabbitai Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Preserve the cluster-profile fallback for CUSTOMER_SUBSCRIPTION.

If Line 15 finds ${SHARED_DIR}/aro-hcp-slot.env but that file does not export CUSTOMER_SUBSCRIPTION, Line 24 aborts under set -u. The new branch makes the env file optional for discovery, but mandatory for this value.

Proposed fix 
 env_file="${SHARED_DIR}/aro-hcp-slot.env"
 if [[ -f "${env_file}" ]]; then
     # shellcheck disable=SC1090
     source "${env_file}"
+    export CUSTOMER_SUBSCRIPTION="${CUSTOMER_SUBSCRIPTION:-$(< "${CLUSTER_PROFILE_DIR}/subscription-name")}"
     export LOCATION="${SELECTED_LOCATION:-${LOCATION:-}}"
 else
-    export CUSTOMER_SUBSCRIPTION; CUSTOMER_SUBSCRIPTION=$(cat "${CLUSTER_PROFILE_DIR}/subscription-name")
+    export CUSTOMER_SUBSCRIPTION="$(< "${CLUSTER_PROFILE_DIR}/subscription-name")"
 fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/aro-hcp/test/persistent/aro-hcp-test-persistent-commands.sh`
around lines 14 - 20, When sourcing env_file="${SHARED_DIR}/aro-hcp-slot.env",
preserve the existing fallback behavior for CUSTOMER_SUBSCRIPTION so the script
won't fail under set -u if the env file doesn't set that variable: after
sourcing (inside the branch where env_file exists) export CUSTOMER_SUBSCRIPTION
using the current CUSTOMER_SUBSCRIPTION if present, otherwise read the
subscription from the cluster profile file (the same file used in the else
branch, referenced by CLUSTER_PROFILE_DIR/subscription-name); update the logic
in aro-hcp-test-persistent-commands.sh to perform this conditional export so
both the env file and the cluster-profile fallback are honored.

fi

az login --service-principal -u "${AZURE_CLIENT_ID}" -p "${AZURE_CLIENT_SECRET}" --tenant "${AZURE_TENANT_ID}" --output none
az account set --subscription "${SUBSCRIPTION_ID}"
az account set --subscription "${CUSTOMER_SUBSCRIPTION}"

if [[ -n "${MULTISTAGE_PARAM_OVERRIDE_LOCATION:-}" ]]; then
export LOCATION="${MULTISTAGE_PARAM_OVERRIDE_LOCATION}"
Expand Down
8 changes: 8 additions & 0 deletions core-services/prow/02_config/_boskos.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,10 @@ resources:
min-count: 1
state: free
type: aro-hcp-int-quota-slice
- names:
- aro-hcp-int-shard0-slot-00
state: free
type: aro-hcp-int-shard0-slot
- names:
- aro-hcp-msi-mock-cs-sp-dev-0
- aro-hcp-msi-mock-cs-sp-dev-1
Expand Down Expand Up @@ -142,6 +146,10 @@ resources:
min-count: 1
state: free
type: aro-hcp-stg-quota-slice
- names:
- aro-hcp-stg-shard0-slot-00
state: free
type: aro-hcp-stg-shard0-slot
- names:
- aro-hcp-test-msi-containers-dev-0
- aro-hcp-test-msi-containers-dev-1
Expand Down
6 changes: 6 additions & 0 deletions core-services/prow/02_config/generate-boskos.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,8 @@
'aro-hcp-dev-shard0-slot': {},
'aro-hcp-dev-shard2-slot': {},
'aro-hcp-dev-shard3-slot': {},
'aro-hcp-int-shard0-slot': {},
'aro-hcp-stg-shard0-slot': {},
# END ARO-HCP E2E SLOT TYPES
'aro-hcp-msi-mock-cs-sp-dev': {},
'equinix-ocp-metal-quota-slice': {
Expand Down Expand Up @@ -783,6 +785,10 @@
CONFIG['aro-hcp-dev-shard2-slot']['aro-hcp-dev-shard2-slot-{i:0>2}'.format(i=i)] = 1
for i in range(1):
CONFIG['aro-hcp-dev-shard3-slot']['aro-hcp-dev-shard3-slot-{i:0>2}'.format(i=i)] = 1
for i in range(1):
CONFIG['aro-hcp-int-shard0-slot']['aro-hcp-int-shard0-slot-{i:0>2}'.format(i=i)] = 1
for i in range(1):
CONFIG['aro-hcp-stg-shard0-slot']['aro-hcp-stg-shard0-slot-{i:0>2}'.format(i=i)] = 1
# END ARO-HCP E2E SLOT RESOURCES
for i in range(20):
CONFIG['aro-hcp-msi-mock-cs-sp-dev']['aro-hcp-msi-mock-cs-sp-dev-{}'.format(i)] = 1
Expand Down