Skip to content

Allow destroy to proceed when EC2 GPU instances cannot start - #9879

Open
rhjcd wants to merge 1 commit into
developmentfrom
fix/ocp4-cluster-destroy-gpu-capacity
Open

Allow destroy to proceed when EC2 GPU instances cannot start#9879
rhjcd wants to merge 1 commit into
developmentfrom
fix/ocp4-cluster-destroy-gpu-capacity

Conversation

@rhjcd

@rhjcd rhjcd commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add ignore_errors: true to the EC2 instance start and wait tasks in ocp4-cluster/ec2_instances_start.yaml
  • GPU instance types (g6.12xlarge, g6.16xlarge) frequently hit InsufficientInstanceCapacity during destroy, blocking cleanup of all downstream resources (VPCs, security groups, EBS volumes)
  • AWS terminates stopped instances without issue -- they do not need to be running first

Impact

  • 84 destroy failures in 30 days for openshift-ai-v3 alone
  • $8,183/mo in leaked cloud costs from orphaned resources
  • Prior art: Ahsen Shah's PR Increase pod status check retries to 300 #9769 used the same pattern for similar GPU capacity issues

Test plan

  • Provision an openshift-ai-v3 environment
  • Verify destroy completes successfully
  • Confirm no regressions for non-GPU ocp4-cluster destroys

GPU instance types (g6.12xlarge, g6.16xlarge) frequently hit
InsufficientInstanceCapacity during destroy, blocking cleanup of
all resources. Instances do not need to be running for AWS to
terminate them -- adding ignore_errors lets the destroy continue.

Addresses 84 destroy failures and $8,183/mo in leaked cloud costs
for openshift-ai-v3 alone.
@rhjcd
rhjcd requested a review from a team as a code owner July 19, 2026 18:58
@rhjcd
rhjcd requested review from rut31337 and wkulhanek July 19, 2026 19:04
@rhjcd

rhjcd commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Tower job evidence

Job ID GUID Instance Type Region Error Date
2628729 fvgzf g6.12xlarge us-east-2a InsufficientInstanceCapacity on StartInstances (max retries: 4) 2026-07-17 13:24 UTC
2628742 khgdd g6.8xlarge us-east-2a Instances started but wait timed out, then capacity_reservation_regions assertion failed 2026-07-17 18:48 UTC
2628777 7xndb g6.16xlarge eu-central-1a Same as 2628742: start succeeded, wait timeout, capacity_reservation_regions assertion 2026-07-18 05:03 UTC

All 3 jobs are on AAP2 controller aap2-prod-us-east-2.aap.infra.demo.redhat.com. The ignore_errors: true fix addresses both failure patterns:

  1. Pattern 1 (job 2628729): Start fails outright with InsufficientInstanceCapacity. Without ignore_errors, the playbook halts and never reaches CloudFormation/VPC cleanup.
  2. Pattern 2 (jobs 2628742, 2628777): Start request accepted but instance never reaches running within the retry window. The wait task hits its retry limit and the playbook continues, but only because ansible moved to the next play. Adding ignore_errors makes this behavior explicit and reliable.

Note: the capacity_reservation_regions assertion in Pattern 2 is a separate issue in the agnosticd.cloud_provider_aws.capacity_reservation collection role and is not addressed by this PR.

@agonzalezrh agonzalezrh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be handle different, this is start not only for destroy but for stopped labs too

contact @rut31337 to find a better solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants