Add CUDN pod churn and CUDN churn periodic tests for 5.0#80485
Add CUDN pod churn and CUDN churn periodic tests for 5.0#80485mohit-sheth wants to merge 1 commit into
Conversation
Add two new 24-node periodic tests to validate kube-burner-ocp CUDN churn support (kube-burner/kube-burner-ocp#458): - cudn-pod-churn-250-24nodes: 50% pod churn with 30m duration - cudn-churn-250-24nodes: 10% CUDN group churn with 3 cycles Temporarily points e2e-benchmarking to mohit-sheth fork with use-fork-kube-burner-ocp branch for pj-rehearse validation. Signed-off-by: Mohit Sheth <msheth@redhat.com>
WalkthroughThis PR adds two new CUDN churn performance test jobs to the OpenShift perfscale CI suite, updating the benchmarking script to use a forked e2e-benchmarking repository pinned to a specific branch, and configuring the new test workloads to run pod and CUDN churn scenarios on 24-node AWS clusters with 250-iteration overrides. ChangesCUDN Churn Test Jobs and Script Configuration
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Suggested labels
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (13 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mohit-sheth The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-aws-5.0-nightly-x86-cudn-pod-churn-250-24nodes |
|
@mohit-sheth: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals. Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@mohit-sheth: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh`:
- Around line 38-39: Hardcoded temporary fork settings (REPO_URL and TAG_OPTION)
lack a revert tracker; update the file to add a TODO comment adjacent to the
REPO_URL and TAG_OPTION declarations that includes the upstream PR/issue URL (or
CI ticket) and a short "revert when merged" note, or reference a scheduled
task/milestone to perform the revert—mention the exact upstream PR number/link
and the intended revert action so future maintainers can find and remove the
fork override.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 532e70f5-63f4-4e29-97ac-bb105a1eb576
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift-eng/ocp-qe-perfscale-ci/openshift-eng-ocp-qe-perfscale-ci-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (2)
ci-operator/config/openshift-eng/ocp-qe-perfscale-ci/openshift-eng-ocp-qe-perfscale-ci-main__aws-5.0-nightly-x86.yamlci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh
| REPO_URL="https://github.com/mohit-sheth/e2e-benchmarking"; | ||
| TAG_OPTION="--branch use-fork-kube-burner-ocp"; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚖️ Poor tradeoff
Temporary fork URL lacks tracking mechanism for revert.
Lines 38-39 hardcode the fork URL and branch as explicitly temporary (per PR description: "Revert e2e-benchmarking fork URL after upstream kube-burner-ocp PR merges"). Without a TODO comment, issue reference, or other tracking, this revert could be forgotten, leaving the repository diverged from upstream.
Recommendation: Add a TODO comment with the upstream PR reference or GitHub issue link to track the revert, or ensure the revert is tied to a scheduled task/milestone.
📝 Suggested fix to add tracking
+# TODO: Revert to cloud-bulldozer after upstream kube-burner-ocp PR merges
+# See: https://github.com/cloud-bulldozer/kube-burner-ocp/pull/458
REPO_URL="https://github.com/mohit-sheth/e2e-benchmarking";
TAG_OPTION="--branch use-fork-kube-burner-ocp";📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| REPO_URL="https://github.com/mohit-sheth/e2e-benchmarking"; | |
| TAG_OPTION="--branch use-fork-kube-burner-ocp"; | |
| # TODO: Revert to cloud-bulldozer after upstream kube-burner-ocp PR merges | |
| # See: https://github.com/cloud-bulldozer/kube-burner-ocp/pull/458 | |
| REPO_URL="https://github.com/mohit-sheth/e2e-benchmarking"; | |
| TAG_OPTION="--branch use-fork-kube-burner-ocp"; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openshift-qe/cudn-density/openshift-qe-cudn-density-commands.sh`
around lines 38 - 39, Hardcoded temporary fork settings (REPO_URL and
TAG_OPTION) lack a revert tracker; update the file to add a TODO comment
adjacent to the REPO_URL and TAG_OPTION declarations that includes the upstream
PR/issue URL (or CI ticket) and a short "revert when merged" note, or reference
a scheduled task/milestone to perform the revert—mention the exact upstream PR
number/link and the intended revert action so future maintainers can find and
remove the fork override.
|
/pj-rehearse periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-aws-5.0-nightly-x86-cudn-churn-250-24nodes |
|
@mohit-sheth: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@mohit-sheth: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Summary
cudn-pod-churn-250-24nodes: 50% pod churn with 30m duration, 1m delaycudn-churn-250-24nodes: 10% CUDN group churn with 3 cycles, 10m durationuse-fork-kube-burner-ocpbranch) for pj-rehearse validationTest plan
Summary by CodeRabbit
This PR adds two new periodic test jobs to the OpenShift perfscale CI infrastructure for validating CUDN (Cluster User Defined Network) churn support in kube-burner-ocp on AWS 5.0 nightly testing.
New Periodic Test Jobs:
Two 24-node test jobs are added to validate CUDN churn scenarios:
cudn-pod-churn-250-24nodes(runs daily at 6 AM UTC): Tests pod churn behavior with 50% churn rate, 30-minute test duration, and 1-minute delay between churn operationscudn-churn-250-24nodes(runs daily at 8 AM UTC): Tests CUDN group churn with 10% churn rate across 3 cycles over a 10-minute durationBoth jobs execute on the
aws-perfscale-qecluster with 21 additional worker nodes, using m6a.2xlarge compute/control-plane instances and r5.2xlarge infra instances. They run the existingopenshift-qe-cudn-densitytest chain with 250 iterations configured per job.Temporary Fork Reference:
The CUDN density test step script (
openshift-qe-cudn-density-commands.sh) is temporarily updated to use a fork of e2e-benchmarking. Instead of cloning from the upstreamcloud-bulldozer/e2e-benchmarkingrepository with dynamic tag selection, the script now clones frommohit-sheth/e2e-benchmarkingon theuse-fork-kube-burner-ocpbranch. This enables testing against in-flight kube-burner-ocp enhancements before they merge upstream. The fork reference is intended to be reverted once the upstream kube-burner-ocp PR (referenced as#458) is merged.Testing Plan:
The PR uses pj-rehearse validation for both new test jobs to ensure they function correctly with the forked kube-burner-ocp changes.