Skip to content

INTEROP-9414: mpiit-data-router-reporter: add best_effort and retry logic - #83291

Closed
amp-rh wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:mpiit-data-router-reporter-best-effort
Closed

INTEROP-9414: mpiit-data-router-reporter: add best_effort and retry logic#83291
amp-rh wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:mpiit-data-router-reporter-best-effort

Conversation

@amp-rh

@amp-rh amp-rh commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The mpiit-data-router-reporter step currently has no retry logic and no best_effort flag, causing a 1-second Data Router 503 to fail an entire multi-hour CI job. This has generated 21 LPINTEROP tickets since Jan 2026 (~monthly Data Router outages).

Changes:

  • Add best_effort: true to the ref YAML, matching medik8s-polarion-reporter
  • Add retry logic (5 attempts, 2-min intervals) modeled on send-results-to-reportportal
  • Align shell options with repo convention (set -euo pipefail, no -x)

Jira: INTEROP-9414

Summary by CodeRabbit

The mpiit-data-router-reporter CI step now tolerates temporary Data Router failures. It retries uploads up to five times with two-minute intervals and enables best_effort execution so brief 503 responses do not interrupt long-running OpenShift CI jobs.

The script now uses repository-standard shell options and removes command tracing. It builds ReportPortal launch attributes once before the upload process.

The mpiit-data-router-reporter step currently has no retry logic and no
best_effort flag, causing a 1-second Data Router 503 to fail an entire
multi-hour CI job. This has generated 21 LPINTEROP tickets since Jan 2026
(~monthly Data Router outages).

Changes:
- Add best_effort: true to the ref YAML, matching medik8s-polarion-reporter
- Add retry logic (5 attempts, 2-min intervals) modeled on
  send-results-to-reportportal
- Align shell options with repo convention (set -euo pipefail, no -x)

Jira: LPINTEROP-6976
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 11, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 11, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

@amp-rh: This pull request references INTEROP-9414 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

The mpiit-data-router-reporter step currently has no retry logic and no best_effort flag, causing a 1-second Data Router 503 to fail an entire multi-hour CI job. This has generated 21 LPINTEROP tickets since Jan 2026 (~monthly Data Router outages).

Changes:

  • Add best_effort: true to the ref YAML, matching medik8s-polarion-reporter
  • Add retry logic (5 attempts, 2-min intervals) modeled on send-results-to-reportportal
  • Align shell options with repo convention (set -euo pipefail, no -x)

Jira: INTEROP-9414

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 9cb9611d-c82e-436b-86de-38d378767682

📥 Commits

Reviewing files that changed from the base of the PR and between f17b1f8 and 3896815.

📒 Files selected for processing (2)
  • ci-operator/step-registry/mpiit/data-router-reporter/mpiit-data-router-reporter-commands.sh
  • ci-operator/step-registry/mpiit/data-router-reporter/mpiit-data-router-reporter-ref.yaml

Walkthrough

The Data Router reporter now uses strict shell error handling, builds launch attributes once, retries uploads up to five times, and fails after exhausting retries. The step configuration enables best-effort execution.

Changes

Data Router Reporter Reliability

Layer / File(s) Summary
Reporter setup and launch attributes
ci-operator/step-registry/mpiit/data-router-reporter/mpiit-data-router-reporter-commands.sh
The script enables undefined-variable and pipeline-failure checks. It constructs ReportPortal launch attributes before uploading.
Upload retries and step execution
ci-operator/step-registry/mpiit/data-router-reporter/mpiit-data-router-reporter-commands.sh, ci-operator/step-registry/mpiit/data-router-reporter/mpiit-data-router-reporter-ref.yaml
Data Router uploads retry up to five times with 120-second delays. The script exits successfully after a successful upload and exits with status 1 after all attempts fail. The step enables best_effort: true.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely identifies the main changes: best-effort handling and retry logic for the mpiit-data-router-reporter step.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes only a shell script and reference YAML; the diff introduces no Ginkgo test constructs or test titles, so the check is not applicable.
Test Structure And Quality ✅ Passed The PR changes only a shell command file and YAML reference; the parent-to-HEAD diff adds no Ginkgo tests or cluster operations subject to this check.
Microshift Test Compatibility ✅ Passed The diff changes only a shell script and step YAML; it adds no Ginkgo e2e tests or MicroShift-incompatible API usage.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR changes only a shell script and reference YAML; the diff adds no Ginkgo e2e tests or test constructs requiring SNO compatibility review.
Topology-Aware Scheduling Compatibility ✅ Passed The diff only changes a CI step script and reference YAML; it adds retry logic and best_effort, with no node selectors, affinity, topology spread, replicas, tolerations, or PDB constraints.
Ote Binary Stdout Contract ✅ Passed The diff changes only a shell script and YAML; it adds no Go OTE binary or main/suite setup code, so the new shell echo statements are outside this check's scope.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The diff changes only a shell uploader and its YAML ref; it adds no Ginkgo e2e tests, IPv4 assumptions, or new external-connectivity test requirements.
No-Weak-Crypto ✅ Passed The parent-to-HEAD diff changes shell control flow, jq metadata, retries, and best_effort only; it introduces no weak crypto, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed The PR adds only retry logic and best_effort; the exact diff and changed YAML contain no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, root, or allowPrivilegeEscalation setting.
No-Sensitive-Data-In-Logs ✅ Passed The diff removes shell tracing and adds only static upload messages with attempt counts; no added log statement prints credentials, tokens, hostnames, or user data.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amp-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 11, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@amp-rh: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-RedHatQE-interop-testing-master-ibm-fusion-access-operator-ocp-4.23-lp-interop-ibm-fips N/A periodic Registry content changed
periodic-ci-quay-quay-tests-master-ocp-4.22-quay-lpGA-lp-ocp-compat-aws-fips N/A periodic Registry content changed
periodic-ci-openshift-pipelines-release-tests-release-v1.21-openshift-pipelines-ocp-4.21-lp-interop-cr-openshift-pipelines-interop-aws N/A periodic Registry content changed
periodic-ci-RedHatQE-interop-testing-master-cnv-odf-ocp-4.21-lp-interop-cr-cnv-component-readiness-aws-ipi-ocp421 N/A periodic Registry content changed
periodic-ci-redhat-developer-gitops-operator-v1.21-gitops-ocp-4.22-lpGA-lp-ocp-compat-cr--gitops--aws N/A periodic Registry content changed
periodic-ci-RedHatQE-interop-testing-master-cnv-odf-ocp-4.23-lp-interop-aws-fips N/A periodic Registry content changed
periodic-ci-oadp-qe-oadp-qe-automation-oadp-1.4-oadp1.4-ocp4.18-lp-interop-oadp-interop-aws N/A periodic Registry content changed
periodic-ci-oadp-qe-oadp-qe-automation-oadp-1.5-oadp1.5-ocp4.20-lp-interop-oadp-interop-aws N/A periodic Registry content changed
periodic-ci-openshift-pipelines-release-tests-release-v1.22-ocp-4.22-lpGA-lp-ocp-compat-cr--openshift-pipelines--aws N/A periodic Registry content changed
periodic-ci-quay-quay-tests-master-ocp-4.22-quay-lpGA-lp-ocp-compat-cr--quay--e2e-tests-aws N/A periodic Registry content changed
periodic-ci-red-hat-storage-ocs-ci-master-ocp-4.22-lpGA-lp-ocp-compat-cr--odf--aws N/A periodic Registry content changed
periodic-ci-stolostron-policy-collection-main-ocp5.0-interop-opp-vsphere N/A periodic Registry content changed
periodic-ci-RedHatQE-interop-testing-master-acm-virt-ocp4.22-p2p-cclm-liv-mig-lp-interop-aws N/A periodic Registry content changed
periodic-ci-oadp-qe-oadp-qe-automation-oadp-1.5-oadp1.5-ocp-4.21-lp-interop-cr-oadp-interop-aws N/A periodic Registry content changed
periodic-ci-oadp-qe-oadp-qe-automation-main-oadp1.4-ocp4.17-lp-interop-oadp-interop-aws N/A periodic Registry content changed
periodic-ci-stackrox-stackrox-master-ocp-4.21-lp-interop-cr-acs-latest-acs-tests-aws N/A periodic Registry content changed
periodic-ci-openshift-pipelines-release-tests-release-v1.21-ocp-4.22-lp-interop-cr-openshift-pipelines-aws N/A periodic Registry content changed
periodic-ci-quay-quay-tests-master-ocp-4.21-quay-cr-quay-e2e-tests-quay314-ocp421-lp-interop-aws N/A periodic Registry content changed
periodic-ci-stackrox-stackrox-release-4.10-ocp-4-22-lpGA-lp-ocp-compat-cr--acs--tests-aws N/A periodic Registry content changed
periodic-ci-RedHatQE-interop-testing-master-acm-cnv-ocp-4.22-p2p-lp-interop-aws N/A periodic Registry content changed
periodic-ci-konveyor-tackle2-ui-release-0.8-mta-ocp-4.21-lp-interop-cr-mta-interop-aws N/A periodic Registry content changed
periodic-ci-stolostron-policy-collection-main-ocp5.0-interop-opp-aws N/A periodic Registry content changed
periodic-ci-red-hat-storage-ocs-ci-master-ocp-4.22-lpGA-lp-ocp-compat-aws-fips N/A periodic Registry content changed
periodic-ci-redhat-developer-gitops-operator-v1.21-gitops-ocp-4.22-lpGA-lp-ocp-compat-aws-fips N/A periodic Registry content changed
periodic-ci-quay-quay-tests-master-ocp-4.23-quay-lp-interop-aws-fips N/A periodic Registry content changed

A total of 51 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@amp-rh

amp-rh commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Superseded.

Reason: Superseded by #83401 (moved to amp-rh fork, applied mpitt fixes)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants