Add new longduration Disconnected CI job for MCO - #82473
Conversation
openshift-ci
Bot
added
the
approved
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a disconnected AWS day-2 setup and workflow for MCO end-to-end testing. It schedules weekly, three-shard long-duration jobs for releases 4.22, 4.23, and 5.0. It also loads shared proxy settings before pull-secret operations. ChangesDisconnected MCO testing
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant PeriodicJob
participant DisconnectedWorkflow
participant Day2Setup
participant MCOE2ETests
PeriodicJob->>DisconnectedWorkflow: Start weekly three-shard job
DisconnectedWorkflow->>Day2Setup: Configure mirror access and image policies
Day2Setup->>DisconnectedWorkflow: Report MCP rollout status
DisconnectedWorkflow->>MCOE2ETests: Run MCO end-to-end tests
Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-1of2 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-1of2 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-2of2 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of2 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
The aws-mco-qe cluster profile has an ssh-privatekey in its vault secret, but it's in a format the SSH client doesn't recognize (Load key: invalid format). The openshift-e2e-aws-disconnected workflow needs the SSH key to be in OpenSSH PEM format (starts with -----BEGIN OPENSSH PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----) because it SSHs into the bastion host to check the mirror registry service. The key in the aws-mco-qe vault might be in a different format (e.g., PKCS#8, PuTTY .ppk, or possibly corrupted/encoded differently). The openshift-org-aws profile has a key in the correct format, which is why other jobs using So either:
|
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of2 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-fips-proxy-longduration-1of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-fips-proxy-longduration-1of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of2 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of2 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of2 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
dc86011 to
d2604d2
Compare
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of2 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of2 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
d2604d2 to
d896e22
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml`:
- Line 415: Resolve the shard-count mismatch by setting shard_count to 2 in both
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml:415-415
and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml:415-415,
preserving the objective’s two-shard configuration, then run make update.
- Line 417: Resolve the aws-mco-qe profile’s cluster-secrets-aws-mco-qe vault
secret by storing a valid literal OpenSSH private key with required newlines and
no passphrase; then run the configured make update generation if CI
configuration changes. Apply this shared profile fix for
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml
lines 401-417 and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
lines 401-417; no direct workflow change is needed unless the cluster is
unavailable, in which case select a compatible profile or workflow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: f06be080-5630-48de-ba58-1df646bc537e
⛔ Files ignored due to path filters (3)
ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (4)
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yamlci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yamlci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yamlci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
| - always_run: false | ||
| as: e2e-aws-mco-disconnected-longduration | ||
| cron: 17 6 * * 6 | ||
| shard_count: 3 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repo files around target:"
git ls-files | rg 'openshift-machine-config-operator-release-4\.2[23]__periodics\.yaml$'
echo
echo "Target sections:"
for f in \
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml \
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
do
echo "--- $f"
nl -ba "$f" | sed -n '380,430p'
done
echo
echo "Search for shard_count and relevant PR objective comments:"
rg -n "shard_count|two shards|two-shard|shard|objective|PR objective" ci-operator/config/openshift/machine-config-operator -S || true
echo
echo "Make/metadata files:"
git ls-files | rg '(^Makefile$|^\.github/workflows/.*\.ya?ml$|zz_generated_metadata|ci-operator)' | head -n 120Repository: openshift/release
Length of output: 592
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repo files around target:"
git ls-files | rg 'openshift-machine-config-operator-release-4\.2[23]__periodics\.yaml$'
echo
echo "Target sections:'
for f in \
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml \
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
do
echo "--- $f"
nl -ba "$f" | sed -n '380,430p'
done
echo
echo "Search for shard_count and relevant PR objective comments:'
rg -n "shard_count|two shards|two-shard|shard|objective|PR objective" ci-operator/config/openshift/machine-config-operator -S || true
echo
echo "Make/metadata files:"
git ls-files | rg '(^Makefile$|^\.github/workflows/.*\.ya?ml$|zz_generated_metadata|ci-operator)' | head -n 120Repository: openshift/release
Length of output: 468
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for p in [
Path("ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml"),
Path("ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml"),
]:
print(f"--- {p}")
lines = p.read_text().splitlines()
start = max(0, 415-30)
end = min(len(lines), 415+10)
for i in range(start, end):
print(f"{i+1:6d}\t{lines[i]}")
print("\nSearch:")
patterns = ["shard_count", "two shards", "two-shard", "three shards", "objective", "PR objective"]
for pat in patterns:
print(f"\n{pat}:")
for p in Path("ci-operator/config/openshift/machine-config-operator").glob("*__periodics.yaml"):
s = p.read_text(errors="ignore")
if pat in s or "shard" in pat:
hits = [n for n,l in enumerate(s.splitlines(),1) if pat in l]
if hits:
print(p)
PY
echo "Candidate generation/make files:"
git ls-files | rg '(^Makefile$|ci-operator/config/openshift/machine-config-operator/zz_generated|openshift-machine-config-operator-release-4\.2[23]__periodics\.yaml$)' || trueRepository: openshift/release
Length of output: 4270
Resolve the shard-count mismatch across both configurations.
The objective states two shards, but both entries generate three shards. Set shard_count: 2 in each periodics file if two shards are intended, or update the objective if three shards are intentional. Run make update after changing these CI entries.
📍 Affects 2 files
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml#L415-L415(this comment)ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L415-L415
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml`
at line 415, Resolve the shard-count mismatch by setting shard_count to 2 in
both
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml:415-415
and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml:415-415,
preserving the objective’s two-shard configuration, then run make update.
Source: Coding guidelines
| cron: 17 6 * * 6 | ||
| shard_count: 3 | ||
| steps: | ||
| cluster_profile: aws-mco-qe |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Target files:\n'
fd -a 'openshift-machine-config-operator-release-4.(22|23)__periodics\.yaml$' . || true
printf '\nGit status/diff stat:\n'
git status --short
git diff --stat || true
printf '\nRelevant lines:\n'
for f in $(fd 'openshift-machine-config-operator-release-4.(22|23)__periodics\.yaml$' .); do
echo "--- $f"
wc -l "$f"
sed -n '405,425p' "$f" | cat -n
done
printf '\nSearch for aws-mco-qe and cluster_profile in machine-config-operator periodic configs:\n'
rg -n 'aws-mco-qe|cluster_profile|ssh-privatekey|ssh-private|privatekey|mirror|bastion|disconnected' ci-operator/config/openshift/machine-config-operator || true
printf '\nSearch for vault references in repo:\n'
rg -n 'vault|vault/|quay.io/app-sre/vault|ssh-private' . || trueRepository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -euo pipefail
TARGETS=(
"ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml"
"ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml"
)
printf 'Target file existence/trimmed relevant sections:\n'
for f in "${TARGETS[@]}"; do
if [ -f "$f" ]; then
echo "--- $f"
rg -n '^cluster_profile:|cluster_profile:|aws-mco-qe|install:|release|tests:' "$f" -C 3
else
echo "MISSING $f"
fi
done
printf '\nCluster profile usage for aws-mco-qe in machine-config-operator configs:\n'
rg -n '^ (aws-mco-qe|mco-qe):|cluster_profile: aws-mco-qe|mco-qe:' ci-operator/config/openshift/machine-config-operator || true
printf '\nCI secret bootstrap entries whose folder name contains mco or mco-qe or aws-mco (bounded):\n'
python3 - <<'PY'
from pathlib import Path
p=Path("core-services/ci-secret-bootstrap/_config.yaml")
text=p.read_text()
lines=text.splitlines()
for i,line in enumerate(lines):
if "aws-mco-qe" in line or ("mco-qe" in line and ":" in line):
print(f"{p}:{i+1}:{line}")
for j in range(max(1,i-6), min(len(lines), i+15)+1):
print(f"{p}:{j+1}:{lines[j]}")
print("---")
PY
printf '\nTarget files in git status (names only):\n'
git status --short | grep -E 'openshift-machine-config-operator-release-4.(22|23)__periodics\.yaml$' || true
printf '\nTarget changed context with git diff (stat and names only):\n'
git diff --stat -- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml || true
git diff -- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml | sed -n '1,120p' || trueRepository: openshift/release
Length of output: 14715
🌐 Web query:
OpenShift CI cluster_profile aws-mco-qe SSH private key invalid format disconnected AWS bastion mirror registry ssh-privatekey
💡 Result:
The "invalid format" error for an ssh-privatekey in an OpenShift CI cluster_profile (such as aws-mco-qe) typically arises from how the private key content is stored and processed within the secret [1][2][3]. When configuring a cluster profile, the ssh-privatekey must contain the literal content of the SSH private key file [1]. The following factors are the most common causes for this error: 1. Missing or Incorrect Newline Characters: SSH keys often require a trailing newline character to be correctly parsed [2]. If the key is stored in a Kubernetes secret without the necessary line breaks (especially at the end of the -----END ... PRIVATE KEY----- block), the system may fail to parse the key format [2]. 2. Improper Encoding/Formatting in the Secret: The ssh-privatekey value must be the raw content of the private key, not a base64-encoded string (unless specifically required by the secret creation method) or a string with escaped newline characters (e.g., "\n" appearing as literal characters instead of actual line breaks) [1][2][4]. If the key was manually pasted or managed through a process that incorrectly handled whitespace or newlines, the format will be rejected [2][4]. 3. Passphrase Protection: SSH private keys used in automated CI/CD environments must not be passphrase-protected [4]. If the key requires a passphrase, the installer or automation tool will fail to load it, often resulting in an error that may be misreported as an invalid format [4]. To resolve this issue: - Verify the key file is in a standard OpenSSH format [1][2]. - Ensure the key does not have a passphrase [4]. - When updating the ssh-privatekey in the cluster_profile secret, ensure the content is stored exactly as it appears in the file, including all necessary newlines [1][2]. If using oc create secret or kubectl, ensure you are correctly pointing to the file so that the tool handles the content correctly, rather than manually pasting the string [5][6]. For disconnected AWS environments, ensure that any bastion host or mirror registry configuration that relies on this key uses the same properly formatted, non-passphrase-protected key to maintain access [7][8][9].
Citations:
- 1: https://docs.ci.openshift.org/how-tos/adding-a-cluster-profile/
- 2: git-clone cluster tasks throwing permission denied public key when using ssh basic-auth tektoncd/pipeline#3221
- 3: https://github.com/sallyom/installer-e2e
- 4: log redaction too aggressive for multi-line output from installer hive#878
- 5: https://bugzilla.redhat.com/show_bug.cgi?id=1819112
- 6: https://www.redhat.com/de/blog/deploying-from-private-git-repositories
- 7: https://docs.redhat.com/ko/documentation/openshift_container_platform/4.5/html/networking/accessing-hosts
- 8: https://docs.redhat.com/ko/documentation/openshift_container_platform/4.18/html/installing_on_aws/installer-provisioned-infrastructure
- 9: https://docs.redhat.com/ko/documentation/openshift_container_platform/4.21/html/disconnected_environments/installing-disconnected-environments
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'ci-secret-bootstrap aws-mco-qe section:\n'
sed -n '1145,1210p' core-services/ci-secret-bootstrap/_config.yaml | cat -n
printf '\nAll aws-mco-qe references:\n'
rg -n 'aws-mco-qe|cluster-secrets-aws-mco-qe' . || true
printf '\nTarget section context both files:\n'
for f in ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml; do
echo "--- $f"
sed -n '390,420p' "$f" | cat -n
done
printf '\nWorkflow/reference for disconnected AWS steps containing aws-mco-qe or MCO profile references:\n'
rg -n 'aws-mco-qe|MCO_CONF|mirror|bastion|ssh-privatekey|cluster-secrets-aws-mco-qe' ci-operator/step ci-operator/config/openshift/machine-config-operator || trueRepository: openshift/release
Length of output: 16274
Resolve the aws-mco-qe SSH-key secret.
aws-mco-qe is used by the disconnected MCO long-duration periodic workflow, but its secret (cluster-secrets-aws-mco-qe) contains an SSH private key rejected as invalid. These jobs use the same profile reference, so fixing the vault secret and regenerating CI config fixes both sites.
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml#L401-L417ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L401-L417
Fix the vault key by storing the literal OpenSSH private-key content with required newlines, no passphrase, and run the configured make update generation if CI config is re-generated.
Or select a compatible profile/workflow if this cluster is not available.
📍 Affects 2 files
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml#L417-L417(this comment)ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L417-L417
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml`
at line 417, Resolve the aws-mco-qe profile’s cluster-secrets-aws-mco-qe vault
secret by storing a valid literal OpenSSH private key with required newlines and
no passphrase; then run the configured make update generation if CI
configuration changes. Apply this shared profile fix for
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml
lines 401-417 and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
lines 401-417; no direct workflow change is needed unless the cluster is
unavailable, in which case select a compatible profile or workflow.
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
d896e22 to
abecfcf
Compare
|
/pj-reherse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 |
1 similar comment
|
/pj-reherse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-1of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-reherse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-reherse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 |
|
@ptalgulk01: job(s): periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-2of2 either don't exist or were not found to be affected, and cannot be rehearsed |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@ptalgulk01: job(s): , either don't exist or were not found to be affected, and cannot be rehearsed |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
5dffc04 to
eb0ae1e
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ptalgulk01 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
removed
the
approved
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
ci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-commands.sh (1)
16-22: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument why tracing is disabled.
Line 17 disables tracing while
registry_credis expanded. Add a comment that states this prevents credential data from appearing in logs when xtrace is enabled.As per coding guidelines, "keep the tracing-disabled scope minimal, add clear comments when disabling tracing."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-commands.sh` around lines 16 - 22, In the tracing control block around registry_cred in the disconnected setup command sequence, add a clear comment explaining that tracing is disabled to prevent expanded registry credentials from appearing in logs, while keeping the existing set +x and restoration scope unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-commands.sh`:
- Around line 90-100: Update the MCP worker rollout loop to capture
status.configuration.target before changes are applied, then require the current
target to differ from that captured value, updatedMachineCount to equal
machineCount, and the pool to have no degraded condition before exiting
successfully. Use the existing worker status polling in the day2 disconnected
setup script and retain the timeout behavior.
---
Nitpick comments:
In
`@ci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-commands.sh`:
- Around line 16-22: In the tracing control block around registry_cred in the
disconnected setup command sequence, add a clear comment explaining that tracing
is disabled to prevent expanded registry credentials from appearing in logs,
while keeping the existing set +x and restoration scope unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 2e68f891-71be-464e-b2d2-2fb203f27ecf
⛔ Files ignored due to path filters (3)
ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (12)
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yamlci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yamlci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yamlci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.shci-operator/step-registry/mco/conf/day2/disconnected-setup/OWNERSci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-commands.shci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-ref.metadata.jsonci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-ref.yamlci-operator/step-registry/ocp/mco/OWNERSci-operator/step-registry/ocp/mco/e2e/aws/disconnected/OWNERSci-operator/step-registry/ocp/mco/e2e/aws/disconnected/ocp-mco-e2e-aws-disconnected-workflow.metadata.jsonci-operator/step-registry/ocp/mco/e2e/aws/disconnected/ocp-mco-e2e-aws-disconnected-workflow.yaml
🚧 Files skipped from review as they are similar to previous changes (3)
- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml
- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-5.0__periodics.yaml
| echo "Waiting for MCP worker to apply configuration (up to 20 minutes)" | ||
| machine_count=$(oc get mcp worker -o=jsonpath='{.status.machineCount}') | ||
| elapsed=0 | ||
| while [[ ${elapsed} -lt 1200 ]]; do | ||
| sleep 20 | ||
| elapsed=$((elapsed + 20)) | ||
| updated=$(oc get mcp worker -o=jsonpath='{.status.updatedMachineCount}') | ||
| echo "Waiting ${elapsed}s — updated ${updated}/${machine_count}" | ||
| if [[ "${updated}" == "${machine_count}" ]]; then | ||
| echo "MCP worker updated successfully" | ||
| exit 0 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="ci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-commands.sh"
echo "== file exists and size =="
wc -l "$file" || true
echo "== relevant sections =="
sed -n '1,180p' "$file" | cat -n
echo "== occurrences of mcp worker/status/rendered-config/degraded =="
rg -n "machine_count|updatedMachineCount|rendered-config|degraded|applied|machine-config" "$file" || true
echo "== git diff stat/name =="
git diff --stat || true
git diff --name-only || true
if git diff -- "$file" >/tmp/mco.diff 2>/dev/null; then
sed -n '1,240p' /tmp/mco.diff | cat -n
fi
echo "== nearby step registry files =="
find ci-operator/step-registry/mco/conf/day2/disconnected-setup -maxdepth 1 -type f -print | sortRepository: openshift/release
Length of output: 5218
🌐 Web query:
OpenShift MachineConfigPool status machineCount updatedMachineCount renderedConfig degraded configGeneration
💡 Result:
In OpenShift, the MachineConfigPool (MCP) status provides observability into the state of the nodes managed by the Machine Config Operator (MCO) [1][2]. Key status fields include: machineCount: The total number of machines currently targeted by the MachineConfigPool [1][2]. updatedMachineCount: The total number of machines within the pool that have successfully applied the current desired MachineConfig [1][2]. When this value equals machineCount, all nodes in the pool are up-to-date [1]. degradedMachineCount: The total number of machines in the pool that are marked as degraded (e.g., failed to apply a configuration) [3][2]. renderedConfig: This field (often appearing as part of the status.configuration structure) refers to the final, combined MachineConfig object generated by the MCO for the pool [1][4]. It represents the "desired" configuration that nodes should be running [1]. configGeneration (observedGeneration): This field reflects the latest generation of the MachineConfigPool specification observed by the controller [1][2]. It is used to track whether the controller has processed the most recent changes made to the MCP object [1]. These fields collectively allow administrators to monitor update progress [1]: - If the current configuration (desired) matches the configuration on a node, the node is considered updated [1]. - If they do not match, the node is either in the process of updating or is degraded [1]. The overall state of the pool is further refined by conditions (such as MachineConfigPoolUpdated, MachineConfigPoolUpdating, or MachineConfigPoolNodeDegraded) that provide human-readable messages explaining why a pool might be in a specific state [4].
Citations:
- 1: https://github.com/openshift/machine-config-operator/blob/main/docs/MachineConfigController.md
- 2: https://docs.redhat.com/en/documentation/openshift_container_platform/4.15/html/machine_apis/machineconfigpool-machineconfiguration-openshift-io-v1
- 3: https://github.com/openshift/machine-config-operator/blob/master/pkg/controller/node/status.go
- 4: https://github.com/openshift/machine-config-operator/blob/master/pkg/operator/status.go
Wait for the new rendered worker configuration to roll out.
The initial updatedMachineCount == machineCount state can describe the old rendered config because machineCount and updatedMachineCount only differ during an in-progress update. If changes have not been observed within the first loop iteration, the script exits successfully and the next step starts before the new pull secret, CA, and mirror policy are applied on workers.
Capture the worker status.configuration.target value before applying changes, then require status.configuration.target to change, updatedMachineCount == machineCount, and no degraded pool condition before exiting successfully.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/mco/conf/day2/disconnected-setup/mco-conf-day2-disconnected-setup-commands.sh`
around lines 90 - 100, Update the MCP worker rollout loop to capture
status.configuration.target before changes are applied, then require the current
target to differ from that captured value, updatedMachineCount to equal
machineCount, and the pool to have no degraded condition before exiting
successfully. Use the existing worker status polling in the day2 disconnected
setup script and retain the timeout behavior.
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
eb0ae1e to
b66904a
Compare
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
b66904a to
16aac36
Compare
|
[REHEARSALNOTIFIER]
A total of 325 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-1of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-2of3 periodic-ci-openshift-machine-config-operator-release-4.23-periodics-e2e-aws-mco-disconnected-longduration-3of3 |
|
@ptalgulk01: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@ptalgulk01: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
New job: e2e-aws-mco-disconnected-longduration added to 3 periodics configs:
Job details:
Step registry fix: Updated mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh to source proxy-conf.sh for disconnected proxy support.
6 generated Prow jobs created across the 3 release versions (2 shards each).
Summary by CodeRabbit
Adds a weekly
e2e-aws-mco-disconnected-longdurationperiodic Prow job to the 4.22, 4.23, and 5.0 MCO release configurations.Runs the job Saturdays at
06:17 UTC(17 6 * * 6) withshard_count: 3on theaws-mco-qecluster profile and executes themco-e2e-testchain via the disconnected AWS workflow.Sets
TechPreviewNoUpgrade, disables audit-log monitoring, configures two compute replicas, applies the disconnected test skip (TEST_SKIPS: [Skipped:Disconnected]), and uses the long-duration MCO suite.Introduces the
ocp-mco-e2e-aws-disconnectedworkflow for disconnected AWS MCO end-to-end runs.Adds and wires a disconnected day-2 setup step (
mco-conf-day2-disconnected-setup) that updates the OpenShift pull-secret for the mirrored proxy Quay registry, trusts the mirror registry CA viaadditionalTrustedCA, and manages image mirroring through ICSP/IDMS forquay.io/openshifttest.Updates the MCO QE robot pull-secret command to source
${SHARED_DIR}/proxy-conf.shwhen present to support disconnected proxy configuration.Adds/updates
OWNERSand workflow step metadata for the new disconnected setup and the MCO component.