Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,22 @@ tests:
test:
- chain: mco-e2e-test
workflow: openshift-e2e-aws-proxy
- always_run: false
as: e2e-aws-mco-disconnected-longduration
cron: 17 6 * * 6
shard_count: 3

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Repo files around target:"
git ls-files | rg 'openshift-machine-config-operator-release-4\.2[23]__periodics\.yaml$'

echo
echo "Target sections:"
for f in \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
do
  echo "--- $f"
  nl -ba "$f" | sed -n '380,430p'
done

echo
echo "Search for shard_count and relevant PR objective comments:"
rg -n "shard_count|two shards|two-shard|shard|objective|PR objective" ci-operator/config/openshift/machine-config-operator -S || true

echo
echo "Make/metadata files:"
git ls-files | rg '(^Makefile$|^\.github/workflows/.*\.ya?ml$|zz_generated_metadata|ci-operator)' | head -n 120

Repository: openshift/release

Length of output: 592


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Repo files around target:"
git ls-files | rg 'openshift-machine-config-operator-release-4\.2[23]__periodics\.yaml$'

echo
echo "Target sections:'
for f in \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml \
  ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
do
  echo "--- $f"
  nl -ba "$f" | sed -n '380,430p'
done

echo
echo "Search for shard_count and relevant PR objective comments:'
rg -n "shard_count|two shards|two-shard|shard|objective|PR objective" ci-operator/config/openshift/machine-config-operator -S || true

echo
echo "Make/metadata files:"
git ls-files | rg '(^Makefile$|^\.github/workflows/.*\.ya?ml$|zz_generated_metadata|ci-operator)' | head -n 120

Repository: openshift/release

Length of output: 468


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
for p in [
    Path("ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml"),
    Path("ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml"),
]:
    print(f"--- {p}")
    lines = p.read_text().splitlines()
    start = max(0, 415-30)
    end = min(len(lines), 415+10)
    for i in range(start, end):
        print(f"{i+1:6d}\t{lines[i]}")

print("\nSearch:")
patterns = ["shard_count", "two shards", "two-shard", "three shards", "objective", "PR objective"]
for pat in patterns:
    print(f"\n{pat}:")
    for p in Path("ci-operator/config/openshift/machine-config-operator").glob("*__periodics.yaml"):
        s = p.read_text(errors="ignore")
        if pat in s or "shard" in pat:
            hits = [n for n,l in enumerate(s.splitlines(),1) if pat in l]
            if hits:
                print(p)
PY

echo "Candidate generation/make files:"
git ls-files | rg '(^Makefile$|ci-operator/config/openshift/machine-config-operator/zz_generated|openshift-machine-config-operator-release-4\.2[23]__periodics\.yaml$)' || true

Repository: openshift/release

Length of output: 4270


Resolve the shard-count mismatch across both configurations.

The objective states two shards, but both entries generate three shards. Set shard_count: 2 in each periodics file if two shards are intended, or update the objective if three shards are intentional. Run make update after changing these CI entries.

📍 Affects 2 files
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml#L415-L415 (this comment)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L415-L415
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml`
at line 415, Resolve the shard-count mismatch by setting shard_count to 2 in
both
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml:415-415
and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml:415-415,
preserving the objective’s two-shard configuration, then run make update.

Source: Coding guidelines

steps:
cluster_profile: aws-mco-qe

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Target files:\n'
fd -a 'openshift-machine-config-operator-release-4.(22|23)__periodics\.yaml$' . || true

printf '\nGit status/diff stat:\n'
git status --short
git diff --stat || true

printf '\nRelevant lines:\n'
for f in $(fd 'openshift-machine-config-operator-release-4.(22|23)__periodics\.yaml$' .); do
  echo "--- $f"
  wc -l "$f"
  sed -n '405,425p' "$f" | cat -n
done

printf '\nSearch for aws-mco-qe and cluster_profile in machine-config-operator periodic configs:\n'
rg -n 'aws-mco-qe|cluster_profile|ssh-privatekey|ssh-private|privatekey|mirror|bastion|disconnected' ci-operator/config/openshift/machine-config-operator || true

printf '\nSearch for vault references in repo:\n'
rg -n 'vault|vault/|quay.io/app-sre/vault|ssh-private' . || true

Repository: openshift/release

Length of output: 50374


🏁 Script executed:

#!/bin/bash
set -euo pipefail

TARGETS=(
  "ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml"
  "ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml"
)

printf 'Target file existence/trimmed relevant sections:\n'
for f in "${TARGETS[@]}"; do
  if [ -f "$f" ]; then
    echo "--- $f"
    rg -n '^cluster_profile:|cluster_profile:|aws-mco-qe|install:|release|tests:' "$f" -C 3
  else
    echo "MISSING $f"
  fi
done

printf '\nCluster profile usage for aws-mco-qe in machine-config-operator configs:\n'
rg -n '^  (aws-mco-qe|mco-qe):|cluster_profile: aws-mco-qe|mco-qe:' ci-operator/config/openshift/machine-config-operator || true

printf '\nCI secret bootstrap entries whose folder name contains mco or mco-qe or aws-mco (bounded):\n'
python3 - <<'PY'
from pathlib import Path
p=Path("core-services/ci-secret-bootstrap/_config.yaml")
text=p.read_text()
lines=text.splitlines()
for i,line in enumerate(lines):
    if "aws-mco-qe" in line or ("mco-qe" in line and ":" in line):
        print(f"{p}:{i+1}:{line}")
        for j in range(max(1,i-6), min(len(lines), i+15)+1):
            print(f"{p}:{j+1}:{lines[j]}")
        print("---")
PY

printf '\nTarget files in git status (names only):\n'
git status --short | grep -E 'openshift-machine-config-operator-release-4.(22|23)__periodics\.yaml$' || true

printf '\nTarget changed context with git diff (stat and names only):\n'
git diff --stat -- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml || true
git diff -- ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml | sed -n '1,120p' || true

Repository: openshift/release

Length of output: 14715


🌐 Web query:

OpenShift CI cluster_profile aws-mco-qe SSH private key invalid format disconnected AWS bastion mirror registry ssh-privatekey

💡 Result:

The "invalid format" error for an ssh-privatekey in an OpenShift CI cluster_profile (such as aws-mco-qe) typically arises from how the private key content is stored and processed within the secret [1][2][3]. When configuring a cluster profile, the ssh-privatekey must contain the literal content of the SSH private key file [1]. The following factors are the most common causes for this error: 1. Missing or Incorrect Newline Characters: SSH keys often require a trailing newline character to be correctly parsed [2]. If the key is stored in a Kubernetes secret without the necessary line breaks (especially at the end of the -----END ... PRIVATE KEY----- block), the system may fail to parse the key format [2]. 2. Improper Encoding/Formatting in the Secret: The ssh-privatekey value must be the raw content of the private key, not a base64-encoded string (unless specifically required by the secret creation method) or a string with escaped newline characters (e.g., "\n" appearing as literal characters instead of actual line breaks) [1][2][4]. If the key was manually pasted or managed through a process that incorrectly handled whitespace or newlines, the format will be rejected [2][4]. 3. Passphrase Protection: SSH private keys used in automated CI/CD environments must not be passphrase-protected [4]. If the key requires a passphrase, the installer or automation tool will fail to load it, often resulting in an error that may be misreported as an invalid format [4]. To resolve this issue: - Verify the key file is in a standard OpenSSH format [1][2]. - Ensure the key does not have a passphrase [4]. - When updating the ssh-privatekey in the cluster_profile secret, ensure the content is stored exactly as it appears in the file, including all necessary newlines [1][2]. If using oc create secret or kubectl, ensure you are correctly pointing to the file so that the tool handles the content correctly, rather than manually pasting the string [5][6]. For disconnected AWS environments, ensure that any bastion host or mirror registry configuration that relies on this key uses the same properly formatted, non-passphrase-protected key to maintain access [7][8][9].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'ci-secret-bootstrap aws-mco-qe section:\n'
sed -n '1145,1210p' core-services/ci-secret-bootstrap/_config.yaml | cat -n

printf '\nAll aws-mco-qe references:\n'
rg -n 'aws-mco-qe|cluster-secrets-aws-mco-qe' . || true

printf '\nTarget section context both files:\n'
for f in ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml; do
  echo "--- $f"
  sed -n '390,420p' "$f" | cat -n
done

printf '\nWorkflow/reference for disconnected AWS steps containing aws-mco-qe or MCO profile references:\n'
rg -n 'aws-mco-qe|MCO_CONF|mirror|bastion|ssh-privatekey|cluster-secrets-aws-mco-qe' ci-operator/step ci-operator/config/openshift/machine-config-operator || true

Repository: openshift/release

Length of output: 16274


Resolve the aws-mco-qe SSH-key secret.

aws-mco-qe is used by the disconnected MCO long-duration periodic workflow, but its secret (cluster-secrets-aws-mco-qe) contains an SSH private key rejected as invalid. These jobs use the same profile reference, so fixing the vault secret and regenerating CI config fixes both sites.

  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml#L401-L417
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L401-L417

Fix the vault key by storing the literal OpenSSH private-key content with required newlines, no passphrase, and run the configured make update generation if CI config is re-generated.

Or select a compatible profile/workflow if this cluster is not available.

📍 Affects 2 files
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml#L417-L417 (this comment)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml#L417-L417
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml`
at line 417, Resolve the aws-mco-qe profile’s cluster-secrets-aws-mco-qe vault
secret by storing a valid literal OpenSSH private key with required newlines and
no passphrase; then run the configured make update generation if CI
configuration changes. Apply this shared profile fix for
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.22__periodics.yaml
lines 401-417 and
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-release-4.23__periodics.yaml
lines 401-417; no direct workflow change is needed unless the cluster is
unavailable, in which case select a compatible profile or workflow.

env:
BASE_DOMAIN: ocp-mco-qe.devcluster.openshift.com
COMPUTE_NODE_REPLICAS: "2"
FEATURE_SET: TechPreviewNoUpgrade
TEST_ARGS: --disable-monitor=audit-log-analyzer
TEST_SKIPS: \[Skipped:Disconnected\]
TEST_SUITE: openshift/machine-config-operator/longduration
test:
- chain: mco-e2e-test
workflow: ocp-mco-e2e-aws-disconnected
- always_run: false
as: e2e-aws-ovn-day1-ocl
capabilities:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,22 @@ tests:
test:
- chain: mco-e2e-test
workflow: openshift-e2e-aws-proxy
- always_run: false
as: e2e-aws-mco-disconnected-longduration
cron: 17 6 * * 6
shard_count: 3
steps:
cluster_profile: aws-mco-qe
env:
BASE_DOMAIN: ocp-mco-qe.devcluster.openshift.com
COMPUTE_NODE_REPLICAS: "2"
FEATURE_SET: TechPreviewNoUpgrade
TEST_ARGS: --disable-monitor=audit-log-analyzer
TEST_SKIPS: \[Skipped:Disconnected\]
TEST_SUITE: openshift/machine-config-operator/longduration
test:
- chain: mco-e2e-test
workflow: ocp-mco-e2e-aws-disconnected
- always_run: false
as: e2e-aws-ovn-day1-ocl
capabilities:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,22 @@ tests:
test:
- chain: mco-e2e-test
workflow: openshift-e2e-aws-proxy
- always_run: false
as: e2e-aws-mco-disconnected-longduration
cron: 17 6 * * 6
shard_count: 3
steps:
cluster_profile: aws-mco-qe
env:
BASE_DOMAIN: ocp-mco-qe.devcluster.openshift.com
COMPUTE_NODE_REPLICAS: "2"
FEATURE_SET: TechPreviewNoUpgrade
TEST_ARGS: --disable-monitor=audit-log-analyzer
TEST_SKIPS: \[Skipped:Disconnected\]
TEST_SUITE: openshift/machine-config-operator/longduration
test:
- chain: mco-e2e-test
workflow: ocp-mco-e2e-aws-disconnected
- always_run: false
as: e2e-aws-ovn-day1-ocl
capabilities:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,273 @@ periodics:
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cron: 17 6 * * 6
decorate: true
decoration_config:
sparse_checkout_files:
- .ci-operator.yaml
- Dockerfile.rhel7
extra_refs:
- base_ref: release-4.22
org: openshift
repo: machine-config-operator
sparse_checkout_files:
- .ci-operator.yaml
- Dockerfile.rhel7
labels:
ci-operator.openshift.io/cloud: aws
ci-operator.openshift.io/cloud-cluster-profile: aws-mco-qe
ci-operator.openshift.io/variant: periodics
ci.openshift.io/generator: prowgen
job-release: "4.22"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-1of3
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --multi-stage-param=SHARD_ARGS="--shard-count 3 --shard-id 1"
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=e2e-aws-mco-disconnected-longduration
- --variant=periodics
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/ci-pull-credentials
name: ci-pull-credentials
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: ci-pull-credentials
secret:
secretName: ci-pull-credentials
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cron: 17 6 * * 6
decorate: true
decoration_config:
sparse_checkout_files:
- .ci-operator.yaml
- Dockerfile.rhel7
extra_refs:
- base_ref: release-4.22
org: openshift
repo: machine-config-operator
sparse_checkout_files:
- .ci-operator.yaml
- Dockerfile.rhel7
labels:
ci-operator.openshift.io/cloud: aws
ci-operator.openshift.io/cloud-cluster-profile: aws-mco-qe
ci-operator.openshift.io/variant: periodics
ci.openshift.io/generator: prowgen
job-release: "4.22"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-2of3
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --multi-stage-param=SHARD_ARGS="--shard-count 3 --shard-id 2"
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=e2e-aws-mco-disconnected-longduration
- --variant=periodics
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/ci-pull-credentials
name: ci-pull-credentials
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: ci-pull-credentials
secret:
secretName: ci-pull-credentials
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cron: 17 6 * * 6
decorate: true
decoration_config:
sparse_checkout_files:
- .ci-operator.yaml
- Dockerfile.rhel7
extra_refs:
- base_ref: release-4.22
org: openshift
repo: machine-config-operator
sparse_checkout_files:
- .ci-operator.yaml
- Dockerfile.rhel7
labels:
ci-operator.openshift.io/cloud: aws
ci-operator.openshift.io/cloud-cluster-profile: aws-mco-qe
ci-operator.openshift.io/variant: periodics
ci.openshift.io/generator: prowgen
job-release: "4.22"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-aws-mco-disconnected-longduration-3of3
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --multi-stage-param=SHARD_ARGS="--shard-count 3 --shard-id 3"
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=e2e-aws-mco-disconnected-longduration
- --variant=periodics
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/ci-pull-credentials
name: ci-pull-credentials
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: ci-pull-credentials
secret:
secretName: ci-pull-credentials
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
decorate: true
Expand Down
Loading