Skip to content

Conversation

isabella-janssen
Copy link
Member

@isabella-janssen isabella-janssen commented Mar 31, 2025

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and RELOADEDCRIO should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-110-176.us-west-2.compute.internal
Name:         ip-10-0-110-176.us-west-2.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
  Creation Timestamp:  2025-04-03T22:08:41Z
  Generation:          2
  Owner References:
    API Version:     v1
    Kind:            Node
    Name:            ip-10-0-110-176.us-west-2.compute.internal
    UID:             f3d946e9-24d4-4b96-b4a7-4536d79c73a8
  Resource Version:  18006
  UID:               7f03657f-d52a-4a48-b835-370f22fc773a
Spec:
  Config Version:
    Desired:  rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
  Node:
    Name:  ip-10-0-110-176.us-west-2.compute.internal
  Pool:
    Name:  worker
Status:
  Conditions:
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               All pinned image sets complete
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsProgressing
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               This node has not yet entered the UpdatePrepared phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  UpdatePrepared
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               This node has not yet entered the UpdateExecuted phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  UpdateExecuted
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               This node has not yet entered the UpdatePostActionComplete phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  UpdatePostActionComplete
    Last Transition Time:  2025-04-03T22:09:55Z
    Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: Uncordoned Node as part of completing upgrade phase
    Reason:                Uncordoned
    Status:                False
    Type:                  UpdateComplete
    Last Transition Time:  2025-04-03T22:09:55Z
    Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: In desired config . Resumed normal operations.
    Reason:                Resumed
    Status:                False
    Type:                  Resumed
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               This node has not yet entered the Drained phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  Drained
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               This node has not yet entered the AppliedFilesAndOS phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  AppliedFilesAndOS
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               This node has not yet entered the Cordoned phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  Cordoned
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               This node has not yet entered the RebootedNode phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  RebootedNode
    Last Transition Time:  2025-04-03T22:09:55Z
    Message:               Node ip-10-0-110-176.us-west-2.compute.internal Updated
    Reason:                Updated
    Status:                True
    Type:                  Updated
    Last Transition Time:  2025-04-03T22:09:55Z
    Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: UnCordoned node. The node is reporting Unschedulable = false
    Reason:                UpdateCompleteUncordoned
    Status:                False
    Type:                  Uncordoned
    Last Transition Time:  2025-04-03T22:08:41Z
    Message:               All is good
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsDegraded
  Config Version:
    Current:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
    Desired:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
  Observed Generation:  3
Events:                 <none>
  1. Apply a valid MC and watch the MCN object throughout the update to see MachineConfigNodeUpdateRebooted as a parent phase.
  • Note that there are now two update paths:
    • If an update requires a reboot, the post action complete phase will not transition through the unknown --> true --> false phase, only the rebooted node phase will.
    • If an update does not require a reboot, the post action complete phase will transition through the unknown --> true --> false phase, and the rebooted node phase will not.
  • To see the following outcomes, apply an MC that requires a node reboot.
  • To see the post action complete phase transition, trigger an update that will not cause a reboot of the node (ex. a node disruption policy update).

Before the update

    Last Transition Time:  2025-04-03T22:08:12Z
    Message:               This node has not yet entered the RebootedNode phase
    Reason:                NotYetOccurred
    Status:                False
    Type:                  RebootedNode

During the reboot

    Message:               Upgrade requires a reboot.
    Reason:                RebootedNode
    Status:                Unknown
    Type:                  RebootedNode

After successful reboot

    Message:               Node has rebooted
    Reason:                RebootedNode
    Status:                True
    Type:                  RebootedNode
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  name: test-pinned
  labels:
    machineconfiguration.openshift.io/role: "worker"
spec:
  pinnedImages:
   - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
   - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
  Conditions:
    Last Transition Time:  2025-04-03T12:27:06Z
    Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
    Reason:                ImagePrefetch
    Status:                True
    Type:                  PinnedImageSetsProgressing
...
    Last Transition Time:  2025-04-03T12:27:07Z
    Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
    Reason:                PrefetchFailed
    Status:                True
    Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
    Desired Generation:            2
    Last Failed Generation:        2
    Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
    Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  name: test-pinned
  labels:
    machineconfiguration.openshift.io/role: "worker"
spec:
  pinnedImages:
   - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
   - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
  Conditions:
    Last Transition Time:  2025-04-03T11:53:06Z
    Message:               All pinned image sets complete
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsProgressing
...
    Last Transition Time:  2025-04-03T11:53:06Z
    Message:               All is good
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsDegraded
...
  Pinned Image Sets:
    Current Generation:            1
    Desired Generation:            1
    Last Failed Generation:        0
    Last Failed Generation Error:  
    Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 31, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it
Launch cluster with this code in tech preview & observe updated fields in MCN objects.

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 31, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.
    UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.
$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.
  • UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.
$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.
  • UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.
$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
  • The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.
$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-21-192.ec2.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T16:01:49Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-21-192.ec2.internal
   UID:             f5db5ed9-70f7-4e53-8e8c-934f8af55a42
 Resource Version:  18523
 UID:               4f9a956a-1ed9-4a1b-bbb0-5955c59c0570
Spec:
 Config Version:
   Desired:  rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Node:
   Name:  ip-10-0-21-192.ec2.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateCompatible phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateCompatible
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the ReloadedCRIO phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  ReloadedCRIO
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Node ip-10-0-21-192.ec2.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
   Desired:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.
  • UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.
$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
  • The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.
$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-21-192.ec2.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T16:01:49Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-21-192.ec2.internal
   UID:             f5db5ed9-70f7-4e53-8e8c-934f8af55a42
 Resource Version:  18523
 UID:               4f9a956a-1ed9-4a1b-bbb0-5955c59c0570
Spec:
 Config Version:
   Desired:  rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Node:
   Name:  ip-10-0-21-192.ec2.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateCompatible phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateCompatible
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the ReloadedCRIO phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  ReloadedCRIO
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Node ip-10-0-21-192.ec2.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
   Desired:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-21-192.ec2.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T16:01:49Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-21-192.ec2.internal
   UID:             f5db5ed9-70f7-4e53-8e8c-934f8af55a42
 Resource Version:  18523
 UID:               4f9a956a-1ed9-4a1b-bbb0-5955c59c0570
Spec:
 Config Version:
   Desired:  rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Node:
   Name:  ip-10-0-21-192.ec2.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateCompatible phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateCompatible
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the ReloadedCRIO phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  ReloadedCRIO
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Node ip-10-0-21-192.ec2.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
   Desired:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-21-192.ec2.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T16:01:49Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-21-192.ec2.internal
   UID:             f5db5ed9-70f7-4e53-8e8c-934f8af55a42
 Resource Version:  18523
 UID:               4f9a956a-1ed9-4a1b-bbb0-5955c59c0570
Spec:
 Config Version:
   Desired:  rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Node:
   Name:  ip-10-0-21-192.ec2.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the UpdateCompatible phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateCompatible
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               This node has not yet entered the ReloadedCRIO phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  ReloadedCRIO
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Node ip-10-0-21-192.ec2.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T16:02:56Z
   Message:               Action during update to rendered-worker-7123333eae329450f3da4a71e20ea7ee: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T16:01:57Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
   Desired:            rendered-worker-7123333eae329450f3da4a71e20ea7ee
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 31, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 2, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                        POOLNAME    DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 2, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

  • Version bumps
  • Pull in changes made to the MCN API.
    • Remove references to statuses MachineConfigNodeUpdateCompatible and MachineConfigNodeUpdateReloaded.
    • Remove references to LastFailedGenerationErrors field. Add new LastFailedGenerationError field.
    • Generalize the message when MachineConfigNodePinnedImageSetsDegraded condition is True.

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@isabella-janssen isabella-janssen force-pushed the mco-1544-apiBump branch 3 times, most recently from 132646e to cdae5f0 Compare April 2, 2025 20:20
@RishabhSaini
Copy link
Contributor

RishabhSaini commented Apr 2, 2025

@isabella-janssen you would need to remove this as well since the MCN.Spec.PIS does not exist anymore:
https://github.com/openshift/machine-config-operator/pull/4962/files#diff-be61a7dfe9d23b169ca1c6cba268550fbfd3146163faaebafa3635bd3f759f27L1178-R1181

func getPinnedImageSetSpecForPools(pools []*mcfgv1.MachineConfigPool) []mcfgv1alpha1.MachineConfigNodeSpecPinnedImageSet {

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2025
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 2, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code with tech preview enabled. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should report as before.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.


- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should report as before.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.


- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.


- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:            1
   Desired Generation:            1
   Last Failed Generation:        0
   Last Failed Generation Error:  
   Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

TODO:

  • Add info about change to having MachineConfigNodeUpdateRebooted be a parent phase and how to test.
  • Update the MCN enhancement to reflect the MachineConfigNodeUpdateRebooted change.

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:            1
   Desired Generation:            1
   Last Failed Generation:        0
   Last Failed Generation Error:  
   Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

TODO:

  • Add info about change to having MachineConfigNodeUpdateRebooted be a parent phase and how to test.
  • Update the MCN enhancement to reflect the MachineConfigNodeUpdateRebooted change.
  • Squash commits

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-21-192.ec2.internal
Name:         ip-10-0-5-15.us-west-1.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-03-31T19:27:51Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-5-15.us-west-1.compute.internal
   UID:             33658233-b500-46f2-a488-0c017fb4918e
 Resource Version:  26919
 UID:               bf38d7f7-9b19-4c70-86bc-b9646daee606
Spec:
 Config Version:
   Desired:  rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Node:
   Name:  ip-10-0-5-15.us-west-1.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Node ip-10-0-5-15.us-west-1.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-03-31T19:28:21Z
   Message:               Action during update to rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsDegraded phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsDegraded
   Last Transition Time:  2025-03-31T19:28:11Z
   Message:               This node has not yet entered the PinnedImageSetsProgressing phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  PinnedImageSetsProgressing
 Config Version:
   Current:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
   Desired:            rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99
 Observed Generation:  3
Events:                 <none>
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:            1
   Desired Generation:            1
   Last Failed Generation:        0
   Last Failed Generation Error:  
   Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

TODO:

  • Add info about change to having MachineConfigNodeUpdateRebooted be a parent phase and how to test.
  • Update the MCN enhancement to reflect the MachineConfigNodeUpdateRebooted change.
  • Squash commits

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-110-176.us-west-2.compute.internal
Name:         ip-10-0-110-176.us-west-2.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-04-03T22:08:41Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-110-176.us-west-2.compute.internal
   UID:             f3d946e9-24d4-4b96-b4a7-4536d79c73a8
 Resource Version:  18006
 UID:               7f03657f-d52a-4a48-b835-370f22fc773a
Spec:
 Config Version:
   Desired:  rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Node:
   Name:  ip-10-0-110-176.us-west-2.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Node ip-10-0-110-176.us-west-2.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
   Desired:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Observed Generation:  3
Events:                 <none>
  1. Apply a valid MC and watch the MCN object throughout the update to see MachineConfigNodeUpdateRebooted as a parent phase.

Before the update

   Last Transition Time:  2025-04-03T22:08:12Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode

During the reboot

   Message:               Upgrade requires a reboot.
   Reason:                RebootedNode
   Status:                Unknown
   Type:                  RebootedNode

After successful reboot

   Message:               Node has rebooted
   Reason:                RebootedNode
   Status:                True
   Type:                  RebootedNode
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:            1
   Desired Generation:            1
   Last Failed Generation:        0
   Last Failed Generation Error:  
   Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 3, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

TODO:

  • Add info about change to having MachineConfigNodeUpdateRebooted be a parent phase and how to test.
  • Update the MCN enhancement to reflect the MachineConfigNodeUpdateRebooted change.
  • Squash commits

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-110-176.us-west-2.compute.internal
Name:         ip-10-0-110-176.us-west-2.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-04-03T22:08:41Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-110-176.us-west-2.compute.internal
   UID:             f3d946e9-24d4-4b96-b4a7-4536d79c73a8
 Resource Version:  18006
 UID:               7f03657f-d52a-4a48-b835-370f22fc773a
Spec:
 Config Version:
   Desired:  rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Node:
   Name:  ip-10-0-110-176.us-west-2.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Node ip-10-0-110-176.us-west-2.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
   Desired:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Observed Generation:  3
Events:                 <none>
  1. Apply a valid MC and watch the MCN object throughout the update to see MachineConfigNodeUpdateRebooted as a parent phase.

Before the update

   Last Transition Time:  2025-04-03T22:08:12Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode

During the reboot

   Message:               Upgrade requires a reboot.
   Reason:                RebootedNode
   Status:                Unknown
   Type:                  RebootedNode

After successful reboot

   Message:               Node has rebooted
   Reason:                RebootedNode
   Status:                True
   Type:                  RebootedNode
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:            1
   Desired Generation:            1
   Last Failed Generation:        0
   Last Failed Generation Error:  
   Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 4, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-110-176.us-west-2.compute.internal
Name:         ip-10-0-110-176.us-west-2.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-04-03T22:08:41Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-110-176.us-west-2.compute.internal
   UID:             f3d946e9-24d4-4b96-b4a7-4536d79c73a8
 Resource Version:  18006
 UID:               7f03657f-d52a-4a48-b835-370f22fc773a
Spec:
 Config Version:
   Desired:  rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Node:
   Name:  ip-10-0-110-176.us-west-2.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Node ip-10-0-110-176.us-west-2.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
   Desired:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Observed Generation:  3
Events:                 <none>
  1. Apply a valid MC and watch the MCN object throughout the update to see MachineConfigNodeUpdateRebooted as a parent phase.

Before the update

   Last Transition Time:  2025-04-03T22:08:12Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode

During the reboot

   Message:               Upgrade requires a reboot.
   Reason:                RebootedNode
   Status:                Unknown
   Type:                  RebootedNode

After successful reboot

   Message:               Node has rebooted
   Reason:                RebootedNode
   Status:                True
   Type:                  RebootedNode
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:            1
   Desired Generation:            1
   Last Failed Generation:        0
   Last Failed Generation Error:  
   Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link
Contributor

@djoshy djoshy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me, thanks for the fixups!

Will do final lgtm tag after pre merge QE testing

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 4, 2025
@isabella-janssen
Copy link
Member Author

/retest-required

2 similar comments
@isabella-janssen
Copy link
Member Author

/retest-required

@isabella-janssen
Copy link
Member Author

/retest-required

@ptalgulk01
Copy link

Pre-merge verification steps:
Verified using IPI based AWS TP cluster

  • Applied the custom MCP
Custom MCP template
$oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: ""
EOF
machineconfigpool.machineconfiguration.openshift.io/infra created

$ oc label node/ip-10-0-118-194.us-east-2.compute.internal node-role.kubernetes.io/infra=
node/ip-10-0-118-194.us-east-2.compute.internal labeled

  • Check the infra label is added in MCN too
$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-118-194.us-east-2.compute.internal   infra      rendered-infra-da792628ef260f08e7cd8e3c007f3676    rendered-infra-da792628ef260f08e7cd8e3c007f3676    True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-119-206.us-east-2.compute.internal   master     rendered-master-c21ec021d9c9a6c7429aa1f2c5b5041e   rendered-master-c21ec021d9c9a6c7429aa1f2c5b5041e   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-120-15.us-east-2.compute.internal    worker     rendered-worker-da792628ef260f08e7cd8e3c007f3676   rendered-worker-da792628ef260f08e7cd8e3c007f3676   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-122-238.us-east-2.compute.internal   master     rendered-master-c21ec021d9c9a6c7429aa1f2c5b5041e   rendered-master-c21ec021d9c9a6c7429aa1f2c5b5041e   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-35-23.us-east-2.compute.internal     worker     rendered-worker-da792628ef260f08e7cd8e3c007f3676   rendered-worker-da792628ef260f08e7cd8e3c007f3676   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-55-132.us-east-2.compute.internal    master     rendered-master-c21ec021d9c9a6c7429aa1f2c5b5041e   rendered-master-c21ec021d9c9a6c7429aa1f2c5b5041e   True      False            False            False                      False            False     False               False          False         False          False
  • The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.
$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o name)
do
   n=${node/node\//}
   echo $n
      oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="UpdateCompatible")]}' | jq;
oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="ReloadedCRIO")]}' | jq;
done
ip-10-0-22-61.us-west-2.compute.internal
ip-10-0-76-141.us-west-2.compute.internal
ip-10-0-80-224.us-west-2.compute.internal

$ for node in $(oc get nodes -l node-role.kubernetes.io/master -o name)
do
   n=${node/node\//}
   echo $n
      oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="UpdateCompatible")]}' | jq;
oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="ReloadedCRIO")]}' | jq;
done
ip-10-0-16-88.us-west-2.compute.internal
ip-10-0-67-129.us-west-2.compute.internal
ip-10-0-82-37.us-west-2.compute.internal
  • Apply the MC on worker pool
MC template
 oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: test-file
spec:
  config:
    ignition:
      version: 3.4.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,dGVzdA==
        filesystem: root
        mode: 420
        path: /etc/test-file.test
EOF
machineconfig.machineconfiguration.openshift.io/test-file created
  • Check the status before, during and after update
Logs Before
$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o name)
do
     n=${node/node\//}
     echo $n
     oc get machineconfignode $n
     oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="RebootedNode")]}' | jq
done
ip-10-0-118-194.us-east-2.compute.internal
NAME                                         POOLNAME   DESIREDCONFIG                                     CURRENTCONFIG                                     UPDATED
ip-10-0-118-194.us-east-2.compute.internal   infra      rendered-infra-bdb870f430ab906759dac44c2dd69453   rendered-infra-da792628ef260f08e7cd8e3c007f3676   False
{
  "lastTransitionTime": "2025-04-08T14:28:30Z",
  "message": "This node has not yet entered the RebootedNode phase",
  "reason": "NotYetOccurred",
  "status": "False",
  "type": "RebootedNode"
}
…..
During
$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o name)
do
     n=${node/node\//}
     echo $n
     oc get machineconfignode $n -o wide
     oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="RebootedNode")]}' | jq
done
ip-10-0-118-194.us-east-2.compute.internal
NAME                                         POOLNAME   DESIREDCONFIG                                     CURRENTCONFIG                                     UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-118-194.us-east-2.compute.internal   infra      rendered-infra-bdb870f430ab906759dac44c2dd69453   rendered-infra-da792628ef260f08e7cd8e3c007f3676   False     True             True             False                      False            False     True                True           True          Unknown        False
{
  "lastTransitionTime": "2025-04-08T16:50:17Z",
  "message": "Upgrade requires a reboot.",
  "reason": "RebootedNode",
  "status": "Unknown",
  "type": "RebootedNode"
}
……
After
$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o name)
do
     n=${node/node\//}
     echo $n
     oc get machineconfignode $n -o wide
     oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="RebootedNode")]}' | jq
done
ip-10-0-118-194.us-east-2.compute.internal
NAME                                         POOLNAME   DESIREDCONFIG                                     CURRENTCONFIG                                     UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-118-194.us-east-2.compute.internal   infra      rendered-infra-bdb870f430ab906759dac44c2dd69453   rendered-infra-bdb870f430ab906759dac44c2dd69453   True      False            False            False                      False            False     False               False          False         False          False
{
  "lastTransitionTime": "2025-04-08T16:51:36Z",
  "message": "Action during update to rendered-infra-bdb870f430ab906759dac44c2dd69453: Node has rebooted",
  "reason": "RebootedNode",
  "status": "False",
  "type": "RebootedNode"
}
….
  • Apply the wrong MC
Wrong MC template
 oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: infra
  name: test-file-infra
spec:
  config:
    ignition:
      version: 3.4.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,dGVzdA==
        filesystem: root
        mode: 420
        path: /etc2/test-file.test
EOF
machineconfig.machineconfiguration.openshift.io/test-file-infra created
$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-bdb870f430ab906759dac44c2dd69453    False     True       True       1              0                   0                     1                      102m
master   rendered-master-c21ec021d9c9a6c7429aa1f2c5b5041e   True      False      False      3              3                   3                     0                      159m
worker   rendered-worker-bdb870f430ab906759dac44c2dd69453   True      False      False      2              2                   2                     0                      159m
  • Check No logs related to degrade is seen in MCN
oc describe machineconfignode ip-10-0-31-154.us-west-2.compute.internal
Name:         ip-10-0-31-154.us-west-2.compute.internal
Namespace:    
Labels:       
Annotations:  
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
  Creation Timestamp:  2025-04-09T03:55:38Z
  Generation:          6
  Owner References:
    API Version:     v1
    Kind:            Node
    Name:            ip-10-0-31-154.us-west-2.compute.internal
    UID:             dde5edc3-565b-4e43-9427-403cc315c46e
  Resource Version:  79694
  UID:               277b2f1b-2c40-4f6e-8d12-30cb8aaaca9f
Spec:
  Config Version:
    Desired:  rendered-infra-d58c96878f0bde7ff712d0e5767fdf9b
  Node:
    Name:  ip-10-0-31-154.us-west-2.compute.internal
  Pool:
    Name:  infra
Status:
  Conditions:
    Last Transition Time:  2025-04-09T06:01:53Z
    Message:               All pinned image sets complete
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsProgressing
    Last Transition Time:  2025-04-09T06:13:41Z
    Message:               Update Compatible. Post Cfg Actions: [] Drain Required: true
    Reason:                UpdatePrepared
    Status:                True
    Type:                  UpdatePrepared
    Last Transition Time:  2025-04-09T06:16:21Z
    Message:               Updating the Files and OS on disk as a part of the in progress phase
    Reason:                AppliedFilesAndOS
    Status:                Unknown
    Type:                  UpdateExecuted
    Last Transition Time:  2025-04-09T04:48:47Z
    Message:               Action during update to rendered-infra-77d571521c92751cb13c2ded5ca69f5a: Changes do not require a reboot
    Reason:                None
    Status:                False
    Type:                  UpdatePostActionComplete
    Last Transition Time:  2025-04-09T06:02:05Z
    Message:               Action during update to rendered-infra-2ffa42a87126fb01994b1241825df993: Uncordoned Node as part of completing upgrade phase
    Reason:                Uncordoned
    Status:                False
    Type:                  UpdateComplete
    Last Transition Time:  2025-04-09T06:02:05Z
    Message:               Action during update to rendered-infra-2ffa42a87126fb01994b1241825df993: In desired config rendered-infra-77d571521c92751cb13c2ded5ca69f5a. Resumed normal operations.
    Reason:                Resumed
    Status:                False
    Type:                  Resumed
    Last Transition Time:  2025-04-09T06:16:19Z
    Message:               Drained node. The drain is complete as the desired drainer matches current drainer: drain-rendered-infra-d58c96878f0bde7ff712d0e5767fdf9b
    Reason:                UpdateExecutedDrained
    Status:                True
    Type:                  Drained
    Last Transition Time:  2025-04-09T06:16:21Z
    Message:               Applying files and new OS config to node. OS will not need an update. SSH Keys will not need an update
    Reason:                UpdateExecutedAppliedFilesAndOS
    Status:                Unknown
    Type:                  AppliedFilesAndOS
    Last Transition Time:  2025-04-09T06:13:46Z
    Message:               Cordoned node. The node is reporting Unschedulable = true
    Reason:                UpdateExecutedCordoned
    Status:                True
    Type:                  Cordoned
    Last Transition Time:  2025-04-09T06:02:05Z
    Message:               Action during update to rendered-infra-2ffa42a87126fb01994b1241825df993: Node has rebooted
    Reason:                RebootedNode
    Status:                False
    Type:                  RebootedNode
    Last Transition Time:  2025-04-09T06:13:39Z
    Message:               Node ip-10-0-31-154.us-west-2.compute.internal needs an update
    Reason:                Updated
    Status:                False
    Type:                  Updated
    Last Transition Time:  2025-04-09T06:02:05Z
    Message:               Action during update to rendered-infra-2ffa42a87126fb01994b1241825df993: UnCordoned node. The node is reporting Unschedulable = false
    Reason:                UpdateCompleteUncordoned
    Status:                False
    Type:                  Uncordoned
    Last Transition Time:  2025-04-09T06:01:53Z
    Message:               All is good
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsDegraded
  Config Version:
    Current:            rendered-infra-2ffa42a87126fb01994b1241825df993
    Desired:            rendered-infra-d58c96878f0bde7ff712d0e5767fdf9b
  Observed Generation:  7
Events:                 
  • Apply the PIS on infra pool and check the MCN update
PIS template for infra pool
$ oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  labels:
    machineconfiguration.openshift.io/role: infra
  name: tc-80334-infrar-pinned-images
spec:
  pinnedImages:
  - name: "quay.io/openshifttest/busybox@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019"
  - name: quay.io/openshifttest/alpine@sha256:be92b18a369e989a6e86ac840b7f23ce0052467de551b064796d67280dfa06d5
EOF
pinnedimageset.machineconfiguration.openshift.io/tc-80334-infrar-pinned-images created
- Verify PIS is applied by checking the MCN infra
$ for node in $(oc get nodes -l node-role.kubernetes.io/infra -o name)
do
   n=${node/node\//}
   echo $n
   oc get machineconfignode $n -o wide
   oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="PinnedImageSetsProgressing")]}' | jq
done
ip-10-0-118-194.us-east-2.compute.internal
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-118-194.us-east-2.compute.internal   infra      rendered-infra-da792628ef260f08e7cd8e3c007f3676    rendered-infra-da792628ef260f08e7cd8e3c007f3676    True      False            False            False                      False            False     False               False          False         False          False
{
  "lastTransitionTime": "2025-04-08T15:22:00Z",
  "message": "All pinned image sets complete",
  "reason": "AsExpected",
  "status": "False",
  "type": "PinnedImageSetsProgressing"
}
$ oc debug node/ip-10-0-118-194.us-east-2.compute.internal -- chroot /host crictl images --pinned
Starting pod/ip-10-0-118-194us-east-2computeinternal-debug-4c45w ...
To use host binaries, run `chroot /host`
IMAGE                                                    TAG                 IMAGE ID            SIZE                PINNED
….
quay.io/openshifttest/alpine                                           45683da4f97c2       5.87MB              true
quay.io/openshifttest/busybox                                          b97242f89c8a2       1.45MB              true
  • Apply the wrong PIS (applied on infra pool)
Wrong PIS template
oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  labels:
    machineconfiguration.openshift.io/role: infra
  name: tc-80334-infrar-pinned-images-wrong
spec:
  pinnedImages:
  - name: "quay.io/openshifttest/fake@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019"
EOF
pinnedimageset.machineconfiguration.openshift.io/tc-80334-infrar-pinned-images-wrong created
$  for node in $(oc get nodes -l node-role.kubernetes.io/infra-o name)
do
   n=${node/node\//}
   echo $n
   oc get machineconfignode $n -o wide
   oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="PinnedImageSetsProgressing")]}' | jq;
  oc get machineconfignode $n -ojsonpath='{.status.conditions[?(@.type=="PinnedImageSetsDegraded")]}' | jq
done
ip-10-0-22-61.us-west-2.compute.internal
NAME                                       POOLNAME   DESIREDCONFIG                                     CURRENTCONFIG                                     UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-22-61.us-west-2.compute.internal   infra      rendered-infra-4dbfd2b60e0d199db92d1d1e04067a6b   rendered-infra-4dbfd2b60e0d199db92d1d1e04067a6b   True      False            False            False                      False            False     False               False          False         False          False
{
  "lastTransitionTime": "2025-04-09T08:51:10Z",
  "message": "node is prefetching images: ip-10-0-22-61.us-west-2.compute.internal",
  "reason": "ImagePrefetch",
  "status": "True",
  "type": "PinnedImageSetsProgressing"
}
{
  "lastTransitionTime": "2025-04-09T08:51:12Z",
  "message": "One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details",
  "reason": "PrefetchFailed",
  "status": "True",
  "type": "PinnedImageSetsDegraded"
}
$ oc describe machineconfignode ip-10-0-118-194.us-east-2.compute.internal
…….
   Pinned Image Sets:
    Desired Generation:            1
    Last Failed Generation:        1
    Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/openshifttest/fake@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019": exit status 125
    Name:                          tc-80334-infrar-pinned-images-wrong
Events:                            

@djoshy
Copy link
Contributor

djoshy commented Apr 10, 2025

/lgtm

feel free to unhold after QE tag has been applied

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 10, 2025
Copy link
Contributor

openshift-ci bot commented Apr 10, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, isabella-janssen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [djoshy,isabella-janssen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ptalgulk01
Copy link

Working on TC fix to adapt as per the changes once PR is merged will create the PR helping to run successful CI job.
Adding QE-approved label

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Apr 10, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Apr 10, 2025

@isabella-janssen: This pull request references MCO-1544 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

- What I did

- How to verify it

  1. Launch 4.19 cluster with this code and the API code part of MCO-1630: Generate v1 crds for PIS api#2257 with tech preview enabled. Note that the API PR version is needed due to the v1 CRD deployments. Using ClusterBot, an example launch command is:
launch 4.19,openshift/machine-config-operator#4962,openshift/api#2257 aws,techpreview
  1. Observe updated fields in MCN objects.

UPDATECOMPATIBLE and REBOOTEDNODE should not be in the output of oc get machineconfignode -o wide.

$ oc get machineconfignode -o wide
NAME                                         POOLNAME   DESIREDCONFIG                                      CURRENTCONFIG                                      UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED   UPDATEDFILESANDOS   CORDONEDNODE   DRAINEDNODE   REBOOTEDNODE   UNCORDONEDNODE
ip-10-0-125-181.us-west-1.compute.internal   master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-29-132.us-west-1.compute.internal    master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-5-15.us-west-1.compute.internal      worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-235.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-59-8.us-west-1.compute.internal      master     rendered-master-d51baee5aae85f41def583ae900f00d3   rendered-master-d51baee5aae85f41def583ae900f00d3   True      False            False            False                      False            False     False               False          False         False          False
ip-10-0-92-170.us-west-1.compute.internal    worker     rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   rendered-worker-1ddf0545a57e0ea7eaebd1fb6a15bc99   True      False            False            False                      False            False     False               False          False         False          False

The conditions list of a MachineConfigNode should not include items with type UpdateCompatible or ReloadedCRIO.

$ oc describe machineconfignode/ip-10-0-110-176.us-west-2.compute.internal
Name:         ip-10-0-110-176.us-west-2.compute.internal
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1alpha1
Kind:         MachineConfigNode
Metadata:
 Creation Timestamp:  2025-04-03T22:08:41Z
 Generation:          2
 Owner References:
   API Version:     v1
   Kind:            Node
   Name:            ip-10-0-110-176.us-west-2.compute.internal
   UID:             f3d946e9-24d4-4b96-b4a7-4536d79c73a8
 Resource Version:  18006
 UID:               7f03657f-d52a-4a48-b835-370f22fc773a
Spec:
 Config Version:
   Desired:  rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Node:
   Name:  ip-10-0-110-176.us-west-2.compute.internal
 Pool:
   Name:  worker
Status:
 Conditions:
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePrepared phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePrepared
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdateExecuted phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdateExecuted
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the UpdatePostActionComplete phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  UpdatePostActionComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: Uncordoned Node as part of completing upgrade phase
   Reason:                Uncordoned
   Status:                False
   Type:                  UpdateComplete
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: In desired config . Resumed normal operations.
   Reason:                Resumed
   Status:                False
   Type:                  Resumed
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Drained phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Drained
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the AppliedFilesAndOS phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  AppliedFilesAndOS
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the Cordoned phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  Cordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Node ip-10-0-110-176.us-west-2.compute.internal Updated
   Reason:                Updated
   Status:                True
   Type:                  Updated
   Last Transition Time:  2025-04-03T22:09:55Z
   Message:               Action during update to rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb: UnCordoned node. The node is reporting Unschedulable = false
   Reason:                UpdateCompleteUncordoned
   Status:                False
   Type:                  Uncordoned
   Last Transition Time:  2025-04-03T22:08:41Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
 Config Version:
   Current:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
   Desired:            rendered-worker-8c82b2aa30e5b7d400201f40ebd660fb
 Observed Generation:  3
Events:                 <none>
  1. Apply a valid MC and watch the MCN object throughout the update to see MachineConfigNodeUpdateRebooted as a parent phase.

Before the update

   Last Transition Time:  2025-04-03T22:08:12Z
   Message:               This node has not yet entered the RebootedNode phase
   Reason:                NotYetOccurred
   Status:                False
   Type:                  RebootedNode

During the reboot

   Message:               Upgrade requires a reboot.
   Reason:                RebootedNode
   Status:                Unknown
   Type:                  RebootedNode

After successful reboot

   Message:               Node has rebooted
   Reason:                RebootedNode
   Status:                True
   Type:                  RebootedNode
  1. Apply an invalid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079
  - name: quay.io/rh-ee-ijanssen/machine-config-operator@sha256:fd3692eff21338e900a244dfe62152c959b84d73f2dd4503893de0f3fae61b0b

The error message in the PinnedImageSetsDegraded MCN conditions and PIS status reference should now be a generalized message and the LastFailedGenerationErrors field should now be a string type field called LastFailedGenerationError.

$ oc describe machineconfignode ip-10-0-36-180.us-east-2.compute.internal
...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T12:27:06Z
   Message:               node is prefetching images: ip-10-0-36-180.us-east-2.compute.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T12:27:07Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            2
   Last Failed Generation:        2
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/rh-ee-ijanssen/machine-config-operator@sha256:65d3a308767b1773b6e3ead2ec1bcae499dde6ef085753d7e20e685f78841079": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS, such as the one below.
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "worker"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-release@sha256:513cf1028aa1a021fa73d0601427a0fbcf6d212b88aaf9d76d4e4841a061e44e
  - name: quay.io/openshift-release-dev/ocp-release@sha256:61eae2d261e54d1b8a0e05f6b5326228b00468364563745eed88460af04f909b

The MCN conditions and PIS status reference should report as before.
Note that the new MCN API validation requires that LastFailedGenerationError is populated when a LastFailedGeneration value is reported. Since the default LastFailedGeneration of 0 always showed on a successful PIS application (as tested in cluster version 4.19.0-0.nightly-arm64-2025-03-21-080021), an empty string for LastFailedGenerationError is now also being reported. This will not be fixed in this PR as it is a result of an existing bug (see OCPBUG-54592).

...
Status:
 Conditions:
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-04-03T11:53:06Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:            1
   Desired Generation:            1
   Last Failed Generation:        0
   Last Failed Generation Error:  
   Name:                          test-pinned
...

- Description for the changelog
MCO-1544: Update MCO code to use MCN’s updated v1alpha1 API

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@djoshy
Copy link
Contributor

djoshy commented Apr 10, 2025

/unhold

/retest-required

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 10, 2025
Copy link
Contributor

openshift-ci bot commented Apr 10, 2025

@isabella-janssen: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-azure-ovn-upgrade-out-of-change e275bbe link false /test e2e-azure-ovn-upgrade-out-of-change
ci/prow/bootstrap-unit e275bbe link false /test bootstrap-unit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@isabella-janssen
Copy link
Member Author

/retest-required

@openshift-merge-bot openshift-merge-bot bot merged commit 88b9bb6 into openshift:main Apr 10, 2025
15 of 17 checks passed
@isabella-janssen isabella-janssen deleted the mco-1544-apiBump branch April 10, 2025 21:51
@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: ose-machine-config-operator
This PR has been included in build ose-machine-config-operator-container-v4.20.0-202504102338.p0.g88b9bb6.assembly.stream.el9.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants