diff --git a/AGENTS.md b/AGENTS.md index 21983ae491..939542b215 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -110,6 +110,7 @@ All `It`/`DescribeTable` specs should be labelled using constants from `tests/in | Disruption | `disruption:destructive`, `disruption:nondestructive` | | Component | `component:controller`, `component:remediation`, `component:webhook`, `component:metrics`, `component:post-deploy` | | Platform | `platform:aws`, `platform:baremetal`, `platform:any` | +| Topology | `topology:control-plane`, `topology:minimal-worker`, `topology:zero-worker` | ### Key Conventions diff --git a/go.mod b/go.mod index 81a6dc32b6..356429a21d 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( k8s.io/apimachinery v0.35.2 k8s.io/client-go v12.0.0+incompatible k8s.io/klog/v2 v2.130.1 - k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 // indirect + k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 sigs.k8s.io/controller-runtime v0.22.5 ) @@ -126,6 +126,7 @@ require ( ) require ( + github.com/medik8s/common v1.17.0 github.com/medik8s/node-maintenance-operator v0.21.0 github.com/rh-ecosystem-edge/eco-goinfra v0.0.0-20260504223242-6b9a2708aabf k8s.io/apiextensions-apiserver v0.34.7 diff --git a/go.sum b/go.sum index e6bf6e7e38..9116868768 100644 --- a/go.sum +++ b/go.sum @@ -239,6 +239,8 @@ github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJ github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/maruel/natural v1.1.1 h1:Hja7XhhmvEFhcByqDoHz9QZbkWey+COd9xWfCfn1ioo= github.com/maruel/natural v1.1.1/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg= +github.com/medik8s/common v1.17.0 h1:AmJKx0tzqGZF27Ot0A4ak85q0F0zqUkVyCvYmm67rtY= +github.com/medik8s/common v1.17.0/go.mod h1:A9jYldC6PZcAuBowNNm712FqWdASB2ey5Vjp8MYN/PY= github.com/medik8s/node-maintenance-operator v0.21.0 h1:9Xxe/1KM0qj1D3S4YI57qi6D/1BmfKa2y59UovBw/mQ= github.com/medik8s/node-maintenance-operator v0.21.0/go.mod h1:sRYMxJxeOR0J9d0oMvuakpH6/09zFt3w07XMAd+/1/s= github.com/mfridman/tparse v0.18.0 h1:wh6dzOKaIwkUGyKgOntDW4liXSo37qg5AXbIhkMV3vE= diff --git a/tests/far-operator/README.md b/tests/far-operator/README.md index 456f87c88b..85f69a0352 100644 --- a/tests/far-operator/README.md +++ b/tests/far-operator/README.md @@ -199,3 +199,42 @@ Validates the full customer upgrade path: install GA FAR from redhat-operators o - **Env vars (optional, have defaults)**: `MEDIK8S_OPERATOR_PACKAGE` (default: `fence-agents-remediation`), `MEDIK8S_TARGET_CHANNEL` (default: `stable`) - **Standalone**: `ginkgo --label-filter="far && tier:upgrade" ./tests/far-operator/...` - **Pass criteria**: FAR deployment Ready on OCP N-1, OCP upgrade completes (Progressing=False, Available=True, Degraded=False), FAR deployment Ready after OCP upgrade, FAR CSV in Succeeded phase after catalog switch (new CSV if Konflux version is higher than GA, same CSV if versions match), controller image changes after operator upgrade (skipped on version parity), remediation succeeds after OCP upgrade and after catalog switch (node rebooted via boot ID change, node recovers to Ready), workload pods evicted after each fencing cycle + +### 17. Remediate a Control Plane Node and Verify etcd Quorum Preservation ([OCP-90217](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-90217)) + +Fences any control plane node via fence_aws and verifies etcd quorum is preserved. Checks etcd ClusterOperator health before and after remediation. Creates a test workload pod pinned to the CP target node, then verifies it is evicted after fencing. Confirms the fenced CP node reboots (boot ID change), returns to Ready, the etcd ClusterOperator recovers (Available=True, Degraded=False), and the FAR CR reaches its terminal status conditions (Processing=False, FenceAgentActionSucceeded=True, Succeeded=True). FAR lifecycle Events are deliberately not asserted on a control-plane target: Kubernetes Events are best-effort and can be dropped during the apiserver/etcd disruption the CP reboot causes, so the durable CR status conditions prove the outcome instead (the full Event bundle is still asserted on the worker specs). + +- **Operators**: FAR +- **Cluster**: AWS IPI, 3 control plane nodes (etcd quorum requires majority) +- **Storage**: None +- **Environment**: Connected +- **Labels**: `tier:acceptance`, `disruption:destructive`, `platform:aws`, `frequency:weekly`, `component:remediation`, `topology:control-plane` +- **Env vars (required)**: AWS credentials provisioned by the `medik8s-aws-credentials` CI step +- **Standalone**: `ginkgo --label-filter="far && topology:control-plane" ./tests/far-operator/...` +- **Pass criteria**: etcd ClusterOperator healthy before and after remediation, CP node rebooted (boot ID change), CP node returns to Ready, workload pod evicted from CP node, FAR CR status conditions Processing=False, FenceAgentActionSucceeded=True, Succeeded=True (FAR lifecycle Events are not asserted on a control-plane target - they are best-effort and can be dropped during the CP reboot's apiserver/etcd disruption) + +### 18. Fence Leader and Complete FAR Remediation with Only 2 Schedulable Workers ([OCP-90218](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-90218)) + +Simulates a minimal 2-worker topology by cordoning extra workers, then fences the FAR leader node via fence_aws, forcing leader election failover to the surviving worker. Verifies the surviving worker takes over leadership and completes remediation despite degraded capacity (at least 1 FAR replica stays Running), workloads are evicted, and FAR recovers to 2 replicas once the cordoned capacity is restored (the fenced node stays NoSchedule-tainted until its CR is deleted, so the recovered replica lands on an uncordoned worker). + +- **Operators**: FAR +- **Cluster**: AWS IPI, 2+ worker nodes (extra workers are cordoned to simulate topology) +- **Storage**: None +- **Environment**: Connected +- **Labels**: `tier:acceptance`, `disruption:destructive`, `platform:aws`, `frequency:weekly`, `component:remediation`, `topology:minimal-worker` +- **Env vars (required)**: AWS credentials provisioned by the `medik8s-aws-credentials` CI step +- **Standalone**: `ginkgo --label-filter="far && topology:minimal-worker" ./tests/far-operator/...` +- **Pass criteria**: At least 1 FAR replica Running during degraded capacity, leader node rebooted (boot ID change), leader node returns to Ready, workload pod evicted, FAR recovers to 2 replicas after schedulable capacity is restored + +### 19. Verify FAR Deployment Unavailability with Zero Schedulable Workers ([OCP-90308](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-90308)) + +Validates FAR operator behavior when all worker nodes are cordoned, simulating a 0-worker topology. Cordons all workers, deletes FAR pods to force reschedule attempts, verifies the deployment reports 0 Ready replicas, then uncordons workers and verifies FAR recovers to full replica count. + +- **Operators**: FAR +- **Cluster**: AWS IPI, 1+ worker nodes (all cordoned to simulate 0-worker topology) +- **Storage**: None +- **Environment**: Connected +- **Labels**: `tier:acceptance`, `disruption:destructive`, `platform:aws`, `frequency:weekly`, `component:controller`, `topology:zero-worker` +- **Env vars (required)**: AWS credentials provisioned by the `medik8s-aws-credentials` CI step +- **Standalone**: `ginkgo --label-filter="far && topology:zero-worker" ./tests/far-operator/...` +- **Pass criteria**: FAR deployment Ready before test, FAR deployment has 0 Ready replicas after pods deleted on cordoned workers, FAR deployment recovers to 2 Ready replicas after uncordoning diff --git a/tests/far-operator/internal/farparams/const.go b/tests/far-operator/internal/farparams/const.go index 2fd886d280..c4ea3f152c 100644 --- a/tests/far-operator/internal/farparams/const.go +++ b/tests/far-operator/internal/farparams/const.go @@ -64,9 +64,6 @@ const ( // NodeReadyTimeout is how long to wait for a node to become Ready after reboot. NodeReadyTimeout = 10 * time.Minute - // NodeNotReadyTimeout is how long to wait for a node to become NotReady after kubelet stop. - NodeNotReadyTimeout = 5 * time.Minute - // NodeRebootTimeout is how long to wait for a node reboot to complete. NodeRebootTimeout = 6 * time.Minute @@ -76,6 +73,15 @@ const ( // FARConditionTimeout is how long to wait for a FAR CR condition to appear. FARConditionTimeout = 2 * time.Minute + // EventVerifyTimeout is how long to wait for Kubernetes lifecycle events to + // appear. Longer than FARConditionTimeout because event delivery lags the CR + // condition and the shared, rate-limited API client can throttle event List + // calls during the destructive suite. + EventVerifyTimeout = 5 * time.Minute + // EventVerifyInterval is the poll interval for event verification. Longer than + // DefaultPollInterval to reduce List pressure on the shared rate-limited client. + EventVerifyInterval = 10 * time.Second + // RemediationCRDeletionTimeout is how long to wait for a FAR/FART CR to be fully deleted. RemediationCRDeletionTimeout = 2 * time.Minute @@ -123,6 +129,24 @@ const ( // SharedCredentialsSecretName is the Secret created by the test suite to hold // fence agent credentials in the format expected by SharedSecretName. SharedCredentialsSecretName = "far-test-shared-credentials" + + // MinControlPlaneNodes is the minimum Ready CP nodes needed for safe CP remediation. + MinControlPlaneNodes = 3 + + // CPRebootTimeout is how long to wait for a CP node reboot (slower than worker due to etcd). + CPRebootTimeout = 10 * time.Minute + // CPNodeReadyTimeout is how long to wait for a CP node to return Ready after reboot. + CPNodeReadyTimeout = 12 * time.Minute + // EtcdRejoinTimeout is how long to wait for etcd ClusterOperator to recover. + EtcdRejoinTimeout = 10 * time.Minute + + // MinWorkersForDestructiveTests is the minimum Ready workers for standard destructive tests. + MinWorkersForDestructiveTests = 3 + // MinWorkersForTwoWorkerTest is the minimum Ready workers for the 2-worker topology test. + MinWorkersForTwoWorkerTest = 2 + + // TestCordonAnnotation marks nodes cordoned by the test for cleanup identification. + TestCordonAnnotation = "system-tests.medik8s.io/cordoned-for-topology-test" ) // WorkloadTestImage is the container image used for test workload pods. diff --git a/tests/far-operator/internal/farutils/cluster.go b/tests/far-operator/internal/farutils/cluster.go index 46314aa6a4..43c038ea9c 100644 --- a/tests/far-operator/internal/farutils/cluster.go +++ b/tests/far-operator/internal/farutils/cluster.go @@ -3,7 +3,9 @@ package farutils import ( "context" "fmt" + "time" + commonlabels "github.com/medik8s/common/pkg/labels" configv1 "github.com/openshift/api/config/v1" corev1 "k8s.io/api/core/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -53,11 +55,12 @@ func GetAWSCredentials( } // BuildAWSNodeParameters builds the --plug node parameter map for fence_aws -// from the list of worker nodes. +// from all Ready nodes (workers and control plane) so that both worker and +// CP remediation tests can use the same parameter map. func BuildAWSNodeParameters(ctx context.Context, k8sClient client.Client) (map[string]map[string]string, error) { nodeList := &corev1.NodeList{} - if err := k8sClient.List(ctx, nodeList, client.MatchingLabels{"node-role.kubernetes.io/worker": ""}); err != nil { - return nil, fmt.Errorf("failed to list worker nodes: %w", err) + if err := k8sClient.List(ctx, nodeList); err != nil { + return nil, fmt.Errorf("failed to list nodes: %w", err) } plugMap := make(map[string]string) @@ -65,21 +68,155 @@ func BuildAWSNodeParameters(ctx context.Context, k8sClient client.Client) (map[s for i := range nodeList.Items { node := &nodeList.Items[i] - if node.Spec.Unschedulable || !helpers.IsNodeReady(node) { + if !helpers.IsNodeReady(node) { continue } instanceID, err := helpers.ExtractAWSInstanceID(node) if err != nil { - return nil, fmt.Errorf("ready worker %s has invalid providerID: %w", node.Name, err) + return nil, fmt.Errorf("ready node %s has invalid providerID: %w", node.Name, err) } plugMap[node.Name] = instanceID } if len(plugMap) == 0 { - return nil, fmt.Errorf("no worker nodes with valid AWS providerID") + return nil, fmt.Errorf("no nodes with valid AWS providerID") } return map[string]map[string]string{farparams.NodeIdentifierAWS: plugMap}, nil } + +// GetReadyControlPlaneNodes returns all Ready control plane nodes. +func GetReadyControlPlaneNodes(ctx context.Context, k8sClient client.Client) ([]corev1.Node, error) { + nodeList := &corev1.NodeList{} + if err := k8sClient.List(ctx, nodeList); err != nil { + return nil, fmt.Errorf("failed to list nodes: %w", err) + } + + var cpNodes []corev1.Node + + for i := range nodeList.Items { + node := &nodeList.Items[i] + + _, hasCP := node.Labels[commonlabels.ControlPlaneRole] + _, hasMaster := node.Labels[commonlabels.MasterRole] + + if (hasCP || hasMaster) && helpers.IsNodeReady(node) { + cpNodes = append(cpNodes, *node) + } + } + + return cpNodes, nil +} + +// SelectControlPlaneNode returns a Ready CP node that is not in the exclude list. +func SelectControlPlaneNode( + ctx context.Context, k8sClient client.Client, excludeNodes ...string, +) (*corev1.Node, error) { + cpNodes, err := GetReadyControlPlaneNodes(ctx, k8sClient) + if err != nil { + return nil, err + } + + excluded := make(map[string]bool, len(excludeNodes)) + for _, name := range excludeNodes { + excluded[name] = true + } + + for i := range cpNodes { + if !excluded[cpNodes[i].Name] { + return &cpNodes[i], nil + } + } + + return nil, fmt.Errorf("no eligible CP node found (excluded: %v)", excludeNodes) +} + +// CordonExtraWorkers cordons all Ready worker nodes except those in keepNames. +// Returns the names of nodes that were cordoned by this function. +func CordonExtraWorkers( + ctx context.Context, k8sClient client.Client, keepNames []string, +) ([]string, error) { + keepSet := make(map[string]bool, len(keepNames)) + for _, name := range keepNames { + keepSet[name] = true + } + + nodeList := &corev1.NodeList{} + if err := k8sClient.List(ctx, nodeList, + client.MatchingLabels{commonlabels.WorkerRole: ""}); err != nil { + return nil, fmt.Errorf("failed to list worker nodes: %w", err) + } + + var cordoned []string + + for i := range nodeList.Items { + node := &nodeList.Items[i] + + if keepSet[node.Name] || node.Spec.Unschedulable || !helpers.IsNodeReady(node) { + continue + } + + patch := client.MergeFrom(node.DeepCopy()) + node.Spec.Unschedulable = true + + if node.Annotations == nil { + node.Annotations = make(map[string]string) + } + + node.Annotations[farparams.TestCordonAnnotation] = "true" + + if err := k8sClient.Patch(ctx, node, patch); err != nil { + return cordoned, fmt.Errorf("failed to cordon node %s: %w", node.Name, err) + } + + cordoned = append(cordoned, node.Name) + } + + return cordoned, nil +} + +// UncordonNodes restores schedulability for the given node names. +func UncordonNodes( + ctx context.Context, k8sClient client.Client, nodeNames []string, + logf ...func(string, ...interface{}), +) { + log := func(format string, args ...interface{}) { + if len(logf) > 0 { + logf[0](format, args...) + } + } + + for _, name := range nodeNames { + node := &corev1.Node{} + if err := k8sClient.Get(ctx, client.ObjectKey{Name: name}, node); err != nil { + log("WARNING: failed to get node %s for uncordon: %v\n", name, err) + + continue + } + + if !node.Spec.Unschedulable { + continue + } + + patch := client.MergeFrom(node.DeepCopy()) + node.Spec.Unschedulable = false + delete(node.Annotations, farparams.TestCordonAnnotation) + + if err := k8sClient.Patch(ctx, node, patch); err != nil { + log("WARNING: failed to uncordon node %s: %v\n", name, err) + } + } +} + +// WaitForClusterOperatorHealthy polls until the named ClusterOperator reports +// Available=True, Progressing=False, Degraded=False. +func WaitForClusterOperatorHealthy( + ctx context.Context, k8sClient client.Client, + operatorName string, timeout time.Duration, + logf func(string, ...interface{}), +) error { + return helpers.WaitForClusterOperatorHealthy(ctx, k8sClient, operatorName, timeout, + farparams.DefaultPollInterval, logf) +} diff --git a/tests/far-operator/tests/far_controller_lifecycle.go b/tests/far-operator/tests/far_controller_lifecycle.go index 0b0bfa7fd4..2383b24d62 100644 --- a/tests/far-operator/tests/far_controller_lifecycle.go +++ b/tests/far-operator/tests/far_controller_lifecycle.go @@ -45,25 +45,39 @@ var _ = Describe("FAR Controller Lifecycle Tests", func() { By("Getting the current active FAR controller pod") - pods, err := farutils.GetFARControllerPods(ctx, APIClient) - Expect(err).ToNot(HaveOccurred()) - Expect(pods).ToNot(BeEmpty(), "No running FAR controller pods found") + // Leader election is eventually-consistent: a prior destructive spec + // (the 0-worker test deletes FAR pods; fencing specs evict them) can + // leave the controller Lease naming a pod that no longer exists until + // re-election updates it. Retry until the Lease resolves to a live, + // Running controller pod instead of failing on a one-shot lookup. + var ( + oldLeaderPod *corev1.Pod + oldLeaderNode string + ) - oldLeaderNode, err := farutils.GetActiveFARControllerNode(ctx, APIClient) - Expect(err).ToNot(HaveOccurred()) + Eventually(func(assertion Gomega) { + pods, err := farutils.GetFARControllerPods(ctx, APIClient) + assertion.Expect(err).ToNot(HaveOccurred()) + assertion.Expect(pods).ToNot(BeEmpty(), "No running FAR controller pods found") + + leaderNode, err := farutils.GetActiveFARControllerNode(ctx, APIClient) + assertion.Expect(err).ToNot(HaveOccurred()) - var oldLeaderPod *corev1.Pod + oldLeaderPod = nil - for i := range pods { - if pods[i].Spec.NodeName == oldLeaderNode { - oldLeaderPod = &pods[i] + for i := range pods { + if pods[i].Spec.NodeName == leaderNode { + oldLeaderPod = &pods[i] - break + break + } } - } - Expect(oldLeaderPod).ToNot(BeNil(), - "Could not find controller pod on leader node %s", oldLeaderNode) + assertion.Expect(oldLeaderPod).ToNot(BeNil(), + "Lease leader node %s has no Running controller pod yet", leaderNode) + oldLeaderNode = leaderNode + }, farparams.ControllerHandoverTimeout, farparams.DefaultPollInterval).Should(Succeed(), + "controller lease did not resolve to a live controller pod") oldPodName := oldLeaderPod.Name GinkgoWriter.Printf("Active controller pod: %s on node %s\n", diff --git a/tests/far-operator/tests/far_destructive.go b/tests/far-operator/tests/far_destructive.go index 548234b7d4..55c9083cec 100644 --- a/tests/far-operator/tests/far_destructive.go +++ b/tests/far-operator/tests/far_destructive.go @@ -3,6 +3,7 @@ package tests import ( "context" "fmt" + "time" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" @@ -103,16 +104,6 @@ var _ = Describe("FAR Destructive Tests", Expect(farDeployment.IsReady(medik8sparams.DefaultTimeout)).To(BeTrue(), "FAR deployment is not Ready") - By("Verifying at least 3 Ready worker nodes") - - workerCount, err := helpers.CountReadyWorkerNodes(ctx, APIClient) - Expect(err).ToNot(HaveOccurred()) - // 3 workers: FAR leader (excluded from fencing) + target (fenced/rebooted) + - // at least 1 spare to keep the cluster schedulable while the target is down. - Expect(workerCount).To( - BeNumerically(">=", 3), - "Destructive tests require at least 3 Ready worker nodes") - By("Reading AWS credentials from CCO Secret") awsAccessKey, awsSecretKey, err := farutils.GetAWSCredentials( @@ -298,6 +289,17 @@ var _ = Describe("FAR Destructive Tests", Context("Standalone FAR remediation", func() { BeforeEach(func() { + By("Verifying minimum Ready worker nodes for destructive tests") + + workerCount, err := helpers.CountReadyWorkerNodes(ctx, APIClient) + Expect(err).ToNot(HaveOccurred()) + + if workerCount < farparams.MinWorkersForDestructiveTests { + Skip(fmt.Sprintf( + "Standalone destructive tests require at least %d Ready worker nodes, found %d", + farparams.MinWorkersForDestructiveTests, workerCount)) + } + By("Verifying FAR controller is Ready before test") farDeployment, err := deployment.Pull( @@ -343,39 +345,7 @@ var _ = Describe("FAR Destructive Tests", By("Creating a test workload pod pinned to " + targetNode.Name) - workloadPod = &corev1.Pod{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "far-workload-test-", - Namespace: medik8sparams.OperatorNs, - }, - Spec: corev1.PodSpec{ - NodeName: targetNode.Name, - RestartPolicy: corev1.RestartPolicyAlways, - Containers: []corev1.Container{{ - Name: "workload", - Image: farparams.WorkloadTestImage, - Command: []string{"sleep", "infinity"}, - }}, - }, - } - - Expect(APIClient.Create(ctx, workloadPod)).To(Succeed()) - DeferCleanup(func() { - _ = APIClient.Delete(ctx, workloadPod) - }) - - By("Waiting for workload pod to be Running") - - Eventually(func() corev1.PodPhase { - pod := &corev1.Pod{} - if err := APIClient.Get(ctx, client.ObjectKey{ - Name: workloadPod.Name, Namespace: workloadPod.Namespace, - }, pod); err != nil { - return corev1.PodPending - } - - return pod.Status.Phase - }, farparams.WorkloadPodReadyTimeout, farparams.DefaultPollInterval).Should(Equal(corev1.PodRunning)) + workloadPod = createWorkloadPod(ctx, APIClient, targetNode.Name) }) JustAfterEach(func() { @@ -434,7 +404,7 @@ var _ = Describe("FAR Destructive Tests", {Reason: farparams.FAREventFenceAgentSucceeded, Type: corev1.EventTypeNormal}, {Reason: farparams.FAREventRemediationFinished, Type: corev1.EventTypeNormal}, }, - farparams.FARConditionTimeout, farparams.DefaultPollInterval, + farparams.EventVerifyTimeout, farparams.EventVerifyInterval, )).To(Succeed(), "FAR lifecycle events not found on CR %s", targetNode.Name) By("Verifying remediation completion event on Node") @@ -448,7 +418,7 @@ var _ = Describe("FAR Destructive Tests", []helpers.EventExpectation{ {Reason: farparams.FAREventNodeRemediationCompleted, Type: corev1.EventTypeNormal}, }, - farparams.FARConditionTimeout, farparams.DefaultPollInterval, + farparams.EventVerifyTimeout, farparams.EventVerifyInterval, )).To(Succeed(), "NodeRemediationCompleted event not found on node %s", targetNode.Name) }) @@ -599,39 +569,7 @@ var _ = Describe("FAR Destructive Tests", By("Creating a test workload pod pinned to " + targetNode.Name) - workloadPod := &corev1.Pod{ - ObjectMeta: metav1.ObjectMeta{ - GenerateName: "far-workload-test-", - Namespace: medik8sparams.OperatorNs, - }, - Spec: corev1.PodSpec{ - NodeName: targetNode.Name, - RestartPolicy: corev1.RestartPolicyAlways, - Containers: []corev1.Container{{ - Name: "workload", - Image: farparams.WorkloadTestImage, - Command: []string{"sleep", "infinity"}, - }}, - }, - } - - Expect(APIClient.Create(ctx, workloadPod)).To(Succeed()) - DeferCleanup(func() { - _ = APIClient.Delete(ctx, workloadPod) - }) - - By("Waiting for workload pod to be Running") - - Eventually(func() corev1.PodPhase { - pod := &corev1.Pod{} - if err := APIClient.Get(ctx, client.ObjectKey{ - Name: workloadPod.Name, Namespace: workloadPod.Namespace, - }, pod); err != nil { - return corev1.PodPending - } - - return pod.Status.Phase - }, farparams.WorkloadPodReadyTimeout, farparams.DefaultPollInterval).Should(Equal(corev1.PodRunning)) + workloadPod := createWorkloadPod(ctx, APIClient, targetNode.Name) By("Recording pre-reboot lease holder for failover verification") @@ -697,7 +635,7 @@ var _ = Describe("FAR Destructive Tests", {Reason: farparams.FAREventFenceAgentSucceeded, Type: corev1.EventTypeNormal}, {Reason: farparams.FAREventRemediationFinished, Type: corev1.EventTypeNormal}, }, - farparams.FARConditionTimeout, farparams.DefaultPollInterval, + farparams.EventVerifyTimeout, farparams.EventVerifyInterval, )).To(Succeed(), "FAR lifecycle events not found on CR after leader failover") By("Verifying remediation completion event on Node after failover") @@ -711,7 +649,7 @@ var _ = Describe("FAR Destructive Tests", []helpers.EventExpectation{ {Reason: farparams.FAREventNodeRemediationCompleted, Type: corev1.EventTypeNormal}, }, - farparams.FARConditionTimeout, farparams.DefaultPollInterval, + farparams.EventVerifyTimeout, farparams.EventVerifyInterval, )).To(Succeed(), "NodeRemediationCompleted event not found on node after leader failover") By("Verifying workload pod was evicted from leader node") @@ -736,6 +674,501 @@ var _ = Describe("FAR Destructive Tests", // stopping kubelet and waiting for NHC to detect the unhealthy // node and create a FAR CR automatically. }) + + Context("control plane node target", + Label(labels.TopologyControlPlane), + func() { + It("should remediate a control plane node and preserve etcd quorum", + Label(labels.TierAcceptance, labels.ComponentRemediation), + reportxml.ID("90217"), + func() { + By("Verifying at least 3 Ready control plane nodes for etcd quorum") + + cpNodes, err := farutils.GetReadyControlPlaneNodes(ctx, APIClient) + Expect(err).ToNot(HaveOccurred()) + + if len(cpNodes) < farparams.MinControlPlaneNodes { + Skip(fmt.Sprintf( + "CP remediation requires at least %d Ready CP nodes, found %d", + farparams.MinControlPlaneNodes, len(cpNodes))) + } + + By("Verifying etcd ClusterOperator is healthy before remediation") + + Expect(farutils.WaitForClusterOperatorHealthy( + ctx, APIClient, "etcd", medik8sparams.DefaultTimeout, + GinkgoWriter.Printf)).To(Succeed(), + "etcd ClusterOperator not healthy before test") + + By("Selecting a CP node as remediation target") + + cpTarget, err := farutils.SelectControlPlaneNode(ctx, APIClient) + Expect(err).ToNot(HaveOccurred()) + + targetNode = cpTarget + GinkgoWriter.Printf("CP remediation target: %s\n", targetNode.Name) + + By("Cleaning CRI-O overlay storage on " + targetNode.Name) + removeWorkloadImage(ctx, targetNode.Name) + + By("Recording boot ID before remediation") + + oldBootID, err := farutils.GetNodeBootIDFromAPI(ctx, APIClient, targetNode.Name) + Expect(err).ToNot(HaveOccurred()) + + By("Creating a test workload pod pinned to CP node " + targetNode.Name) + + workloadPod := createWorkloadPod(ctx, APIClient, targetNode.Name) + + By("Creating FAR CR targeting CP node " + targetNode.Name) + + farCR := buildFARUnstructured(targetNode.Name, fenceAgent, sharedParams, nodeParams) + createFARCR(ctx, APIClient, farCR) + + currentFARName = targetNode.Name + + waitForRemediationWithTimeouts(ctx, APIClient, targetNode.Name, oldBootID, + farparams.CPRebootTimeout, farparams.CPNodeReadyTimeout) + + By("Verifying workload pod was evicted from fenced CP node") + + Eventually(func() bool { + pod := &corev1.Pod{} + getErr := APIClient.Get(ctx, client.ObjectKey{ + Name: workloadPod.Name, Namespace: workloadPod.Namespace, + }, pod) + + return k8serrors.IsNotFound(getErr) || pod.DeletionTimestamp != nil + }, farparams.WorkloadEvictionTimeout, farparams.DefaultPollInterval).Should(BeTrue(), + "Workload pod was not evicted from CP node after remediation") + + By("Verifying etcd ClusterOperator recovered after CP node rejoin") + + Expect(farutils.WaitForClusterOperatorHealthy( + ctx, APIClient, "etcd", farparams.EtcdRejoinTimeout, + GinkgoWriter.Printf)).To(Succeed(), + "etcd ClusterOperator did not recover after CP node reboot") + + By("Verifying FAR CR status conditions (fence action + overall outcome)") + + // The fence agent's success is verified via the DURABLE + // FenceAgentActionSucceeded status condition rather than the + // transient FenceAgentSucceeded Kubernetes Event. That event is + // emitted the instant fencing triggers the control-plane reboot, + // exactly when the target node's apiserver/etcd is disrupted, so + // best-effort event delivery can drop it (see the note below). + // Status conditions are persisted on the CR and updated with retry, + // so they survive the disruption. + expectedConditions := map[string]string{ + farparams.FARConditionProcessing: string(metav1.ConditionFalse), + farparams.FARConditionFenceAgentSucceeded: string(metav1.ConditionTrue), + farparams.FARConditionSucceeded: string(metav1.ConditionTrue), + } + + Eventually(func(assertion Gomega) { + farObj := &unstructured.Unstructured{} + farObj.SetGroupVersionKind(farGVK) + assertion.Expect(APIClient.Get(ctx, client.ObjectKey{ + Name: targetNode.Name, Namespace: medik8sparams.OperatorNs, + }, farObj)).To(Succeed()) + + conditions, found, condErr := unstructured.NestedSlice( + farObj.Object, "status", "conditions") + assertion.Expect(condErr).ToNot(HaveOccurred()) + assertion.Expect(found).To(BeTrue(), "FAR CR has no status.conditions") + + for condType, expectedStatus := range expectedConditions { + condFound := false + + for _, c := range conditions { + condMap, ok := c.(map[string]interface{}) + if !ok { + continue + } + + if condMap["type"] == condType { + condFound = true + + assertion.Expect(condMap["status"]).To(Equal(expectedStatus), + "Condition %s has unexpected status", condType) + + break + } + } + + assertion.Expect(condFound).To(BeTrue(), + "Condition %s not found in FAR CR status", condType) + } + }, farparams.FARConditionTimeout, farparams.DefaultPollInterval).Should(Succeed()) + + // FAR lifecycle Events are intentionally NOT asserted on a + // control-plane target. Kubernetes Events are best-effort: + // client-go's broadcaster drops them when its queue is full and + // the sink drops them after a bounded retry limit, so delivery is + // never guaranteed. Fencing a control-plane node reboots it and + // briefly disrupts apiserver/etcd (quorum drops to 2/3 with a short + // write stall), so ANY Event emitted across that window can be lost + // even when the remediation fully succeeds - including + // RemediationFinished (emitted as the node rejoins) and the Node's + // NodeRemediationCompleted. This spec previously flaked + // non-deterministically on whichever Event happened to be dropped + // (FenceAgentSucceeded, then RemediationFinished on later runs). The + // remediation outcome is instead proven above by the DURABLE FAR CR + // status conditions (Processing=False, FenceAgentActionSucceeded=True, + // Succeeded=True) plus observable cluster state (boot-ID change, node + // Ready, workload eviction, etcd ClusterOperator recovery). The full + // Event bundle is still asserted on the worker specs, where fencing + // does not disrupt the control plane and Events are reliable. + }) + }) + + Context("minimal 2-worker topology", + Label(labels.TopologyMinimalWorker), + func() { + It("should fence the leader and complete remediation with only 2 schedulable workers", + Label(labels.TierAcceptance, labels.ComponentRemediation), + reportxml.ID("90218"), + func() { + By("Verifying at least 2 Ready worker nodes") + + workerCount, err := helpers.CountReadyWorkerNodes(ctx, APIClient) + Expect(err).ToNot(HaveOccurred()) + + if workerCount < farparams.MinWorkersForTwoWorkerTest { + Skip(fmt.Sprintf( + "2-worker test requires at least %d Ready workers, found %d", + farparams.MinWorkersForTwoWorkerTest, workerCount)) + } + + By("Finding active leader node to use as fencing target") + + Eventually(func() error { + var leaderErr error + + leaderNode, leaderErr = farutils.GetActiveFARControllerNode(ctx, APIClient) + + return leaderErr + }, farparams.ControllerHandoverTimeout, farparams.DefaultPollInterval).Should(Succeed()) + + By("Selecting a survivor worker (non-leader) to keep schedulable") + + survivor, err := helpers.SelectWorkerNode(ctx, APIClient, leaderNode) + Expect(err).ToNot(HaveOccurred()) + + keepNames := []string{leaderNode, survivor.Name} + + // Declared at It scope so it can be uncordoned inline (to restore + // schedulable capacity before verifying FAR recovery) and again via + // DeferCleanup as an idempotent safety net. + var cordonedNodes []string + + if workerCount > farparams.MinWorkersForTwoWorkerTest { + By("Cordoning extra workers to simulate 2-worker topology") + + var cordonErr error + + cordonedNodes, cordonErr = farutils.CordonExtraWorkers(ctx, APIClient, keepNames) + + DeferCleanup(func() { + if len(cordonedNodes) > 0 { + By("Restoring topology: uncordoning extra worker nodes") + farutils.UncordonNodes(ctx, APIClient, cordonedNodes, GinkgoWriter.Printf) + } + }) + Expect(cordonErr).ToNot(HaveOccurred()) + GinkgoWriter.Printf("Cordoned %d extra workers: %v\n", + len(cordonedNodes), cordonedNodes) + } + + leaderNodeObj, err := helpers.SelectWorkerNode(ctx, APIClient, survivor.Name) + Expect(err).ToNot(HaveOccurred()) + Expect(leaderNodeObj.Name).To(Equal(leaderNode), + "Expected leader node %s but got %s", leaderNode, leaderNodeObj.Name) + + targetNode = leaderNodeObj + GinkgoWriter.Printf("2-worker target (leader): %s, survivor: %s\n", + targetNode.Name, survivor.Name) + + By("Cleaning CRI-O overlay storage on " + targetNode.Name) + removeWorkloadImage(ctx, targetNode.Name) + + By("Recording boot ID before remediation") + + oldBootID, err := farutils.GetNodeBootIDFromAPI(ctx, APIClient, targetNode.Name) + Expect(err).ToNot(HaveOccurred()) + + By("Creating a test workload pod pinned to " + targetNode.Name) + + workloadPod := createWorkloadPod(ctx, APIClient, targetNode.Name) + + By("Recording pre-reboot lease holder for failover verification") + + preRebootLease := &coordinationv1.Lease{} + Expect(APIClient.Get(ctx, client.ObjectKey{ + Name: farparams.ControllerLeaseName, + Namespace: medik8sparams.OperatorNs, + }, preRebootLease)).To(Succeed()) + Expect(preRebootLease.Spec.HolderIdentity).ToNot(BeNil(), + "Lease has no holder before fencing the leader") + oldLeaderHolder := *preRebootLease.Spec.HolderIdentity + GinkgoWriter.Printf("Pre-reboot lease holder: %s\n", oldLeaderHolder) + + By("Creating FAR CR targeting leader node " + targetNode.Name) + + farCR := buildFARUnstructured(targetNode.Name, fenceAgent, sharedParams, nodeParams) + createFARCR(ctx, APIClient, farCR) + + currentFARName = targetNode.Name + + // ReadyReplicas is the deployment-level signal that the leader pod became + // NotReady. It is more reliable than the node Ready condition, which flips + // to NotReady before the pod on that node stops counting as Ready. A + // transient Pull error returns ExpectedReplicas (full capacity) so it + // neither false-detects a drop below nor fails the survival check. + readyReplicas := func() int32 { + dep, depErr := deployment.Pull( + APIClient, farparams.OperatorDeploymentName, + medik8sparams.OperatorNs) + if depErr != nil { + return farparams.ExpectedReplicas + } + + return dep.Object.Status.ReadyReplicas + } + + By("Waiting for FAR to reflect reduced capacity (leader replica no longer Ready)") + + // Gate the survival check on the leader replica actually dropping, so it + // runs during genuine degraded capacity instead of passing on the first + // poll while both replicas are still Ready. The drop is durably observable + // because on this 2-worker topology anti-affinity keeps the replacement + // replica Pending, so ReadyReplicas stays at 1 for the reboot window. + Eventually(readyReplicas, medik8sparams.DefaultTimeout, farparams.DefaultPollInterval).Should( + BeNumerically("<", farparams.ExpectedReplicas), + "FAR ReadyReplicas never dropped below %d after fencing the leader node", + farparams.ExpectedReplicas) + + By("Verifying at least 1 FAR replica stays Running throughout degraded capacity") + + Consistently(readyReplicas, farparams.ControllerHandoverTimeout, farparams.DefaultPollInterval).Should( + BeNumerically(">=", 1), + "FAR dropped below 1 Ready replica during degraded capacity") + + By("Verifying controller lease transferred to the survivor during degraded capacity") + + // Assert leadership moved off the fenced leader WHILE it is still down: + // the fenced node is the only other controller host, so a changed holder + // here must be the survivor. This mirrors OCP-70638's HolderIdentity check + // but runs during the degraded window, so it proves transfer TO the survivor + // rather than only that the old pod identity is gone after the node recovers. + Eventually(func(assertion Gomega) { + lease := &coordinationv1.Lease{} + assertion.Expect(APIClient.Get(ctx, client.ObjectKey{ + Name: farparams.ControllerLeaseName, + Namespace: medik8sparams.OperatorNs, + }, lease)).To(Succeed()) + assertion.Expect(lease.Spec.HolderIdentity).ToNot(BeNil(), + "Lease has no holder during degraded capacity") + + if lease.Spec.HolderIdentity != nil { + assertion.Expect(*lease.Spec.HolderIdentity).ToNot(Equal(oldLeaderHolder), + "Lease is still held by pre-reboot pod %s", oldLeaderHolder) + } + }, farparams.ControllerHandoverTimeout, farparams.DefaultPollInterval).Should(Succeed(), + "Controller lease did not transfer to the survivor during degraded capacity") + + waitForRemediation(ctx, APIClient, targetNode.Name, oldBootID) + + By("Verifying workload pod was evicted from fenced worker") + + Eventually(func() bool { + pod := &corev1.Pod{} + getErr := APIClient.Get(ctx, client.ObjectKey{ + Name: workloadPod.Name, Namespace: workloadPod.Namespace, + }, pod) + + return k8serrors.IsNotFound(getErr) || pod.DeletionTimestamp != nil + }, farparams.WorkloadEvictionTimeout, farparams.DefaultPollInterval).Should(BeTrue(), + "Workload pod was not evicted after remediation in 2-worker topology") + + By("Verifying FAR CR reached Succeeded after leader failover") + + // Mirrors the standalone (OCP-67015) and CP tests: without the + // Succeeded assertion a CR stuck in Processing would pass silently. + expectedConditions := map[string]string{ + farparams.FARConditionProcessing: string(metav1.ConditionFalse), + farparams.FARConditionFenceAgentSucceeded: string(metav1.ConditionTrue), + farparams.FARConditionSucceeded: string(metav1.ConditionTrue), + } + + Eventually(func(assertion Gomega) { + farObj := &unstructured.Unstructured{} + farObj.SetGroupVersionKind(farGVK) + assertion.Expect(APIClient.Get(ctx, client.ObjectKey{ + Name: targetNode.Name, Namespace: medik8sparams.OperatorNs, + }, farObj)).To(Succeed()) + + conditions, found, condErr := unstructured.NestedSlice( + farObj.Object, "status", "conditions") + assertion.Expect(condErr).ToNot(HaveOccurred()) + assertion.Expect(found).To(BeTrue(), "FAR CR has no status.conditions") + + for condType, expectedStatus := range expectedConditions { + condFound := false + + for _, c := range conditions { + condMap, ok := c.(map[string]interface{}) + if !ok { + continue + } + + if condMap["type"] == condType { + condFound = true + + assertion.Expect(condMap["status"]).To(Equal(expectedStatus), + "Condition %s has unexpected status", condType) + + break + } + } + + assertion.Expect(condFound).To(BeTrue(), + "Condition %s not found in FAR CR status", condType) + } + }, farparams.FARConditionTimeout, farparams.DefaultPollInterval).Should(Succeed()) + + // Restore schedulable capacity so FAR can recover its second replica. + // The fenced leader node stays NoSchedule-tainted until its CR is deleted + // (in JustAfterEach), so recovery to 2 replicas relies on the uncordoned + // extra worker(s). On a cluster with exactly 2 workers there is nothing to + // uncordon and full recovery only completes after CR deletion; this test + // targets the 3+-worker CI topology. + if len(cordonedNodes) > 0 { + By("Restoring schedulable capacity: uncordoning extra worker(s)") + farutils.UncordonNodes(ctx, APIClient, cordonedNodes, GinkgoWriter.Printf) + // Clear so the DeferCleanup safety net does not re-uncordon + // already-restored nodes (matches the 0-worker test). + cordonedNodes = nil + } + + By("Verifying FAR recovers to 2 Ready replicas after capacity is restored") + + Eventually(func() int32 { + dep, depErr := deployment.Pull( + APIClient, farparams.OperatorDeploymentName, + medik8sparams.OperatorNs) + if depErr != nil { + return 0 + } + + return dep.Object.Status.ReadyReplicas + }, medik8sparams.DefaultTimeout, farparams.DefaultPollInterval).Should( + BeNumerically(">=", farparams.ExpectedReplicas), + "FAR should recover to %d Ready replicas after capacity is restored", + farparams.ExpectedReplicas) + }) + }) + + Context("0-worker topology", + Label(labels.TopologyZeroWorker), + func() { + It("should report FAR deployment unavailable when no workers are schedulable", + Label(labels.TierAcceptance, labels.ComponentController), + reportxml.ID("90308"), + func() { + By("Verifying FAR deployment is Ready before test") + + farDeployment, err := deployment.Pull( + APIClient, farparams.OperatorDeploymentName, medik8sparams.OperatorNs) + Expect(err).ToNot(HaveOccurred()) + Expect(farDeployment.IsReady(medik8sparams.DefaultTimeout)).To(BeTrue(), + "FAR deployment is not Ready before 0-worker test") + + By("Verifying at least 1 Ready worker node exists to cordon") + + workerCount, err := helpers.CountReadyWorkerNodes(ctx, APIClient) + Expect(err).ToNot(HaveOccurred()) + + if workerCount < 1 { + Skip(fmt.Sprintf( + "0-worker test requires at least 1 Ready worker to cordon, found %d", + workerCount)) + } + + By("Cordoning all Ready worker nodes to simulate 0-worker topology") + + cordonedNodes, err := farutils.CordonExtraWorkers(ctx, APIClient, nil) + + DeferCleanup(func() { + if len(cordonedNodes) > 0 { + By("Restoring topology: uncordoning all worker nodes") + farutils.UncordonNodes(ctx, APIClient, cordonedNodes, GinkgoWriter.Printf) + } + }) + Expect(err).ToNot(HaveOccurred()) + + // Skip (do not fail) if nothing was cordoned: the upfront count + // found workers, so an empty result here means the topology changed + // during setup (a worker went NotReady between the count and the + // cordon). Failing on that race is the hard-fail this replaced. + if len(cordonedNodes) == 0 { + Skip("0-worker test found no Ready workers to cordon " + + "(topology changed during setup)") + } + + GinkgoWriter.Printf("Cordoned all %d workers: %v\n", + len(cordonedNodes), cordonedNodes) + + By("Deleting FAR controller pods to trigger reschedule on cordoned nodes") + + podList := &corev1.PodList{} + Expect(APIClient.List(ctx, podList, + client.InNamespace(medik8sparams.OperatorNs), + client.MatchingLabels(farparams.OperatorControllerPodLabels), + )).To(Succeed()) + + for i := range podList.Items { + Expect(APIClient.Delete(ctx, &podList.Items[i])).To(Succeed()) + GinkgoWriter.Printf("Deleted FAR pod %s\n", podList.Items[i].Name) + } + + By("Verifying FAR deployment has 0 Ready replicas") + + Eventually(func() int32 { + dep, depErr := deployment.Pull( + APIClient, farparams.OperatorDeploymentName, + medik8sparams.OperatorNs) + if depErr != nil { + return -1 + } + + return dep.Object.Status.ReadyReplicas + }, medik8sparams.DefaultTimeout, farparams.DefaultPollInterval).Should( + BeNumerically("==", 0), + "FAR deployment should have 0 Ready replicas with no schedulable workers") + + By("Uncordoning workers and verifying FAR recovery") + + farutils.UncordonNodes(ctx, APIClient, cordonedNodes, GinkgoWriter.Printf) + cordonedNodes = nil + + By("Verifying FAR deployment recovers to 2 Ready replicas") + + Eventually(func() int32 { + dep, depErr := deployment.Pull( + APIClient, farparams.OperatorDeploymentName, + medik8sparams.OperatorNs) + if depErr != nil { + return 0 + } + + return dep.Object.Status.ReadyReplicas + }, medik8sparams.DefaultTimeout, farparams.DefaultPollInterval).Should( + BeNumerically(">=", farparams.ExpectedReplicas), + "FAR deployment should recover to %d Ready replicas after workers are uncordoned", + farparams.ExpectedReplicas) + }) + }) }) func buildFARUnstructured( @@ -801,18 +1234,35 @@ func waitForRemediation( ctx context.Context, k8sClient client.Client, nodeName, oldBootID string, ) { + GinkgoHelper() + + waitForRemediationWithTimeouts(ctx, k8sClient, nodeName, oldBootID, + farparams.NodeRebootTimeout, farparams.NodeReadyTimeout) +} + +func waitForRemediationWithTimeouts( + ctx context.Context, k8sClient client.Client, + nodeName, oldBootID string, + rebootTimeout, readyTimeout time.Duration, +) { + // GinkgoHelper() attributes any failure below to the calling It block, + // correctly whether this is invoked directly (CP test) or through the + // waitForRemediation wrapper (which adds a stack frame). It composes + // across nesting depth, unlike a fixed ExpectWithOffset. + GinkgoHelper() + By("Waiting for node to reboot") Expect(farutils.WaitForNodeReboot( ctx, k8sClient, nodeName, oldBootID, - farparams.NodeRebootTimeout, GinkgoWriter.Printf)).To(Succeed(), + rebootTimeout, GinkgoWriter.Printf)).To(Succeed(), "Node %s did not reboot", nodeName) By("Waiting for node to become Ready") Expect(farutils.WaitForNodeReady( ctx, k8sClient, nodeName, - farparams.NodeReadyTimeout, GinkgoWriter.Printf)).To(Succeed(), + readyTimeout, GinkgoWriter.Printf)).To(Succeed(), "Node %s did not become Ready after reboot", nodeName) } @@ -945,6 +1395,47 @@ func logPodDiagnostics(ctx context.Context, k8sClient client.Client, pod *corev1 } } +func createWorkloadPod( + ctx context.Context, k8sClient client.Client, nodeName string, +) *corev1.Pod { + workloadPod := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{ + GenerateName: "far-workload-test-", + Namespace: medik8sparams.OperatorNs, + }, + Spec: corev1.PodSpec{ + NodeName: nodeName, + RestartPolicy: corev1.RestartPolicyAlways, + Containers: []corev1.Container{{ + Name: "workload", + Image: farparams.WorkloadTestImage, + Command: []string{"sleep", "infinity"}, + }}, + }, + } + + ExpectWithOffset(1, k8sClient.Create(ctx, workloadPod)).To(Succeed()) + DeferCleanup(func() { + if err := k8sClient.Delete(ctx, workloadPod); err != nil && !k8serrors.IsNotFound(err) { + GinkgoWriter.Printf("WARNING: failed to delete workload pod %s: %v\n", + workloadPod.Name, err) + } + }) + + EventuallyWithOffset(1, func() corev1.PodPhase { + pod := &corev1.Pod{} + if err := k8sClient.Get(ctx, client.ObjectKey{ + Name: workloadPod.Name, Namespace: workloadPod.Namespace, + }, pod); err != nil { + return corev1.PodPending + } + + return pod.Status.Phase + }, farparams.WorkloadPodReadyTimeout, farparams.DefaultPollInterval).Should(Equal(corev1.PodRunning)) + + return workloadPod +} + func removeWorkloadImage(ctx context.Context, nodeName string) { GinkgoWriter.Printf("Removing workload image from node %s to prevent corrupt overlay layers\n", nodeName) diff --git a/tests/internal/helpers/clusteroperator.go b/tests/internal/helpers/clusteroperator.go new file mode 100644 index 0000000000..56b2082e92 --- /dev/null +++ b/tests/internal/helpers/clusteroperator.go @@ -0,0 +1,58 @@ +package helpers + +import ( + "context" + "time" + + configv1 "github.com/openshift/api/config/v1" + "k8s.io/apimachinery/pkg/util/wait" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +// WaitForClusterOperatorHealthy polls until the named ClusterOperator reports +// Available=True, Progressing=False, Degraded=False. +func WaitForClusterOperatorHealthy( + ctx context.Context, k8sClient client.Client, + operatorName string, timeout, pollInterval time.Duration, + logf func(string, ...interface{}), +) error { + log := func(format string, args ...interface{}) { + if logf != nil { + logf(format, args...) + } + } + + return wait.PollUntilContextTimeout(ctx, pollInterval, timeout, true, + func(ctx context.Context) (bool, error) { + clusterOperator := &configv1.ClusterOperator{} + if err := k8sClient.Get(ctx, client.ObjectKey{Name: operatorName}, clusterOperator); err != nil { + log("WARNING: failed to get ClusterOperator %s: %v\n", operatorName, err) + + return false, nil + } + + available := false + progressing := true + degraded := true + + for _, cond := range clusterOperator.Status.Conditions { + switch cond.Type { //nolint:exhaustive + case configv1.OperatorAvailable: + available = cond.Status == configv1.ConditionTrue + case configv1.OperatorProgressing: + progressing = cond.Status == configv1.ConditionTrue + case configv1.OperatorDegraded: + degraded = cond.Status == configv1.ConditionTrue + } + } + + if available && !progressing && !degraded { + return true, nil + } + + log("ClusterOperator %s: Available=%v Progressing=%v Degraded=%v\n", + operatorName, available, progressing, degraded) + + return false, nil + }) +} diff --git a/tests/internal/labels/labels.go b/tests/internal/labels/labels.go index 5f2ebd1704..724a5105d4 100644 --- a/tests/internal/labels/labels.go +++ b/tests/internal/labels/labels.go @@ -63,4 +63,11 @@ const ( PlatformBareMetal = "platform:baremetal" // PlatformAny marks tests that run on any platform. PlatformAny = "platform:any" + + // TopologyControlPlane marks tests targeting control plane nodes. + TopologyControlPlane = "topology:control-plane" + // TopologyMinimalWorker marks tests with reduced worker count. + TopologyMinimalWorker = "topology:minimal-worker" + // TopologyZeroWorker marks tests with zero schedulable workers. + TopologyZeroWorker = "topology:zero-worker" ) diff --git a/vendor/github.com/medik8s/common/LICENSE b/vendor/github.com/medik8s/common/LICENSE new file mode 100644 index 0000000000..261eeb9e9f --- /dev/null +++ b/vendor/github.com/medik8s/common/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/vendor/github.com/medik8s/common/pkg/labels/labels.go b/vendor/github.com/medik8s/common/pkg/labels/labels.go new file mode 100644 index 0000000000..87409558ef --- /dev/null +++ b/vendor/github.com/medik8s/common/pkg/labels/labels.go @@ -0,0 +1,14 @@ +package labels + +const ( + // WorkerRole is the role label of worker nodes + WorkerRole = "node-role.kubernetes.io/worker" + // MasterRole is the old role label of control plane nodes + MasterRole = "node-role.kubernetes.io/master" + // ControlPlaneRole is the new role label of control plane nodes + ControlPlaneRole = "node-role.kubernetes.io/control-plane" + // DefaultTemplate label indicates to third party tools (e.g. UI) the default remediation template in case several exists. + DefaultTemplate = "remediation.medik8s.io/default-template" + // ExcludeFromRemediation label would be put on a node with value "true" in order to indicate this node should not be remediated. + ExcludeFromRemediation = "remediation.medik8s.io/exclude-from-remediation" +) diff --git a/vendor/modules.txt b/vendor/modules.txt index 88de41d222..0e7b6fb060 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -210,6 +210,9 @@ github.com/lib/pq/scram # github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de ## explicit github.com/liggitt/tabwriter +# github.com/medik8s/common v1.17.0 +## explicit; go 1.20 +github.com/medik8s/common/pkg/labels # github.com/medik8s/node-maintenance-operator v0.21.0 ## explicit; go 1.25.0 github.com/medik8s/node-maintenance-operator/api/v1beta1