far: Add CP remediation, 2-worker leader failover, and 0-worker topology destructive tests

AGENTS.md

@@ -110,6 +110,7 @@ All `It`/`DescribeTable` specs should be labelled using constants from `tests/in
 | Disruption | `disruption:destructive`, `disruption:nondestructive`                                                              |
 | Component  | `component:controller`, `component:remediation`, `component:webhook`, `component:metrics`, `component:post-deploy` |
 | Platform   | `platform:aws`, `platform:baremetal`, `platform:any`                                                               |
+| Topology   | `topology:control-plane`, `topology:minimal-worker`, `topology:zero-worker`                                        |
 
 ### Key Conventions
 

go.mod

@@ -24,7 +24,7 @@ require (
 	k8s.io/apimachinery v0.35.2
 	k8s.io/client-go v12.0.0+incompatible
 	k8s.io/klog/v2 v2.130.1
-	k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 // indirect
+	k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2
 	sigs.k8s.io/controller-runtime v0.22.5
 )
 
@@ -126,6 +126,7 @@ require (
 )
 
 require (
+	github.com/medik8s/common v1.17.0
 	github.com/medik8s/node-maintenance-operator v0.21.0
 	github.com/rh-ecosystem-edge/eco-goinfra v0.0.0-20260504223242-6b9a2708aabf
 	k8s.io/apiextensions-apiserver v0.34.7

go.sum

@@ -239,6 +239,8 @@ github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJ
 github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
 github.com/maruel/natural v1.1.1 h1:Hja7XhhmvEFhcByqDoHz9QZbkWey+COd9xWfCfn1ioo=
 github.com/maruel/natural v1.1.1/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg=
+github.com/medik8s/common v1.17.0 h1:AmJKx0tzqGZF27Ot0A4ak85q0F0zqUkVyCvYmm67rtY=
+github.com/medik8s/common v1.17.0/go.mod h1:A9jYldC6PZcAuBowNNm712FqWdASB2ey5Vjp8MYN/PY=
 github.com/medik8s/node-maintenance-operator v0.21.0 h1:9Xxe/1KM0qj1D3S4YI57qi6D/1BmfKa2y59UovBw/mQ=
 github.com/medik8s/node-maintenance-operator v0.21.0/go.mod h1:sRYMxJxeOR0J9d0oMvuakpH6/09zFt3w07XMAd+/1/s=
 github.com/mfridman/tparse v0.18.0 h1:wh6dzOKaIwkUGyKgOntDW4liXSo37qg5AXbIhkMV3vE=

tests/far-operator/README.md

@@ -199,3 +199,42 @@ Validates the full customer upgrade path: install GA FAR from redhat-operators o
 - **Env vars (optional, have defaults)**: `MEDIK8S_OPERATOR_PACKAGE` (default: `fence-agents-remediation`), `MEDIK8S_TARGET_CHANNEL` (default: `stable`)
 - **Standalone**: `ginkgo --label-filter="far && tier:upgrade" ./tests/far-operator/...`
 - **Pass criteria**: FAR deployment Ready on OCP N-1, OCP upgrade completes (Progressing=False, Available=True, Degraded=False), FAR deployment Ready after OCP upgrade, FAR CSV in Succeeded phase after catalog switch (new CSV if Konflux version is higher than GA, same CSV if versions match), controller image changes after operator upgrade (skipped on version parity), remediation succeeds after OCP upgrade and after catalog switch (node rebooted via boot ID change, node recovers to Ready), workload pods evicted after each fencing cycle
+
+### 17. Remediate a Control Plane Node and Verify etcd Quorum Preservation ([OCP-90217](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-90217))
+
+Fences any control plane node via fence_aws and verifies etcd quorum is preserved. Checks etcd ClusterOperator health before and after remediation. Creates a test workload pod pinned to the CP target node, then verifies it is evicted after fencing. Confirms the fenced CP node reboots (boot ID change), returns to Ready, the etcd ClusterOperator recovers (Available=True, Degraded=False), and the FAR CR reaches its terminal status conditions (Processing=False, FenceAgentActionSucceeded=True, Succeeded=True). FAR lifecycle Events are deliberately not asserted on a control-plane target: Kubernetes Events are best-effort and can be dropped during the apiserver/etcd disruption the CP reboot causes, so the durable CR status conditions prove the outcome instead (the full Event bundle is still asserted on the worker specs).
+
+- **Operators**: FAR
+- **Cluster**: AWS IPI, 3 control plane nodes (etcd quorum requires majority)
+- **Storage**: None
+- **Environment**: Connected
+- **Labels**: `tier:acceptance`, `disruption:destructive`, `platform:aws`, `frequency:weekly`, `component:remediation`, `topology:control-plane`
+- **Env vars (required)**: AWS credentials provisioned by the `medik8s-aws-credentials` CI step
+- **Standalone**: `ginkgo --label-filter="far && topology:control-plane" ./tests/far-operator/...`
+- **Pass criteria**: etcd ClusterOperator healthy before and after remediation, CP node rebooted (boot ID change), CP node returns to Ready, workload pod evicted from CP node, FAR CR status conditions Processing=False, FenceAgentActionSucceeded=True, Succeeded=True (FAR lifecycle Events are not asserted on a control-plane target - they are best-effort and can be dropped during the CP reboot's apiserver/etcd disruption)
+
+### 18. Fence Leader and Complete FAR Remediation with Only 2 Schedulable Workers ([OCP-90218](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-90218))
+
+Simulates a minimal 2-worker topology by cordoning extra workers, then fences the FAR leader node via fence_aws, forcing leader election failover to the surviving worker. Verifies the surviving worker takes over leadership and completes remediation despite degraded capacity (at least 1 FAR replica stays Running), workloads are evicted, and FAR recovers to 2 replicas once the cordoned capacity is restored (the fenced node stays NoSchedule-tainted until its CR is deleted, so the recovered replica lands on an uncordoned worker).
+
+- **Operators**: FAR
+- **Cluster**: AWS IPI, 2+ worker nodes (extra workers are cordoned to simulate topology)
+- **Storage**: None
+- **Environment**: Connected
+- **Labels**: `tier:acceptance`, `disruption:destructive`, `platform:aws`, `frequency:weekly`, `component:remediation`, `topology:minimal-worker`
+- **Env vars (required)**: AWS credentials provisioned by the `medik8s-aws-credentials` CI step
+- **Standalone**: `ginkgo --label-filter="far && topology:minimal-worker" ./tests/far-operator/...`
+- **Pass criteria**: At least 1 FAR replica Running during degraded capacity, leader node rebooted (boot ID change), leader node returns to Ready, workload pod evicted, FAR recovers to 2 replicas after schedulable capacity is restored
+
+### 19. Verify FAR Deployment Unavailability with Zero Schedulable Workers ([OCP-90308](https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-90308))
+
+Validates FAR operator behavior when all worker nodes are cordoned, simulating a 0-worker topology. Cordons all workers, deletes FAR pods to force reschedule attempts, verifies the deployment reports 0 Ready replicas, then uncordons workers and verifies FAR recovers to full replica count.
+
+- **Operators**: FAR
+- **Cluster**: AWS IPI, 1+ worker nodes (all cordoned to simulate 0-worker topology)
+- **Storage**: None
+- **Environment**: Connected
+- **Labels**: `tier:acceptance`, `disruption:destructive`, `platform:aws`, `frequency:weekly`, `component:controller`, `topology:zero-worker`
+- **Env vars (required)**: AWS credentials provisioned by the `medik8s-aws-credentials` CI step
+- **Standalone**: `ginkgo --label-filter="far && topology:zero-worker" ./tests/far-operator/...`
+- **Pass criteria**: FAR deployment Ready before test, FAR deployment has 0 Ready replicas after pods deleted on cordoned workers, FAR deployment recovers to 2 Ready replicas after uncordoning

tests/far-operator/internal/farparams/const.go

@@ -64,9 +64,6 @@ const (
 	// NodeReadyTimeout is how long to wait for a node to become Ready after reboot.
 	NodeReadyTimeout = 10 * time.Minute
 
-	// NodeNotReadyTimeout is how long to wait for a node to become NotReady after kubelet stop.
-	NodeNotReadyTimeout = 5 * time.Minute
-
 	// NodeRebootTimeout is how long to wait for a node reboot to complete.
 	NodeRebootTimeout = 6 * time.Minute
 
@@ -76,6 +73,15 @@ const (
 	// FARConditionTimeout is how long to wait for a FAR CR condition to appear.
 	FARConditionTimeout = 2 * time.Minute
 
+	// EventVerifyTimeout is how long to wait for Kubernetes lifecycle events to
+	// appear. Longer than FARConditionTimeout because event delivery lags the CR
+	// condition and the shared, rate-limited API client can throttle event List
+	// calls during the destructive suite.
+	EventVerifyTimeout = 5 * time.Minute
+	// EventVerifyInterval is the poll interval for event verification. Longer than
+	// DefaultPollInterval to reduce List pressure on the shared rate-limited client.
+	EventVerifyInterval = 10 * time.Second
+
 	// RemediationCRDeletionTimeout is how long to wait for a FAR/FART CR to be fully deleted.
 	RemediationCRDeletionTimeout = 2 * time.Minute
 
@@ -123,6 +129,24 @@ const (
 	// SharedCredentialsSecretName is the Secret created by the test suite to hold
 	// fence agent credentials in the format expected by SharedSecretName.
 	SharedCredentialsSecretName = "far-test-shared-credentials"
+
+	// MinControlPlaneNodes is the minimum Ready CP nodes needed for safe CP remediation.
+	MinControlPlaneNodes = 3
+
+	// CPRebootTimeout is how long to wait for a CP node reboot (slower than worker due to etcd).
+	CPRebootTimeout = 10 * time.Minute
+	// CPNodeReadyTimeout is how long to wait for a CP node to return Ready after reboot.
+	CPNodeReadyTimeout = 12 * time.Minute
+	// EtcdRejoinTimeout is how long to wait for etcd ClusterOperator to recover.
+	EtcdRejoinTimeout = 10 * time.Minute
+
+	// MinWorkersForDestructiveTests is the minimum Ready workers for standard destructive tests.
+	MinWorkersForDestructiveTests = 3
+	// MinWorkersForTwoWorkerTest is the minimum Ready workers for the 2-worker topology test.
+	MinWorkersForTwoWorkerTest = 2
+
+	// TestCordonAnnotation marks nodes cordoned by the test for cleanup identification.
+	TestCordonAnnotation = "system-tests.medik8s.io/cordoned-for-topology-test"
 )
 
 // WorkloadTestImage is the container image used for test workload pods.

tests/far-operator/internal/farutils/cluster.go

@@ -3,7 +3,9 @@ package farutils
 import (
 	"context"
 	"fmt"
+	"time"
 
+	commonlabels "github.com/medik8s/common/pkg/labels"
 	configv1 "github.com/openshift/api/config/v1"
 	corev1 "k8s.io/api/core/v1"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -53,33 +55,168 @@ func GetAWSCredentials(
 }
 
 // BuildAWSNodeParameters builds the --plug node parameter map for fence_aws
-// from the list of worker nodes.
+// from all Ready nodes (workers and control plane) so that both worker and
+// CP remediation tests can use the same parameter map.
 func BuildAWSNodeParameters(ctx context.Context, k8sClient client.Client) (map[string]map[string]string, error) {
 	nodeList := &corev1.NodeList{}
-	if err := k8sClient.List(ctx, nodeList, client.MatchingLabels{"node-role.kubernetes.io/worker": ""}); err != nil {
-		return nil, fmt.Errorf("failed to list worker nodes: %w", err)
+	if err := k8sClient.List(ctx, nodeList); err != nil {
+		return nil, fmt.Errorf("failed to list nodes: %w", err)
 	}
 
 	plugMap := make(map[string]string)
 
 	for i := range nodeList.Items {
 		node := &nodeList.Items[i]
 
-		if node.Spec.Unschedulable || !helpers.IsNodeReady(node) {
+		if !helpers.IsNodeReady(node) {
 			continue
 		}
 
 		instanceID, err := helpers.ExtractAWSInstanceID(node)
 		if err != nil {
-			return nil, fmt.Errorf("ready worker %s has invalid providerID: %w", node.Name, err)
+			return nil, fmt.Errorf("ready node %s has invalid providerID: %w", node.Name, err)
 		}
 
 		plugMap[node.Name] = instanceID
 	}
 
 	if len(plugMap) == 0 {
-		return nil, fmt.Errorf("no worker nodes with valid AWS providerID")
+		return nil, fmt.Errorf("no nodes with valid AWS providerID")
 	}
 
 	return map[string]map[string]string{farparams.NodeIdentifierAWS: plugMap}, nil
 }
+
+// GetReadyControlPlaneNodes returns all Ready control plane nodes.
+func GetReadyControlPlaneNodes(ctx context.Context, k8sClient client.Client) ([]corev1.Node, error) {
+	nodeList := &corev1.NodeList{}
+	if err := k8sClient.List(ctx, nodeList); err != nil {
+		return nil, fmt.Errorf("failed to list nodes: %w", err)
+	}
+
+	var cpNodes []corev1.Node
+
+	for i := range nodeList.Items {
+		node := &nodeList.Items[i]
+
+		_, hasCP := node.Labels[commonlabels.ControlPlaneRole]
+		_, hasMaster := node.Labels[commonlabels.MasterRole]
+
+		if (hasCP || hasMaster) && helpers.IsNodeReady(node) {
+			cpNodes = append(cpNodes, *node)
+		}
+	}
+
+	return cpNodes, nil
+}
+
+// SelectControlPlaneNode returns a Ready CP node that is not in the exclude list.
+func SelectControlPlaneNode(
+	ctx context.Context, k8sClient client.Client, excludeNodes ...string,
+) (*corev1.Node, error) {
+	cpNodes, err := GetReadyControlPlaneNodes(ctx, k8sClient)
+	if err != nil {
+		return nil, err
+	}
+
+	excluded := make(map[string]bool, len(excludeNodes))
+	for _, name := range excludeNodes {
+		excluded[name] = true
+	}
+
+	for i := range cpNodes {
+		if !excluded[cpNodes[i].Name] {
+			return &cpNodes[i], nil
+		}
+	}
+
+	return nil, fmt.Errorf("no eligible CP node found (excluded: %v)", excludeNodes)
+}
+
+// CordonExtraWorkers cordons all Ready worker nodes except those in keepNames.
+// Returns the names of nodes that were cordoned by this function.
+func CordonExtraWorkers(
+	ctx context.Context, k8sClient client.Client, keepNames []string,
+) ([]string, error) {
+	keepSet := make(map[string]bool, len(keepNames))
+	for _, name := range keepNames {
+		keepSet[name] = true
+	}
+
+	nodeList := &corev1.NodeList{}
+	if err := k8sClient.List(ctx, nodeList,
+		client.MatchingLabels{commonlabels.WorkerRole: ""}); err != nil {
+		return nil, fmt.Errorf("failed to list worker nodes: %w", err)
+	}
+
+	var cordoned []string
+
+	for i := range nodeList.Items {
+		node := &nodeList.Items[i]
+
+		if keepSet[node.Name] || node.Spec.Unschedulable || !helpers.IsNodeReady(node) {
+			continue
+		}
+
+		patch := client.MergeFrom(node.DeepCopy())
+		node.Spec.Unschedulable = true
+
+		if node.Annotations == nil {
+			node.Annotations = make(map[string]string)
+		}
+
+		node.Annotations[farparams.TestCordonAnnotation] = "true"
+
+		if err := k8sClient.Patch(ctx, node, patch); err != nil {
+			return cordoned, fmt.Errorf("failed to cordon node %s: %w", node.Name, err)
+		}
+
+		cordoned = append(cordoned, node.Name)
+	}
+
+	return cordoned, nil
+}
+
+// UncordonNodes restores schedulability for the given node names.
+func UncordonNodes(
+	ctx context.Context, k8sClient client.Client, nodeNames []string,
+	logf ...func(string, ...interface{}),
+) {
+	log := func(format string, args ...interface{}) {
+		if len(logf) > 0 {
+			logf[0](format, args...)
+		}
+	}
+
+	for _, name := range nodeNames {
+		node := &corev1.Node{}
+		if err := k8sClient.Get(ctx, client.ObjectKey{Name: name}, node); err != nil {
+			log("WARNING: failed to get node %s for uncordon: %v\n", name, err)
+
+			continue
+		}
+
+		if !node.Spec.Unschedulable {
+			continue
+		}
+
+		patch := client.MergeFrom(node.DeepCopy())
+		node.Spec.Unschedulable = false
+		delete(node.Annotations, farparams.TestCordonAnnotation)
+
+		if err := k8sClient.Patch(ctx, node, patch); err != nil {
+			log("WARNING: failed to uncordon node %s: %v\n", name, err)
+		}
+	}
+}
+
+// WaitForClusterOperatorHealthy polls until the named ClusterOperator reports
+// Available=True, Progressing=False, Degraded=False.
+func WaitForClusterOperatorHealthy(
+	ctx context.Context, k8sClient client.Client,
+	operatorName string, timeout time.Duration,
+	logf func(string, ...interface{}),
+) error {
+	return helpers.WaitForClusterOperatorHealthy(ctx, k8sClient, operatorName, timeout,
+		farparams.DefaultPollInterval, logf)
+}

tests/far-operator/tests/far_controller_lifecycle.go

@@ -45,25 +45,39 @@ var _ = Describe("FAR Controller Lifecycle Tests",
 			func() {
 				By("Getting the current active FAR controller pod")
 
-				pods, err := farutils.GetFARControllerPods(ctx, APIClient)
-				Expect(err).ToNot(HaveOccurred())
-				Expect(pods).ToNot(BeEmpty(), "No running FAR controller pods found")
+				// Leader election is eventually-consistent: a prior destructive spec
+				// (the 0-worker test deletes FAR pods; fencing specs evict them) can
+				// leave the controller Lease naming a pod that no longer exists until
+				// re-election updates it. Retry until the Lease resolves to a live,
+				// Running controller pod instead of failing on a one-shot lookup.
+				var (
+					oldLeaderPod  *corev1.Pod
+					oldLeaderNode string
+				)
 
-				oldLeaderNode, err := farutils.GetActiveFARControllerNode(ctx, APIClient)
-				Expect(err).ToNot(HaveOccurred())
+				Eventually(func(assertion Gomega) {
+					pods, err := farutils.GetFARControllerPods(ctx, APIClient)
+					assertion.Expect(err).ToNot(HaveOccurred())
+					assertion.Expect(pods).ToNot(BeEmpty(), "No running FAR controller pods found")
+
+					leaderNode, err := farutils.GetActiveFARControllerNode(ctx, APIClient)
+					assertion.Expect(err).ToNot(HaveOccurred())
 
-				var oldLeaderPod *corev1.Pod
+					oldLeaderPod = nil
 
-				for i := range pods {
-					if pods[i].Spec.NodeName == oldLeaderNode {
-						oldLeaderPod = &pods[i]
+					for i := range pods {
+						if pods[i].Spec.NodeName == leaderNode {
+							oldLeaderPod = &pods[i]
 
-						break
+							break
+						}
 					}
-				}
 
-				Expect(oldLeaderPod).ToNot(BeNil(),
-					"Could not find controller pod on leader node %s", oldLeaderNode)
+					assertion.Expect(oldLeaderPod).ToNot(BeNil(),
+						"Lease leader node %s has no Running controller pod yet", leaderNode)
+					oldLeaderNode = leaderNode
+				}, farparams.ControllerHandoverTimeout, farparams.DefaultPollInterval).Should(Succeed(),
+					"controller lease did not resolve to a live controller pod")
 
 				oldPodName := oldLeaderPod.Name
 				GinkgoWriter.Printf("Active controller pod: %s on node %s\n",