Custom taints and toleration node operation #9920

vkathole · 2024-06-07T10:27:17Z

No description provided.

ocs_ci/ocs/resources/pod.py

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

ocs_ci/ocs/resources/pod.py

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: vkathole-t26
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: vkathole-o1
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: vkathole-o1
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

Shrivaibavi · 2024-10-10T17:32:53Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+                if "config" in subscription_data.get("spec", {}):
+                    params = '[{"op": "remove", "path": "/spec/config"}]'
+                    sub_obj.patch(resource_name=sub, params=params, format_type="json")
+            time.sleep(180)


are we not supposed to remove the tolerations from the rook-ceph operator configmap and ocsinitializations.ocs.openshift.io too ??

They are not added in that place now, we are adding it to storagecluster yaml only

I see you have added tolerations on configmap and ocsint in func apply_custom_taint_and_toleration() and also since this test has the ability to run on all ODF versions >, < 4.16, the cleanup should be according to that. Removing toleration just from storagecluster might not be right for version < 4.16

Please check and add cleanup accordingly.

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

Shrivaibavi · 2024-10-10T17:35:36Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+    def test_negative_custom_taint(self, nodes):
+        """
+        Test runs the following steps
+        1. Taint odf nodes with non-ocs taint


Suggested change

1. Taint odf nodes with non-ocs taint

1. Taint odf worker nodes with non-ocs taint

Shrivaibavi · 2024-10-10T17:38:13Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+
+        assert not wait_for_pods_to_be_running(
+            timeout=120, sleep=15
+        ), "Pods are running when they should not be."


are we expecting all pods to go in a bad state ?

I see we apply tolerations on storagecluster and subscription other than ODF, are we sure all pods will not be running if the toleration is just not applied properly on sub when we are setting it properly on storagecluster ? Please check the scenario again. if we are setting the toleration properly on storagecluster few pods should be up and running.

No, we are not expecting all pods in bad state, some in pending and some in running state, wait_for_pods_to_be_running will fail with even one pod in pending state. so assert not wait_for_pods_to_be_running will work fine for that.

instead of checking for all the pods we can check for pods that are affected by the subscription changes. This will not be a blocker for the merge though.

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: vkathole-m4
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_negative_custom_taint
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

openshift-ci · 2024-11-12T08:09:33Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vkathole

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-11-12T08:09:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vkathole

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Vishakha Kathole <[email protected]>

Signed-off-by: vkathole <[email protected]>

ocs-ci

PR validation on existing cluster

Cluster Name: vkathole-m11
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_non_ocs_taint_and_tolerations tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_reboot_on_tainted_node tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_negative_custom_taint
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

Signed-off-by: vkathole <[email protected]>

ocs-ci

PR validation on existing cluster

Cluster Name: vkathole-m11
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_non_ocs_taint_and_tolerations tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_reboot_on_tainted_node tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_negative_custom_taint
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_non_ocs_taint_and_tolerations tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_reboot_on_tainted_node tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_negative_custom_taint
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job UNSTABLE (some or all tests failed).

Signed-off-by: vkathole <[email protected]>

PrasadDesala · 2024-11-13T10:05:44Z

ocs_ci/helpers/helpers.py

@@ -5392,3 +5401,130 @@ def verify_performance_profile_change(perf_profile):
    ), f"Performance profile is not updated successfully to {perf_profile}"
    logger.info(f"Performance profile successfully got updated to {perf_profile} mode")
    return True
+
+
+def apply_custom_taint_and_toleration(taint_label):


you can have the custom taint label defined here in the arg

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_non_ocs_taint_and_tolerations tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_reboot_on_tainted_node tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_negative_custom_taint
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job PASSED.

Signed-off-by: vkathole <[email protected]>

ocs-ci

Unknown PR validation on existing cluster

Cluster Name: vkathole-m11
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_non_ocs_taint_and_tolerations tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_reboot_on_tainted_node tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_negative_custom_taint
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job state: ABORTED.

ocs-ci

PR validation

Cluster Name:
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_non_ocs_taint_and_tolerations tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_reboot_on_tainted_node tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py::TestNonOCSTaintAndTolerations::test_negative_custom_taint
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job FAILED (installation failed, tests not executed).

Shrivaibavi · 2024-11-14T10:52:20Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+                if "config" in subscription_data.get("spec", {}):
+                    params = '[{"op": "remove", "path": "/spec/config"}]'
+                    sub_obj.patch(resource_name=sub, params=params, format_type="json")
+            time.sleep(180)


I see you have added tolerations on configmap and ocsint in func apply_custom_taint_and_toleration() and also since this test has the ability to run on all ODF versions >, < 4.16, the cleanup should be according to that. Removing toleration just from storagecluster might not be right for version < 4.16

Shrivaibavi · 2024-11-14T10:53:13Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+                if "config" in subscription_data.get("spec", {}):
+                    params = '[{"op": "remove", "path": "/spec/config"}]'
+                    sub_obj.patch(resource_name=sub, params=params, format_type="json")
+            time.sleep(180)


Please check and add cleanup accordingly.

Shrivaibavi · 2024-11-14T10:56:36Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+        logger.info(
+            "After adding toleration wait for some time for pods to respin as expected"
+        )
+        time.sleep(300)


Reinitializing the pod variable again would be the best approach instead of sleep. Otherwise reduce the sleep time and timeout in line 214.

Shrivaibavi · 2024-11-14T11:09:41Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+from tests.functional.z_cluster.nodes.test_node_replacement_proactive import (
+    delete_and_create_osd_node,
+    select_osd_node_name,
+)


We might have to modify
@pytest.mark.polarion_id("OCS-2705") markers accordingly. I see many tests added to this class. so markers to be modified accordingly

Shrivaibavi · 2024-11-14T11:33:42Z

tests/functional/z_cluster/nodes/test_non_ocs_taint_and_toleration.py

+
+        assert not wait_for_pods_to_be_running(
+            timeout=120, sleep=15
+        ), "Pods are running when they should not be."


instead of checking for all the pods we can check for pods that are affected by the subscription changes. This will not be a blocker for the merge though.

vkathole requested review from a team as code owners June 7, 2024 10:27

pull-request-size bot added the size/L PR that changes 100-499 lines label Jun 7, 2024

vkathole added team/e2e E2E team related issues/PRs and removed size/L PR that changes 100-499 lines labels Jun 7, 2024

vkathole self-assigned this Jun 7, 2024

openshift-merge-robot added the needs-rebase label Jun 14, 2024

Shrivaibavi reviewed Jul 31, 2024

View reviewed changes

PrasadDesala reviewed Aug 1, 2024

View reviewed changes

vkathole force-pushed the custom_taints_node_operation branch from 273c9cf to ee87c8d Compare September 19, 2024 10:38

pull-request-size bot added the size/M PR that changes 30-99 lines label Sep 19, 2024

openshift-merge-robot removed the needs-rebase label Sep 19, 2024

ocs-ci reviewed Sep 20, 2024

View reviewed changes

pull-request-size bot added size/L PR that changes 100-499 lines and removed size/M PR that changes 30-99 lines labels Sep 26, 2024

ocs-ci reviewed Sep 26, 2024

View reviewed changes

ocs-ci reviewed Oct 1, 2024

View reviewed changes

ocs-ci reviewed Oct 2, 2024

View reviewed changes

ocs-ci reviewed Oct 3, 2024

View reviewed changes

ocs-ci reviewed Oct 6, 2024

View reviewed changes

ocs-ci reviewed Oct 10, 2024

View reviewed changes

PrasadDesala reviewed Oct 10, 2024

View reviewed changes

Shrivaibavi reviewed Oct 10, 2024

View reviewed changes

Shrivaibavi added Squad/Brown and removed Squad/Brown labels Oct 10, 2024

Shrivaibavi requested review from a team and removed request for a team October 10, 2024 17:51

ocs-ci reviewed Oct 23, 2024

View reviewed changes

ocs-ci reviewed Nov 5, 2024

View reviewed changes

pull-request-size bot added the size/XL label Nov 12, 2024

openshift-merge-robot added the needs-rebase label Nov 12, 2024

vkathole added 10 commits November 12, 2024 13:42

Support custom taints node operation

1d16bfe

Signed-off-by: Vishakha Kathole <[email protected]>

Add changes

a733299

Signed-off-by: vkathole <[email protected]>

Fix test

1cd4ec2

Signed-off-by: vkathole <[email protected]>

add negative test case

7699001

Signed-off-by: vkathole <[email protected]>

Fix test

82fa976

Signed-off-by: vkathole <[email protected]>

change for ipi cluster

844f84e

Signed-off-by: vkathole <[email protected]>

add exception

5d5cd1d

Signed-off-by: vkathole <[email protected]>

increase timeout

d09eba1

Signed-off-by: vkathole <[email protected]>

include review comments

ec9704c

Signed-off-by: vkathole <[email protected]>

move function to helpers

bc093fd

Signed-off-by: vkathole <[email protected]>

vkathole force-pushed the custom_taints_node_operation branch from 49b3b71 to bc093fd Compare November 12, 2024 08:17

openshift-merge-robot removed the needs-rebase label Nov 12, 2024

vkathole added 3 commits November 12, 2024 13:52

correct docstrings

37efd8d

Signed-off-by: vkathole <[email protected]>

fix python pr check

305cc36

Signed-off-by: vkathole <[email protected]>

fix pr check

2eb2080

Signed-off-by: vkathole <[email protected]>

ocs-ci reviewed Nov 12, 2024

View reviewed changes

correct toleration

f10f06b

Signed-off-by: vkathole <[email protected]>

ocs-ci reviewed Nov 12, 2024

View reviewed changes

fix

2a91dab

Signed-off-by: vkathole <[email protected]>

PrasadDesala reviewed Nov 13, 2024

View reviewed changes

ocs-ci reviewed Nov 13, 2024

View reviewed changes

define taint_label

ae58c43

Signed-off-by: vkathole <[email protected]>

ocs-ci reviewed Nov 14, 2024

View reviewed changes

Shrivaibavi reviewed Nov 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom taints and toleration node operation #9920

Custom taints and toleration node operation #9920

vkathole commented Jun 7, 2024

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

Shrivaibavi Oct 10, 2024

vkathole Nov 5, 2024

Shrivaibavi Nov 14, 2024

Shrivaibavi Nov 14, 2024

Shrivaibavi Oct 10, 2024

vkathole Nov 5, 2024

Shrivaibavi Oct 10, 2024

Shrivaibavi Oct 10, 2024

vkathole Nov 5, 2024

Shrivaibavi Nov 14, 2024

ocs-ci left a comment

ocs-ci left a comment

openshift-ci bot commented Nov 12, 2024

openshift-ci bot commented Nov 12, 2024

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

PrasadDesala Nov 13, 2024

vkathole Nov 14, 2024

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

Shrivaibavi Nov 14, 2024

Shrivaibavi Nov 14, 2024

Shrivaibavi Nov 14, 2024

Shrivaibavi Nov 14, 2024

Shrivaibavi Nov 14, 2024

	1. Taint odf nodes with non-ocs taint
	1. Taint odf worker nodes with non-ocs taint

Custom taints and toleration node operation #9920

Are you sure you want to change the base?

Custom taints and toleration node operation #9920

Conversation

vkathole commented Jun 7, 2024

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 12, 2024

openshift-ci bot commented Nov 12, 2024

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment