feat: add fault [Node Resource Exhaustion] (SPAI-414)

documentation/library/faults/README.md

@@ -23,6 +23,7 @@ A fault is a solvable issue injected into an environment to create an incident.
 | [Misconfigured Kubernetes Workload Container Readiness Probe](./misconfigured-kubernetes-workload-container-readiness-probe.md) | Kubernetes |
 | [Modified Kubernetes Workload Container Environment Variable](./modified-kubernetes-workload-container-environment-variable.md) | Kubernetes |
 | [Modified Target Port Kubernetes Service](./modified-target-port-kubernetes-service.md) | Kubernetes |
+| [Node Resource Exhaustion](./node-resource-exhaustion.md) | Kubernetes |
 | [Nonexistent Kubernetes Workload Container Image](./nonexistent-kubernetes-workload-container-image.md) | Kubernetes |
 | [Nonexistent Kubernetes Workload Node](./nonexistent-kubernetes-workload-node.md) | Kubernetes |
 | [Nonexistent Kubernetes Workload Persistent Volume Claim](./nonexistent-kubernetes-workload-persistent-volume-claim.md) | Kubernetes |

documentation/library/faults/node-resource-exhaustion.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Node Resource Exhaustion
+
+| [Source](../../../../scenarios/sre/library/indexes/faults/node-resource-exhaustion.json) | [Schema](../../../../schemas/json/faults/node-resource-exhaustion.json) | [Implementation](../../../../scenarios/sre/project/roles/faults/tasks/inject_node_resource_exhaustion.yaml) | Kubernetes | Deployment, Performance |
+
+This fault injects excessive resource requests into a workload, causing nodes to run out of allocatable resources.
+
+## Expectation
+
+The faulted workload pods will consume excessive resources, causing other pods on the same node to be evicted or fail to schedule.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodNotReady
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
+- https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/
+- https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
+- https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/

documentation/library/scenarios/README.md

@@ -73,3 +73,4 @@ _For statistics on the distrubution spread of scenarios, please see [here](./sta
 | [102](./sre/102.md) | sre | medium |
 | [105](./sre/105.md) | sre | medium |
 | [114](./sre/114.md) | sre | low |
+| [121](./sre/121.md) | sre | medium |

documentation/library/scenarios/sre/121.md

@@ -0,0 +1,42 @@
+> [!NOTE]
+> This file is autogenerated. Please do not edit.
+
+# Scenario 121
+
+| [Source](../../../../scenarios/sre/library/indexes/scenarios/121.json) | Medium Complexity | Kubernetes | Deployment, Performance |
+
+## Description
+
+This scenario simulates node resource exhaustion by injecting excessive resource requests into the OpenTelemetry Demo's `product-catalog` service.
+
+## Applications
+
+- [OpenTelemetry Demo](../../applications/opentelemetry-demo.md)
+
+## Faults
+
+- [Node Resource Exhaustion](../../../../scenarios/sre/docs/faults.md#Node-Resource-Exhaustion)
+
+## Solution Guides
+
+Below is a noncomprehensive list of step by step guides which can lead to remediating the one or more problems in the scenario.
+
+A **solution** represents the process of fixing an issue created by a fault. To remediate the issues in the scenarios, follow the solutions in order of appearance.
+
+A solution can have multiple **variants**. A variant is a different process of deriving the solution. If multiple variants are presented, pick only one to solve the problem.
+
+### Solution 1
+
+#### Variant 1
+
+1. Revert the resource request changes to restore normal resource usage.
+```shell
+kubectl -n {{ args.kubernetesObject.metadata.namespace }} rollout undo {{ args.kubernetesObject.kind | lower }}/{{ args.kubernetesObject.metadata.name }}
+```
+
+#### Variant 2
+
+1. Manually edit the manifest and set resource requests to appropriate values.
+```shell
+kubectl -n {{ args.kubernetesObject.metadata.namespace }} edit {{ args.kubernetesObject.kind | lower }} {{ args.kubernetesObject.metadata.name }}
+```

documentation/library/scenarios/statistics.md

@@ -7,16 +7,16 @@
 
 | BookInfo | OpenTelemetry Demo |
 | --- | --- |
-| 5 | 61 |
+| 5 | 36 |
 
 ## Category Distribution
 
 | FinOps | SRE |
 | --- | --- |
-| 2 | 64 |
+| 2 | 39 |
 
 ## Complexity Distribution
 
 | Low | Medium | High |
 | --- | --- | --- |
-| 17 | 46 | 3 |
+| 17 | 23 | 1 |

scenarios/sre/library/indexes/faults/37.json

@@ -0,0 +1,92 @@
+{
+    "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/fault.json",
+    "alerts": {
+        "application": [
+            "KubePodNotReady"
+        ]
+    },
+    "arguments": {
+        "jsonSchema": {
+            "properties": {
+                "kubernetesObject": {
+                    "properties": {
+                        "apiVersion": {
+                            "enum": [
+                                "apps/v1"
+                            ],
+                            "type": "string"
+                        },
+                        "kind": {
+                            "enum": [
+                                "Deployment",
+                                "StatefulSet"
+                            ],
+                            "type": "string"
+                        },
+                        "metadata": {
+                            "properties": {
+                                "name": {
+                                    "type": "string"
+                                },
+                                "namespace": {
+                                    "type": "string"
+                                }
+                            },
+                            "required": [
+                                "name",
+                                "namespace"
+                            ],
+                            "type": "object"
+                        }
+                    },
+                    "required": [
+                        "apiVersion",
+                        "kind",
+                        "metadata"
+                    ],
+                    "type": "object"
+                }
+            },
+            "required": [
+                "kubernetesObject"
+            ],
+            "type": "object"
+        }
+    },
+    "description": "This fault injects excessive resource requests into a workload, causing nodes to run out of allocatable resources.",
+    "expectation": "The faulted workload pods will consume excessive resources, causing other pods on the same node to be evicted or fail to schedule.",
+    "id": "node-resource-exhaustion",
+    "index": 37,
+    "name": "Node Resource Exhaustion",
+    "platform": "Kubernetes",
+    "resources": [
+        "https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/",
+        "https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/",
+        "https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/",
+        "https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/"
+    ],
+    "solutions": {
+        "templates": [
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} rollout undo {{ args.kubernetesObject.kind | lower }}/{{ args.kubernetesObject.metadata.name }}",
+                        "text": "Revert the resource request changes to restore normal resource usage."
+                    }
+                ]
+            },
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} edit {{ args.kubernetesObject.kind | lower }} {{ args.kubernetesObject.metadata.name }}",
+                        "text": "Manually edit the manifest and set resource requests to appropriate values."
+                    }
+                ]
+            }
+        ]
+    },
+    "tags": [
+        "Deployment",
+        "Performance"
+    ]
+}

scenarios/sre/library/indexes/scenarios/121.json

@@ -0,0 +1,64 @@
+{
+    "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/scenario.json",
+    "alerts": [
+        "KubePodNotReady"
+    ],
+    "category": "sre",
+    "complexity": "medium",
+    "description": "This scenario simulates node resource exhaustion by injecting excessive resource requests into the OpenTelemetry Demo's `product-catalog` service.",
+    "disruptions": [
+        {
+            "injections": [
+                {
+                    "args": {
+                        "kubernetesObject": {
+                            "apiVersion": "apps/v1",
+                            "kind": "Deployment",
+                            "metadata": {
+                                "name": "product-catalog",
+                                "namespace": "otel-demo"
+                            }
+                        }
+                    },
+                    "id": "node-resource-exhaustion"
+                }
+            ]
+        }
+    ],
+    "environment": {
+        "applications": [
+            {
+                "id": "opentelemetry-demo"
+            }
+        ]
+    },
+    "id": 121,
+    "index": 121,
+    "platforms": [
+        "Kubernetes"
+    ],
+    "solutions": [
+        [
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} rollout undo {{ args.kubernetesObject.kind | lower }}/{{ args.kubernetesObject.metadata.name }}",
+                        "text": "Revert the resource request changes to restore normal resource usage."
+                    }
+                ]
+            },
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} edit {{ args.kubernetesObject.kind | lower }} {{ args.kubernetesObject.metadata.name }}",
+                        "text": "Manually edit the manifest and set resource requests to appropriate values."
+                    }
+                ]
+            }
+        ]
+    ],
+    "tags": [
+        "Deployment",
+        "Performance"
+    ]
+}

scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/converge.yml

@@ -0,0 +1,20 @@
+---
+- name: Run fault injection test
+  hosts:
+    - localhost
+  tasks:
+    - name: Import faults role
+      ansible.builtin.import_role:
+        name: faults
+      vars:
+        faults_cluster:
+          kubeconfig: "{{ cluster.kubeconfig }}"
+        faults_injection_tasks:
+          - id: node-resource-exhaustion
+            args:
+kubernetesObject:
+  apiVersion: apps/v1
+  kind: Deployment
+  metadata:
+    name: test-workload
+    namespace: node-resource-exhaustion-test

scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/create.yml

@@ -0,0 +1,48 @@
+---
+- name: Create test resources
+  hosts:
+    - localhost
+  tasks:
+    - name: Create test namespace
+      kubernetes.core.k8s:
+        kubeconfig: "{{ cluster.kubeconfig }}"
+        resource_definition:
+          apiVersion: v1
+          kind: Namespace
+          metadata:
+            name: node-resource-exhaustion-test
+        state: present
+
+    - name: Create test deployment
+      kubernetes.core.k8s:
+        kubeconfig: "{{ cluster.kubeconfig }}"
+        resource_definition:
+          apiVersion: apps/v1
+          kind: Deployment
+          metadata:
+            name: test-workload
+            namespace: node-resource-exhaustion-test
+          spec:
+            replicas: 1
+            selector:
+              matchLabels:
+                app: test-workload
+            template:
+              metadata:
+                labels:
+                  app: test-workload
+              spec:
+                containers:
+                  - name: server
+                    image: registry.access.redhat.com/ubi10-minimal:10.2-1781509346
+                    command:
+                      - /bin/sh
+                    args:
+                      - -c
+                      - "sleep infinity"
+                    ports:
+                      - containerPort: 8080
+                        name: http
+        state: present
+        wait: true
+        wait_timeout: 120

scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/destroy.yml

@@ -0,0 +1,15 @@
+---
+- name: Destroy test resources
+  hosts:
+    - localhost
+  tasks:
+    - name: Delete test namespace
+      kubernetes.core.k8s:
+        kubeconfig: "{{ cluster.kubeconfig }}"
+        resource_definition:
+          apiVersion: v1
+          kind: Namespace
+          metadata:
+            name: node-resource-exhaustion-test
+        state: absent
+        wait: true

scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/molecule.yml

@@ -0,0 +1,24 @@
+dependency:
+  name: galaxy
+ansible:
+  env:
+    ANSIBLE_ROLES_PATH: ../../..
+  executor:
+    backend: ansible-playbook
+    args:
+      ansible_playbook:
+      - --inventory=${MOLECULE_PROJECT_DIRECTORY}/../../../inventory
+      - --tags=inject_faults,untagged
+  playbooks:
+    create: create.yml
+    converge: converge.yml
+    destroy: destroy.yml
+    verify: verify.yml
+scenario:
+  name: inject_node_resource_exhaustion
+  test_sequence:
+  - syntax
+  - create
+  - converge
+  - verify
+  - destroy

scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/verify.yml

@@ -0,0 +1,21 @@
+---
+- name: Verify that fault injection succeeded
+  hosts:
+    - localhost
+  tasks:
+    - name: Retrieve pods
+      kubernetes.core.k8s_info:
+        kubeconfig: "{{ cluster.kubeconfig }}"
+        api_version: v1
+        kind: Pod
+        namespace: node-resource-exhaustion-test
+        label_selectors:
+          - app=test-workload
+      register: faults_pods
+
+    - name: Validate pod is in Pending state
+      ansible.builtin.assert:
+        that:
+          - faults_pods.resources | length > 0
+        fail_msg: No pods found. Fault injection may have failed.
+        success_msg: Pod found in expected Pending state.