feat: add fault [Exhausted Etcd Database Storage] (SPAI-414)

documentation/library/faults/README.md

@@ -11,6 +11,7 @@ A fault is a solvable issue injected into an environment to create an incident.
 | [Crashing Kubernetes Workload Init Container](./crashing-kubernetes-workload-init-container.md) | Kubernetes |
 | [Deleted Kubernetes Service](./deleted-kubernetes-service.md) | Kubernetes |
 | [Disabled Istio Ambient Mode Kubernetes Namespace](./disabled-istio-ambient-mode-kubernetes-namespace.md) | Kubernetes |
+| [Exhausted Etcd Database Storage](./exhausted-etcd-database-storage.md) | Kubernetes |
 | [Failing Name Resolution Kubernetes Workload DNS Policy](./failing-name-resolution-kubernetes-workload-dns-policy.md) | Kubernetes |
 | [Hanging Kubernetes Workload Init Container](./hanging-kubernetes-workload-init-container.md) | Kubernetes |
 | [Ingress Port Blocking Network Policy](./ingress-port-blocking-network-policy.md) | Kubernetes |

documentation/library/faults/exhausted-etcd-database-storage.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Exhausted Etcd Database Storage
+
+| [Source](../../../../scenarios/sre/library/indexes/faults/exhausted-etcd-database-storage.json) | [Schema](../../../../schemas/json/faults/exhausted-etcd-database-storage.json) | [Implementation](../../../../scenarios/sre/project/roles/faults/tasks/inject_exhausted_etcd_database_storage.yaml) | Kubernetes | Performance |
+
+This fault creates etcd storage pressure by writing a large number of ConfigMaps, causing increased API server latency and degraded cluster performance. This simulates real-world incidents where etcd storage growth causes progressive degradation of cluster operations.
+
+## Expectation
+
+The etcd database will experience increased storage pressure. The Kubernetes API server will exhibit higher latency for read and write operations.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Golden Signal Alerts
+
+- HighRequestLatency
+
+## Resources
+
+- https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
+- https://etcd.io/docs/v3.5/op-guide/maintenance/
+- https://kubernetes.io/docs/concepts/overview/components/#etcd
+- https://kubernetes.io/docs/tasks/debug/debug-cluster/

documentation/library/scenarios/README.md

@@ -73,3 +73,4 @@ _For statistics on the distrubution spread of scenarios, please see [here](./sta
 | [102](./sre/102.md) | sre | medium |
 | [105](./sre/105.md) | sre | medium |
 | [114](./sre/114.md) | sre | low |
+| [116](./sre/116.md) | sre | high |

documentation/library/scenarios/sre/116.md

@@ -0,0 +1,43 @@
+> [!NOTE]
+> This file is autogenerated. Please do not edit.
+
+# Scenario 116
+
+| [Source](../../../../scenarios/sre/library/indexes/scenarios/116.json) | High Complexity | Kubernetes | Performance |
+
+## Description
+
+This scenario simulates etcd storage pressure by filling the cluster's etcd database with ConfigMaps, causing increased API server latency and degrading cluster-wide operations including the OpenTelemetry Demo application.
+
+## Applications
+
+- [OpenTelemetry Demo](../../applications/opentelemetry-demo.md)
+
+## Faults
+
+- [Exhausted Etcd Database Storage](../../../../scenarios/sre/docs/faults.md#Exhausted-Etcd-Database-Storage)
+
+## Solution Guides
+
+Below is a noncomprehensive list of step by step guides which can lead to remediating the one or more problems in the scenario.
+
+A **solution** represents the process of fixing an issue created by a fault. To remediate the issues in the scenarios, follow the solutions in order of appearance.
+
+A solution can have multiple **variants**. A variant is a different process of deriving the solution. If multiple variants are presented, pick only one to solve the problem.
+
+### Solution 1
+
+#### Variant 1
+
+1. Remove the ConfigMaps that are consuming etcd storage.
+```shell
+kubectl -n otel-demo delete configmap -l app=etcd-storage-fill
+```
+
+2. Compact and defragment the etcd database to reclaim storage space.
+
+#### Variant 2
+
+1. Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage.
+
+2. Increase the etcd storage quota if the current limit is too low for the cluster workload.

documentation/library/scenarios/statistics.md

@@ -7,16 +7,16 @@
 
 | BookInfo | OpenTelemetry Demo |
 | --- | --- |
-| 5 | 61 |
+| 5 | 36 |
 
 ## Category Distribution
 
 | FinOps | SRE |
 | --- | --- |
-| 2 | 64 |
+| 2 | 39 |
 
 ## Complexity Distribution
 
 | Low | Medium | High |
 | --- | --- | --- |
-| 17 | 46 | 3 |
+| 17 | 22 | 2 |

scenarios/sre/library/indexes/faults/32.json

@@ -0,0 +1,96 @@
+{
+    "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/fault.json",
+    "alerts": {
+        "goldenSignal": [
+            "HighRequestLatency"
+        ]
+    },
+    "arguments": {
+        "jsonSchema": {
+            "properties": {
+                "kubernetesObject": {
+                    "properties": {
+                        "apiVersion": {
+                            "enum": [
+                                "apps/v1"
+                            ],
+                            "type": "string"
+                        },
+                        "kind": {
+                            "enum": [
+                                "Deployment",
+                                "StatefulSet"
+                            ],
+                            "type": "string"
+                        },
+                        "metadata": {
+                            "properties": {
+                                "name": {
+                                    "type": "string"
+                                },
+                                "namespace": {
+                                    "type": "string"
+                                }
+                            },
+                            "required": [
+                                "name",
+                                "namespace"
+                            ],
+                            "type": "object"
+                        }
+                    },
+                    "required": [
+                        "apiVersion",
+                        "kind",
+                        "metadata"
+                    ],
+                    "type": "object"
+                }
+            },
+            "required": [
+                "kubernetesObject"
+            ],
+            "type": "object"
+        }
+    },
+    "description": "This fault creates etcd storage pressure by writing a large number of ConfigMaps, causing increased API server latency and degraded cluster performance. This simulates real-world incidents where etcd storage growth causes progressive degradation of cluster operations.",
+    "expectation": "The etcd database will experience increased storage pressure. The Kubernetes API server will exhibit higher latency for read and write operations.",
+    "id": "exhausted-etcd-database-storage",
+    "index": 32,
+    "name": "Exhausted Etcd Database Storage",
+    "platform": "Kubernetes",
+    "resources": [
+        "https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/",
+        "https://etcd.io/docs/v3.5/op-guide/maintenance/",
+        "https://kubernetes.io/docs/concepts/overview/components/#etcd",
+        "https://kubernetes.io/docs/tasks/debug/debug-cluster/"
+    ],
+    "solutions": {
+        "templates": [
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} delete configmap -l app=etcd-storage-fill",
+                        "text": "Remove the ConfigMaps that are consuming etcd storage."
+                    },
+                    {
+                        "text": "Compact and defragment the etcd database to reclaim storage space."
+                    }
+                ]
+            },
+            {
+                "steps": [
+                    {
+                        "text": "Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage."
+                    },
+                    {
+                        "text": "Increase the etcd storage quota if the current limit is too low for the cluster workload."
+                    }
+                ]
+            }
+        ]
+    },
+    "tags": [
+        "Performance"
+    ]
+}

scenarios/sre/library/indexes/scenarios/116.json

@@ -0,0 +1,68 @@
+{
+    "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/scenario.json",
+    "alerts": [
+        "HighRequestLatency"
+    ],
+    "category": "sre",
+    "complexity": "high",
+    "description": "This scenario simulates etcd storage pressure by filling the cluster's etcd database with ConfigMaps, causing increased API server latency and degrading cluster-wide operations including the OpenTelemetry Demo application.",
+    "disruptions": [
+        {
+            "injections": [
+                {
+                    "args": {
+                        "kubernetesObject": {
+                            "apiVersion": "apps/v1",
+                            "kind": "Deployment",
+                            "metadata": {
+                                "name": "frontend",
+                                "namespace": "otel-demo"
+                            }
+                        }
+                    },
+                    "id": "exhausted-etcd-database-storage"
+                }
+            ]
+        }
+    ],
+    "environment": {
+        "applications": [
+            {
+                "id": "opentelemetry-demo"
+            }
+        ]
+    },
+    "id": 116,
+    "index": 116,
+    "platforms": [
+        "Kubernetes"
+    ],
+    "solutions": [
+        [
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n otel-demo delete configmap -l app=etcd-storage-fill",
+                        "text": "Remove the ConfigMaps that are consuming etcd storage."
+                    },
+                    {
+                        "text": "Compact and defragment the etcd database to reclaim storage space."
+                    }
+                ]
+            },
+            {
+                "steps": [
+                    {
+                        "text": "Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage."
+                    },
+                    {
+                        "text": "Increase the etcd storage quota if the current limit is too low for the cluster workload."
+                    }
+                ]
+            }
+        ]
+    ],
+    "tags": [
+        "Performance"
+    ]
+}

scenarios/sre/project/roles/faults/meta/argument_specs.yaml

@@ -24,6 +24,7 @@ argument_specs:
               - crashing-kubernetes-workload-init-container
               - deleted-kubernetes-service
               - disabled-istio-ambient-mode-kubernetes-namespace
+              - exhausted-etcd-database-storage
               - failing-name-resolution-kubernetes-workload-dns-policy
               - hanging-kubernetes-workload-init-container
               - ingress-port-blocking-network-policy

scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/converge.yml

@@ -0,0 +1,20 @@
+---
+- name: Run fault injection test
+  hosts:
+    - localhost
+  tasks:
+    - name: Import faults role
+      ansible.builtin.import_role:
+        name: faults
+      vars:
+        faults_cluster:
+          kubeconfig: "{{ cluster.kubeconfig }}"
+        faults_injection_tasks:
+          - id: exhausted-etcd-database-storage
+            args:
+              kubernetesObject:
+                apiVersion: apps/v1
+                kind: Deployment
+                metadata:
+                  name: test-workload
+                  namespace: exhausted-etcd-database-storage-test

scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/create.yml

@@ -0,0 +1,48 @@
+---
+- name: Create test resources
+  hosts:
+    - localhost
+  tasks:
+    - name: Create test namespace
+      kubernetes.core.k8s:
+        kubeconfig: "{{ cluster.kubeconfig }}"
+        resource_definition:
+          apiVersion: v1
+          kind: Namespace
+          metadata:
+            name: exhausted-etcd-database-storage-test
+        state: present
+
+    - name: Create test deployment
+      kubernetes.core.k8s:
+        kubeconfig: "{{ cluster.kubeconfig }}"
+        resource_definition:
+          apiVersion: apps/v1
+          kind: Deployment
+          metadata:
+            name: test-workload
+            namespace: exhausted-etcd-database-storage-test
+          spec:
+            replicas: 1
+            selector:
+              matchLabels:
+                app: test-workload
+            template:
+              metadata:
+                labels:
+                  app: test-workload
+              spec:
+                containers:
+                  - name: server
+                    image: registry.access.redhat.com/ubi10-minimal:10.2-1781509346
+                    command:
+                      - /bin/sh
+                    args:
+                      - -c
+                      - "sleep infinity"
+                    ports:
+                      - containerPort: 8080
+                        name: http
+        state: present
+        wait: true
+        wait_timeout: 120

scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/destroy.yml

@@ -0,0 +1,15 @@
+---
+- name: Destroy test resources
+  hosts:
+    - localhost
+  tasks:
+    - name: Delete test namespace
+      kubernetes.core.k8s:
+        kubeconfig: "{{ cluster.kubeconfig }}"
+        resource_definition:
+          apiVersion: v1
+          kind: Namespace
+          metadata:
+            name: exhausted-etcd-database-storage-test
+        state: absent
+        wait: true

scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/molecule.yml

@@ -0,0 +1,27 @@
+---
+dependency:
+  name: galaxy
+
+ansible:
+  env:
+    ANSIBLE_ROLES_PATH: ../../..
+  executor:
+    backend: ansible-playbook
+    args:
+      ansible_playbook:
+        - --inventory=${MOLECULE_PROJECT_DIRECTORY}/../../../inventory
+        - --tags=inject_faults,untagged
+  playbooks:
+    create: create.yml
+    converge: converge.yml
+    destroy: destroy.yml
+    verify: verify.yml
+
+scenario:
+  name: inject_exhausted_etcd_database_storage
+  test_sequence:
+    - syntax
+    - create
+    - converge
+    - verify
+    - destroy