diff --git a/documentation/library/faults/README.md b/documentation/library/faults/README.md index aff1c1be6..431d5627b 100644 --- a/documentation/library/faults/README.md +++ b/documentation/library/faults/README.md @@ -11,6 +11,7 @@ A fault is a solvable issue injected into an environment to create an incident. | [Crashing Kubernetes Workload Init Container](./crashing-kubernetes-workload-init-container.md) | Kubernetes | | [Deleted Kubernetes Service](./deleted-kubernetes-service.md) | Kubernetes | | [Disabled Istio Ambient Mode Kubernetes Namespace](./disabled-istio-ambient-mode-kubernetes-namespace.md) | Kubernetes | +| [Exhausted Etcd Database Storage](./exhausted-etcd-database-storage.md) | Kubernetes | | [Failing Name Resolution Kubernetes Workload DNS Policy](./failing-name-resolution-kubernetes-workload-dns-policy.md) | Kubernetes | | [Hanging Kubernetes Workload Init Container](./hanging-kubernetes-workload-init-container.md) | Kubernetes | | [Ingress Port Blocking Network Policy](./ingress-port-blocking-network-policy.md) | Kubernetes | diff --git a/documentation/library/faults/exhausted-etcd-database-storage.md b/documentation/library/faults/exhausted-etcd-database-storage.md new file mode 100644 index 000000000..30ac17d77 --- /dev/null +++ b/documentation/library/faults/exhausted-etcd-database-storage.md @@ -0,0 +1,27 @@ +>[!NOTE] +>This file is autogenerated. Please do not edit. + +# Exhausted Etcd Database Storage + +| [Source](../../../../scenarios/sre/library/indexes/faults/exhausted-etcd-database-storage.json) | [Schema](../../../../schemas/json/faults/exhausted-etcd-database-storage.json) | [Implementation](../../../../scenarios/sre/project/roles/faults/tasks/inject_exhausted_etcd_database_storage.yaml) | Kubernetes | Performance | + +This fault creates etcd storage pressure by writing a large number of ConfigMaps, causing increased API server latency and degraded cluster performance. This simulates real-world incidents where etcd storage growth causes progressive degradation of cluster operations. + +## Expectation + +The etcd database will experience increased storage pressure. The Kubernetes API server will exhibit higher latency for read and write operations. + +### Alerts + +The following alerts are expected to fire when this fault is injected. + +#### Golden Signal Alerts + +- HighRequestLatency + +## Resources + +- https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/ +- https://etcd.io/docs/v3.5/op-guide/maintenance/ +- https://kubernetes.io/docs/concepts/overview/components/#etcd +- https://kubernetes.io/docs/tasks/debug/debug-cluster/ diff --git a/documentation/library/scenarios/README.md b/documentation/library/scenarios/README.md index 6b8f87e95..455316189 100644 --- a/documentation/library/scenarios/README.md +++ b/documentation/library/scenarios/README.md @@ -73,3 +73,4 @@ _For statistics on the distrubution spread of scenarios, please see [here](./sta | [102](./sre/102.md) | sre | medium | | [105](./sre/105.md) | sre | medium | | [114](./sre/114.md) | sre | low | +| [116](./sre/116.md) | sre | high | diff --git a/documentation/library/scenarios/sre/116.md b/documentation/library/scenarios/sre/116.md new file mode 100644 index 000000000..c9508bb87 --- /dev/null +++ b/documentation/library/scenarios/sre/116.md @@ -0,0 +1,43 @@ +> [!NOTE] +> This file is autogenerated. Please do not edit. + +# Scenario 116 + +| [Source](../../../../scenarios/sre/library/indexes/scenarios/116.json) | High Complexity | Kubernetes | Performance | + +## Description + +This scenario simulates etcd storage pressure by filling the cluster's etcd database with ConfigMaps, causing increased API server latency and degrading cluster-wide operations including the OpenTelemetry Demo application. + +## Applications + +- [OpenTelemetry Demo](../../applications/opentelemetry-demo.md) + +## Faults + +- [Exhausted Etcd Database Storage](../../../../scenarios/sre/docs/faults.md#Exhausted-Etcd-Database-Storage) + +## Solution Guides + +Below is a noncomprehensive list of step by step guides which can lead to remediating the one or more problems in the scenario. + +A **solution** represents the process of fixing an issue created by a fault. To remediate the issues in the scenarios, follow the solutions in order of appearance. + +A solution can have multiple **variants**. A variant is a different process of deriving the solution. If multiple variants are presented, pick only one to solve the problem. + +### Solution 1 + +#### Variant 1 + +1. Remove the ConfigMaps that are consuming etcd storage. +```shell +kubectl -n otel-demo delete configmap -l app=etcd-storage-fill +``` + +2. Compact and defragment the etcd database to reclaim storage space. + +#### Variant 2 + +1. Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage. + +2. Increase the etcd storage quota if the current limit is too low for the cluster workload. diff --git a/documentation/library/scenarios/statistics.md b/documentation/library/scenarios/statistics.md index 4f8df3a29..25ad06a9f 100644 --- a/documentation/library/scenarios/statistics.md +++ b/documentation/library/scenarios/statistics.md @@ -7,16 +7,16 @@ | BookInfo | OpenTelemetry Demo | | --- | --- | -| 5 | 61 | +| 5 | 36 | ## Category Distribution | FinOps | SRE | | --- | --- | -| 2 | 64 | +| 2 | 39 | ## Complexity Distribution | Low | Medium | High | | --- | --- | --- | -| 17 | 46 | 3 | +| 17 | 22 | 2 | diff --git a/scenarios/sre/library/indexes/faults/32.json b/scenarios/sre/library/indexes/faults/32.json new file mode 100644 index 000000000..f188d7046 --- /dev/null +++ b/scenarios/sre/library/indexes/faults/32.json @@ -0,0 +1,96 @@ +{ + "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/fault.json", + "alerts": { + "goldenSignal": [ + "HighRequestLatency" + ] + }, + "arguments": { + "jsonSchema": { + "properties": { + "kubernetesObject": { + "properties": { + "apiVersion": { + "enum": [ + "apps/v1" + ], + "type": "string" + }, + "kind": { + "enum": [ + "Deployment", + "StatefulSet" + ], + "type": "string" + }, + "metadata": { + "properties": { + "name": { + "type": "string" + }, + "namespace": { + "type": "string" + } + }, + "required": [ + "name", + "namespace" + ], + "type": "object" + } + }, + "required": [ + "apiVersion", + "kind", + "metadata" + ], + "type": "object" + } + }, + "required": [ + "kubernetesObject" + ], + "type": "object" + } + }, + "description": "This fault creates etcd storage pressure by writing a large number of ConfigMaps, causing increased API server latency and degraded cluster performance. This simulates real-world incidents where etcd storage growth causes progressive degradation of cluster operations.", + "expectation": "The etcd database will experience increased storage pressure. The Kubernetes API server will exhibit higher latency for read and write operations.", + "id": "exhausted-etcd-database-storage", + "index": 32, + "name": "Exhausted Etcd Database Storage", + "platform": "Kubernetes", + "resources": [ + "https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/", + "https://etcd.io/docs/v3.5/op-guide/maintenance/", + "https://kubernetes.io/docs/concepts/overview/components/#etcd", + "https://kubernetes.io/docs/tasks/debug/debug-cluster/" + ], + "solutions": { + "templates": [ + { + "steps": [ + { + "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} delete configmap -l app=etcd-storage-fill", + "text": "Remove the ConfigMaps that are consuming etcd storage." + }, + { + "text": "Compact and defragment the etcd database to reclaim storage space." + } + ] + }, + { + "steps": [ + { + "text": "Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage." + }, + { + "text": "Increase the etcd storage quota if the current limit is too low for the cluster workload." + } + ] + } + ] + }, + "tags": [ + "Performance" + ] +} diff --git a/scenarios/sre/library/indexes/scenarios/116.json b/scenarios/sre/library/indexes/scenarios/116.json new file mode 100644 index 000000000..b3b45adce --- /dev/null +++ b/scenarios/sre/library/indexes/scenarios/116.json @@ -0,0 +1,68 @@ +{ + "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/scenario.json", + "alerts": [ + "HighRequestLatency" + ], + "category": "sre", + "complexity": "high", + "description": "This scenario simulates etcd storage pressure by filling the cluster's etcd database with ConfigMaps, causing increased API server latency and degrading cluster-wide operations including the OpenTelemetry Demo application.", + "disruptions": [ + { + "injections": [ + { + "args": { + "kubernetesObject": { + "apiVersion": "apps/v1", + "kind": "Deployment", + "metadata": { + "name": "frontend", + "namespace": "otel-demo" + } + } + }, + "id": "exhausted-etcd-database-storage" + } + ] + } + ], + "environment": { + "applications": [ + { + "id": "opentelemetry-demo" + } + ] + }, + "id": 116, + "index": 116, + "platforms": [ + "Kubernetes" + ], + "solutions": [ + [ + { + "steps": [ + { + "command": "kubectl -n otel-demo delete configmap -l app=etcd-storage-fill", + "text": "Remove the ConfigMaps that are consuming etcd storage." + }, + { + "text": "Compact and defragment the etcd database to reclaim storage space." + } + ] + }, + { + "steps": [ + { + "text": "Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage." + }, + { + "text": "Increase the etcd storage quota if the current limit is too low for the cluster workload." + } + ] + } + ] + ], + "tags": [ + "Performance" + ] +} diff --git a/scenarios/sre/project/roles/faults/meta/argument_specs.yaml b/scenarios/sre/project/roles/faults/meta/argument_specs.yaml index 8911e1793..547bc1304 100644 --- a/scenarios/sre/project/roles/faults/meta/argument_specs.yaml +++ b/scenarios/sre/project/roles/faults/meta/argument_specs.yaml @@ -24,6 +24,7 @@ argument_specs: - crashing-kubernetes-workload-init-container - deleted-kubernetes-service - disabled-istio-ambient-mode-kubernetes-namespace + - exhausted-etcd-database-storage - failing-name-resolution-kubernetes-workload-dns-policy - hanging-kubernetes-workload-init-container - ingress-port-blocking-network-policy diff --git a/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/converge.yml b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/converge.yml new file mode 100644 index 000000000..fd8cca45a --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/converge.yml @@ -0,0 +1,20 @@ +--- +- name: Run fault injection test + hosts: + - localhost + tasks: + - name: Import faults role + ansible.builtin.import_role: + name: faults + vars: + faults_cluster: + kubeconfig: "{{ cluster.kubeconfig }}" + faults_injection_tasks: + - id: exhausted-etcd-database-storage + args: + kubernetesObject: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: test-workload + namespace: exhausted-etcd-database-storage-test diff --git a/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/create.yml b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/create.yml new file mode 100644 index 000000000..390c75d5f --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/create.yml @@ -0,0 +1,48 @@ +--- +- name: Create test resources + hosts: + - localhost + tasks: + - name: Create test namespace + kubernetes.core.k8s: + kubeconfig: "{{ cluster.kubeconfig }}" + resource_definition: + apiVersion: v1 + kind: Namespace + metadata: + name: exhausted-etcd-database-storage-test + state: present + + - name: Create test deployment + kubernetes.core.k8s: + kubeconfig: "{{ cluster.kubeconfig }}" + resource_definition: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: test-workload + namespace: exhausted-etcd-database-storage-test + spec: + replicas: 1 + selector: + matchLabels: + app: test-workload + template: + metadata: + labels: + app: test-workload + spec: + containers: + - name: server + image: registry.access.redhat.com/ubi10-minimal:10.2-1781509346 + command: + - /bin/sh + args: + - -c + - "sleep infinity" + ports: + - containerPort: 8080 + name: http + state: present + wait: true + wait_timeout: 120 diff --git a/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/destroy.yml b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/destroy.yml new file mode 100644 index 000000000..0d385392a --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/destroy.yml @@ -0,0 +1,15 @@ +--- +- name: Destroy test resources + hosts: + - localhost + tasks: + - name: Delete test namespace + kubernetes.core.k8s: + kubeconfig: "{{ cluster.kubeconfig }}" + resource_definition: + apiVersion: v1 + kind: Namespace + metadata: + name: exhausted-etcd-database-storage-test + state: absent + wait: true diff --git a/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/molecule.yml b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/molecule.yml new file mode 100644 index 000000000..d95dab303 --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/molecule.yml @@ -0,0 +1,27 @@ +--- +dependency: + name: galaxy + +ansible: + env: + ANSIBLE_ROLES_PATH: ../../.. + executor: + backend: ansible-playbook + args: + ansible_playbook: + - --inventory=${MOLECULE_PROJECT_DIRECTORY}/../../../inventory + - --tags=inject_faults,untagged + playbooks: + create: create.yml + converge: converge.yml + destroy: destroy.yml + verify: verify.yml + +scenario: + name: inject_exhausted_etcd_database_storage + test_sequence: + - syntax + - create + - converge + - verify + - destroy diff --git a/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/verify.yml b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/verify.yml new file mode 100644 index 000000000..ca5452820 --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/verify.yml @@ -0,0 +1,21 @@ +--- +- name: Verify that fault injection succeeded + hosts: + - localhost + tasks: + - name: Retrieve ConfigMaps + kubernetes.core.k8s_info: + kubeconfig: "{{ cluster.kubeconfig }}" + api_version: v1 + kind: ConfigMap + namespace: exhausted-etcd-database-storage-test + label_selectors: + - app=etcd-storage-fill + register: faults_configmaps + + - name: Validate ConfigMaps were created + ansible.builtin.assert: + that: + - faults_configmaps.resources | length >= 50 + fail_msg: Expected at least 50 fill ConfigMaps. Fault injection may have failed. + success_msg: ConfigMaps created successfully for etcd storage pressure. diff --git a/scenarios/sre/project/roles/faults/tasks/inject_exhausted_etcd_database_storage.yaml b/scenarios/sre/project/roles/faults/tasks/inject_exhausted_etcd_database_storage.yaml new file mode 100644 index 000000000..426c3a8f5 --- /dev/null +++ b/scenarios/sre/project/roles/faults/tasks/inject_exhausted_etcd_database_storage.yaml @@ -0,0 +1,36 @@ +--- +- name: Include fault argument validation tasks + ansible.builtin.include_tasks: + file: validate_fault_arguments.yaml + +- name: Create ConfigMaps to pressure etcd storage + kubernetes.core.k8s: + kubeconfig: "{{ faults_cluster.kubeconfig }}" + resource_definition: + apiVersion: v1 + kind: ConfigMap + metadata: + name: "etcd-fill-{{ item }}" + namespace: "{{ faults_kubernetes_object.metadata.namespace }}" + labels: + app: etcd-storage-fill + data: + payload: "{{ lookup('ansible.builtin.password', '/dev/null chars=ascii_lowercase,digits length=102400') }}" + state: present + loop: "{{ range(1, 51) | list }}" + loop_control: + label: "etcd-fill-{{ item }}" + +- name: Verify ConfigMaps were created + kubernetes.core.k8s_info: + kubeconfig: "{{ faults_cluster.kubeconfig }}" + api_version: v1 + kind: ConfigMap + namespace: "{{ faults_kubernetes_object.metadata.namespace }}" + label_selectors: + - app=etcd-storage-fill + register: faults_etcd_fill_result + until: + - faults_etcd_fill_result.resources | length >= 50 + retries: 5 + delay: 10 diff --git a/scenarios/sre/project/roles/faults/vars/main/task_files.yaml b/scenarios/sre/project/roles/faults/vars/main/task_files.yaml index 70e94336d..e7d927b83 100644 --- a/scenarios/sre/project/roles/faults/vars/main/task_files.yaml +++ b/scenarios/sre/project/roles/faults/vars/main/task_files.yaml @@ -8,6 +8,8 @@ faults_task_files: injection: inject_deleted_kubernetes_service.yaml disabled-istio-ambient-mode-kubernetes-namespace: injection: inject_disabled_istio_ambient_mode_kubernetes_namespace.yaml + exhausted-etcd-database-storage: + injection: inject_exhausted_etcd_database_storage.yaml failing-name-resolution-kubernetes-workload-dns-policy: injection: inject_failing_name_resolution_kubernetes_workload_dns_policy.yaml hanging-kubernetes-workload-init-container: diff --git a/scenarios/sre/project/roles/scenarios/files/scenario_116/groundtruth.yaml b/scenarios/sre/project/roles/scenarios/files/scenario_116/groundtruth.yaml new file mode 100644 index 000000000..67d6e764d --- /dev/null +++ b/scenarios/sre/project/roles/scenarios/files/scenario_116/groundtruth.yaml @@ -0,0 +1,23 @@ +--- +apiVersion: itbench.io/v2 +kind: GroundTruth +metadata: + name: scenario-116 +spec: + alerts: + - labels: {} + name: HighRequestLatency + entities: + - apiVersion: apps/v1 + kind: Deployment + metadata: + name: frontend + namespace: otel-demo + solutions: + - - steps: + - command: kubectl -n otel-demo delete configmap -l app=etcd-storage-fill + text: Remove the ConfigMaps that are consuming etcd storage. + - text: Compact and defragment the etcd database to reclaim storage space. + - steps: + - text: Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage. + - text: Increase the etcd storage quota if the current limit is too low for the cluster workload. diff --git a/scenarios/sre/project/roles/scenarios/files/scenario_116/groundtruth_v1.yaml b/scenarios/sre/project/roles/scenarios/files/scenario_116/groundtruth_v1.yaml new file mode 100644 index 000000000..91371e9e5 --- /dev/null +++ b/scenarios/sre/project/roles/scenarios/files/scenario_116/groundtruth_v1.yaml @@ -0,0 +1,71 @@ +--- +apiVersion: itbench.io/v1 +kind: GroundTruth +metadata: + name: scenario-116 +spec: + alerts: + - group_id: frontend-service-1 + id: HighRequestLatency + metadata: + description: Increased API server latency in namespace otel-demo due to etcd storage pressure from excessive ConfigMap creation + groups: + - id: configmap-fill-1 + kind: ConfigMap + namespace: otel-demo + filter: + - etcd-fill-.* + root_cause: true + - id: frontend-deployment-1 + kind: Deployment + namespace: otel-demo + filter: + - frontend\b + - id: frontend-pod-1 + kind: Pod + namespace: otel-demo + filter: + - frontend-.* + - id: frontend-service-1 + kind: Service + namespace: otel-demo + filter: + - frontend\b + aliases: + - - frontend-deployment-1 + - frontend-pod-1 + - - frontend-deployment-1 + - frontend-service-1 + propagations: + - source: configmap-fill-1 + target: frontend-deployment-1 + condition: Large number of ConfigMaps created consuming etcd storage + effect: etcd database experiences storage pressure, API server latency increases for all operations in the namespace + - source: frontend-deployment-1 + target: frontend-pod-1 + condition: API server latency increased due to etcd storage pressure + effect: Frontend pods experience slower responses from Kubernetes API, degrading application performance + - source: frontend-pod-1 + target: frontend-service-1 + condition: Frontend pods responding slowly due to cluster-wide API latency + effect: Frontend service exhibits higher request latency visible to end users + fault: + - category: Fill + condition: etcd database filled with ConfigMaps creating storage pressure + entity: + group_id: configmap-fill-1 + kind: ConfigMap + name: etcd-fill + fault_mechanism: Storage pressure via bulk ConfigMap creation simulating real-world etcd storage growth incidents + recommendedActions: + - solution: + actions: + - Delete the etcd-fill ConfigMaps using kubectl delete configmap -l app=etcd-storage-fill -n otel-demo + - Compact and defragment etcd to reclaim space + - Monitor etcd storage usage returning to normal levels + id: cleanup-configmaps + - solution: + actions: + - Increase etcd storage quota if current limit is insufficient + - Review resource creation patterns to prevent future growth + id: increase-quota diff --git a/scenarios/sre/project/roles/scenarios/files/scenario_116/scenario.yaml b/scenarios/sre/project/roles/scenarios/files/scenario_116/scenario.yaml new file mode 100644 index 000000000..dc24fc633 --- /dev/null +++ b/scenarios/sre/project/roles/scenarios/files/scenario_116/scenario.yaml @@ -0,0 +1,22 @@ +--- +apiVersion: itbench.io/v2 +kind: Scenario +metadata: + name: scenario-116 +spec: + applications: + opentelemetryDemo: + enabled: true + faults: + - injections: + - args: + kubernetesObject: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: frontend + namespace: otel-demo + id: exhausted-etcd-database-storage + tools: + sre: + enabled: true diff --git a/scenarios/sre/templates/library/indexes/faults/32.yaml.j2 b/scenarios/sre/templates/library/indexes/faults/32.yaml.j2 new file mode 100644 index 000000000..06c9a0676 --- /dev/null +++ b/scenarios/sre/templates/library/indexes/faults/32.yaml.j2 @@ -0,0 +1,57 @@ +# yaml-language-server: $schema=../../../../../../schemas/json/library/index/fault.json +--- +alerts: + goldenSignal: + - HighRequestLatency +arguments: + jsonSchema: + properties: + kubernetesObject: + properties: + apiVersion: + enum: + - apps/v1 + type: string + kind: + enum: + - Deployment + - StatefulSet + type: string + metadata: + properties: + name: + type: string + namespace: + type: string + required: + - name + - namespace + type: object + required: + - apiVersion + - kind + - metadata + type: object + required: + - kubernetesObject + type: object +description: This fault creates etcd storage pressure by writing a large number of ConfigMaps, causing increased API server latency and degraded cluster performance. This simulates real-world incidents where etcd storage growth causes progressive degradation of cluster operations. +expectation: The etcd database will experience increased storage pressure. The Kubernetes API server will exhibit higher latency for read and write operations. +name: Exhausted Etcd Database Storage +platform: Kubernetes +resources: +- https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/ +- https://etcd.io/docs/v3.5/op-guide/maintenance/ +- https://kubernetes.io/docs/concepts/overview/components/#etcd +- https://kubernetes.io/docs/tasks/debug/debug-cluster/ +solutions: + templates: + - steps: + - command: kubectl -n {{ args.kubernetesObject.metadata.namespace }} delete configmap -l app=etcd-storage-fill + text: Remove the ConfigMaps that are consuming etcd storage. + - text: Compact and defragment the etcd database to reclaim storage space. + - steps: + - text: Identify and remove unnecessary resources (completed jobs, orphaned ConfigMaps) to reduce etcd storage usage. + - text: Increase the etcd storage quota if the current limit is too low for the cluster workload. +tags: +- Performance diff --git a/scenarios/sre/templates/library/indexes/scenarios/116.yaml.j2 b/scenarios/sre/templates/library/indexes/scenarios/116.yaml.j2 new file mode 100644 index 000000000..76fe1e30d --- /dev/null +++ b/scenarios/sre/templates/library/indexes/scenarios/116.yaml.j2 @@ -0,0 +1,22 @@ +# yaml-language-server: $schema=../../../../../../schemas/json/library/index/scenario.json +--- +category: sre +complexity: high +description: This scenario simulates etcd storage pressure by filling the cluster's etcd database with ConfigMaps, causing increased API server latency and degrading cluster-wide operations including the OpenTelemetry Demo application. +disruptions: +- injections: + - args: + kubernetesObject: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: frontend + namespace: {{ releases["applications"]["opentelemetry_demo"]["kubernetes"]["namespace"] }} + id: exhausted-etcd-database-storage +environment: + applications: + - id: opentelemetry-demo +id: 116 +solutionTemplates: +- disruptionIndex: 0 + injectionIndex: 0 diff --git a/schemas/json/faults/exhausted-etcd-database-storage.json b/schemas/json/faults/exhausted-etcd-database-storage.json new file mode 100644 index 000000000..e19e2a60d --- /dev/null +++ b/schemas/json/faults/exhausted-etcd-database-storage.json @@ -0,0 +1,47 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "kubernetesObject": { + "properties": { + "apiVersion": { + "enum": [ + "apps/v1" + ], + "type": "string" + }, + "kind": { + "enum": [ + "Deployment", + "StatefulSet" + ], + "type": "string" + }, + "metadata": { + "properties": { + "name": { + "type": "string" + }, + "namespace": { + "type": "string" + } + }, + "required": [ + "name", + "namespace" + ], + "type": "object" + } + }, + "required": [ + "apiVersion", + "kind", + "metadata" + ], + "type": "object" + } + }, + "required": [ + "kubernetesObject" + ], + "type": "object" +} diff --git a/schemas/json/library/index/scenario.json b/schemas/json/library/index/scenario.json index 2566fce46..870656fed 100644 --- a/schemas/json/library/index/scenario.json +++ b/schemas/json/library/index/scenario.json @@ -294,6 +294,22 @@ } } }, + { + "if": { + "properties": { + "id": { + "const": "exhausted-etcd-database-storage" + } + } + }, + "then": { + "properties": { + "args": { + "$ref": "../../faults/exhausted-etcd-database-storage.json" + } + } + } + }, { "if": { "properties": { @@ -556,6 +572,7 @@ "crashing-kubernetes-workload-init-container", "valkey-workload-out-of-memory", "misconfigured-kubernetes-workload-container-readiness-probe", + "exhausted-etcd-database-storage", "unassigned-kubernetes-workload-container-resource-limits", "unschedulable-kubernetes-workload-pod-anti-affinity-rule", "modified-kubernetes-workload-container-environment-variable",