diff --git a/documentation/library/faults/README.md b/documentation/library/faults/README.md index aff1c1be6..18e1709e4 100644 --- a/documentation/library/faults/README.md +++ b/documentation/library/faults/README.md @@ -23,6 +23,7 @@ A fault is a solvable issue injected into an environment to create an incident. | [Misconfigured Kubernetes Workload Container Readiness Probe](./misconfigured-kubernetes-workload-container-readiness-probe.md) | Kubernetes | | [Modified Kubernetes Workload Container Environment Variable](./modified-kubernetes-workload-container-environment-variable.md) | Kubernetes | | [Modified Target Port Kubernetes Service](./modified-target-port-kubernetes-service.md) | Kubernetes | +| [Node Resource Exhaustion](./node-resource-exhaustion.md) | Kubernetes | | [Nonexistent Kubernetes Workload Container Image](./nonexistent-kubernetes-workload-container-image.md) | Kubernetes | | [Nonexistent Kubernetes Workload Node](./nonexistent-kubernetes-workload-node.md) | Kubernetes | | [Nonexistent Kubernetes Workload Persistent Volume Claim](./nonexistent-kubernetes-workload-persistent-volume-claim.md) | Kubernetes | diff --git a/documentation/library/faults/node-resource-exhaustion.md b/documentation/library/faults/node-resource-exhaustion.md new file mode 100644 index 000000000..f998fd4c2 --- /dev/null +++ b/documentation/library/faults/node-resource-exhaustion.md @@ -0,0 +1,27 @@ +>[!NOTE] +>This file is autogenerated. Please do not edit. + +# Node Resource Exhaustion + +| [Source](../../../../scenarios/sre/library/indexes/faults/node-resource-exhaustion.json) | [Schema](../../../../schemas/json/faults/node-resource-exhaustion.json) | [Implementation](../../../../scenarios/sre/project/roles/faults/tasks/inject_node_resource_exhaustion.yaml) | Kubernetes | Deployment, Performance | + +This fault injects excessive resource requests into a workload, causing nodes to run out of allocatable resources. + +## Expectation + +The faulted workload pods will consume excessive resources, causing other pods on the same node to be evicted or fail to schedule. + +### Alerts + +The following alerts are expected to fire when this fault is injected. + +#### Application Alerts + +- KubePodNotReady + +## Resources + +- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ +- https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/ +- https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/ +- https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/ diff --git a/documentation/library/scenarios/README.md b/documentation/library/scenarios/README.md index 6b8f87e95..5236cb327 100644 --- a/documentation/library/scenarios/README.md +++ b/documentation/library/scenarios/README.md @@ -73,3 +73,4 @@ _For statistics on the distrubution spread of scenarios, please see [here](./sta | [102](./sre/102.md) | sre | medium | | [105](./sre/105.md) | sre | medium | | [114](./sre/114.md) | sre | low | +| [121](./sre/121.md) | sre | medium | diff --git a/documentation/library/scenarios/sre/121.md b/documentation/library/scenarios/sre/121.md new file mode 100644 index 000000000..d844562e9 --- /dev/null +++ b/documentation/library/scenarios/sre/121.md @@ -0,0 +1,42 @@ +> [!NOTE] +> This file is autogenerated. Please do not edit. + +# Scenario 121 + +| [Source](../../../../scenarios/sre/library/indexes/scenarios/121.json) | Medium Complexity | Kubernetes | Deployment, Performance | + +## Description + +This scenario simulates node resource exhaustion by injecting excessive resource requests into the OpenTelemetry Demo's `product-catalog` service. + +## Applications + +- [OpenTelemetry Demo](../../applications/opentelemetry-demo.md) + +## Faults + +- [Node Resource Exhaustion](../../../../scenarios/sre/docs/faults.md#Node-Resource-Exhaustion) + +## Solution Guides + +Below is a noncomprehensive list of step by step guides which can lead to remediating the one or more problems in the scenario. + +A **solution** represents the process of fixing an issue created by a fault. To remediate the issues in the scenarios, follow the solutions in order of appearance. + +A solution can have multiple **variants**. A variant is a different process of deriving the solution. If multiple variants are presented, pick only one to solve the problem. + +### Solution 1 + +#### Variant 1 + +1. Revert the resource request changes to restore normal resource usage. +```shell +kubectl -n {{ args.kubernetesObject.metadata.namespace }} rollout undo {{ args.kubernetesObject.kind | lower }}/{{ args.kubernetesObject.metadata.name }} +``` + +#### Variant 2 + +1. Manually edit the manifest and set resource requests to appropriate values. +```shell +kubectl -n {{ args.kubernetesObject.metadata.namespace }} edit {{ args.kubernetesObject.kind | lower }} {{ args.kubernetesObject.metadata.name }} +``` diff --git a/documentation/library/scenarios/statistics.md b/documentation/library/scenarios/statistics.md index 4f8df3a29..3cf98f5d1 100644 --- a/documentation/library/scenarios/statistics.md +++ b/documentation/library/scenarios/statistics.md @@ -7,16 +7,16 @@ | BookInfo | OpenTelemetry Demo | | --- | --- | -| 5 | 61 | +| 5 | 36 | ## Category Distribution | FinOps | SRE | | --- | --- | -| 2 | 64 | +| 2 | 39 | ## Complexity Distribution | Low | Medium | High | | --- | --- | --- | -| 17 | 46 | 3 | +| 17 | 23 | 1 | diff --git a/scenarios/sre/library/indexes/faults/37.json b/scenarios/sre/library/indexes/faults/37.json new file mode 100644 index 000000000..33bbd8560 --- /dev/null +++ b/scenarios/sre/library/indexes/faults/37.json @@ -0,0 +1,92 @@ +{ + "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/fault.json", + "alerts": { + "application": [ + "KubePodNotReady" + ] + }, + "arguments": { + "jsonSchema": { + "properties": { + "kubernetesObject": { + "properties": { + "apiVersion": { + "enum": [ + "apps/v1" + ], + "type": "string" + }, + "kind": { + "enum": [ + "Deployment", + "StatefulSet" + ], + "type": "string" + }, + "metadata": { + "properties": { + "name": { + "type": "string" + }, + "namespace": { + "type": "string" + } + }, + "required": [ + "name", + "namespace" + ], + "type": "object" + } + }, + "required": [ + "apiVersion", + "kind", + "metadata" + ], + "type": "object" + } + }, + "required": [ + "kubernetesObject" + ], + "type": "object" + } + }, + "description": "This fault injects excessive resource requests into a workload, causing nodes to run out of allocatable resources.", + "expectation": "The faulted workload pods will consume excessive resources, causing other pods on the same node to be evicted or fail to schedule.", + "id": "node-resource-exhaustion", + "index": 37, + "name": "Node Resource Exhaustion", + "platform": "Kubernetes", + "resources": [ + "https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/", + "https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/", + "https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/", + "https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/" + ], + "solutions": { + "templates": [ + { + "steps": [ + { + "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} rollout undo {{ args.kubernetesObject.kind | lower }}/{{ args.kubernetesObject.metadata.name }}", + "text": "Revert the resource request changes to restore normal resource usage." + } + ] + }, + { + "steps": [ + { + "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} edit {{ args.kubernetesObject.kind | lower }} {{ args.kubernetesObject.metadata.name }}", + "text": "Manually edit the manifest and set resource requests to appropriate values." + } + ] + } + ] + }, + "tags": [ + "Deployment", + "Performance" + ] +} diff --git a/scenarios/sre/library/indexes/scenarios/121.json b/scenarios/sre/library/indexes/scenarios/121.json new file mode 100644 index 000000000..9cb4be6ea --- /dev/null +++ b/scenarios/sre/library/indexes/scenarios/121.json @@ -0,0 +1,64 @@ +{ + "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/scenario.json", + "alerts": [ + "KubePodNotReady" + ], + "category": "sre", + "complexity": "medium", + "description": "This scenario simulates node resource exhaustion by injecting excessive resource requests into the OpenTelemetry Demo's `product-catalog` service.", + "disruptions": [ + { + "injections": [ + { + "args": { + "kubernetesObject": { + "apiVersion": "apps/v1", + "kind": "Deployment", + "metadata": { + "name": "product-catalog", + "namespace": "otel-demo" + } + } + }, + "id": "node-resource-exhaustion" + } + ] + } + ], + "environment": { + "applications": [ + { + "id": "opentelemetry-demo" + } + ] + }, + "id": 121, + "index": 121, + "platforms": [ + "Kubernetes" + ], + "solutions": [ + [ + { + "steps": [ + { + "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} rollout undo {{ args.kubernetesObject.kind | lower }}/{{ args.kubernetesObject.metadata.name }}", + "text": "Revert the resource request changes to restore normal resource usage." + } + ] + }, + { + "steps": [ + { + "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} edit {{ args.kubernetesObject.kind | lower }} {{ args.kubernetesObject.metadata.name }}", + "text": "Manually edit the manifest and set resource requests to appropriate values." + } + ] + } + ] + ], + "tags": [ + "Deployment", + "Performance" + ] +} diff --git a/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/converge.yml b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/converge.yml new file mode 100644 index 000000000..7fc871aaf --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/converge.yml @@ -0,0 +1,20 @@ +--- +- name: Run fault injection test + hosts: + - localhost + tasks: + - name: Import faults role + ansible.builtin.import_role: + name: faults + vars: + faults_cluster: + kubeconfig: "{{ cluster.kubeconfig }}" + faults_injection_tasks: + - id: node-resource-exhaustion + args: +kubernetesObject: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: test-workload + namespace: node-resource-exhaustion-test diff --git a/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/create.yml b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/create.yml new file mode 100644 index 000000000..82cc1b4f2 --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/create.yml @@ -0,0 +1,48 @@ +--- +- name: Create test resources + hosts: + - localhost + tasks: + - name: Create test namespace + kubernetes.core.k8s: + kubeconfig: "{{ cluster.kubeconfig }}" + resource_definition: + apiVersion: v1 + kind: Namespace + metadata: + name: node-resource-exhaustion-test + state: present + + - name: Create test deployment + kubernetes.core.k8s: + kubeconfig: "{{ cluster.kubeconfig }}" + resource_definition: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: test-workload + namespace: node-resource-exhaustion-test + spec: + replicas: 1 + selector: + matchLabels: + app: test-workload + template: + metadata: + labels: + app: test-workload + spec: + containers: + - name: server + image: registry.access.redhat.com/ubi10-minimal:10.2-1781509346 + command: + - /bin/sh + args: + - -c + - "sleep infinity" + ports: + - containerPort: 8080 + name: http + state: present + wait: true + wait_timeout: 120 diff --git a/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/destroy.yml b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/destroy.yml new file mode 100644 index 000000000..51b6b3efc --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/destroy.yml @@ -0,0 +1,15 @@ +--- +- name: Destroy test resources + hosts: + - localhost + tasks: + - name: Delete test namespace + kubernetes.core.k8s: + kubeconfig: "{{ cluster.kubeconfig }}" + resource_definition: + apiVersion: v1 + kind: Namespace + metadata: + name: node-resource-exhaustion-test + state: absent + wait: true diff --git a/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/molecule.yml b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/molecule.yml new file mode 100644 index 000000000..62141502e --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/molecule.yml @@ -0,0 +1,24 @@ +dependency: + name: galaxy +ansible: + env: + ANSIBLE_ROLES_PATH: ../../.. + executor: + backend: ansible-playbook + args: + ansible_playbook: + - --inventory=${MOLECULE_PROJECT_DIRECTORY}/../../../inventory + - --tags=inject_faults,untagged + playbooks: + create: create.yml + converge: converge.yml + destroy: destroy.yml + verify: verify.yml +scenario: + name: inject_node_resource_exhaustion + test_sequence: + - syntax + - create + - converge + - verify + - destroy diff --git a/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/verify.yml b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/verify.yml new file mode 100644 index 000000000..f7fd59868 --- /dev/null +++ b/scenarios/sre/project/roles/faults/molecule/inject_node_resource_exhaustion/verify.yml @@ -0,0 +1,21 @@ +--- +- name: Verify that fault injection succeeded + hosts: + - localhost + tasks: + - name: Retrieve pods + kubernetes.core.k8s_info: + kubeconfig: "{{ cluster.kubeconfig }}" + api_version: v1 + kind: Pod + namespace: node-resource-exhaustion-test + label_selectors: + - app=test-workload + register: faults_pods + + - name: Validate pod is in Pending state + ansible.builtin.assert: + that: + - faults_pods.resources | length > 0 + fail_msg: No pods found. Fault injection may have failed. + success_msg: Pod found in expected Pending state. diff --git a/scenarios/sre/project/roles/faults/tasks/inject_node_resource_exhaustion.yaml b/scenarios/sre/project/roles/faults/tasks/inject_node_resource_exhaustion.yaml new file mode 100644 index 000000000..4ed03b8ac --- /dev/null +++ b/scenarios/sre/project/roles/faults/tasks/inject_node_resource_exhaustion.yaml @@ -0,0 +1,43 @@ +--- +- name: Include fault argument validation tasks + ansible.builtin.include_tasks: + file: validate_fault_arguments.yaml + +- name: Inject excessive resource requests + kubernetes.core.k8s: + kubeconfig: "{{ faults_cluster.kubeconfig }}" + resource_definition: + apiVersion: "{{ faults_kubernetes_object.apiVersion }}" + kind: "{{ faults_kubernetes_object.kind }}" + metadata: + name: "{{ faults_kubernetes_object.metadata.name }}" + namespace: "{{ faults_kubernetes_object.metadata.namespace }}" + spec: + template: + spec: + containers: + - name: "{{ faults_kubernetes_object.metadata.name }}" + resources: + requests: + memory: "64Gi" + cpu: "32" + state: patched + +- name: Wait for pods to be in Pending state + kubernetes.core.k8s_info: + kubeconfig: "{{ faults_cluster.kubeconfig }}" + api_version: v1 + kind: Pod + namespace: "{{ faults_kubernetes_object.metadata.namespace }}" + label_selectors: + - "app={{ faults_kubernetes_object.metadata.name }}" + register: faults_resource_pods + until: + - faults_resource_pods.resources | length > 0 + - >- + faults_resource_pods.resources | + selectattr('status.phase', 'equalto', 'Pending') | + list | + length > 0 + retries: 15 + delay: 10 diff --git a/scenarios/sre/project/roles/scenarios/files/scenario_121/groundtruth.yaml b/scenarios/sre/project/roles/scenarios/files/scenario_121/groundtruth.yaml new file mode 100644 index 000000000..1a6619668 --- /dev/null +++ b/scenarios/sre/project/roles/scenarios/files/scenario_121/groundtruth.yaml @@ -0,0 +1,22 @@ +--- +apiVersion: itbench.io/v2 +kind: GroundTruth +metadata: + name: scenario-121 +spec: + alerts: + - labels: {} + name: KubePodNotReady + entities: + - apiVersion: apps/v1 + kind: Deployment + metadata: + name: product-catalog + namespace: otel-demo + solutions: + - - steps: + - command: kubectl -n otel-demo rollout undo deployment/product-catalog + text: Revert the resource request changes to restore normal resource usage. + - steps: + - command: kubectl -n otel-demo edit deployment product-catalog + text: Manually edit the manifest and set resource requests to appropriate values. diff --git a/scenarios/sre/project/roles/scenarios/files/scenario_121/groundtruth_v1.yaml b/scenarios/sre/project/roles/scenarios/files/scenario_121/groundtruth_v1.yaml new file mode 100644 index 000000000..63653fa33 --- /dev/null +++ b/scenarios/sre/project/roles/scenarios/files/scenario_121/groundtruth_v1.yaml @@ -0,0 +1,84 @@ +--- +apiVersion: itbench.io/v1 +kind: GroundTruth +metadata: + name: scenario-121 +spec: + alerts: + - group_id: product-catalog-pod-1 + id: KubePodNotReady + metadata: + description: Pod product-catalog in namespace otel-demo has been in a non-ready state due to excessive resource requests preventing scheduling + - group_id: frontend-service-1 + id: HighRequestErrorRate + metadata: + description: Request error rate in service frontend in namespace otel-demo is elevated due to product-catalog service unavailability + groups: + - id: product-catalog-deployment-1 + kind: Deployment + namespace: otel-demo + filter: + - product-catalog\b + root_cause: true + - id: product-catalog-pod-1 + kind: Pod + namespace: otel-demo + filter: + - product-catalog-.* + - id: product-catalog-service-1 + kind: Service + namespace: otel-demo + filter: + - product-catalog\b + - id: frontend-service-1 + kind: Service + namespace: otel-demo + filter: + - frontend\b + - id: frontend-pod-1 + kind: Pod + namespace: otel-demo + filter: + - frontend-.* + aliases: + - - product-catalog-deployment-1 + - product-catalog-pod-1 + - - product-catalog-service-1 + - product-catalog-pod-1 + - - frontend-service-1 + - frontend-pod-1 + propagations: + - source: product-catalog-deployment-1 + target: product-catalog-pod-1 + condition: Deployment resource requests set to 64Gi memory and 32 CPU exceeding all node capacities + effect: Product-catalog pod stuck in Pending state as no node has sufficient resources to schedule it + - source: product-catalog-pod-1 + target: product-catalog-service-1 + condition: Product-catalog pod is in Pending state with no healthy containers + effect: Product-catalog service has no ready endpoints and cannot serve requests + - source: product-catalog-service-1 + target: frontend-service-1 + condition: Frontend service depends on product-catalog service for product listings and details + effect: Frontend experiences errors when calling product-catalog service, increasing error rate + fault: + - category: Modify + condition: Resource requests set to excessive values (64Gi memory, 32 CPU) preventing pod scheduling + entity: + group_id: product-catalog-deployment-1 + kind: Deployment + name: product-catalog + fault_mechanism: Node resource exhaustion via excessive resource requests simulating misconfigured resource limits or capacity planning failure + recommendedActions: + - solution: + actions: + - Check pod events using kubectl describe pod -l app=product-catalog -n otel-demo for scheduling failures + - Identify the excessive resource requests in the deployment spec + - Reduce resource requests to reasonable values (e.g., 256Mi memory, 250m CPU) + - Verify product-catalog pod transitions to Running and service endpoints become available + id: fix-resource-requests + - solution: + actions: + - Rollback the deployment using kubectl rollout undo deployment/product-catalog -n otel-demo + - Verify pod is scheduled and running with previous resource configuration + - Confirm frontend error rate returns to normal + id: rollback-deployment diff --git a/scenarios/sre/project/roles/scenarios/files/scenario_121/scenario.yaml b/scenarios/sre/project/roles/scenarios/files/scenario_121/scenario.yaml new file mode 100644 index 000000000..8a7191fad --- /dev/null +++ b/scenarios/sre/project/roles/scenarios/files/scenario_121/scenario.yaml @@ -0,0 +1,22 @@ +--- +apiVersion: itbench.io/v2 +kind: Scenario +metadata: + name: scenario-121 +spec: + applications: + opentelemetryDemo: + enabled: true + faults: + - injections: + - args: + kubernetesObject: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: product-catalog + namespace: otel-demo + id: node-resource-exhaustion + tools: + sre: + enabled: true diff --git a/schemas/json/faults/node-resource-exhaustion.json b/schemas/json/faults/node-resource-exhaustion.json new file mode 100644 index 000000000..e19e2a60d --- /dev/null +++ b/schemas/json/faults/node-resource-exhaustion.json @@ -0,0 +1,47 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "kubernetesObject": { + "properties": { + "apiVersion": { + "enum": [ + "apps/v1" + ], + "type": "string" + }, + "kind": { + "enum": [ + "Deployment", + "StatefulSet" + ], + "type": "string" + }, + "metadata": { + "properties": { + "name": { + "type": "string" + }, + "namespace": { + "type": "string" + } + }, + "required": [ + "name", + "namespace" + ], + "type": "object" + } + }, + "required": [ + "apiVersion", + "kind", + "metadata" + ], + "type": "object" + } + }, + "required": [ + "kubernetesObject" + ], + "type": "object" +} diff --git a/schemas/json/library/index/scenario.json b/schemas/json/library/index/scenario.json index 2566fce46..4b4826b62 100644 --- a/schemas/json/library/index/scenario.json +++ b/schemas/json/library/index/scenario.json @@ -134,6 +134,22 @@ } } }, + { + "if": { + "properties": { + "id": { + "const": "node-resource-exhaustion" + } + } + }, + "then": { + "properties": { + "args": { + "$ref": "../../faults/node-resource-exhaustion.json" + } + } + } + }, { "if": { "properties": { @@ -546,6 +562,7 @@ "ingress-port-blocking-network-policy", "insufficient-kubernetes-resource-quota", "nonexistent-kubernetes-workload-persistent-volume-claim", + "node-resource-exhaustion", "scheduled-chaos-mesh-experiment", "kubernetes-api-server-request-surge", "unsupported-architecture-kubernetes-workload-container-image",