feat: add fault [Exhausted Etcd Database Storage] (SPAI-414)

[Aliciapet11]

Summary

• Add new fault type exhausted-etcd-database-storage and scenario 116 derived from 6 real Konflux Web RCA incidents
• Fault mechanism: Creates 200 large ConfigMaps to fill etcd storage
• Includes molecule test for CI validation

New Files

• Fault definition: scenarios/sre/library/indexes/faults/32.json
• Injection task: scenarios/sre/project/roles/faults/tasks/inject_exhausted_etcd_database_storage.yaml
• Molecule test: scenarios/sre/project/roles/faults/molecule/inject_exhausted_etcd_database_storage/
• Scenario index: scenarios/sre/library/indexes/scenarios/116.json

Test plan

• make validate-library passes
• Molecule test provided (inject_exhausted_etcd_database_storage)
• Live cluster test with OpenTelemetry Demo

🤖 Generated with Claude Code

[Red-GV]

I'm sorry for how long it took for me to get to this one @Aliciapet11 .

I'm a bit concerned with this one. I like the scenario itself, but my problem is the clean up. As I understand it, once etcd is full, kubectl will stop working. This means that we cannot actually do our teardown sequence without ensuring that the cluster is fixed. Which would mean two things:

1. We'd have to supply the code for fixing it in order to be part of the automation. While we do supply instruction sequences to fix the fault, providing a whole code/playbook to undo this fault seems a bit unorthodox. In addition, from what I'm seeing for fixes for this one, the agent would need ssh access to the control plane node in order to run these compaction commands. This is not permission level we'd like to give to the agents running these tests at this time. Also, it looks like running on kind (docker, podman, etc.) have different procedures compared to running on AWS or another cloud provider.
2. We'd have to destroy the cluster. Generally, after a run is over, we don't destroy the cluster but ensure we have cleaned all of our files and modifications from it. Thus returning it back to the state it was before the benchmark started. If we don't supply the fix via (1), then we're left with this nuclear option being the only course correction.

Unless I'm wrong with my understand (completely possible), I'm not sure if we can support this scenario on the bench (though it is a good/interesting one).

[Aliciapet11]

Hi @Red-GV, thanks for the thorough feedback — you were right about the cleanup issue. I've reworked this to use a degradation approach instead of full exhaustion:

Key changes:

• Reduced from 200×900KB to 50×100KB ConfigMaps — enough to create storage pressure and latency, but kubectl still works for cleanup
• Changed alert from KubePodNotReady to HighRequestLatency (degradation, not outage)
• Added kubectl delete configmap -l app=etcd-storage-fill commands to the solutions — cleanup is now straightforward since the API server is still responsive
• Allowed Deployment kind in the schema (not just StatefulSet) — targeting frontend deployment in otel-demo instead of an etcd StatefulSet
• Registered the fault in argument_specs and task_files
• Added Jinja2 templates for both the fault and scenario definitions

The teardown sequence should work fine now since we're not locking out the API server. Let me know what you think!