Skip to content

Add security benchmark suite and performance overhead measurements #9

@vgudur-dev

Description

@vgudur-dev

Description

Create a benchmark suite that measures:

  1. Detection accuracy against known memory poisoning attack patterns
  2. False positive rates on benign memory operations
  3. Performance overhead (latency added per read/write operation)

Motivation

Enterprise adoption requires quantified security guarantees and performance impact data. This benchmark will:

  • Provide data for the README and documentation
  • Enable regression testing as new detectors are added
  • Give users confidence in production deployment

Proposed Implementation

Create benchmarks/ directory with:

1. benchmarks/security_accuracy.py

  • Test against a corpus of known attack patterns (prompt injection variants, obfuscated secrets, etc.)
  • Measure true positive rate, false positive rate, and false negative rate
  • Report results as a confusion matrix

2. benchmarks/performance.py

  • Measure latency overhead per operation (read, write, snapshot, rollback)
  • Test with varying policy complexity (1 rule vs 10 rules)
  • Test with varying value sizes (100B, 1KB, 10KB, 100KB)
  • Compare InMemoryStore vs Redis backend

3. benchmarks/attack_corpus/

  • Collection of memory poisoning payloads for testing
  • Categorized by attack type (injection, leakage, tampering, churn)

Acceptance Criteria

  • Security accuracy benchmark with reproducible results
  • Performance benchmark with latency measurements
  • Attack corpus with 50+ test payloads
  • Results table in README showing detection rates
  • CI job that runs benchmarks on each release

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestsecuritySecurity-related improvements

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions