feat: add optional rust-based acceleration engine & security gateway

[Mr-Charvaka]

Summary

This PR introduces the KORA Rust Acceleration & Security Engine, providing an optional high-performance execution and validation backend written in Rust, connected via PyO3 FFI.

To prevent adoption friction for maintainers, it is designed with a graceful fallback policy:

• Set KORA_USE_RUST=1 in the environment to route normalization, validation, and task loop execution through Rust.
• If the environment variable is not set, or the compiled library is missing, it falls back to native Python seamlessly, ensuring zero breaking changes.

Key Additions

1. kora-rust/ Crate:
   • A Kahn topological scheduler and stable graph validator in Rust.
   • Real-time, SIEM-compliant JSON telemetry logging to standard output.
   • A security gateway that recursively redacts PII (emails, SSNs, phone numbers) and critical developer keys (OpenAI keys, AWS keys, JWTs, GitHub tokens, and Bearer tokens) before adapter routing.
   • Luhn-algorithm card detection to prevent redacting random digit series.
2. PyO3 Python Bindings: Exposes validation, normalization, and execution functions to Python.
3. Differential Fuzzer: Custom test suite (cargo test --test differential_fuzz) running 10,000 mutated inputs to ensure 100% engine stability (0 crashes, 0 panics).
4. Criterion Benches: Micro-benchmarking targets (cargo bench) reporting nanosecond scheduling latency.

Benchmarks (Reproducible via python benchmark.py)

Latency average:

• 100 tasks: Python 124.5 ms | Rust 2.0 ms (62.0x speedup)
• 10,000 tasks: Python 20.46s | Rust 0.25s (81.5x speedup)

Peak RSS Memory savings:

• 10,000 tasks: Python 204.8 MB | Rust 197.1 MB (7.68 MB saved in execution, 23.45 MB saved in validation).

See METHODOLOGY.md for detailed telemetry, memory, and FFI startup latency analysis.

[hkalbertkim]

Thanks for the contribution. I reviewed this PR, but I don't think it is ready to merge yet because it changes runtime behavior and adds a new Rust/PyO3 execution path without enough packaging, compatibility, and test coverage.

Could you please revise it with the following changes?

1. Remove local absolute file URLs from METHODOLOGY.md, for example file:///d:/.... Public docs should use repository-relative links only.

2. Add a supported packaging path for kora_rust if this is meant to be installable from the Python package. Right now pyproject.toml does not include the Rust extension build integration, an optional extra, or clear install instructions.

3. Do not make the Rust path return ok: True unconditionally. The KORA_USE_RUST=1 path in run_graph() needs to preserve the existing Python executor contract, including failure handling, events, outputs, final, and timing fields.

4. Add focused tests for:

   • KORA_USE_RUST=1
   • fallback behavior when kora_rust is not installed
   • error propagation from the Rust path
   • parity with the existing Python executor contract for at least one deterministic graph

5. Please soften or remove the benchmark and security claims unless they are reproducible in CI or clearly marked as local experimental results. Claims like speedups, SIEM-ready telemetry, and security gateway behavior need tests or clearly documented limitations.

6. Keep the first version smaller if possible. A good first mergeable version would be:

   • Rust crate scaffold
   • one narrow validation/normalization function
   • packaging/install path
   • parity tests
   • no broad benchmark or enterprise security claims yet

I could not run Rust tests locally because cargo is not available in my environment. I did run the Python executor/task IR tests on this branch, and they failed, so please make sure the existing Python test suite passes before the next review.

[Mr-Charvaka]

@hkalbertkim Thaks for reviewing and providing the insights!