feat: add opt-in exact-match caching for LLM responses

[zhongxuanwang-nv]

Overview

First implementation stage of a five-stage response-cache series (exact match → logical keys → tool-result cache → semantic → streaming semantic), with the standalone documentation in #405. This stage is independently reviewable.

Adds an opt-in, exact-match response cache for managed LLM calls as a response_cache section of the Adaptive plugin config. Repeated eligible calls can be served from an in-memory or Redis store instead of re-running the provider. The feature is off until the section is present; by default, only explicitly deterministic requests (temperature = 0) are cacheable. Runtime backend failures fail open to a live call, while invalid configuration is rejected.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

• Keying (crates/adaptive/src/response_cache/key.rs): keys use a SHA-256 fingerprint of the RFC 8785-canonicalized, normalized request. Provider surface detection and guarded raw-body fallback avoid incorrect reuse when a decode would be lossy. Provider, namespace, and schema version are folded into the key; only allowlisted headers participate.
• Eligibility and storage (crates/adaptive/src/response_cache/intercept.rs): incomplete, error, stateful, unparseable, or otherwise unsafe calls bypass or remain unstored. Nondeterministic requests bypass unless cache_nondeterministic = true is set explicitly.
• Streaming: live misses tee provider-native chunks while assembling a replayable aggregate. Only naturally completed, faithfully replayable streams are stored. Early close and cleanup errors remain idempotent, close upstream exactly once, and prevent cache writes.
• Store (crates/adaptive/src/response_cache/store.rs): bounded in-memory storage and feature-gated Redis storage with expiry and operation deadlines. Redis initialization/storage failures disable or bypass the optional cache rather than blocking managed calls.
• Gateway integration (crates/cli/src/gateway): upstream failures remain verbatim for the client but are not cacheable; post-success runtime rejections still surface as errors. Buffered and streaming cache hits use the correct content type.
• Config and bindings: Rust, Python, Node.js, and Go expose the same safe default (cache_nondeterministic = false) and validation behavior. nemo-relay doctor reports configuration and backend reachability.
• Observability: response_cache hit/miss/bypass marks include fingerprints, reasons, and savings without request or response bodies.
• Review follow-up: restored the close-aware streaming regressions dropped by an earlier branch replacement, added subscriber-visible nondeterministic bypass assertions, added Redis initialization fail-open coverage, and strengthened the benchmark determinism assertion.

Validation completed on the final branch head:

• RUST_TEST_THREADS=1 just test-rust
• cargo clippy --workspace --all-targets -- -D warnings
• just test-python — 542 passed
• just test-node — 286 passed
• just test-go
• uv run pre-commit run --all-files — all hooks pass on the final tree

Where should the reviewer start?

1. crates/adaptive/src/response_cache/key.rs — key derivation and guarded normalization.
2. crates/adaptive/tests/integration/response_cache_tests.rs — end-to-end eligibility, replay, and stream-lifecycle behavior.
3. crates/cli/src/gateway — the upstream-failure and cached-response boundary.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to feat(codec): add a provider codec factory and route built-in consumers through it #396 — keying, replay, and savings reporting route through the provider codec factory.

Summary by CodeRabbit

• New Features
  • Added opt-in exact-match LLM response caching (buffered + streaming) with TTL, namespace/priority, bypass controls, header allowlisting, and in-memory/Redis backends.
  • Extended configuration and editor/schema support across Rust, Go, JavaScript/TypeScript, and Python, including cache telemetry (hit/miss/bypass) and savings metadata.
  • Added CLI response-cache health/reachability checks with safe fail-open when initialization can’t complete.
• Bug Fixes
  • Improved gateway replay by capturing upstream body bytes, enforcing strict JSON parsing rules, and preserving correct headers and SSE content type.
• Tests
  • Expanded unit/integration/coverage/benchmark coverage for keying, replay safety, marks, gateway behavior, and storage backends.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds an opt-in exact-match LLM response cache with canonical keying, TTL-backed in-memory or Redis storage, buffered and streaming interception, provider-native replay, telemetry, configuration validation, gateway handling, and Rust, Go, Python, and Node APIs.

Changes

Response cache feature

Layer / File(s)	Summary
Configuration and public contracts crates/adaptive/src/config.rs, crates/adaptive/src/response_cache/*, crates/adaptive/src/lib.rs, crates/node/*, go/nemo_relay/*, python/nemo_relay/*, crates/adaptive/Cargo.toml	Adds response-cache configuration, backend specifications, editor metadata, public exports, language bindings, factory helpers, and integration-test targets.
Validation, storage, and runtime registration crates/adaptive/src/plugin_component.rs, crates/adaptive/src/runtime/validation.rs, crates/adaptive/src/response_cache/store.rs, crates/adaptive/src/runtime/features.rs, crates/adaptive/src/redis.rs	Validates cache settings, implements bounded in-memory and optional Redis stores, exposes backend health checks, and registers shared buffered and streaming intercepts.
Keying, interception, and streaming replay crates/adaptive/src/response_cache/key.rs, crates/adaptive/src/response_cache/intercept.rs, crates/adaptive/src/response_cache/replay.rs	Builds canonical SHA-256 keys with cacheability safeguards, handles buffered and streaming hits or misses, aggregates eligible streams, and replays supported provider-native chunk formats.
Savings, diagnostics, and gateway responses crates/adaptive/src/response_cache/mark.rs, crates/cli/src/diagnostics/mod.rs, crates/cli/src/gateway/mod.rs, crates/cli/tests/coverage/shared/*	Reports cache decisions and savings, checks backend reachability, preserves captured upstream responses, serializes short-circuited JSON, and supplies gateway and doctor coverage.
Adaptive integration coverage crates/adaptive/tests/integration/*, crates/adaptive/tests/unit/config_tests.rs, crates/adaptive/tests/unit/response_cache/*	Tests exact-match reuse, bypass rules, validation, telemetry, streaming replay exclusions, cross-mode reuse, Redis sharing, cache isolation, storage behavior, key safety, and latency behavior.
Cross-language configuration validation go/nemo_relay/*, python/tests/test_adaptive_config.py	Tests typed configuration round trips, default factories, serialization, and invalid response-cache diagnostics across Go and Python surfaces.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant Gateway
  participant ResponseCacheIntercept
  participant CacheStore
  participant Provider
  Client->>Gateway: submit LLM request
  Gateway->>ResponseCacheIntercept: execute buffered or streaming request
  ResponseCacheIntercept->>CacheStore: lookup canonical key
  alt cache hit
    CacheStore-->>ResponseCacheIntercept: cached response aggregate
    ResponseCacheIntercept-->>Gateway: return response or replay chunks
  else cache miss
    ResponseCacheIntercept->>Provider: execute upstream request
    Provider-->>ResponseCacheIntercept: response or stream
    ResponseCacheIntercept->>CacheStore: store eligible result
    ResponseCacheIntercept-->>Gateway: return live response or stream
  end
  Gateway-->>Client: response with appropriate content type

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows Conventional Commits and accurately summarizes the PR's exact-match response-cache feature.
Description check	✅ Passed	The description includes the required Overview, Details, reviewer-start, and related-issues sections with substantial implementation detail.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-404.docs.buildwithfern.com/nemo/relay/dev

[github-actions]

License Diff

Compared against origin/main.

Lockfile license changes

Lockfile License Changes

Rust

Added

• None

Removed

• None

Updated/Changed

• None

Node

Added

• None

Removed

• None

Updated/Changed

• None

Python

Added

• None

Removed

• None

Updated/Changed

• None

Status output

[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (449 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (367 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (105 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/main into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (449 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (367 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (105 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done

[willkill07]

One issue found that should be addressed.

[zhongxuanwang-nv]

/ok to test 5975aa6

[zhongxuanwang-nv]

/merge