Benchmark KV-affine Responses routing and token artifacts

[franciscojavierarceo]

Parent tracker: #69
Design and initial evidence: #65

Summary

Build end-to-end benchmark coverage for both KV-affine Responses routing and later portable token-artifact execution across realistic agentic workloads.

Established baseline

In the initial two-replica GPT-OSS-20B previous_response_id benchmark, precise routing increased mean continuation cache hits from 49.54% to 99.08% and reduced mean continuation latency from 811.0 ms to 316.5 ms relative to load-only routing. Approximate and precise routing were effectively tied in the idle deterministic workload.

Scope

• Compare load-only, approximate-prefix, and precise Responses routing while holding prompts, ordering, vLLM processes, APC configuration, and cache state controls constant.
• Run concurrent conversations, gateway tool loops, vLLM-native tool loops, branching histories, renderer/template/tool changes, event lag, eviction pressure, restarts, and active-active EPP.
• Separate warm correct-pod hits, wrong-pod misses, HBM hits, CPU/offload loads, shared-storage loads, and full recomputation.
• Verify /tokenize and inference prompt-token parity for every tested renderer/model family.
• Add server-side timings for agentic rehydration, llm-d render/index/score/queue, vLLM scheduler/prefill/decode, and persistence.
• After portable artifacts exist, compare full logical requests with artifact execution while holding routing and KV residency constant.
• Include GPT-OSS/Harmony plus non-Harmony Responses profiles and realistic short-, medium-, and long-context distributions.

Required output

• TTFT, total latency, input/generated/cached token counts, cache-hit ratio, selected endpoint, matched blocks, queue/load score, fallback reason, and confidence intervals.
• Raw machine-readable samples plus reproducible configuration, image/commit identities, and summarized graphs.
• Separate routing benefit from request-transfer/rendering benefit.
• Report neutral and negative results, including cases where precise and approximate routing are tied.

Acceptance criteria

• Precise routing materially outperforms load-only routing under concurrent agent/tool traffic without changing prompt token IDs or output semantics.
• Tests identify workloads where exact routing is more reliable than approximate pseudo-token identity, or explicitly conclude that no measurable difference was observed.
• Wrong-pod, eviction, restart, event-lag, and tiered-storage behavior is quantified rather than hidden in one aggregate hit metric.
• Portable artifacts preserve routing/cache-hit performance and show whether transfer/render/tokenization savings justify their operational complexity.
• Results are suitable for direct inclusion in ADR-04 and upstream project PRs.