Build a long-context KV policy evaluation harness

[randomvariable]

Objective

Build one evaluation harness for long-context KV selection/compression policies before choosing an implementation.

Status

Partial. 23533f449915fda4f2ec027b4e9aaa54f44df4ad ships an offline paged-KV replay harness with deterministic coverage for dense prefix reuse, pool pressure, shared ownership, and hybrid full/sliding-window boundaries. That is a foundation, not the evaluation matrix: no recorded GQA/hybrid/reasoning traces, policy-quality results, or predeclared promotion thresholds are checked in yet.

Evidence tranche: test-only harness now proves physical free-path ownership/refcount decrement for same-step shared blocks (2 -> 1 -> 0), and hybrid shared-prefix-boundary propagation through physical cache layout, prefix hits, and allocation counters, including a cleared-boundary comparison.

Next gate: publish reproducible traces and baseline metrics, then evaluate one policy against fixed retained-quality, memory, and latency/goodput thresholds.

Candidate methods

• RDKV joint eviction/quantization (existing fork issue Prototype rate-distortion KV cache eviction plus quantization #5): https://arxiv.org/abs/2605.08317
• EpiKV reasoning-trace eviction: https://arxiv.org/abs/2606.26472
• SelKV GQA-aware merge/drop compression: https://arxiv.org/abs/2607.16213
• DynaCalKV low-rank head grouping: https://arxiv.org/abs/2607.24331
• LOCKS page-local key summaries: https://arxiv.org/abs/2607.24555
• KARA sliding-window-aware reasoning compression: https://arxiv.org/abs/2607.01237

Applicability is scope-dependent

DynaCalKV, LOCKS, and KARA were verified against the executing code. All three are inapplicable to MLA and applicable to the GQA/SWA scope this issue targets — so the GQA scoping below is load-bearing, not incidental:

Candidate	MLA scope	GQA/SWA scope
DynaCalKV	Redundant — MLA KV is already a low-rank latent	Applicable — no low-rank projection in the GQA path (vllm/v1/attention/backends/flash_attn.py:133)
LOCKS	Redundant — the DSv4 indexer already does select-then-exact	Applicable — GQA attention receives the full block table with no selection stage (flash_attn.py:904)
KARA	Blocked — no arbitrary-token compaction in the KV manager	Applicable — an implementation gap, not an architectural bar

The KV manager frees or nulls whole blocks only (vllm/v1/core/single_type_kv_cache_manager.py:595); sub-block token compaction is the shared prerequisite for KARA-style work.

In-tree precedent for block-sparse GQA selection

Block-sparse GQA page selection already exists for one model family: vllm/models/minimax_m3/common/indexer.py scores KV blocks and selects top-k plus fixed init/local blocks, common/sparse_attention.py consumes only selected blocks, and amd/ops/sparse_pa.py provides a ROCm path.

Caveat: M3's index heads are trained parameters, and the indexer deliberately mirrors DeepSeek-V4's separate-indexer structure. LOCKS proposes training-free page summaries. The reusable part is the mechanism — side cache, block scoring, top-k selection, block-sparse consumption, working on both targets — not the scoring policy. Treat the M3 path as the reference implementation to measure against, not a drop-in.

Model scope and cache-contract boundary

Initial harness covers block-addressable attention KV only: dense GQA/MQA and sliding-window/global hybrids. MLA requires a dedicated trace adapter for its compressed latent representation and sparse-index metadata. DeltaNet, Kimi Delta Attention, Mamba-family recurrent state, and convolutional state are out of scope unless separate state-transition adapters are defined.

Candidate rows: Hy3 or Granite 4.1 for GQA; Gemma 4, Laguna S 2.1, Step-3.7-Flash, or North-Mini-Code for sliding/global; GLM-5.2 or DeepSeek-V4-Flash for MLA/sparse-MLA only once a compatible trace adapter exists. Do not use Qwen3.6, Kimi K3, Nemotron 3 Ultra/Nano, Inkling, or LFM2.5 as generic paged-KV baselines for their recurrent, linear, or convolutional layers.

Add representative 128K, 256K, 500K, and 1M rows. Measure full-attention retention separately from naturally bounded SWA state. MiMo-V2.5 and Mistral Small 4 make 1M-token architecture rows concrete; Step-3.7-Flash and North-Mini-Code add mixed SWA/global cases. Issue #22 owns cross-request admission and cache-sharing correctness for combined multimodal/hybrid state.

Related fork work

• Prototype rate-distortion KV cache eviction plus quantization #5 tracks an RDKV-specific joint eviction/quantization experiment. This issue provides the common evaluation harness and comparison surface; it does not duplicate that implementation candidate.

Rationale

These methods compete for the same fork surface and have model-specific caveats. Creating separate implementation issues now would commit to unvalidated paper claims and duplicate cache-policy plumbing. Evaluate them under one page/block-policy interface first.

This framing held up under verification: candidates that failed elsewhere failed because they were assessed against the wrong attention class, or because the mechanism already existed in the executing path. A harness that measures rather than assumes addresses both failure modes.

Proposed work

• Define an offline/replay harness over vLLM paged-KV traces with pluggable score, evict, merge, low-rank, and page-summary policies.
• Separate GQA/MQA/global layers from sliding-window and MLA layouts.
• Measure retained KV bytes, KV read traffic, decode latency, quality, and speculative acceptance.
• Use exact attention over retained blocks for first experiments.
• Promote only methods that show a useful quality/performance frontier into implementation issues.

Acceptance criteria

• Reproducible traces and baseline metrics exist for at least one GQA model, one sliding/global hybrid, and one reasoning workload.
• Policies cannot corrupt cache ownership or address mapping.
• Results include quality and performance, not memory reduction alone.
• Before running the matrix, define baseline policies plus minimum retained quality, memory reduction, and latency/goodput improvement required to open an implementation issue.