[RFC]: Session-centric KV-cache orchestration over typed session identity

[vMaroon]

Motivation.

Agentic workloads are stateful programs, and their KV-cache is the program's working memory - tool definitions reused every turn, prefixes shared across parallel branches, context carried across turns and pauses. Serving them well across a fleet is a state-orchestration problem: which session state to keep, where to place it, when to move it, which program it belongs to. That orchestration belongs above the engine, in a control plane (llm-d, Dynamo) whose indexer becomes the cluster's memory map - while the engine stays a policy-unaware mechanism that materializes KV, reports what it did, and executes intent-free directives.

Typed session identity is the first primitive that split needs, and it is in flight: #48049 proposes a request-level session_id that engine internals and connectors read instead of overloading request_id or extra_args, with the implementation open in #48048. This RFC builds directly on it.

But session_id makes identity readable inside the engine; session-centric orchestration by an external control plane needs two more things the engine must expose and accept, both intent-free:

• Visibility in session coordinates. KV-cache events today report block hashes with no session or lineage. A control plane can read session_id on the request, but it cannot see which stored blocks belong to that session, or when they were evicted, without recomputing engine hashes — the coupling that makes the prefix-cache path mirror engine internals. Echoing the session coordinates on KV events closes that gap: the indexer maps KV lifecycle to sessions directly, no tokenization, no hash mirroring.
• Directives in session coordinates. Keeping a paused session warm — and later offloading, moving, or prefetching its state — are the actual orchestration actions. The retention API ([RFC]: Context-Aware KV-Cache Retention API (Prioritized Evictions) #37003) is the first of them; it becomes addressable at session granularity once the coordinates exist on both the visibility and directive sides.

One coordinate is coarser than orchestration needs. session_id scopes a lineage; it does not locate a position within it, which is what fork-safe, recompute-stable KV addressing requires. This RFC adds a second, content-derived coordinate — continuation_id, a chain position whose equality holds exactly when content prefixes are byte-identical (same model, same cache_salt) — carried and echoed the same way, opaque to the engine.

The identity all of this rests on is already declared at the application edge and already plumbed by #48049: Anthropic Messages carries cache_control block markers and the conversation array; OpenAI Chat and Responses carry prompt_cache_key, prompt_cache_retention, previous_response_id, and conversation objects. The control plane sets session_id from those where present and derives it from content where absent; continuation_id is always content-derived, with declared ids aliasing into it. The engine parses none of it.

Proposed Change.

The ask is small: a continuation_id request coordinate beside session_id, echoed with it on block-store events, and the directive channel aligned to the same coordinates. Everything else is framing and roadmap.

Coordinates

• session_id (from [RFC][Feature]: First-class session id for conversation aware routing #48049) — the session scope, constant across a session. Adopt as-is; no competing field.
• continuation_id — a content-derived chain position, the addressable node. Carried on the request beside session_id, opaque to the engine, recomputable by the control plane from any resent prefix. Proposed as a scalar field mirroring session_id — the minimal form; if [RFC][Feature]: First-class session id for conversation aware routing #48049's RequestMetadata direction lands, it moves into that structure's "cache lineage" slot.

The two are orthogonal and both load-bearing. Being content-derived, continuation_id is identical across sessions exactly where their content is: a shared prefix, a fork's common ancestor. Neither session_id nor an engine-reported token offset can supply that — an offset locates a block within one request; a block shared by several sessions, or the same position reached by two of them, has no common name unless the name is the content. That position is also what well-posed removals of shared blocks need, and it is the only one that is tokenizer-free and engine-hash-free — minted by the control plane from bytes rather than read back from the engine.

Visibility (engine -> control plane)

V1a — first increment:

• Echo session_id and continuation_id on BlockStored events, alongside the existing block hashes.
• Coordinates never feed block hashing — zero effect on dedup, prefix matching, or reuse. They do enter event equality (msgspec generates __eq__ over all fields; __hash__ must follow), which is safe today: the only in-tree consumer of event identity is KVEventAggregator, and the events that cross it carry no labels. The connector labeling follow-up (next bullet) must keep that true — labels derived once, scheduler-side, so replica workers emit identical events — or move the aggregator to a label-blind key.
• Scope: labels cover the engine's prefix-cache events (medium="GPU"), where the block pool has the Request in scope. Connector-emitted events do not and stay unlabeled in V1a — offload tiers (MEDIUM_CPU / MEDIUM_FS / MEDIUM_OBJ, surfaced scheduler-side via the connector's take_events) and worker-side events (LMCache, Mooncake, aggregated via KVEventAggregator). Labeling them means plumbing coordinates through connector metadata — a named follow-up, not this ask.
• Label semantics follow the report mode — part of the consumer contract. Under the default kv_cache_report_mode="incremental", a store event fires per admission — duplicate blocks under one hash and re-admission after eviction can repeat a hash — each carrying its admitting request's labels. Under "full" (already on main: rides sampling_params.extra_args, drives emit_cached_block_events on prefix-cache hits), reuse events also carry the touching request's labels — the consumer keeps a hash-to-sessions multimap: exactly the cross-session reuse visibility this RFC is after, and how a restarted indexer rebuilds coverage from live traffic.
• BlockRemoved stays hash-keyed in V1a; the consumer correlates removals to sessions via the store labels it already saw. Removals mirror admissions — one per admitted copy on the engine path — so a consumer either balances stores against removals for exact residency, or invalidates on the first removal, which errs safe (under-coverage, never phantom coverage). No engine-side per-session accounting.
• The one footgun: every serving entrypoint must thread the coordinates explicitly, and a missed path silently drops labels. Guard each with a test.

V1b — coordinates on removals:

• Attribute each cached block hash to the continuation_id that first admitted it (a per-hash side map on the block pool), so a removal emits that continuation_id plus the prefix length the hash covers, alongside the group_idx and medium it already carries. Attribution is per-hash, not per-block — a shared, refcounted block carries several hashes from different continuations, while each engine prefix-cache removal carries exactly one hash, so "the admitting continuation" is well-defined. And it survives recompute: an evicting- or recomputing-request label would change owners after the first eviction-and-recompute cycle; the admission label re-derives identically when the same content is re-admitted.

Event mode: toward hash-free session coordinates

Block hashes stay on the wire by default: existing consumers (precise prefix-cache indexers, KV connectors) key on them, and unlabeled traffic has no coordinates to carry. But once removals carry node coordinates and a prefix length, the hash is redundant for a session-coordinate consumer. Consumers may rely on the invariant: byte-identical content prefixes carry equal continuation_ids. On the wire the equality surfaces only where chain heads coincide, since each request labels its stores with its own position; interior-position sharing resolves through the coordinate ancestry the control plane minted. A removal names the admitting position; the consumer invalidates every session whose coverage includes it.

So the end-state is a per-request report mode, not a config flag and not a breaking change — a coordinates value on the existing kv_cache_report_mode:

• Default — hashes on, as today; labels ride alongside. Nothing changes on the wire for deployments where no request opts in.
• Opt-in, per request — that request's events are emitted in session coordinates only: no hashes, no token_ids (most of the wire savings — on long contexts token content dominates event size), coverage addressed as (session_id, continuation_id, token range). A control plane that consumes only session coordinates stops carrying a correlation substrate it no longer reads. Unlabeled traffic cannot opt in. A removal has no request at the emit site, so its shape follows the admitting request's recorded mode.

The engine keeps hashing internally for its own dedup and for unlabeled traffic; this is a reporting-language choice on the wire, decoupled from block management. Consumers that key on block hashes stay with default-mode traffic — coordinate mode reports token ranges without block identity, and on a mixed stream a hash-keyed consumer does not see coordinate-mode admissions, a constraint of the opt-in increment. The language addresses prefix-positional identity, matching the engine's own prefix-only hashing; mid-sequence reuse of displaced content would extend it with a new coordinate, not reinterpret continuation_id. The point is a path toward one coordinate language rather than carrying both indefinitely.

Directives (control plane -> engine)

• Retention (RFC [RFC]: Context-Aware KV-Cache Retention API (Prioritized Evictions) #37003 / PR [Feature] Context-Aware KV-Cache Retention API (#37003) #38514) is the directive-down counterpart, keyed on the same coordinates: an eviction-priority bias with optional TTL, no pinning. It and the visibility channel land better together — the BlockStored labels build the session-to-residency map a keyed retention directive then acts on.
• Offload, move, discard, and prefetch are the rest of the directive family, addressable in the same coordinates as the roadmap progresses. Not in this RFC's ask.

Boundary

The engine carries two opaque coordinates, echoes them on events, and applies an eviction-priority bias. It does not parse APIs, does not know what a session is, and does not evaluate policy — all orchestration intelligence stays in the control plane, the same boundary #48049 draws for identity, extended to lifecycle and directives. And the coordinates are identity, never semantics: no context types, priorities, or lifecycle classes ride engine events; semantic tagging stays in the control plane's indexer, and lifecycle typing reaches the engine only as directives.

Trust follows the same line: the engine validates only shape (non-empty string, length-capped). The carrying request fields and headers (X-Session-ID / X-Continuation-ID) are client-controllable on an exposed engine, so in a control-plane deployment the gateway mints the coordinates and strips client-supplied values at the edge. Mis-minted labels corrupt attribution on the event stream — never block hashing or reuse, which are label-blind. Once directives key on the coordinates, label trust becomes load-bearing (a mis-minted coordinate can steer eviction priority) — exactly why minting belongs to the gateway, not the client.

Robustness

• The event stream needs a restart signal, or a crashed and restarted engine leaves phantom coverage in the indexer. Concretely: an appended publisher_epoch: str | None on EventBatch — a nonce minted at publisher startup, per endpoint. EventBatch is array_like and msgspec tolerates an appended trailing field, so existing consumers decode unchanged. Until it lands, the fallback is treating a seq regression on the ZMQ stream as a reset (sequence numbers restart at zero with the process) — which fires only once a new event arrives, and a consumer that reconnects late can miss entirely.

Prototype

A V1a echo on BlockStored with consumer-side correlation exists on a fork, validated end-to-end against the llm-d router. Link to follow.

Sketch: proposed API changes

Grounded against current main @ 5c342876a (kv_events.py, block_pool.py, kv_cache_manager.py, config/kv_events.py) and #48048. Dependency: #48048 is unmerged, so Request.session_id / EngineCoreRequest.session_id live only on its branch; continuation_id is new here. Invariant: the labels never feed block hashing (generate_block_hash_extra_keys, ExternalBlockHash).

1. Event structs (vllm/distributed/kv_events.py). KVCacheEvent encodes as a tagged msgpack map (no array_like on it), so appended optional fields are ignored by older consumers and default-filled when absent; with omit_defaults, unlabeled traffic stays byte-identical to today. Append after the current last fields and mirror into __hash__:

class BlockStored(KVCacheEvent):
    ...  # block_hashes, parent_block_hash, token_ids, block_size, lora_id, medium,
         # lora_name, extra_keys, group_idx, kv_cache_spec_kind, kv_cache_spec_sliding_window
    session_id: str | None = None         # new (appended last)
    continuation_id: str | None = None    # new (appended last)

class BlockRemoved(KVCacheEvent):
    block_hashes: list[ExternalBlockHash]
    medium: str | None
    group_idx: int | None = None
    # V1b; set only by the engine prefix-cache emit path (one hash per event).
    continuation_id: str | None = None    # admitting continuation of the single hash
    num_tokens: int | None = None         # prefix length the hash covers, from admission

Scalar continuation_id, not a parallel list: _emit_block_removed_events emits exactly one hash per event, while connector/offload removals batch hashes per group (kv_connector/v1/offloading/events.py) and leave the fields None. (continuation_id, num_tokens) names the admitted block whose prefix ends at num_tokens — what makes hash-free removals precise rather than continuation-coarse. The consumer invalidates that block's span (width from the admission store's block_size), never truncates beyond it. Removals attribute to a content position, not a session — an admitting session_id would be arbitrary for a shared block; consumers resolve position-to-sessions from the coverage they built from stores.

2. Store-time echo (vllm/v1/core/block_pool.py, three sites). Patch _build_block_stored_event (shared by cache_full_blocks and emit_cached_block_events) and the direct construction in cache_partial_block — editing only the helper leaves partial-block events unlabeled:

BlockStored(..., session_id=request.session_id, continuation_id=request.continuation_id)

All three sites have the request in scope (session_id today via #48048; continuation_id once step 4 lands): admission and partial stores carry the admitter, reuse stores the toucher — the V1a semantics with no extra logic.

3. Eviction attribution — per-hash side map (V1b). Not a per-block stash: KVCacheBlock is @dataclass(slots=True) with no room for it, and the removal path receives bare hashes after reset_hash(). Attribute per hash:

# BlockPool.__init__, beside cached_block_hash_to_block:
self.hash_to_continuation: dict[BlockHashWithGroupId, tuple[str | None, int | None]] = {}

# _insert_block_hash gains a continuation_id param (default None). Populate only when
# self.enable_kv_cache_events, after the existing early returns; setdefault => the
# continuation that FIRST admitted this hash wins. num_tokens is already a param.
if self.enable_kv_cache_events:
    self.hash_to_continuation.setdefault(
        block_hash_with_group_id, (continuation_id, num_tokens))

# _emit_block_removed_events (iterating BlockHashWithGroupId keys, the map's key type):
# pop only when the hash actually left the cache. Duplicate blocks under one hash
# (BlockHashToBlockMap NOTE #1) emit a BlockRemoved while another block still serves
# the hash; popping there would strand the final removal unlabeled.
if self.cached_block_hash_to_block.get_one_block(block_hash_with_group_id) is None:
    cid, num_tokens = self.hash_to_continuation.pop(
        block_hash_with_group_id, (None, None))
else:
    cid, num_tokens = self.hash_to_continuation.get(
        block_hash_with_group_id, (None, None))
BlockRemoved(..., continuation_id=cid, num_tokens=num_tokens)

# reset_prefix_cache: self.hash_to_continuation.clear()

The pop condition leans on an ordering invariant that holds on main and must stay pinned: _remove_cached_block_hashes deletes the (hash, block) pair before _emit_block_removed_events runs, and events are constructed synchronously at removal — deferred emission would read post-removal map state. The enable_kv_cache_events gate is load-bearing too: pops happen only on the event path, so unconditional population would leak when events are off. Callers: cache_full_blocks and cache_partial_block pass request.continuation_id; move_block_hashes has no request, emits no events, and pops nothing — its re-insert setdefault no-ops on the still-present key, so attribution survives a move. emit_cached_block_events never calls _insert_block_hash, so reuse traffic cannot perturb attribution. The attribution is first admission exactly — not "the prefix's owner" or "last user"; the event stream cannot answer those for a shared block.

4. Request-path plumbing for continuation_id — mirror #48048's chain exactly. Protocol fields on chat/completions/responses; a resolver beside GenerateBaseServing._get_session_id (vllm/entrypoints/generate/base/serving.py) with the same precedence — body field, then X-Continuation-ID header, then vllm_xargs fallback — and the same shape validation plus a length cap; the generate / add_request path (vllm/engine/protocol.py, async_llm.py, llm_engine.py); vllm/v1/engine/input_processor.py; EngineCoreRequest (vllm/v1/engine/__init__.py); Request (ctor + from_engine_core_request). Parallel-sampling children get it via copy(request). #48048 also threads the Rust frontend (request types, chat/completions converters, resolve_session_id) — parity there is part of the cost, roughly the same touch-set again. The labels stay out of renderers/ and inputs/preprocess.py.

5. Coordinate report mode — per-request, not a config flag. A field on KVEventsConfig is the wrong home twice over: EventPublisherFactory.create forwards config fields as publisher-constructor kwargs (it asdict()s and pops only publisher and enable_kv_cache_events), and BlockPool receives only a boolean — a config flag needs new plumbing on both sides. The existing per-request kv_cache_report_mode knob is the right precedent and granularity:

# extra_args["kv_cache_report_mode"]: "incremental" (default) | "full" | "coordinates"
# "coordinates": labeled events with block_hashes=[] and token_ids=[],
# plus a start token offset.

The mode is recorded beside the V1b side map's label. The exact coordinate-mode event shape and mixed-stream semantics are the open design points of the V1b/V2 increment (block_hashes and token_ids are required fields, so the mode emits them empty rather than absent); V1a is labels-alongside only.

Tests.

• Labels pass every serving entrypoint — chat, completions, responses, generate, beam-search, batch, Rust frontend — and reach BlockStored.
• Labels never enter block hashes: identical prefixes under different labels still share blocks and emit one admission store.
• Engine removal events stay single-hash: an emit-site test pins one hash per BlockRemoved on the prefix-cache path — the invariant scalar attribution rests on.
• V1b: admit, share from a second session, evict — removal attributes to the first admitter; duplicate-block eviction keeps the final removal labeled; move_block_hashes then evict keeps attribution; reset_prefix_cache clears the map; events-disabled runs leave the map empty.
• Old-consumer decode of labeled events (unknown map keys ignored) and unlabeled events byte-identical to the current wire format.

Feedback Period.

2 weeks.

CC List.

@tlrmchlsmth @ywang96 @karen-sy @PeaBrane @bongwoobak @hyeongyun0916

Any Other Things.

Companion documents.

Public design docs this RFC's primitives serve. They are control-plane vision, not proposed vLLM behavior — what a control plane does with the coordinates once the engine reports and accepts them.

• llm-d Agentic Inference Northstar — the top-level vision: where today's inference stacks fall short of agentic workloads, and the envisioned directions.
• KV-Cache Orchestration for Agentic Inference — KV-cache orchestration as a control plane above the engine: retention, offload, and move as intent-free primitives, the indexer as the cluster's semantic memory map, and the boundary that keeps all policy out of the model server. Where the directive family in this RFC comes from.
• Session-Graph Orchestration for Agentic Inference — the session and its session-blocks as the control-plane abstraction, reuse of shared blocks across sessions, and alignment with application-layer signals (cache_control, prompt_cache_key, previous_response_id). Where the session coordinates and the API alignment come from.
• Session-Centric Prefix-Cache Affinity and Session Keeping — the concrete proposal these engine primitives serve.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[franciscojavierarceo]

Thanks for writing this up. I’m working on the agentic Responses path in vllm-project/agentic-api and have been testing the end-to-end lifecycle across agentic-api, llm-d-router, and vLLM on a two-replica GPT-OSS-20B deployment.

I definitely support additive session/continuation labels and the llm-d-policy/vLLM-mechanism boundary. One challenge is treating a request-level, byte-derived continuation_id as interchangeable with block identity.

At the cache layer, vLLM reuse is determined by the parent hash, rendered token IDs, LoRA/MM/prompt-embedding/cache-salt extra keys, cache group, and execution configuration. That makes continuation_id useful as a logical or policy coordinate, but not sufficient evidence that two requests can reuse the same KV blocks.

There is a similar granularity issue on the event side: one request-level continuation label covers many blocks, while a shared block may benefit multiple sessions or branches. “First admitting continuation” seems useful as provenance, but not as ownership or a complete removal address.

I’d recommend retaining block hashes, adding engine instance/epoch semantics, and treating session coordinates as lifecycle metadata for retention, offload, move, and prefetch rather than exact locality identity. The control plane can then maintain a many-to-many mapping from sessions and continuations to the exact resident extents reported by vLLM.

Some context for the routing terminology in our results:

• Cache-blind load routing selects a replica from load and queue signals without considering where the reusable KV prefix is located.
• Estimated prefix routing remembers which replica previously received a matching prompt and predicts that its blocks are still resident. It does not receive actual eviction truth from vLLM.
• Event-backed exact routing renders the model-visible token sequence, computes its block keys, and compares them with the blocks each vLLM replica last reported through KV-cache events.

In our small sequential experiment, cache-blind routing returned 6 of 12 continuations to the warm replica, with a 49.54% mean KV hit rate. Both cache-aware approaches returned 12 of 12, with a 99.08% mean hit rate.

That validates exact placement and shows that an estimate can also work in a simple idle workload. It does not test whether useful blocks survive concurrent load, eviction, or tool-call pauses—the lifecycle problem this RFC is aiming at. My view is that session coordinates are the right additional abstraction for that problem, while exact block identity should remain the placement and residency substrate.

We documented the full lifecycle, experiment, and remaining work in ADR-04: KV-affine continuation for agentic Responses workloads, with the implementation and benchmark context in agentic-api PR #65.

I'll be joining the agentic-inference working group to see if we can collaborate together on this because I think there's a lot of impact we can have together to work through this. 🤠

[chaunceyjiang]

Typed session_id and continuation_id fields inside EngineCore make sense to
me. For the Responses API, though, clients should not need to send or manage
them.

The frontend already has enough information to derive them:

conversation.id      -> session_id
previous_response_id -> look up the parent and inherit its session_id
current response.id  -> continuation_id
prompt_cache_key     -> cache affinity only

It can resolve these fields into an internal ResponseLineage and pass the
result to EngineCoreRequest. There is no need to add either field to the
public Responses request or response schema.

This gives the engine the typed coordinates proposed here without requiring
changes to existing Responses clients or OpenAI SDKs. For a stateless request
with no conversation or previous response, the session can remain unset or be
scoped to that request. KV reuse should continue to fall back to the existing
block-hash-based APC rather than trying to infer a session from matching
content.

I see the boundary as:

standard Responses API
-> frontend-derived ResponseLineage
-> internal session_id / continuation_id
-> block-hash-verified KV operations

That keeps the feature transparent to clients and leaves block hashes as the
source of truth for cache correctness.

[franciscojavierarceo]

For the Responses API, though, clients should not need to send or manage
them.

The front end already has enough information to derive them:

💯