[KV Connector] Add per-layer canonical KV page mappings for parallelism-agnostic offload#48408
Open
Etelis wants to merge 4 commits into
Open
[KV Connector] Add per-layer canonical KV page mappings for parallelism-agnostic offload#48408Etelis wants to merge 4 commits into
Etelis wants to merge 4 commits into
Conversation
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
This was referenced Jul 12, 2026
Etelis
marked this pull request as ready for review
July 14, 2026 09:33
Etelis
requested review from
ApostaC,
NickLucche,
ivanium,
orozery and
xuechendi
as code owners
July 14, 2026 09:33
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Etelis
marked this pull request as draft
July 14, 2026 09:34
All parallelism reasoning moves to vllm/v1/kv_offload/sharding.py, covering TP/DCP/PCP, packed and split KV layouts, with single-writer election. Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Etelis
changed the title
Etelis
marked this pull request as ready for review
July 18, 2026 19:42
1 task
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records on
CanonicalKVCacheRef, per layer, how the worker's KV page maps into acanonical (parallelism-free) page: the full offloaded block, all KV heads, all
block_size * dcp * pcptokens. A cache offloaded under one parallel config canlater be rearranged or consumed under another.
All parallelism reasoning lives in
vllm/v1/kv_offload/sharding.py; the connectorand everything downstream consume byte mappings (
MappedRun/CanonicalPageMapping)only.
Certified layouts:
total_cp_rank, chunked bycp_kv_cache_interleave_sizeparallel_invariant=False(fail closed)Store runs are verified at startup to tile each canonical page exactly once across
the worker group. Mappings are inert metadata in this PR; the CPU layout consumer
lands in the follow-up.
Test plan
pytest tests/v1/kv_offload/test_kv_sharding.py(26 tests, no GPU): hand-computedplacement offsets for all four layouts, DCP/PCP tiling, writer election, cross-TP
and CP byte round-trips, fail-closed gates.