Skip to content

[KV Offload] Carry request provenance through stored events - #52103

Draft
Change72 wants to merge 1 commit into
vllm-project:mainfrom
Change72:codex/kv-offload-event-provenance
Draft

[KV Offload] Carry request provenance through stored events#52103
Change72 wants to merge 1 commit into
vllm-project:mainfrom
Change72:codex/kv-offload-event-provenance

Conversation

@Change72

Copy link
Copy Markdown
Contributor

Purpose

Carry request provenance through asynchronous KV-offload completions so locally initiated stores and promotions can emit full BlockStored events.

The current tracker snapshots full payloads at scheduler lookup/store time and keeps them in a global map. This PR instead keeps a lazy event locator with the request and builds the payload when the raw stored event arrives.

Request + KV-group geometry
            |
     ReqContext event state
            |
   existing async job paths
            |
CPU / FS / OBJ completion
            |
OffloadingEvent(key, ReqContext)
            |
      full BlockStored

Changes

  • Attach request-scoped event state to ReqContext.
  • Lazily resolve a stored key to its full-chunk or partial-tail boundary.
  • Carry the exact ReqContext through CPU, FS, OBJ, cascade, and promotion completion.
  • Use context identity, not request ID, when an old completion is finalized.
  • Generate GPU-equivalent per-block extra_keys, including multimodal and prompt-embedding inputs.
  • Keep placeholders for external work, unsupported cache shapes, and unsafe token-mutating requests.
  • Keep only detached chunk hashes for the existing CPU removal contract; [RFC]: KV offload event path refactor — provenance-carrying events and key-only removals #49413 PR 2 removes that table.
  • Preserve queued CPU removals and their detached expansion metadata across reset.
  • Move context ownership from active work to the raw stored event, then release it after translation; failed submissions and shutdown unwind their owners.

Scope

This PR does not change:

  • OffloadKey = (block_hash, group_idx);
  • prepare_store admission or batching;
  • worker transfer metadata;
  • the legacy expanded-removal contract; or
  • the public KV-event schema.

Single-key admission, k-way ordering, and #44865 remain separate work.

This PR is based on the KV-group block-granularity fix merged in #51614. Token-mutating paths affected by #49377 keep placeholder payloads until their hash and offload state are safe.

Duplicate-work check

No open PR above carries request provenance through async stored-event completion.

Tests

All validation ran on the workstation against 3d204dfdaa:

.venv/bin/python -m pytest \
  tests/v1/kv_connector/unit/offloading_connector/test_events.py -q
32 passed

.venv/bin/python -m pytest \
  tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py -q
122 passed

.venv/bin/python -m pytest \
  tests/v1/kv_offload/cpu/test_manager.py \
  tests/v1/kv_offload/tiering/test_async_lookup.py \
  tests/v1/kv_offload/tiering/test_fs_tier.py \
  tests/v1/kv_offload/tiering/test_obj_tier.py \
  tests/v1/kv_offload/tiering/test_tiering_offloading.py -q
182 passed

pre-commit run ruff-format --files \
  $(git diff --name-only origin/main...HEAD -- '*.py')
Passed

pre-commit run ruff-check --files \
  $(git diff --name-only origin/main...HEAD -- '*.py')
Passed

pre-commit run mypy-3.12 --hook-stage manual \
  --files $(git diff --name-only origin/main...HEAD -- vllm)
Passed

Model evaluation: not applicable. This changes event metadata and lifecycle only, not model execution or output.

AI assistance

AI assistance was used to trace the event lifecycle, implement the change, and draft tests. The human submitter must review every changed line and these test results before submission.

Co-authored-by: OpenAI Codex <noreply@openai.com>

Signed-off-by: Change72 <changg@nvidia.com>
@mergify mergify Bot added the kv-connector label Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant