[Bugfix][KV Offload] Propagate EAGLE mode to SimpleCPU coordinator - #49071
Conversation
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <codex@openai.com>
|
@claude review |
There was a problem hiding this comment.
LGTM. This correctly mirrors the existing GPU-side pattern (scheduler.py derives use_eagle from speculative_config.use_eagle()) for the SimpleCPU coordinator, fixing a real inconsistency where the CPU path was hard-coded to False.
What was reviewed: the one-line diff in vllm/v1/simple_kv_offload/manager.py, confirmed the derivation logic matches the equivalent GPU coordinator setup in vllm/v1/core/sched/scheduler.py, and checked that use_eagle flows correctly into get_kv_cache_coordinator with no other call sites needing updates.
Extended reasoning...
Overview
Single-file, single-purpose change to vllm/v1/simple_kv_offload/manager.py: replaces a hard-coded use_eagle=False with a value derived from vllm_config.speculative_config.use_eagle() when constructing the CPU-side KVCacheCoordinator for SimpleCPUOffloadConnector. Also removes a stale TODO comment and a blank line (cosmetic).
Security risks
None. This is an internal correctness fix affecting speculative-decoding cache semantics; it does not touch auth, networking, serialization of untrusted input, or any privilege boundary.
Level of scrutiny
Low-to-moderate is appropriate. The change is a one-line logic fix with no new code paths, no API changes, and no behavior change for non-EAGLE configurations (spec_config is None → use_eagle stays False, same as before). I verified the derivation mirrors the established GPU-side pattern in vllm/v1/core/sched/scheduler.py (self.use_eagle = speculative_config.use_eagle()), which gives confidence the fix is semantically correct and consistent with the rest of the codebase.
Other factors
The bug-hunting system found no issues. The PR includes unit test results (29 passed) and a GSM8K eval run for the EAGLE3 case, which is good practice for a change affecting model correctness. No outstanding reviewer comments to address. The scope is narrow and self-contained, matching the approval bar for simple, mechanical fixes.
…llm-project#49071) Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <codex@openai.com>
…llm-project#49071) Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: aarushjain29 <Aarushi.Jain2@amd.com>
…llm-project#49071) Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <codex@openai.com>
Purpose
Propagate EAGLE-family speculative decoding state to the CPU KV-cache coordinator used by
SimpleCPUOffloadConnector.The GPU-side coordinator derives
use_eaglefromVllmConfig.speculative_config, but the SimpleCPU coordinator hard-coded it toFalse. As a result, CPU prefix-cache lookup did not apply EAGLE's final-block handling and could reuse the volatile trailing block instead of recomputing the hidden state required by the drafter.This change derives the flag from
speculative_config.use_eagle()and passes it toget_kv_cache_coordinator, keeping CPU and GPU cache semantics aligned for EAGLE, EAGLE3, MTP, DFlash, and DSpark.Duplicate-work check
No linked issue was provided. Searches for
simple_kv_offload EAGLE,simple CPU offload EAGLE,offload eagle, andSimpleCPUOffloadSchedulerfound no open PR implementing this hard-coded-flag fix. The nearest open work (#47891, #48911, and #49025) concerns the native OffloadingConnector or different EAGLE/offload failure modes.Test Plan
SimpleCPUOffloadSchedulermarks its CPU cache manager withuse_eagle=True.SimpleCPUOffloadConnector, reset the GPU prefix cache after draining eager stores, and rerun to exercise CPU-backed reuse.Test Result
.venv/bin/python -m pytest tests/v1/simple_kv_offload/test_scheduler.py -q: 29 passed..venv/bin/pre-commit run --files vllm/v1/simple_kv_offload/manager.py: passed.AI assistance
OpenAI Codex was used to review, rebase, validate, and prepare this change. The human submitter must review every changed line and be prepared to explain and defend the change end-to-end.
Essential Elements of an Effective PR Description Checklist