Skip to content

[Bugfix][KV Offload] Propagate EAGLE mode to SimpleCPU coordinator - #49071

Merged
Isotr0py merged 2 commits into
vllm-project:mainfrom
ivanium:fix/simple-offload-eagle
Jul 20, 2026
Merged

[Bugfix][KV Offload] Propagate EAGLE mode to SimpleCPU coordinator#49071
Isotr0py merged 2 commits into
vllm-project:mainfrom
ivanium:fix/simple-offload-eagle

Conversation

@ivanium

@ivanium ivanium commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Propagate EAGLE-family speculative decoding state to the CPU KV-cache coordinator used by SimpleCPUOffloadConnector.

The GPU-side coordinator derives use_eagle from VllmConfig.speculative_config, but the SimpleCPU coordinator hard-coded it to False. As a result, CPU prefix-cache lookup did not apply EAGLE's final-block handling and could reuse the volatile trailing block instead of recomputing the hidden state required by the drafter.

This change derives the flag from speculative_config.use_eagle() and passes it to get_kv_cache_coordinator, keeping CPU and GPU cache semantics aligned for EAGLE, EAGLE3, MTP, DFlash, and DSpark.

Duplicate-work check

No linked issue was provided. Searches for simple_kv_offload EAGLE, simple CPU offload EAGLE, offload eagle, and SimpleCPUOffloadScheduler found no open PR implementing this hard-coded-flag fix. The nearest open work (#47891, #48911, and #49025) concerns the native OffloadingConnector or different EAGLE/offload failure modes.

Test Plan

  • Run the focused SimpleCPU scheduler unit suite.
  • Run changed-file pre-commit hooks.
  • Directly verify that an EAGLE-configured SimpleCPUOffloadScheduler marks its CPU cache manager with use_eagle=True.
  • Run 200-question, 5-shot GSM8K with Llama 3.1 8B + EAGLE3 + SimpleCPUOffloadConnector, reset the GPU prefix cache after draining eager stores, and rerun to exercise CPU-backed reuse.

Test Result

  • .venv/bin/python -m pytest tests/v1/simple_kv_offload/test_scheduler.py -q: 29 passed.
  • .venv/bin/pre-commit run --files vllm/v1/simple_kv_offload/manager.py: passed.
  • Direct EAGLE propagation probe: passed.
  • GSM8K pass 1: 0.765 accuracy, 0.000 invalid rate (200 questions, 5-shot).
  • GSM8K pass 2 after successful GPU prefix-cache reset: 0.765 accuracy, 0.000 invalid rate.

AI assistance

OpenAI Codex was used to review, rebase, validate, and prepare this change. The human submitter must review every changed line and be prepared to explain and defend the change end-to-end.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR is explained.
  • The test plan and commands are provided.
  • Unit and model-evaluation results are reported.
  • No documentation update is needed for this internal correctness fix.

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
@mergify mergify Bot added v1 bug Something isn't working labels Jul 19, 2026
@ivanium
ivanium marked this pull request as ready for review July 19, 2026 04:20

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@ivanium
ivanium requested a review from NickLucche July 19, 2026 04:36
@ivanium

ivanium commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This correctly mirrors the existing GPU-side pattern (scheduler.py derives use_eagle from speculative_config.use_eagle()) for the SimpleCPU coordinator, fixing a real inconsistency where the CPU path was hard-coded to False.

What was reviewed: the one-line diff in vllm/v1/simple_kv_offload/manager.py, confirmed the derivation logic matches the equivalent GPU coordinator setup in vllm/v1/core/sched/scheduler.py, and checked that use_eagle flows correctly into get_kv_cache_coordinator with no other call sites needing updates.

Extended reasoning...

Overview

Single-file, single-purpose change to vllm/v1/simple_kv_offload/manager.py: replaces a hard-coded use_eagle=False with a value derived from vllm_config.speculative_config.use_eagle() when constructing the CPU-side KVCacheCoordinator for SimpleCPUOffloadConnector. Also removes a stale TODO comment and a blank line (cosmetic).

Security risks

None. This is an internal correctness fix affecting speculative-decoding cache semantics; it does not touch auth, networking, serialization of untrusted input, or any privilege boundary.

Level of scrutiny

Low-to-moderate is appropriate. The change is a one-line logic fix with no new code paths, no API changes, and no behavior change for non-EAGLE configurations (spec_config is None → use_eagle stays False, same as before). I verified the derivation mirrors the established GPU-side pattern in vllm/v1/core/sched/scheduler.py (self.use_eagle = speculative_config.use_eagle()), which gives confidence the fix is semantically correct and consistent with the rest of the codebase.

Other factors

The bug-hunting system found no issues. The PR includes unit test results (29 passed) and a GSM8K eval run for the EAGLE3 case, which is good practice for a change affecting model correctness. No outstanding reviewer comments to address. The scope is narrow and self-contained, matching the approval bar for simple, mechanical fixes.

@Isotr0py
Isotr0py enabled auto-merge (squash) July 20, 2026 04:21
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 20, 2026
@Isotr0py
Isotr0py merged commit 9bc266d into vllm-project:main Jul 20, 2026
83 checks passed
@ivanium
ivanium deleted the fix/simple-offload-eagle branch July 20, 2026 08:50
ArjunPakhan pushed a commit to ArjunPakhan/vllm that referenced this pull request Jul 21, 2026
…llm-project#49071)

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
aarushjain29 pushed a commit to ROCm/vllm that referenced this pull request Jul 21, 2026
…llm-project#49071)

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: aarushjain29 <Aarushi.Jain2@amd.com>
edwinlim0919 pushed a commit to chaeminlim-mb/vllm that referenced this pull request Jul 29, 2026
…llm-project#49071)

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants