eagle3 cb impl with top-1 proposal #2740

songbell · 2025-09-17T07:28:21Z

eagle3 CB impl
Tickets: CVS-173358
ref code: https://github.com/SafeAILab/EAGLE

Copilot

Pull Request Overview

This PR implements Eagle speculative decoding functionality for top-1 proposal generation. The implementation adds support for Eagle3 mode, which enables accelerated text generation through speculative decoding with hidden state sharing between main and draft models.

Key changes include:

Added Eagle decoding implementation with model transformation pipelines for hidden state extraction
Integrated safetensor parsing for Eagle3 configuration data (d2t mappings)
Extended continuous batching pipeline to support Eagle mode with hidden state management

Reviewed Changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tools/continuous_batching/accuracy/continuous_batching_eagle_decoding.cpp	New Eagle decoding accuracy test tool
src/cpp/src/speculative_decoding/speculative_decoding_impl.hpp	Added Eagle decoding class definitions and model transformation passes
src/cpp/src/speculative_decoding/speculative_decoding_impl.cpp	Core Eagle decoding implementation with model transformations
src/cpp/src/continuous_batching/pipeline.cpp	Integration of Eagle mode into pipeline construction
src/cpp/src/continuous_batching/model_runner.hpp	Added hidden state management functionality
samples/cpp/text_generation/eagle_speculative_lm.cpp	New Eagle speculative decoding sample
src/cpp/src/safe_tensor_wrapper.hpp	New safetensor parsing utilities

Comments suppressed due to low confidence (1)

src/cpp/src/continuous_batching/model_runner.hpp:1

This appears to be modifying the token index without bounds checking on the d2t array. Add bounds checking to prevent potential buffer overflow.

// Copyright (C) 2023-2025 Intel Corporation

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/cpp/src/speculative_decoding/speculative_decoding_impl.cpp

src/cpp/src/continuous_batching/model_runner.hpp

tools/continuous_batching/accuracy/CMakeLists.txt

src/cpp/src/speculative_decoding/continuous_batching_for_speculative_decoding_impl.cpp

src/cpp/src/sequence_group.cpp

src/cpp/src/continuous_batching/pipeline.cpp

Signed-off-by: fishbell <[email protected]>

samples/cpp/text_generation/greedy_causal_lm.cpp

tools/continuous_batching/accuracy/continuous_batching_speculative_decoding.cpp

tools/continuous_batching/benchmark/continuous_batching_benchmark.cpp

samples/cpp/text_generation/eagle_speculative_lm.cpp

Copilot

Pull Request Overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 10 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tools/llm_bench/task/text_generation.py

tools/llm_bench/llm_bench_utils/ov_utils.py

tools/continuous_batching/benchmark/continuous_batching_benchmark.cpp

tools/continuous_batching/accuracy/continuous_batching_speculative_decoding.cpp

src/cpp/src/speculative_decoding/speculative_decoding_impl.cpp

src/cpp/src/speculative_decoding/continuous_batching_for_speculative_decoding_impl.cpp

src/cpp/src/continuous_batching/pipeline.cpp

src/cpp/src/continuous_batching/model_runner.hpp

samples/cpp/text_generation/greedy_causal_lm.cpp

Copilot

Pull Request Overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/python_tests/test_continuous_batching.py

src/cpp/src/continuous_batching/model_runner.hpp

Copilot

Pull Request Overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/python_tests/utils/hugging_face.py

src/cpp/src/speculative_decoding/speculative_decoding_eagle3_impl.cpp

src/cpp/src/continuous_batching/pipeline.cpp

src/cpp/src/continuous_batching/model_runner.hpp

rkazants · 2025-11-12T07:28:44Z

Please do NOT merge until we will have more-less clear vision about solution in optimum-intel side. The model preparation can affect IRs and their structure so that the proposed solution in this PR will be incompatible.

Now we have no the full architectural proposal for eagle3 and we need to pass it for model preparation in arch review.

@peterchen-intel, did you have internal code-review here? Is the code clean enough?

Best regards,
Roman

peterchen-intel · 2025-11-12T08:34:57Z

Please do NOT merge until we will have more-less clear vision about solution in optimum-intel side. The model preparation can affect IRs and their structure so that the proposed solution in this PR will be incompatible.

Now we have no the full architectural proposal for eagle3 and we need to pass it for model preparation in arch review.

@peterchen-intel, did you have internal code-review here? Is the code clean enough?

Best regards, Roman

@rkazants 50 comments from all reviewers (including 12 from copilot) have been resolved, the code is clean now.

src/cpp/src/continuous_batching/pipeline.cpp

Copilot

Pull Request Overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/cpp/src/speculative_decoding/speculative_decoding_impl.cpp

src/cpp/src/speculative_decoding/speculative_decoding_eagle3_impl.cpp

src/cpp/src/continuous_batching/pipeline.cpp

src/cpp/src/continuous_batching/model_runner.hpp

src/cpp/src/speculative_decoding/speculative_decoding_eagle3_impl.cpp

tests/python_tests/test_continuous_batching.py

Copilot

Pull Request Overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/python_tests/utils/hugging_face.py

tests/python_tests/test_continuous_batching.py

src/cpp/src/speculative_decoding/speculative_decoding_impl.cpp

src/cpp/src/speculative_decoding/speculative_decoding_eagle3_impl.cpp

src/cpp/src/continuous_batching/pipeline.cpp

src/cpp/src/continuous_batching/model_runner.hpp

src/cpp/src/sampling/sampler.cpp

.github/workflows/windows.yml

…genai into bell/eagle_cb_impl

…vino.genai into bell/eagle_cb_impl

Signed-off-by: fishbell <[email protected]>

.github/workflows/linux.yml

src/cpp/src/continuous_batching/model_runner.hpp

src/cpp/src/continuous_batching/pipeline.cpp

src/cpp/src/speculative_decoding/speculative_decoding_eagle3_impl.cpp

MaximProshin · 2025-11-18T08:27:37Z

Please do NOT merge until we will have more-less clear vision about solution in optimum-intel side. The model preparation can affect IRs and their structure so that the proposed solution in this PR will be incompatible.

Now we have no the full architectural proposal for eagle3 and we need to pass it for model preparation in arch review.

@peterchen-intel, did you have internal code-review here? Is the code clean enough?

Best regards, Roman

This is not a blocking comment and will be addressed separately.

Copilot

Pull Request Overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/python_tests/test_continuous_batching.py

tests/python_tests/samples/test_speculative_decoding_lm.py

src/cpp/src/continuous_batching/pipeline.cpp

src/cpp/src/continuous_batching/model_runner.hpp

Copilot

Pull Request Overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/python_tests/samples/test_speculative_decoding_lm.py

src/cpp/src/speculative_decoding/speculative_decoding_eagle3_impl.cpp

src/cpp/src/continuous_batching/pipeline.cpp

src/cpp/src/continuous_batching/model_runner.hpp

Signed-off-by: xuchen-intel <[email protected]>

Co-authored-by: Copilot <[email protected]>

songbell · 2025-11-21T08:54:03Z

replace with #3055

…genai into bell/eagle_cb_impl

Signed-off-by: xuchen-intel <[email protected]>

Copilot AI review requested due to automatic review settings September 17, 2025 07:28

songbell changed the title ~~eagle impl with top-1 proposal~~ eagle3 cb impl with top-1 proposal Sep 17, 2025

Copilot AI reviewed Sep 17, 2025

View reviewed changes

songbell added 2 commits September 17, 2025 23:24

eagle impl with top-1 proposal

f56435a

Signed-off-by: fishbell <[email protected]>

enable cb benchmark for eagle3

1d74c03

Signed-off-by: fishbell <[email protected]>

github-actions bot added the category: llm_bench Label for tool/llm_bench folder label Sep 18, 2025

songbell added 7 commits September 18, 2025 18:02

add benchmarking, apply copilot review comments

0b09a65

Signed-off-by: fishbell <[email protected]>

fix case build failure

b36ecf7

Signed-off-by: fishbell <[email protected]>

fix SDL

d86e5a7

Signed-off-by: fishbell <[email protected]>

typo

f0aa2c7

Signed-off-by: fishbell <[email protected]>

opt hidden state transfer with ROI tensor

bbfa8ad

Signed-off-by: fishbell <[email protected]>

opt roi copy interface

35e6376

Signed-off-by: fishbell <[email protected]>

Merge branch 'master' into bell/eagle_cb_impl

f6a8872

peterchen-intel reviewed Sep 21, 2025

View reviewed changes

samples/cpp/text_generation/greedy_causal_lm.cpp Show resolved Hide resolved

tools/continuous_batching/accuracy/continuous_batching_speculative_decoding.cpp Show resolved Hide resolved

tools/continuous_batching/benchmark/continuous_batching_benchmark.cpp Outdated Show resolved Hide resolved

peterchen-intel requested review from wangleis and xufang-lisa September 21, 2025 02:42

peterchen-intel assigned Wovchena Sep 21, 2025

peterchen-intel mentioned this pull request Sep 21, 2025

[do not review]eagle cb pipeline impl #2540

Draft

Wovchena requested a review from Copilot September 22, 2025 06:19

xufang-lisa reviewed Sep 22, 2025

View reviewed changes

samples/cpp/text_generation/eagle_speculative_lm.cpp Outdated Show resolved Hide resolved

Copilot AI reviewed Sep 22, 2025

View reviewed changes

peterchen-intel requested a review from sunxiaoxia2022 November 11, 2025 02:40

Copilot AI reviewed Nov 11, 2025

View reviewed changes

tests/python_tests/test_continuous_batching.py Show resolved Hide resolved

tests/python_tests/test_continuous_batching.py Outdated Show resolved Hide resolved

src/cpp/src/continuous_batching/model_runner.hpp Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings November 12, 2025 04:17

Copilot AI reviewed Nov 12, 2025

View reviewed changes

rkazants added the do_not_merge label Nov 12, 2025

peterchen-intel approved these changes Nov 12, 2025

View reviewed changes

src/cpp/src/continuous_batching/pipeline.cpp Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings November 12, 2025 08:36

Copilot AI reviewed Nov 12, 2025

View reviewed changes

peterchen-intel requested a review from Copilot November 12, 2025 08:38

peterchen-intel removed this from the 2025.4 milestone Nov 12, 2025

Copilot AI reviewed Nov 12, 2025

View reviewed changes

songbell and others added 5 commits November 12, 2025 20:17

Merge branch 'master' of https://github.com/openvinotoolkit/openvino.…

b6abe7f

…genai into bell/eagle_cb_impl

Merge branch 'bell/eagle_cb_impl' of https://github.com/songbell/open…

ee263f9

…vino.genai into bell/eagle_cb_impl

move the def out of class

ad09392

Signed-off-by: fishbell <[email protected]>

With explicit commit id in the ref link.

a3e431f

formatging

b168f41

Signed-off-by: fishbell <[email protected]>

xipingyan reviewed Nov 13, 2025

View reviewed changes

Copilot AI review requested due to automatic review settings November 19, 2025 03:26

Copilot AI reviewed Nov 19, 2025

View reviewed changes

Copilot AI review requested due to automatic review settings November 19, 2025 03:30

Copilot AI reviewed Nov 19, 2025

View reviewed changes

xuchen-intel and others added 2 commits November 19, 2025 19:28

apply review comments

3af7823

Signed-off-by: xuchen-intel <[email protected]>

Update tests/python_tests/samples/test_speculative_decoding_lm.py

e413561

Co-authored-by: Copilot <[email protected]>

songbell closed this Nov 21, 2025

xuchen-intel added 2 commits November 21, 2025 17:58

Merge branch 'master' of https://github.com/openvinotoolkit/openvino.…

1d963cf

…genai into bell/eagle_cb_impl

use new logger

bd28b9b

Signed-off-by: xuchen-intel <[email protected]>

eagle3 cb impl with top-1 proposal #2740

eagle3 cb impl with top-1 proposal #2740

Uh oh!

Conversation

songbell commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rkazants commented Nov 12, 2025

Uh oh!

peterchen-intel commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

songbell commented Sep 17, 2025 •

edited

Loading

peterchen-intel commented Nov 12, 2025 •

edited

Loading