diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 7233ddd3e..abd014ed9 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -677,5 +677,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1791](https://github.com/mudler/vllm.cpp/issues/1791) | `SPEC-DSPARK` | **`scripts/dspark-paired-e2e.sh`'s `settle()` can never break early, so a wait for the GPU to drain always spends its full 360 s however fast the box actually drains.** The same idiom as [#1734](https://github.com/mudler/vllm.cpp/issues/1734), found by sweeping `scripts/` for it: `grep -c .` with an `\|\| echo 0` fallback makes `$n` the two-line string `0\n0`, so `[ "$n" -eq 0 ] && break` answers `integer expression expected` and returns 2 instead of deciding. The BUSY half of the guard works -- a positive count exits 0 and the fallback does not fire -- so only the FREE half is dead, and the failure is in the safe direction, which is why it was paid in silence. This is the FIFTH diagnosis of the idiom in this tree: `scripts/cpu-x86-llamacpp-floor.sh` already carries the removal and the reason in a comment, and a comment in one file is not reachable from another. FIXED IN FLOW with #1734: `\|\| true` keeps grep's own `0` and swallows only its status. The recurrence gate is `TheIdiomIsGoneFromEveryShellScript` in `tests/scripts/test_ltx25_ab_memwatch.py`, which sweeps every `scripts/*.sh` for a counting `grep`/`pgrep` paired with an `\|\| echo` fallback outside a comment; run against `27d8bfa70` it names all three live instances, this one included. It is a TRIPWIRE and says so: it reads text, and a `wc -l` with the same fallback walks past it | bug | | [#1796](https://github.com/mudler/vllm.cpp/issues/1796) | `SPEC-DFLASH2` | **[#1456](https://github.com/mudler/vllm.cpp/issues/1456)'s conclusion is retracted AT THE ARTIFACT, off-GPU, and the DFlash2 speed ratio's denominator inherits the retraction: `0.8016987337853048` MAY BE FLATTERING US.** #1456 measured a real source build at `CUDA_ARCHS=12.0` emitting `sm_80` for `_vllm_fa2_C` and `sm_75` for `_vllm_fa3_C`, and concluded the GB10 oracle has no `FLASH_ATTN` denominator. **The arch measurement stands; the conclusion does not, and no lease was needed to show it.** Both staged oracle wheels were opened with `zipfile` and walked as fatbinaries on the CPU dev box: `0.1.dev1+g66e5414c6` (this row's oracle, `/mnt/nas_share/rc/oracle-dflash2/`) and `0.1.dev1+g555967922` (the parity pin, `/mnt/nas_share/rc/oracle-vllm/`) each carry 76 FA2 fatbinaries and 192 FA3 fatbinaries, and **every one holds a PTX image beside its SASS image** -- arch 80 for FA2, arch 75 for FA3. The first FA2 PTX payload is zstd and decompresses to `.version 9.0` / `.target sm_80` for `flash_fwd_hdim128_bf16_causal_sm80`. That is the `+PTX` half of `FA2_ARCHS "8.0+PTX"` and it is the mechanism by which the module CAN reach sm_121 -- the artifact establishes a necessary condition, forward-JITtable code being shipped, and that the JIT then ran is an inference from that plus a run selecting `FLASH_ATTN` and generating; `cudaErrorUnsupportedPtxVersion` is the OPPOSITE failure, PTX ISA newer than the driver, which `.version 9.0` under driver 580.173.02 is not. vLLM says it directly too: `FlashAttentionBackend.supports_compute_capability` returns `capability >= DeviceCapability(8, 0)` (`vllm/v1/attention/backends/flash_attn.py:251-252` in the staged wheel). **THE DOWNSTREAM IS THE THING TO CHASE.** #1456's body records that the DFlash2 speed gate's denominator "will be vLLM pinned to `TRITON_ATTN`, by developer decision on 2026-08-20", explicitly because of that conclusion. AGENTS.md requires vLLM's PRODUCTION configuration as the denominator, and on this box vLLM's own auto-selection picks `FLASH_ATTN`: the 0.8017 gate run's log carries BOTH paths in one process, the forced branch taking `TRITON_ATTN` for the 27B target (`cuda.py:426`) and the auto branch taking `FLASH_ATTN` out of four valid backends (`cuda.py:486`), in `/mnt/nas_share/rc/dflash2-1673/out-n1673b/m-gate.log`. **So if `TRITON_ATTN` is the slower backend -- which vLLM's own priority ordering IMPLIES rather than states, and which nothing here measures -- then 16.279 tok/s is too LOW and the ratio is too HIGH. An error in our own favour is the one nobody chases, so it is recorded with its sign.** The ratio is NOT withdrawn and no denominator is substituted, because a wave must not replace a denominator the developer declared. **The five FA layers of [#1685](https://github.com/mudler/vllm.cpp/issues/1685) fall out of the same read**: `vllm/v1/worker/gpu/spec_decode/dflash/utils.py:31-46` sets the draft's `backend=speculative_config.attention_backend` UNCONDITIONALLY, so a harness that sets only the engine backend leaves the draft at `None` and the draft re-runs auto-selection; the siblings `dspark/utils.py:24-28` and `gemma4/speculator.py:66-89` both carry the target's backend through, and dspark's comment names this hazard by name. NOT reconciled in flow, and the reason is precise: what remains is ONE lease running vLLM against itself on this workload with `attention_backend=FLASH_ATTN` against `TRITON_ATTN`, each read back off the built engine, and **no GPU lease is authorised for this session**, so it is PENDING. Recommended collapse, recommended and not taken: #1456, [#1679](https://github.com/mudler/vllm.cpp/issues/1679) and #1685 observe ONE mechanism on two wheels, and #1685 is the one attached to a live number and should survive. Owed under `## Owed` O33 of [the DFlash2 spec](specs/dflash2-spec-decode.md), with caveat 5 beside the ratio in the same file | verification | | [#1794](https://github.com/mudler/vllm.cpp/issues/1794) | `LTX25-DIT-ATTN-ARM-PARSE` | **#1551 renamed the values of `VLLM_LTX2_DIT_FLASH_ATTN` and the two #1549-era LTX-2.5 A/B harnesses kept exporting the old ones, so three committed arms name a rung they do not select -- and one of the three says nothing.** At [#1549](https://github.com/mudler/vllm.cpp/issues/1549) the knob was BINARY: `=0` selected `vt::Attention` and every other value, unset included, selected `vt::AttentionDenseFlash`. [#1551](https://github.com/mudler/vllm.cpp/issues/1551) made it THREE-WAY, moved the unset default up a rung to `vt::AttentionDenseFa2` and gave the flash rung the exact spelling `flash`; it touched neither harness. Read at `27d8bfa70`: `scripts/ltx25-dit-attn-flash-pixel-ab.sh` exports `=1` for both its `flash` and its `flash-ctl` arms, and `scripts/ltx25-dit-attn-flash-ab.sh` leaves the variable UNSET for its `flash` arm -- all three of which selected `AttentionDenseFlash` before #1551 and select `AttentionDenseFa2` after it. `scripts/ltx25-dit-attn-fa2-hd128-ab.sh` is CORRECT and is the contrast that makes the other two legible: written after the rename, it exports `flash` / `""` / `0` and carries `assert_arm_op`. THE ASYMMETRY IS THE ISSUE: the pixel harness's `arm_report` counts `op=18` and `op=21` in the arm's own log and exits 46 on `ROUTING_BAD`, so its two `=1` arms abort -- after the render, an hour of a four-hour lease spent to learn a literal went stale -- while `ltx25-dit-attn-flash-ab.sh` has NO such assertion at all: its phase `[F]` PRINTS the op-provider selections and asserts nothing, so its `flash` arm renders FA-2, is reduced under the label `flash`, and the ratio it publishes is FA-2 against naive while the file's own header says flash against naive. THE RECORDED NUMBERS ARE NOT INVALIDATED: `.agents/specs/ltx25-dit-attn-flash.md` section 10 records `knob=1` announcing `op=21` and `ROUTING_OK=flash`, so the binary that ran it is one where `=1` still selected flash; what is wrong is the file as it stands against the binary `main` builds today. WHY A GATE AND NOT A REVIEWER: both harnesses were TRUE when written and both became false in a commit that touched neither file, because nothing connected the literal in a shell script to the literal in the C++ dispatch. FIXED IN FLOW with the red-first case that would have caught it -- `tests/scripts/test_ltx2_dit_attn_knob_arms.py` reads the accepted set out of `ltx2_device.cpp`'s own `std::strcmp(arm, "...")` calls rather than restating it, reads each harness's arm invocations, and asserts BOTH that every value is one the dispatch parses AND that every arm selects the rung its LABEL claims; the second half is the one that matters, because `unset` is a perfectly valid value and was still the wrong arm for a whole row. Registered on the preflight and CI record lanes, and it asserts both registrations itself. Found while fixing [#1751](https://github.com/mudler/vllm.cpp/issues/1751), which is the same defect one level down. Spec [`ltx25-dit-attn-arm-parse.md`](specs/ltx25-dit-attn-arm-parse.md) | bug | +| [#1824](https://github.com/mudler/vllm.cpp/issues/1824) | `SPEC-DFLASH2` | **A configured speculator forces synchronous scheduling; upstream keeps async ON for the Eagle-type family (dflash included).** `model_loader.cpp` resolves `async_scheduling_enabled_` to false whenever `resolved_spec_config_` is present, a deferral recorded at SPEC-MTP I5d ([mtp-spec-decode.md](specs/mtp-spec-decode.md) §2.8's not-ported list). Upstream's polarity at the pin (`vllm/config/vllm.py:1064-1112`) disables async only for a method OUTSIDE `EagleModelTypes ∪ NgramGPUTypes ∪ {"dspark"}` — and `"dflash"` (which DFlash2 rides) and every MTP type are Eagle-type. At c1 spec decode (~360 steps / 2048 tokens) every host-side scheduling cost is serialized into each step, the largest named host-side divergence in the [#1574](https://github.com/mudler/vllm.cpp/issues/1574) gap. W7 under `SPEC-DFLASH2` ports the draft-in-output flow (AsyncScheduler `-1` placeholders, worker-side fill, `update_draft_token_ids_in_output`, the `async_tokens_to_discard` rollback guard) and flips the enable to upstream's method predicate; the GPU TPOT A/B stays owed to the operator. Spec [`spec-decode-async-scheduling.md`](specs/spec-decode-async-scheduling.md) | feature | | [#1754](https://github.com/mudler/vllm.cpp/issues/1754) | `GATE-CI-SITE-HUGO-LANE` | **`agent-record` is RED on `main` and on every pull request cut from it: `test_rendered_benchmark_index_links_resolve_to_emitted_pages` shells out to `hugo` and no job in `ci.yml` installs it, so `subprocess.run` raises `FileNotFoundError` before any assertion runs and `unittest` reports an ERROR.** Introduced by [#1714](https://github.com/mudler/vllm.cpp/pull/1714) at `1db7e59cf`; the scheduled baseline at `deb53c6a3` was green on this job and `08c81a892` and every scheduled run after it is red. FIXED IN FLOW by installing the renderer on the lane -- the same `peaceiris/actions-hugo@v3` action, the same `extended: true`, and the same `0.146.3` pin the `gh-pages` job publishes on -- plus `hugo version` as the step's first line and `tests/scripts/test_ci_site_lane.py` holding the two pins equal and the setup step present and ordered before the render. A SKIP GUARD WAS REJECTED AS THE FIX and is deliberately not in this change: it retires the red by arranging for the assertion to run nowhere, which [PR #1726](https://github.com/mudler/vllm.cpp/pull/1726) demonstrates -- its `agent-record` is SUCCESS having rendered no site. The guard is still wanted for a developer box without Hugo and is separately owned by [#1722](https://github.com/mudler/vllm.cpp/issues/1722) / #1726, which edits a disjoint file, so the two compose. [#1764](https://github.com/mudler/vllm.cpp/issues/1764) names this red as one of its three and is not closed by this change | bug | | [#1828](https://github.com/mudler/vllm.cpp/issues/1828) | `GATE-CI-SITE-HUGO-LANE` | **`tests/scripts/test_check_site.py` asserts a literal `10` rendered benchmark detail links, which is a stored count of `docs/benchmarks/*.md` inside another file, and #1754's lane fix is what ARMS it**: until Hugo was installed the case never reached that line in CI. Measured on `d60692c89` with Hugo 0.146.3+extended: 10 slugs, 10 table hrefs, no duplicated target and no unlinked slug, so the relationship is a bijection and the literal is derivable at read time -- with its own non-vacuity floor, since `sorted(x) == sorted(y)` is satisfied by two empty sets. NOT FIXED IN FLOW: deriving it changes what the case asserts rather than how it spells a number, which `AGENTS.md` routes through the normal row, spec and fresh-review path, and the file already has an open pull request against the same case. Listed under `## Owed` in [gate-ci-site-hugo-lane.md](specs/gate-ci-site-hugo-lane.md) | bug | diff --git a/.agents/specs/dflash2-spec-decode.md b/.agents/specs/dflash2-spec-decode.md index 4e1de151b..ec07b3d27 100644 --- a/.agents/specs/dflash2-spec-decode.md +++ b/.agents/specs/dflash2-spec-decode.md @@ -595,6 +595,17 @@ reviewer who mutates the guarantee rather than reading it. - **W5 — the GGUF drafter arm**, with its lower bound. LANDED 2026-08-20. - **W6 — the gates.** G2 and G3 on a leased GPU against the PR-head oracle, then `## Outcome`. +- **W7 — async scheduling for the Eagle-type speculative family + ([#1824](https://github.com/mudler/vllm.cpp/issues/1824)). LANDED + 2026-08-23.** The engine forced synchronous scheduling under ANY speculator + (a SPEC-MTP I5d deferral); upstream keeps async ON for Eagle-type methods, + dflash included, and at c1 that difference serializes every host-side + scheduling cost into each of ~360 steps — the largest named host-side + divergence in the #1574 gap. W7 ports the draft-in-output flow and flips the + enable to upstream's method predicate. Own spec: + [spec-decode-async-scheduling.md](spec-decode-async-scheduling.md); the c1 + TPOT A/B (async-ON vs `VT_ASYNC_SCHED=0`, same binary, #1574 workload) is + owed there as A1, operator-run under an `rc` lease. **The gate head is reconciled to ONE head here and it is `66e5414c`, which is NOT what G2's rule selects.** W6 wrote that vllm#52816 was "still OPEN on diff --git a/.agents/specs/mtp-spec-decode.md b/.agents/specs/mtp-spec-decode.md index b6dee7937..7db3f68bf 100644 --- a/.agents/specs/mtp-spec-decode.md +++ b/.agents/specs/mtp-spec-decode.md @@ -344,6 +344,16 @@ lookup, the async draft-in-output path (`update_draft_token_ids_in_output`, `scheduler.py:1959`), and the structured-output grammar validation of proposed drafts. +**2026-08-23 update:** the async draft-in-output path — and with it +`num_spec_tokens_to_schedule` / `num_invalid_spec_tokens` and the +AsyncScheduler's `-1` placeholder assignment — LANDED as `SPEC-DFLASH2` W7 +([spec-decode-async-scheduling.md](spec-decode-async-scheduling.md), +[#1824](https://github.com/mudler/vllm.cpp/issues/1824)); the "configured +speculator forces sync scheduling" line this section justified is retired +for the Eagle-type family. The sync scheduler's first-decode-step padding +(`pad_spec_decode`), the dynamic-SD lookup, and the grammar `validate_tokens` +arm stay deferred. + ## 3. GDN linear-state rollback — the mechanism (B5's hard problem, answered) Linear-attention state is not a paged KV you can truncate. Upstream solves diff --git a/.agents/specs/spec-decode-async-scheduling.md b/.agents/specs/spec-decode-async-scheduling.md new file mode 100644 index 000000000..94f0bd6f2 --- /dev/null +++ b/.agents/specs/spec-decode-async-scheduling.md @@ -0,0 +1,328 @@ +# Spec: async scheduling for the Eagle-type speculative family (SPEC-DFLASH2 W7) + +Issue: [#1824](https://github.com/mudler/vllm.cpp/issues/1824). Owning row: +`SPEC-DFLASH2` (engine-matrix.md:177, `ACTIVE`). The deferral this wave +discharges was RECORDED under `SPEC-MTP` +([mtp-spec-decode.md](mtp-spec-decode.md) §2.8 "STILL DEFERRED: ... the async +draft-in-output path (`update_draft_token_ids_in_output`, scheduler.py:1959)"), +but `SPEC-MTP` closed `DONE` on 2026-07-26 with that deferral on the record, and +the row that OWES the result this wave unblocks is `SPEC-DFLASH2`: its W6 speed +gate ([#1574](https://github.com/mudler/vllm.cpp/issues/1574)) names the forced +synchronous scheduling as the largest host-side divergence at c1 spec decode +(~360 steps for 2048 tokens, every host-side scheduling cost serialized into +each step, while our acceptance per step is BETTER than both competitors'). +Reopening a `DONE` row to carry new engine work would be a lifecycle edit that +buys nothing; the wave therefore lands as `SPEC-DFLASH2` W7 and this paragraph +is the required "say why". + +## Scope + +Mirror upstream's polarity: async scheduling stays ON when the configured +speculative method is one upstream keeps async for, and the engine runs the +draft-in-output flow — the scheduler schedules placeholder (`-1`) draft tokens +for step N+1 before step N's output is processed, and the WORKER substitutes +the real draft values it already holds. Concretely: + +1. **Enable flip** (`src/vllm/entrypoints/model_loader.cpp::LoadedEngine`, + the `async_scheduling_enabled_` initializer): drop the blanket + `!resolved_spec_config_.has_value() &&` and refuse async only for the + methods upstream refuses. +2. **AsyncScheduler spec placeholders** + (`src/vllm/v1/core/sched/async_scheduler.cpp::update_after_schedule`): + assign `request.spec_token_ids = [-1] * num_spec_tokens_to_schedule` after + each schedule, mirroring `async_scheduler.py:24-25,43-45` — the half that + was landed as a COUNT only (the header's own DEFERRED list names it). +3. **`Scheduler::update_draft_token_ids_in_output`** + (`scheduler.py:2072-2107` at the pin): trim the worker's drafts to the + scheduled count, grammar-validate when the request advances a grammar, pad + the removed tail with `-1`, rewrite `scheduler_output. + scheduled_spec_decode_tokens`, and record + `scheduler_output.num_invalid_spec_tokens`. +4. **Engine-core draft polarity** (`src/vllm/v1/engine/core.cpp`): `post_step` + pulls drafts only when async scheduling is OFF (`core.py:617`); the + deferred-grammar branch of `step_with_batch_queue` pulls them and calls + `update_draft_token_ids_in_output` (`core.py:718-731`). `post_step` reads the mode off a new + `Scheduler::async_scheduling()` virtual (upstream's `self.async_scheduling` + flag, `core.py:232`) — the resolution product IS the scheduler class, and + `EngineCore` gains no flag. +5. **Worker fill + computed-token correction** + (`src/vllm/v1/worker/gpu/runner.cpp::execute_model`): when async scheduling + is on and a speculator is configured, replace the scheduler's `-1` + placeholders with the runner's own last proposed drafts before the splice, + and correct the scheduler's optimistic `num_computed_tokens` for the + previous step's rejections (§Design D3/D4). The runner learns the resolved + mode through a `set_async_scheduling` setter called by `LoadedEngine`. +6. **Async sampler routing** (`runner.cpp::sample_tokens_async`): with a + speculator configured, degenerate to the synchronous + `sample_tokens` wrapped in a `ReadyModelRunnerOutput` — the rejection + sampler and the propose loop are host-synchronous today, and the + device-resident spec sampler is recorded owed (§Owed A2), not faked. +7. **Preemption hygiene** (`scheduler.cpp::preempt_request`): clear + `spec_token_ids` on preemption, mirroring `scheduler.py:1217-1218` — today's + sync path carries the same latent staleness; the placeholder path makes it + live. +8. **Rollback guard** (`scheduler.cpp::update_from_output`): skip the + spec-rollback block while `request.async_tokens_to_discard > 0` + (`scheduler.py:1670-1675`), so a stale in-flight frame cannot underflow the + counters. + +OUT of scope, named so the next reader can tell staging from silence: + +- The sync scheduler's first-decode-step padding (`scheduler.py:827-843,1022`, + `pad_spec_decode`) — a cudagraph-uniformity optimization, not a correctness + requirement; stays deferred exactly as I2 left it. +- The dynamic-SD lookup (`scheduler.py:1123-1125`): `num_spec_tokens_to_schedule` + is the flat `num_spec_tokens` here. +- The device-resident async spec sampler (upstream's optimistic + `prev_num_draft_len` + GPU correction + on-device draft scatter, + `gpu_model_runner.py:1356-1396,1795-1907`): §Owed A2. +- `disable_padded_drafter_batch`: we do not carry the field; upstream's + default (padded ON) is the only arm we implement, which is also the only arm + compatible with async. + +## Upstream anchors + +Two revisions, both readable in `/home/mudler/_git/vllm`: + +- The parity pin `5559679229bc961848b121ccdeaa8fa5d79bec98` (checked out; all + bare `file:line` below are at the pin). +- The DFlash2 merge `b389ac29465b33f9e9c534df221ea3c129e9793f` (fetched object, + read with `git show`); NOT an ancestor of the pin and the pin is not an + ancestor of it — they are parallel heads and the pin governs wherever both + define the behavior. + +| What | Anchor | +|---|---| +| Enable polarity (default-ON unless the method is refused) | `vllm/config/vllm.py:1064-1112` at the pin: refused iff `method not in EagleModelTypes ∪ NgramGPUTypes ∪ {"dspark"}`; also refuses pooling models and `disable_padded_drafter_batch`. At `b389ac2946` (`vllm/config/vllm.py:1275-1304`) the same list gains `"draft_model"` — the pin governs, so `draft_model` stays sync here, noted in the code | +| The method families | `vllm/config/speculative.py:60-65` at the pin: `EagleModelTypes = eagle, eagle3, extract_hidden_states, MTPModelTypes ("mtp", "qwen3_5_mtp", …), DFlashModelTypes ("dflash")`. DFlash2 has NO method literal of its own at either head — the DFlash2 draft rides method `"dflash"` (`speculative.py:952-956` at `b389ac2946`), which matches OUR classification (`include/vllm/config/speculative.h` — DFlash2 is a draft-architecture distinction under `method == "dflash"`) | +| Placeholder assignment | `vllm/v1/core/sched/async_scheduler.py:14-17,20-25,36-45` (`_spec_token_placeholders`, `num_output_placeholders += num_sampled + cur_num_spec`, `request.spec_token_ids = placeholders`, "We will update the actual spec token ids in the worker process") | +| `num_spec_tokens_to_schedule` | `vllm/v1/core/sched/output.py:244,263`; set at `scheduler.py:1123-1156` | +| Scheduling placeholders (already landed at I2, re-verified live) | `scheduler.py:472-491` (max-tokens early-continue), `:504-507` (`num_tokens_with_spec + num_output_placeholders − num_computed`), `:623-639` (`num_scheduled_spec_tokens` subtracts `num_output_placeholders`) — our `scheduler.cpp:483-506,599-626` | +| Rejection rollback under async | `scheduler.py:1666-1698`: `num_computed_tokens -= num_rejected`, `num_output_placeholders -= num_rejected`, the whole block gated on `request.async_tokens_to_discard == 0` — ours has the first two (`scheduler.cpp:952-976`), lacks the discard guard | +| `update_draft_token_ids_in_output` | `scheduler.py:2072-2107` | +| Engine-core wiring | `core.py:168-170` (`check_for_draft_tokens`), `:232` (`self.async_scheduling`), `:617-620` (post_step guard), `:718-731` (deferred-grammar draft update) | +| Worker substitution (the part we adapt, §Design D3) | `gpu_model_runner.py:1795-1907` (`_prepare_input_ids` draft scatter from the device tensor), `:1356-1396` (optimistic `prev_num_draft_len` correction), `gpu_input_batch.py:505-528` ("token_ids_cpu assigned from spec_token_ids are placeholders and will be overwritten") | +| Preempt clears drafts | `scheduler.py:1203-1225` (`:1217-1218`) | +| Upstream tests | `tests/v1/core/test_async_scheduler.py:332` (`test_no_placeholder_underflow_on_discarded_spec_frame`); `tests/v1/e2e/general/test_async_scheduling.py` (`check_outputs_equal` across async ON/OFF with spec configs — the token-identity contract, GPU-model there, CPU-model here) | + +## Design + +**D1 — the enable condition mirrors the pin's method families.** A new +`SpeculativeConfig::async_scheduling_compatible()` +(`include/vllm/config/speculative.h`) answers the pin's +`vllm.py:1076-1087` predicate over the methods THIS engine implements: +`"mtp"`, `"dflash"`, `"dspark"` (and the `"eagle"`/`"eagle3"` strings the +config type admits) are compatible; `"ngram"` (host n-gram — upstream's async +lane is `ngram_gpu`, a different implementation we do not have) and +`"draft_model"` (refused at the pin; allowed only at `b389ac2946`, which the +pin outranks) are not. `model_loader.cpp` becomes +`async_scheduling_enabled_((!resolved_spec_config_.has_value() || +resolved_spec_config_->async_scheduling_compatible()) && ResolveAsyncEnabled(...))`. +`MakeScheduler` forwards `speculative_config` into `AsyncScheduler` (today the +async arm drops it, which was correct only while async forced no-spec). + +**D2 — the scheduler side is upstream's, verbatim.** Placeholder assignment +in `AsyncScheduler::update_after_schedule` (the count half already landed at +I2 and is live; this adds only the `spec_token_ids` assignment, guarded on +`num_spec_tokens_to_schedule > 0` so the no-speculator path stays +byte-identical); `SchedulerOutput::num_spec_tokens_to_schedule` and +`num_invalid_spec_tokens`; `update_draft_token_ids_in_output`; the +`async_tokens_to_discard` rollback guard; the preempt-time `spec_token_ids` +clear. + +**D3 — the worker fill is host-side, and that is an adaptation with a named +reason.** Upstream's worker keeps the drafts as a DEVICE tensor and scatters +them into `input_ids` on device, because its propose runs without a host sync. +OUR propose (`runner.cpp::propose_drafts` and its dflash/dspark/ngram +branches) materializes the drafts on the HOST every step +(`pending_drafts_`, a `DraftTokenIds`), synchronously with sampling. The +host-resident drafts make the faithful analog of "update the actual spec token +ids in the worker process" a host-side substitution: in +`GPUModelRunner::execute_model`, under `use_async_scheduling_ && spec_on()`, +build a patched copy of `scheduled_spec_decode_tokens` whose values come from +the runner's own last `pending_drafts_` (trimmed to the scheduled count; +looking THROUGH the stash without consuming it, so `take_draft_token_ids` +still serves the engine's deferred-grammar pull), and hand THAT to the +`update_req_spec_token_ids` splice. Placeholders never reach the embedding. +A placeholder row whose drafts are missing from the stash is a refusal +(`VT_CHECK` naming the request), not a silent `-1` embed: placeholders are +only ever assigned to requests that were scheduled and sampled the previous +step (async_scheduler.py:36-45 iterates the scheduled set; preemption clears +them per D2), so a miss is a bug and must say so. + +**D4 — the computed-token correction is exact where upstream's is +optimistic, same adaptation reason.** Under async the scheduler's +`CachedRequestData.num_computed_tokens` for a request whose PREVIOUS step +scheduled drafts may still include that step's rejected drafts (the rollback +runs in `update_from_output`, which under the batch queue happens after the +next `schedule()`). Upstream cannot know the true value host-side without +killing its overlap, so it assumes all-accepted and corrects on device +(`gpu_model_runner.py:1356-1396`). Our runner DID the rejection on the host +last step, so the true value is derivable structurally: +for a cached request present in the persistent batch whose previous step +scheduled drafts (recorded per-slot at splice time), the corrected +`num_computed_tokens_cpu` is `num_tokens_no_spec − 1` — the position of the +newest committed token, which equals the scheduler's value when the rollback +already ran (depth-1 `LLMEngine::step` order) and subtracts exactly +`num_rejected` when it has not (depth-2 batch-queue order). Both engine +orders are live in production (`LLMEngine::step` for vllm-bench/C-ABI-sync, +`AsyncLLM`'s `step_with_batch_queue` for the server), which is why the +correction must be structural rather than "subtract last step's rejections": +a temporal rule is wrong in whichever order it was not written for. +The relation `sent − corrected ∈ {0, prev_num_rejected}` is asserted. + +**D5 — spec sampling under async stays the synchronous path.** In +`sample_tokens_async`, `spec_on()` routes to +`ReadyModelRunnerOutput(sample_tokens(grammar_output))`. This keeps rejection, +write-backs, telemetry, propose, and the device-mirror upload on the one +proven path — token-exactness with the sync scheduler is then by construction +(identical per-step inputs and identical sampling code; only scheduler +bookkeeping timing differs). The costs, stated: the sampled-id D2H overlap +(the ~3.25 ms/step capture) is not realized on spec steps, and the +`combine_sampled_and_draft_tokens` splice must be SKIPPED under +`spec_on()` in `execute_model` — the host arrays are fresh (the sync sampler +wrote them), and the combine's `num_new_sampled_tokens == 1` arithmetic would +overwrite the LAST DRAFT position of a `1+k` row with `last_sampled` +(`prepare_inputs.cpp:285-333` computes `logits_start = query_end − 1`), which +is exactly the async-input-combine corruption I5e RCA'd. Extending the combine +kernel to draft-aware `num_logits = 1 + k` is part of §Owed A2. + +**D6 — what async buys with D5 in place, honestly.** The scheduler-side win +the issue names — schedule N+1 and issue its forward before step N's host-side +`update_from_output` — is real in the depth-2 loop; the GPU-idle capture on +spec steps is NOT realized until A2 lands (the propose loop's own host sync +bounds it today anyway). The CPU tier gates the CONTRACT (token identity, +placeholder arithmetic, reachability); the TPOT delta is the operator's owed +GPU A/B (§Gates G3) and no number is claimed here. + +## Risks + +- **R1 — placeholder/rollback double-count.** The same `num_rejected` rewinds + `num_computed_tokens` and `num_output_placeholders`; an error leaves the + budget formula `num_tokens_with_spec + placeholders − computed` drifting and + either stalls the request or schedules past `max_tokens`. Gated by the + scheduler tests (multi-step arithmetic checked to steady state) and by the + mutation set (§Mutations M2/M3). +- **R2 — the fill substituting the wrong step's drafts.** The stash is + overwritten every sample; a stale row would verify yesterday's drafts and + silently deflate acceptance while staying token-correct (accept-iff-equal + masks it). The engine-level stub test pins the VALUES the worker receives, + not just the counts. +- **R3 — depth-1 vs depth-2 ordering.** D4's structural correction is the + defense; the LoadedEngine identity gate runs BOTH entries (`engine()` and + `async_engine()`). +- **R4 — GDN spec state under async.** The GDN rollback keys off + `num_accepted_tokens` and the spec slot remap, all worker-side and exact; + the identity gate runs on the GDN-hybrid synthetic model, so a divergence + reds as a token mismatch. + +## Tests + +All CPU, red-first, focused suites named per case: + +1. `tests/vllm/v1/test_async_scheduler.cpp` — placeholders: after a spec + schedule, `request.spec_token_ids == [-1]*k` and + `SchedulerOutput.num_spec_tokens_to_schedule == k`; next schedule verifies + the placeholders (`1+k` tokens, placeholder values in + `scheduled_spec_decode_tokens`) and the multi-step arithmetic reaches + steady state under partial acceptance; the ported + `test_no_placeholder_underflow_on_discarded_spec_frame`. +2. `tests/vllm/v1/test_scheduler.cpp` — `update_draft_token_ids_in_output`: + trims to the scheduled count, leaves shorter rows alone, skips + finished/unknown requests, rewrites the map in place. +3. `tests/vllm/v1/test_engine_core_proc.cpp` — a drafting `RunnerStub` under + AsyncScheduler + depth-2: the engine does NOT pull drafts out-of-band + (`take_draft_token_ids` count == 0 under async, > 0 under sync), the stub + receives real draft VALUES via its own fill, and the output token stream is + identical to the sync run of the same stub. +4. `tests/vllm/v1/spec_decode/test_mtp_depth.cpp` (the production-entry gate, + real `LoadedEngine` over the synthetic GDN-hybrid model + real MTP head) — + (a) a spec engine now RESOLVES async ON (the enable flip, red against the + current forced-sync line); (b) token identity: sync arm + (`VT_ASYNC_SCHED=0`) vs async arm (default), both `engine().generate` and + `async_engine().generate`, exact token equality, plus the existing depth + witnesses on every arm so a clamped drafter cannot pass as identity. +5. `tests/vllm/test_scheduler_config.cpp` / + `tests/vllm/config/test_speculative*.cpp` — + `async_scheduling_compatible()` truth table: mtp/dflash/dspark true, + ngram/draft_model false. + +## Gates + +- **G1 (this wave, CPU):** all suites above green; full + `scripts/agent-preflight.sh --staged` green (known-flaky exception + `test_cpu_x86_llamacpp_floor`, #618). +- **G2 (this wave, CPU):** spec-OFF byte-identity — the no-speculator path + through every touched file is inert (placeholder assignment guarded on + `num_spec_tokens_to_schedule > 0`, fill guarded on `spec_on()`, post_step + guard change is a strict narrowing). +- **G3 (OWED, GPU, operator-run):** the c1 DFlash2 TPOT A/B, async-ON vs + `VT_ASYNC_SCHED=0`, same binary, idle box, on the #1574 workload — the + measurement this wave exists for. NOT run here: no GPU lease is authorized + for this session. Recorded under §Owed A1. + +## Mutations (the reviewer's set, written by the implementer as required) + +- **M1 (reachability):** revert the `model_loader.cpp` enable condition to + `!resolved_spec_config_.has_value() && …` — test 4a goes red (the engine + resolves sync). This is the production-call-site deletion for the flip. +- **M2:** delete `request->num_output_placeholders -= num_rejected` in + `update_from_output` — scheduler steady-state case and the identity gate go + red (budget drift). +- **M3:** delete the `async_tokens_to_discard == 0` guard — the ported + underflow test goes red. +- **M4:** make the worker fill a no-op (splice the scheduler's map verbatim) + — the LoadedEngine identity gate is the discriminator: it reds loudly + (`vt: embedding: id out of range`, a `-1` placeholder reaching the embed). + The engine-level stub test (`test_engine_core_proc`'s W7 case) does NOT red + under this mutation: the stub performs its own fill and cannot see the + production runner's fill. +- **M5:** delete D4's computed correction — the depth-2 identity run reds + (positions shift by the rejected count). +- **M6:** in `update_draft_token_ids_in_output`, drop the trim — test 2 reds. + +## Owed + +- **A1 — the GPU A/B (G3).** TPOT/output-tput delta async-ON vs OFF at c1 on + the #1574 DFlash2 workload, operator-run under an `rc` lease. This wave's + claim stops at "the contract is token-exact and reachable"; no speed number + is asserted without it. +- **A2 — the device-resident async spec sampler.** Upstream's overlap on spec + steps needs: device-resident rejection sampling, the propose loop's drafts + staying on device, the draft-aware `combine_sampled_and_draft_tokens` + (`num_logits = 1 + k`), and the optimistic `prev_num_draft_len` + deferred + correction. Owned by `SPEC-DFLASH2` follow-on; issue to be filed if G3 shows + the host-sync bound dominating. +- **A3 — structured output × spec × async.** The multi-row grammar bitmask + under spec decode is deferred upstream-inventory-wide (porting-inventory + §6), and `update_draft_token_ids_in_output`'s grammar `validate_tokens` arm + is deferred WITH it, exactly as the sync `update_draft_token_ids`'s arm + already was (no per-request validate seam exists in this tree). The ported + function's trim / -1-pad / `num_invalid_spec_tokens` halves are live and + unit-tested — a worker can deliver fewer drafts than were scheduled without + any grammar involved. +- **A4 — the `async_tokens_to_discard` producer.** No production path in this + tree sets `async_tokens_to_discard` (`include/vllm/v1/request.h:271`) above + zero: upstream's producer is the reset-prefix-cache force-preempt path, + which is not ported. The W7 rollback guard's false branch + (`scheduler.cpp` `async_tokens_to_discard == 0`) is therefore + production-unreachable until that path lands, and a boundary mutation there + (`== 0` → `<= 1`) survives every suite. The porter of the + reset-prefix-cache force-preempt path owns this gate. + +## Stop conditions + +- The port requiring V2-runner semantics we do not have → `NEEDS_DECISION` + naming the dependency (not hit: upstream's mechanism is V1 + `gpu_model_runner.py`, and the adaptations D3/D4 are host-side). +- Any correctness gate red that a scoped fix cannot make green without + widening scope → `BLOCKED` with the failing case named. + +## Now + +Implementation LANDED in this pull request (spec committed first; commit order +is the proof). CPU tier green: the scheduler placeholder/rollback suites, the +engine-core placeholder/worker-fill contract, and the LoadedEngine token +identity across sync/async through both fronts on the synthetic GDN-hybrid +model with a real MTP head. `SPEC-DFLASH2` stays `ACTIVE`; A1 (the GPU TPOT +A/B) and A2 (the device-resident spec sampler) are the open follow-ons. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 1fb489aef..d05590044 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -196,6 +196,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DFlash2 block-diffusion (dynamic conv + candidate selector) | Qwen3 DFlash2 draft, safetensors or GGUF (bf16 / Q8_0 / Q4_K_M) | Gated against vLLM: 4/4 token-exact, 45/47 draft blocks identical, acceptance identical per prompt. All 7 DFlash2 suites green on `sm_121a`, zero CUDA skips | GREEDY only. Speed 0.8017x vLLM: RECORDED, no floor, NOT a pass (#1562). A GGUF draft is dequantized to bf16 at load ([#1314](https://github.com/mudler/vllm.cpp/issues/1314)) | +| Async scheduling × speculative decoding | any Eagle-type speculator (`mtp`, `dflash`/DFlash2, `dspark`) | `test_mtp_depth` W7 cases: a spec engine resolves async ON and emits the sync scheduler's exact tokens through both engine fronts (depth-1 and the depth-2 batch queue); `test_engine_core_proc` pins the -1-placeholder / worker-fill contract | Mirrors vLLM's polarity (async disabled only OUTSIDE the Eagle-type family): drafts ride as `-1` placeholders the worker fills from its own propose; host `ngram` and `draft_model` stay synchronous. `VT_ASYNC_SCHED=0` rolls back. Spec steps keep the host sampler (device-resident spec sampling owed); the GPU TPOT A/B owed ([#1824](https://github.com/mudler/vllm.cpp/issues/1824)) | | DFlash/DFlash2 shared `lm_head` kept PACKED | a DFlash or DFlash2 draft off an NVFP4 safetensors target | `test_qwen3_dflash2_draft` 36/36 (353): block logits BITWISE equal to `Qwen3_5MTPModel::ComputeLogits` on the same packed head, and `FromModelDir` loads and drafts off one | Widening a head stays refused by name: GGUF `output.weight`, FP8, W4A4. `VT_LMHEAD_FP4=0` rolls back to the refusal. DSpark and the CUDA arm owed ([#1628](https://github.com/mudler/vllm.cpp/issues/1628)) | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/include/vllm/config/speculative.h b/include/vllm/config/speculative.h index b6a223bb8..a5fafddb6 100644 --- a/include/vllm/config/speculative.h +++ b/include/vllm/config/speculative.h @@ -497,6 +497,22 @@ struct SpeculativeConfig { // separate bonus query. bool use_dspark() const { return method == "dspark"; } + // async_scheduling_compatible (SPEC-DFLASH2 W7, #1824): whether async + // scheduling stays ON with this speculative method. Mirrors the pin's + // resolution predicate (vllm/config/vllm.py:1076-1087 @ 555967922): async is + // disabled only for a method OUTSIDE + // EagleModelTypes ∪ NgramGPUTypes ∪ {"dspark"} + // where EagleModelTypes = eagle / eagle3 / extract_hidden_states / every MTP + // type / dflash (speculative.py:60-65). Over the methods THIS engine + // implements that is exactly use_eagle() — mtp, dflash (DFlash2 rides method + // "dflash", same as upstream at b389ac2946 speculative.py:952-956), dspark, + // eagle, eagle3. Host "ngram" is NOT upstream's async lane ("ngram_gpu" is a + // different, GPU-side implementation this engine does not have), and + // "draft_model" is refused AT THE PIN (b389ac2946 later allows it — + // vllm/config/vllm.py:1291 there — but the pin governs; reconcile when the + // pin advances past that commit). + bool async_scheduling_compatible() const { return use_eagle(); } + // NumLookaheadTokens: the scheduler's num_lookahead_tokens for this config // (scheduler.py:275-292). This is the value threaded into allocate_slots so the // verify slots are reserved ahead of time. 0 for a method the scheduler does diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index 4049ae84b..fa3e9c0ea 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -552,10 +552,15 @@ class LoadedEngine { // SchedulerConfig::ResolveAsyncScheduling then the VT_ASYNC_SCHED rollback env. // `is_pooling_model` (ARCH-ONE-SURFACE ROW 6) resolves async OFF for pooling // models (mirror of vllm/config/vllm.py:1068-1073); default false is the - // byte-identical text path. + // byte-identical text path. `spec_decode_incompatible` (SPEC-DFLASH2 W7, + // #1824) resolves async OFF for a speculative method upstream refuses + // (vllm/config/vllm.py:1076-1087 — anything outside the Eagle-type family / + // ngram_gpu / dspark); an Eagle-type speculator passes false and keeps + // async scheduling ON, exactly as upstream. static bool ResolveAsyncEnabled(const vllm::SchedulerConfig& scheduler_config, bool runner_supports_async, - bool is_pooling_model = false); + bool is_pooling_model = false, + bool spec_decode_incompatible = false); // SPEC-MTP I5d: finalize the entrypoint's SpeculativeConfig against the loaded // checkpoint. params.speculative_config carries the CLI method + optional user // k; this re-runs SpeculativeConfig::ResolveMtp with the checkpoint's diff --git a/include/vllm/v1/core/sched/async_scheduler.h b/include/vllm/v1/core/sched/async_scheduler.h index 75266a978..3913b8d74 100644 --- a/include/vllm/v1/core/sched/async_scheduler.h +++ b/include/vllm/v1/core/sched/async_scheduler.h @@ -21,9 +21,16 @@ // count is 0, i.e. under the synchronous Scheduler), so subclassing is the only // delta — no base-class behavior changes for the sync path. // +// SPEC-DFLASH2 W7 (#1824) un-deferred the spec-decode placeholders: with a +// SpeculativeConfig threaded through the ctor, update_after_schedule assigns +// request.spec_token_ids = [-1] * num_spec_tokens_to_schedule after each +// schedule (async_scheduler.py:24-25,43-45) — the NEXT step schedules those +// placeholders as 1+k tokens and the WORKER substitutes the real drafts it +// kept from its own propose ("We will update the actual spec token ids in the +// worker process"). Guarded on num_spec_tokens_to_schedule > 0, so the +// no-speculator path stays byte-identical. +// // DEFERRED (T0, matches upstream structure so re-adding is mechanical): -// - spec-decode placeholders (_spec_token_placeholders / spec_token_ids; -// num_spec_tokens == 0 at T0 so no draft placeholders are added), // - next_decode_eligible_step PP-microbatch cadence (pp_size == 1 → the base // guard stays inert), // - the diffusion num_sampled_tokens_per_step == 0 path (T0 is autoregressive, @@ -45,11 +52,23 @@ namespace vllm::v1 { // ResolveAsyncScheduling, mirroring get_scheduler_cls at scheduler.py:180-189). class AsyncScheduler : public Scheduler { public: + // Same parameters as the base Scheduler. speculative_config (W7 #1824) makes + // the async engine's spec plumbing live: num_lookahead_tokens, the running- + // loop spec budget, and num_spec_tokens_to_schedule for the placeholder + // assignment below. std::nullopt (the default) is the production + // no-speculator path, byte-identical to the pre-W7 ctor. AsyncScheduler(SchedulerConfig scheduler_config, KVCacheConfig kv_cache_config, int block_size, bool enable_caching = false, - StructuredOutputManager* structured_output_manager = nullptr) + StructuredOutputManager* structured_output_manager = nullptr, + std::optional speculative_config = + std::nullopt) : Scheduler(std::move(scheduler_config), std::move(kv_cache_config), - block_size, enable_caching, structured_output_manager) {} + block_size, enable_caching, structured_output_manager, + std::move(speculative_config)) {} + + // The async-scheduling class answers true (read by EngineCore::post_step to + // skip the out-of-band draft pull — core.py:617; see the base declaration). + bool async_scheduling() const override { return true; } protected: // async_scheduler.py:19-49. diff --git a/include/vllm/v1/core/sched/output.h b/include/vllm/v1/core/sched/output.h index 71cc1a503..be387e4c9 100644 --- a/include/vllm/v1/core/sched/output.h +++ b/include/vllm/v1/core/sched/output.h @@ -170,6 +170,22 @@ struct SchedulerOutput { // ENG-ASYNC-SCHED (spec async-serving.md W3). bool pending_structured_output_tokens = false; + // num_spec_tokens_to_schedule (output.py:263 @ 555967922; set at + // scheduler.py:1123-1156): the draft-token count the NEXT step's async + // placeholders take — AsyncScheduler::update_after_schedule assigns + // [-1] * this to each scheduled decode request (async_scheduler.py:24-25, + // 43-45). The flat num_spec_tokens here (the dynamic-SD lookup that can + // shrink it per step is deferred). 0 whenever no speculator is configured, + // which keeps the default path byte-identical. SPEC-DFLASH2 W7 (#1824). + int num_spec_tokens_to_schedule = 0; + + // num_invalid_spec_tokens (output.py:244): req_id -> the count of scheduled + // spec tokens the draft-in-output rewrite could not fill (worker returned + // fewer than scheduled / grammar-filtered) and padded with -1 + // (Scheduler::update_draft_token_ids_in_output, scheduler.py:2072-2107). + // Each rewrite REPLACES the whole map. Empty unless that rewrite ran. + std::map num_invalid_spec_tokens; + // make_empty: an empty step output. static SchedulerOutput make_empty(); }; diff --git a/include/vllm/v1/core/sched/scheduler.h b/include/vllm/v1/core/sched/scheduler.h index 3d7bd72b9..288f0c04a 100644 --- a/include/vllm/v1/core/sched/scheduler.h +++ b/include/vllm/v1/core/sched/scheduler.h @@ -47,9 +47,13 @@ // update_draft_token_ids (scheduler.py:1937). All INERT when no // SpeculativeConfig is supplied (num_lookahead_tokens == 0, every request's // spec_token_ids stays empty), so the default path is byte-identical. -// STILL DEFERRED: the first-decode-step spec PADDING (placeholder -1 drafts / -// num_spec_tokens_to_schedule / num_invalid_spec_tokens), the dynamic-SD -// lookup, and the async draft-in-output path (update_draft_token_ids_in_output). +// SPEC-DFLASH2 W7 (#1824) un-deferred the ASYNC draft-in-output path: +// num_spec_tokens_to_schedule / num_invalid_spec_tokens on SchedulerOutput, +// update_draft_token_ids_in_output (scheduler.py:2072-2107), the +// async_tokens_to_discard rollback guard (:1670-1675) and the preempt-time +// spec_token_ids clear (:1217-1218). STILL DEFERRED: the sync scheduler's +// first-decode-step spec PADDING (pad_spec_decode, scheduler.py:827-843 — a +// cudagraph-uniformity optimization) and the dynamic-SD lookup. // - num_output_placeholders / async scheduling (the early-continue on // max_tokens, next_decode_eligible_step) — treated as 0 / inert. // - DP prefill balancing (throttle_prefills / defer_prefills / @@ -260,6 +264,27 @@ class Scheduler { // runner never produces drafts otherwise). void update_draft_token_ids(const DraftTokenIds& draft_token_ids); + // update_draft_token_ids_in_output (scheduler.py:2072-2107, SPEC-DFLASH2 W7 + // #1824): the ASYNC variant. Under async scheduling the request state holds + // only -1 placeholders (the worker keeps the real drafts), so when the + // deferred (structured-output) sampling path needs real draft values for its + // grammar bitmask, the worker's drafts are rewritten INTO the + // SchedulerOutput's scheduled_spec_decode_tokens: each row is trimmed to the + // scheduled count, a short row is padded back to it with -1, and the invalid + // tail is recorded in scheduler_output.num_invalid_spec_tokens (REPLACED + // whole per call). Unknown / finished requests and requests with no + // scheduled entry are skipped. The grammar validate_tokens arm is deferred + // exactly as in update_draft_token_ids above. + void update_draft_token_ids_in_output(const DraftTokenIds& draft_token_ids, + SchedulerOutput& scheduler_output); + + // async_scheduling: whether this scheduler is the async-scheduling class. + // The engine core's post_step reads it to skip the out-of-band draft pull + // under async scheduling (core.py:617 `not self.async_scheduling`; upstream + // reads the resolved config — here the resolution PRODUCT is the scheduler + // class, model_loader.cpp::MakeScheduler, so the class answers). + virtual bool async_scheduling() const { return false; } + // get_num_unfinished_requests: len(waiting) + len(running) (T0 subset). int get_num_unfinished_requests() const; // get_request_counts: (num_running, num_waiting). @@ -458,6 +483,12 @@ class Scheduler { // it in the ctor via SpeculativeConfig::NumLookaheadTokens (k for MTP). Threaded // into allocate_slots (schedule() lines below). int num_lookahead_tokens_ = 0; + // num_spec_tokens (scheduler.py:241 `self.num_spec_tokens = + // vllm_config.num_speculative_tokens`): the resolved per-step draft count. + // 0 when no SpeculativeConfig is supplied — which keeps + // SchedulerOutput::num_spec_tokens_to_schedule 0 and the AsyncScheduler's + // placeholder assignment inert on the default path. SPEC-DFLASH2 W7 (#1824). + int num_spec_tokens_ = 0; // log_stats (scheduler.py: `self.log_stats = ... not disable_log_stats`, // default True): gate for recording the per-request QUEUED/SCHEDULED/PREEMPTED // engine-core events. ON by default — upstream disable_log_stats defaults diff --git a/include/vllm/v1/engine/core.h b/include/vllm/v1/engine/core.h index ade30f6cb..ef80e51c4 100644 --- a/include/vllm/v1/engine/core.h +++ b/include/vllm/v1/engine/core.h @@ -93,11 +93,13 @@ class EngineCore { // the M1.8 tests building a bare EngineCore; when null, structured output is a // no-op. When provided, it must be the SAME manager the Scheduler was built // with (so get_grammar_bitmask/should_advance see the compiled grammars). - // check_for_draft_tokens (core.py:186-190): whether to pull the drafter's - // out-of-band proposal after each step and feed it to the scheduler. Set true - // ONLY when a speculator is configured AND async scheduling is off (in async - // mode the worker updates drafts itself). Default false = the no-speculator - // production path — post_step is a no-op, so step() is byte-identical. + // check_for_draft_tokens (core.py:166-170): whether a speculator is + // configured at all. post_step pairs it with the scheduler's + // async_scheduling() — under async scheduling the worker updates drafts + // itself and the ONLY engine-side pull is the deferred-grammar + // update_draft_token_ids_in_output rewrite (core.py:718-731; SPEC-DFLASH2 + // W7, #1824). Default false = the no-speculator production path — post_step + // is a no-op, so step() is byte-identical. EngineCore(Scheduler& scheduler, Executor& executor, StructuredOutputManager* structured_output_manager = nullptr, bool check_for_draft_tokens = false) diff --git a/include/vllm/v1/worker/gpu/runner.h b/include/vllm/v1/worker/gpu/runner.h index f6bfe58fc..5b09d4ccc 100644 --- a/include/vllm/v1/worker/gpu/runner.h +++ b/include/vllm/v1/worker/gpu/runner.h @@ -208,18 +208,31 @@ class GPUModelRunner final : public ModelRunnerBase { } // runner_supports_async (mirror of the vLLM compat gate feeding - // SchedulerConfig::ResolveAsyncScheduling — vllm/config/vllm.py:990-1038). TRUE - // iff the runner advertises the placeholder-aware async device path: it is - // engaged (async_input_combine(), from VT_ASYNC_RUNNER / set_async_input_combine) - // AND runs on a backend whose async-output primitives are wired (CUDA, or CPU's - // synchronous degeneration for the CPU gate). DEFAULT ON since the 2026-07-17 - // flip (VT_ASYNC_RUNNER default ON, async_runner_flag.h): with async on this - // returns true, so ResolveAsyncScheduling resolves the AsyncScheduler + mcb=2 by - // default, mirroring vLLM's async-scheduling default. VT_ASYNC_RUNNER=0 rolls the - // runner back to the synchronous host path (returns false → sync Scheduler); - // VT_ASYNC_SCHED=0 rolls only the scheduler back to synchronous in the same binary - // while the runner stays async-capable. - bool runner_supports_async() const override { return async_input_combine_; } + // SchedulerConfig::ResolveAsyncScheduling — vllm/config/vllm.py:990-1038). + // TRUE iff the runner advertises async SCHEDULING capability: the env opt-in + // (VT_ASYNC_RUNNER, default ON since the 2026-07-17 flip) on a backend whose + // async-output primitives are wired (CUDA, or CPU's synchronous + // degeneration for the CPU gate). With it true, ResolveAsyncScheduling + // resolves the AsyncScheduler + mcb=2 by default, mirroring vLLM. + // VT_ASYNC_RUNNER=0 rolls the runner back (false → sync Scheduler); + // VT_ASYNC_SCHED=0 rolls only the scheduler back in the same binary. + // + // SPEC-DFLASH2 W7 (#1824): this is deliberately NOT async_input_combine_ — + // that lever carries the spec veto (the device combine is not draft-aware, + // I5e) and stays OFF under a speculator, while async SCHEDULING now stays ON + // for the Eagle-type family exactly as upstream resolves it + // (vllm/config/vllm.py:1064-1112): the spec engine keeps the synchronous + // host input path and the host spec sampler, and the scheduler still + // overlaps step N+1 with step N's output processing. The two were one flag + // while spec forced sync scheduling; W7 is the commit that splits them. + bool runner_supports_async() const override { return async_sched_supported_; } + + // SPEC-DFLASH2 W7 (#1824): whether the ENGINE resolved async scheduling ON. + // Set once by LoadedEngine after ResolveAsyncEnabled (the runner cannot know + // — the resolution also reads VT_ASYNC_SCHED and the pooling arm); read by + // execute_model's draft-placeholder fill and computed-token correction. + void set_async_scheduling(bool enabled) { use_async_scheduling_ = enabled; } + bool use_async_scheduling() const { return use_async_scheduling_; } // ─── Accessors (for tests + the ordering identity gate) ──────────────────── InputBatch& input_batch() { return input_batch_; } @@ -589,6 +602,19 @@ class GPUModelRunner final : public ModelRunnerBase { // Async-scheduling device-input opt-in (see set_async_input_combine). Default // from VT_ASYNC_RUNNER at construction; OFF keeps the sync host path. bool async_input_combine_ = false; + // Async SCHEDULING capability (see runner_supports_async): the same + // env/backend predicate WITHOUT the spec veto. W7 (#1824). + bool async_sched_supported_ = false; + // Whether the engine resolved async scheduling ON (set_async_scheduling; + // LoadedEngine calls it once after ResolveAsyncEnabled). Gates the + // draft-placeholder fill + computed-token correction in execute_model. + bool use_async_scheduling_ = false; + // W7 (#1824): req_id -> the draft count the PREVIOUS step scheduled for it, + // recorded at splice time under async scheduling. The computed-token + // correction applies only to requests with an entry here (the scheduler's + // num_computed_tokens can carry that step's not-yet-rolled-back rejected + // drafts). Rebuilt every spec step; empty otherwise. + std::map prev_sched_draft_counts_; // ENG-ASYNC-SCHED depth-2 LIFETIME GUARD. sample_tokens_async DEFERS the main // queue's completion to the consuming step's get_output() (one step_with_batch_ // queue call later), so when it returns the previous step's forward / sample / diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index fed839a8d..7527010d3 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -1176,13 +1176,16 @@ vllm::SchedulerConfig LoadedEngine::MakeSchedulerConfig( // (when otherwise compatible). bool LoadedEngine::ResolveAsyncEnabled( const vllm::SchedulerConfig& scheduler_config, bool runner_supports_async, - bool is_pooling_model) { + bool is_pooling_model, bool spec_decode_incompatible) { // Pooling models resolve async scheduling OFF (the mirror of vLLM disabling // it by default for pooling models, vllm/config/vllm.py:1068-1073) — the // landed is_pooling_model arm of ResolveAsyncScheduling, wired here since // ARCH-ONE-SURFACE ROW 6. false (every text arch) is byte-identical. + // spec_decode_incompatible (SPEC-DFLASH2 W7, #1824) is the vllm.py:1076-1087 + // arm: a speculative method OUTSIDE the Eagle-type family resolves OFF; an + // Eagle-type one (mtp/dflash/dspark) passes false and stays ON. return vllm::AsyncSchedulingEnabled(scheduler_config.ResolveAsyncScheduling( - runner_supports_async, is_pooling_model)); + runner_supports_async, is_pooling_model, spec_decode_incompatible)); } std::unique_ptr LoadedEngine::MakeScheduler( @@ -1192,13 +1195,15 @@ std::unique_ptr LoadedEngine::MakeScheduler( vllm::v1::StructuredOutputManager* structured_output_manager, std::optional speculative_config) { if (async_enabled) { - // get_scheduler_cls -> AsyncScheduler (scheduler.py:180-189). SPEC-MTP: the - // async-scheduling draft-in-output path is deferred, so speculation forces - // the synchronous Scheduler below — async_enabled is never true when a - // speculative_config is present. + // get_scheduler_cls -> AsyncScheduler (scheduler.py:180-189). SPEC-DFLASH2 + // W7 (#1824): the speculative_config now rides into the AsyncScheduler — + // an Eagle-type speculator keeps async scheduling and the AsyncScheduler + // needs the config for num_lookahead_tokens, the spec budget, and the + // -1 placeholder assignment. return std::make_unique( std::move(scheduler_config), std::move(kv_cache_config), block_size, - enable_caching, structured_output_manager); + enable_caching, structured_output_manager, + std::move(speculative_config)); } return std::make_unique( std::move(scheduler_config), std::move(kv_cache_config), block_size, @@ -1737,18 +1742,24 @@ LoadedEngine::LoadedEngine(HfConfig config, // step_with_batch_queue; 1 otherwise). Since the 2026-07-17 flip the default // (no env) resolves ON (VT_ASYNC_RUNNER default ON), mirroring vLLM; // VT_ASYNC_RUNNER=0 / VT_ASYNC_SCHED=0 roll back to the synchronous path. - // SPEC-MTP I5d: speculation uses the out-of-band take_draft_token_ids / - // post_step path, which is the SYNCHRONOUS scheduler's contract; the - // async-scheduling draft-in-output variant is deferred (spec §2.5), so a - // configured speculator forces sync scheduling here. - async_scheduling_enabled_(!resolved_spec_config_.has_value() && - ResolveAsyncEnabled( + // SPEC-DFLASH2 W7 (#1824): async scheduling now SURVIVES an Eagle-type + // speculator (mtp/dflash/dspark — vllm/config/vllm.py:1064-1112 at the + // pin), the draft-in-output path having landed: the AsyncScheduler ships + // -1 placeholder drafts, the runner fills them from its own propose, and + // post_step is skipped under async. A method upstream refuses (host + // ngram, draft_model at the pin) still forces the synchronous scheduler + // through the spec_decode_incompatible arm. This line is the production + // reach for the whole wave — the reachability mutation reverts it to the + // pre-W7 `!resolved_spec_config_.has_value() &&` form. + async_scheduling_enabled_(ResolveAsyncEnabled( MakeSchedulerConfig( max_model_len_, params.max_num_seqs > 0 ? params.max_num_seqs : 8, max_num_batched_tokens_, params.policy), runner_.runner_supports_async(), - model_->registration().info.is_pooling_model)), + model_->registration().info.is_pooling_model, + /*spec_decode_incompatible=*/resolved_spec_config_.has_value() && + !resolved_spec_config_->async_scheduling_compatible())), max_concurrent_batches_(MakeSchedulerConfig( max_model_len_, params.max_num_seqs > 0 ? params.max_num_seqs @@ -1867,6 +1878,11 @@ LoadedEngine::LoadedEngine(HfConfig config, std::cerr << "vllm.cpp: Asynchronous scheduling is " << (async_scheduling_enabled_ ? "enabled" : "disabled") << " (max_concurrent_batches=" << max_concurrent_batches_ << ")\n"; + // SPEC-DFLASH2 W7 (#1824): tell the runner which scheduling mode resolved — + // under async + a speculator it fills the scheduler's -1 draft placeholders + // from its own propose and corrects the optimistic num_computed_tokens. + // Before any step runs (WarmupKernels below is the first). + runner_.set_async_scheduling(async_scheduling_enabled_); WarmupKernels(); } diff --git a/src/vllm/v1/core/sched/async_scheduler.cpp b/src/vllm/v1/core/sched/async_scheduler.cpp index 3cb7c1e06..c8b98c36f 100644 --- a/src/vllm/v1/core/sched/async_scheduler.cpp +++ b/src/vllm/v1/core/sched/async_scheduler.cpp @@ -16,12 +16,11 @@ void AsyncScheduler::update_after_schedule(SchedulerOutput& scheduler_output) { // flush finished/preempted sets), then add this step's placeholders. Scheduler::update_after_schedule(scheduler_output); - // async_scheduler.py:20-21. The per-step scheduled draft tokens (empty on the - // default no-speculator path; the first-decode-step PADDING via - // _spec_token_placeholders / num_spec_tokens_to_schedule and the async - // worker-fill assignment `request.spec_token_ids = placeholders` are DEFERRED - // with the async-draft-in-output path — SPEC-MTP I2 lands only the placeholder - // COUNT so num_output_placeholders stays balanced under rejection rollback). + // async_scheduler.py:20-21. The per-step scheduled draft tokens (empty on + // the default no-speculator path). SPEC-DFLASH2 W7 (#1824) landed the async + // worker-fill assignment `request.spec_token_ids = placeholders` below; the + // sync scheduler's first-decode-step PADDING (pad_spec_decode) stays + // deferred. const std::map>& spec_decode_tokens = scheduler_output.scheduled_spec_decode_tokens; for (const auto& [req_id, num_scheduled] : @@ -54,6 +53,19 @@ void AsyncScheduler::update_after_schedule(SchedulerOutput& scheduler_output) { request->num_output_placeholders += num_sampled_tokens_per_step() + cur_num_spec_tokens; + // async_scheduler.py:24-25,43-45 (SPEC-DFLASH2 W7, #1824): assign the NEXT + // step's placeholder drafts. The next schedule() treats them exactly like + // real drafts (1 + k tokens, recorded in scheduled_spec_decode_tokens); + // the WORKER substitutes the real values it kept from its own propose + // ("We will update the actual spec token ids in the worker process"). + // Guarded so the no-speculator path stays byte-identical (the count is 0 + // there, and an empty assignment would still be a write). + if (scheduler_output.num_spec_tokens_to_schedule > 0) { + request->spec_token_ids.assign( + static_cast(scheduler_output.num_spec_tokens_to_schedule), + -1); + } + // async_scheduler.py:46-49 (next_decode_eligible_step, PP microbatching): // pp_size == 1 at T0, so next_decode_eligible_step stays current_step + 1 // which never gates a single-GPU decode; left inert (0) here. diff --git a/src/vllm/v1/core/sched/scheduler.cpp b/src/vllm/v1/core/sched/scheduler.cpp index e75d9578e..4e53f02a9 100644 --- a/src/vllm/v1/core/sched/scheduler.cpp +++ b/src/vllm/v1/core/sched/scheduler.cpp @@ -247,6 +247,9 @@ Scheduler::Scheduler(SchedulerConfig scheduler_config, // from the SpeculativeConfig (k for MTP). Threaded into allocate_slots below. if (speculative_config.has_value()) { num_lookahead_tokens_ = speculative_config->NumLookaheadTokens(); + // num_spec_tokens (scheduler.py:241): the per-step draft count, feeding + // SchedulerOutput::num_spec_tokens_to_schedule (W7 #1824). 0 by default. + num_spec_tokens_ = speculative_config->ResolvedNumSpeculativeTokens(); } // Scheduling policy -> the waiting (FCFS) queue. @@ -331,6 +334,13 @@ void Scheduler::preempt_request(Request* request, double timestamp) { kv_cache_manager->free(*request); request->status = RequestStatus::kPreempted; request->num_computed_tokens = 0; + // scheduler.py:1217-1218 (W7 #1824): drop un-verified drafts — real values + // under the sync flow, -1 placeholders under async. A resumed request + // re-enters through prefill; stale drafts scheduled beside it would verify + // garbage (and, under async, would not pair with any worker-kept drafts). + if (!request->spec_token_ids.empty()) { + request->spec_token_ids.clear(); + } // Upstream Request.num_preemptions (read by PrefixCacheStats.record at // vllm/v1/core/kv_cache_manager.py:239). request->num_preemptions += 1; @@ -825,6 +835,12 @@ SchedulerOutput Scheduler::schedule() { scheduler_output.finished_req_ids = std::move(finished_req_ids); // free_encoder_mm_hashes stays empty (encoder deferred). + // num_spec_tokens_to_schedule (scheduler.py:1123-1156, W7 #1824): the count + // the AsyncScheduler's update_after_schedule below turns into -1 placeholder + // drafts for the NEXT step. Flat num_spec_tokens (dynamic-SD deferred); 0 on + // the no-speculator default. + scheduler_output.num_spec_tokens_to_schedule = num_spec_tokens_; + update_after_schedule(scheduler_output); // Fold this step's prefix-cache lookups into the sliding-window hit-rate @@ -949,9 +965,15 @@ EngineCoreOutputs Scheduler::update_from_output( // num_output_placeholders) is rewound by that many. Inert when the request // had no scheduled drafts (the map lookup misses -> whole block skipped), so // the default path is byte-identical. + // W7 (#1824): skip the whole block while the request is draining stale + // in-flight frames (scheduler.py:1670-1675 `async_tokens_to_discard == 0`): + // a discarded frame's pre-reset rejection count would underflow both + // counters. 0 on the synchronous path and whenever no force-preemption is + // in flight, so the existing arms are unchanged. auto spec_it = scheduler_output.scheduled_spec_decode_tokens.find(req_id); if (spec_it != scheduler_output.scheduled_spec_decode_tokens.end() && - (!new_token_ids.empty() || num_sampled_tokens_per_step_ == 0)) { + (!new_token_ids.empty() || num_sampled_tokens_per_step_ == 0) && + request->async_tokens_to_discard == 0) { const int num_draft_tokens = static_cast(spec_it->second.size()); // num_accepted = generated - num_sampled, floored at 0 so an empty // (aborted / error) output does not underflow (regression: upstream @@ -1285,6 +1307,51 @@ void Scheduler::update_draft_token_ids(const DraftTokenIds& draft_token_ids) { } } +void Scheduler::update_draft_token_ids_in_output( + const DraftTokenIds& draft_token_ids, SchedulerOutput& scheduler_output) { + // scheduler.py:2072-2107 (SPEC-DFLASH2 W7, #1824). The async draft-in-output + // variant: under async scheduling the request state carries only -1 + // placeholders, so the drafts are rewritten INTO the SchedulerOutput the + // deferred (structured-output) sampling path is about to consume. The + // grammar validate_tokens arm (:2096-2098) is deferred exactly as in + // update_draft_token_ids above (no per-request validate seam yet); the -1 + // pad stays REACHABLE without it, because a worker can deliver fewer drafts + // than were scheduled. + std::map num_invalid_spec_tokens; + std::map>& sched_spec_tokens = + scheduler_output.scheduled_spec_decode_tokens; + const std::size_t n = std::min(draft_token_ids.req_ids.size(), + draft_token_ids.draft_token_ids.size()); + for (std::size_t i = 0; i < n; ++i) { + const std::string& req_id = draft_token_ids.req_ids[i]; + const auto req_it = requests.find(req_id); + if (req_it == requests.end() || req_it->second->IsFinished()) { + continue; // the request may have been finished; skip (:2082-2085). + } + const auto sched_it = sched_spec_tokens.find(req_id); + if (sched_it == sched_spec_tokens.end() || sched_it->second.empty()) { + continue; // nothing scheduled for it this step (:2087-2089). + } + const std::size_t orig_num_spec_tokens = sched_it->second.size(); + std::vector spec_token_ids = draft_token_ids.draft_token_ids[i]; + if (spec_token_ids.size() > orig_num_spec_tokens) { + // Trim to the scheduled count (the chunked-prefill case, :2091-2094). + spec_token_ids.resize(orig_num_spec_tokens); + } + if (spec_token_ids.size() < orig_num_spec_tokens) { + // Pad back to the scheduled count with -1 and record the invalid tail; + // the grammar bitmask computation skips the -1 slots (:2099-2103). + const int num_invalid = + static_cast(orig_num_spec_tokens - spec_token_ids.size()); + spec_token_ids.resize(orig_num_spec_tokens, -1); + num_invalid_spec_tokens[req_id] = num_invalid; + } + sched_it->second = std::move(spec_token_ids); + } + // REPLACE the whole map each call (upstream builds a fresh dict, :2075,2107). + scheduler_output.num_invalid_spec_tokens = std::move(num_invalid_spec_tokens); +} + int Scheduler::get_num_unfinished_requests() const { return static_cast(waiting->size()) + static_cast(running.size()); } diff --git a/src/vllm/v1/engine/core.cpp b/src/vllm/v1/engine/core.cpp index b0da6bcc1..1daffe87f 100644 --- a/src/vllm/v1/engine/core.cpp +++ b/src/vllm/v1/engine/core.cpp @@ -107,10 +107,18 @@ std::pair, bool> EngineCore::step() { } void EngineCore::post_step(bool model_executed) { - // core.py:509-517. Under async scheduling the draft token ids are updated in - // the worker process instead (the synchronous step() is the only caller here, - // so `not async_scheduling` holds — the batch-queue path never calls this). - if (!check_for_draft_tokens_ || !model_executed) { + // core.py:509-517 (:617 at 555967922). Under async scheduling the draft + // token ids are updated in the worker process instead (SPEC-DFLASH2 W7, + // #1824): the AsyncScheduler ships -1 placeholders and the runner fills them + // from its own propose, so the out-of-band pull below must NOT run — it + // would overwrite the placeholders with values the scheduler must never + // carry under async. The guard cannot be "which step function called me": + // EngineCoreProc's busy loop AND the depth-1 LLMEngine::step both reach + // here whatever the resolution, so it mirrors upstream's + // `not self.async_scheduling` (the scheduler class IS the resolved value, + // model_loader.cpp::MakeScheduler). + if (!check_for_draft_tokens_ || scheduler_.async_scheduling() || + !model_executed) { return; } std::optional draft_token_ids = executor_.take_draft_token_ids(); @@ -214,6 +222,21 @@ EngineCore::step_with_batch_queue() { // (The runner's execute_model stash from this step is still valid — no other // execute_model ran between it and here.) if (deferred_scheduler_output.has_value()) { + // core.py:718-731 (SPEC-DFLASH2 W7, #1824): with drafts under async + // scheduling, the deferred batch's scheduled_spec_decode_tokens still + // holds the -1 placeholders (the worker fill patches only its own copy). + // Pull the worker's real drafts and rewrite them into the deferred output + // so the grammar bitmask below reads real token ids; a slot the worker + // could not fill is -1-padded and recorded in num_invalid_spec_tokens for + // the bitmask computation to skip. No-op without a speculator. + if (check_for_draft_tokens_) { + std::optional draft_token_ids = + executor_.take_draft_token_ids(); + if (draft_token_ids.has_value()) { + scheduler_.update_draft_token_ids_in_output(*draft_token_ids, + *deferred_scheduler_output); + } + } const std::optional grammar_output = scheduler_.get_grammar_bitmask(*deferred_scheduler_output); std::unique_ptr sampled = diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 75848736c..22195c325 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -401,12 +401,22 @@ GPUModelRunner::GPUModelRunner( // last_sampled token over each decode row's input id with // num_new_sampled_tokens==1; it is NOT spec-aware and would overwrite the // draft token at a verify step's draft position with the committed token. - // Speculative decode already forces SYNC scheduling and gets its drafts + // A speculator therefore keeps the sync HOST INPUT path (its drafts are // spliced into token_ids_cpu by update_req_spec_token_ids + prepare_inputs, - // so force the sync host input path here. Byte-identical for non-spec - // (spec_config_ is nullopt there, so this is AsyncRunnerEnvDefault()). + // and its sampler is the sync one, so the host arrays stay fresh). + // Since SPEC-DFLASH2 W7 (#1824) this veto is INPUT-side only: async + // SCHEDULING stays on for the Eagle-type family via async_sched_supported_ + // below. Byte-identical for non-spec (spec_config_ is nullopt there, so + // this is AsyncRunnerEnvDefault()). async_input_combine_ = AsyncRunnerEnvDefault() && !spec_config_.has_value() && QueueSupportsAsyncInputCombine(queue_); + // SPEC-DFLASH2 W7 (#1824): async SCHEDULING capability is the same + // env/backend predicate WITHOUT the spec veto above — a spec engine keeps + // the sync host input path (the combine is not draft-aware) while still + // advertising the scheduler overlap, mirroring upstream keeping async + // scheduling ON for the Eagle-type family (vllm/config/vllm.py:1064-1112). + async_sched_supported_ = + AsyncRunnerEnvDefault() && QueueSupportsAsyncInputCombine(queue_); // ARCH-ONE-SURFACE ROW 6 (mirror gpu/model_runner.py:368-369): a POOLING // model's runner pools instead of sampling — build the PoolingRunner over // the model-owned Pooler. Null for every text arch (byte-identical). @@ -442,12 +452,22 @@ GPUModelRunner::GPUModelRunner( // last_sampled token over each decode row's input id with // num_new_sampled_tokens==1; it is NOT spec-aware and would overwrite the // draft token at a verify step's draft position with the committed token. - // Speculative decode already forces SYNC scheduling and gets its drafts + // A speculator therefore keeps the sync HOST INPUT path (its drafts are // spliced into token_ids_cpu by update_req_spec_token_ids + prepare_inputs, - // so force the sync host input path here. Byte-identical for non-spec - // (spec_config_ is nullopt there, so this is AsyncRunnerEnvDefault()). + // and its sampler is the sync one, so the host arrays stay fresh). + // Since SPEC-DFLASH2 W7 (#1824) this veto is INPUT-side only: async + // SCHEDULING stays on for the Eagle-type family via async_sched_supported_ + // below. Byte-identical for non-spec (spec_config_ is nullopt there, so + // this is AsyncRunnerEnvDefault()). async_input_combine_ = AsyncRunnerEnvDefault() && !spec_config_.has_value() && QueueSupportsAsyncInputCombine(queue_); + // SPEC-DFLASH2 W7 (#1824): async SCHEDULING capability is the same + // env/backend predicate WITHOUT the spec veto above — a spec engine keeps + // the sync host input path (the combine is not draft-aware) while still + // advertising the scheduler overlap, mirroring upstream keeping async + // scheduling ON for the Eagle-type family (vllm/config/vllm.py:1064-1112). + async_sched_supported_ = + AsyncRunnerEnvDefault() && QueueSupportsAsyncInputCombine(queue_); // ARCH-ONE-SURFACE ROW 6 (mirror gpu/model_runner.py:368-369): a POOLING // model's runner pools instead of sampling — build the PoolingRunner over // the model-owned Pooler. Null for every text arch (byte-identical). @@ -1307,12 +1327,111 @@ std::optional GPUModelRunner::execute_model( // token_ids_cpu after each request's committed prefix (gpu_input_batch.py: // 484-509 update_req_spec_token_ids) so prepare_inputs reads the k draft tokens // at the verify positions. No-op on the default path (empty map / no speculator). + // + // SPEC-DFLASH2 W7 (#1824), the async draft-in-output WORKER half. Under + // async scheduling two things differ from the sync flow, both handled here: + // + // (a) COMPUTED-TOKEN CORRECTION. The scheduler's num_computed_tokens for a + // request whose PREVIOUS step scheduled drafts may still include that + // step's rejected drafts — the rollback runs in update_from_output, + // which the depth-2 loop applies AFTER this schedule. Upstream corrects + // optimistically on-device (gpu_model_runner.py:1356-1396 prev_num_ + // draft_len + the _prepare_inputs GPU correction) because its rejection + // result is not host-visible in time; OUR rejection ran on the host + // last step, so the exact value is STRUCTURAL: the newest committed + // token's position, num_tokens_no_spec - 1. That equals the scheduler's + // value when the rollback already ran (the depth-1 LLMEngine::step + // order) and subtracts exactly num_rejected when it has not (the + // depth-2 batch-queue order) — both orders are live in production, + // which is why the rule is structural rather than temporal. + // + // (b) PLACEHOLDER FILL. The scheduler ships -1 placeholders + // (async_scheduler.py:43-45); the real values are the drafts THIS + // runner proposed at the previous step's sampling (pending_drafts_, + // host-resident because our propose is host-synchronous — the + // device-resident variant is the row's owed A2). The fill patches a + // LOCAL copy for the splice: the engine-side SchedulerOutput keeps its + // placeholders, exactly as upstream's worker-side scatter leaves the + // scheduler's copy untouched (gpu_input_batch.py:520-523). Reads the + // stash WITHOUT consuming it — take_draft_token_ids (the deferred- + // grammar pull) stays the only mover. if (spec_on()) { + if (use_async_scheduling_) { + const CachedRequestData& cached = scheduler_output.scheduled_cached_reqs; + for (int ci = 0; ci < cached.num_reqs(); ++ci) { + const std::string& req_id = cached.req_ids[static_cast(ci)]; + const auto prev_it = prev_sched_draft_counts_.find(req_id); + if (prev_it == prev_sched_draft_counts_.end() || prev_it->second <= 0) { + continue; // no drafts scheduled for it last step: value is exact. + } + const auto idx_it = input_batch_.req_id_to_index.find(req_id); + if (idx_it == input_batch_.req_id_to_index.end()) { + continue; // not in the persistent batch (resumed-as-new path). + } + const int req_index = idx_it->second; + const int sent = + input_batch_.num_computed_tokens_cpu[static_cast(req_index)]; + const int corrected = + input_batch_.num_tokens_no_spec[static_cast(req_index)] - 1; + // sent == corrected (rollback already applied) or exceeds it by at + // most the previous step's rejected count (bounded by its draft + // count). Anything else is a bookkeeping defect — refuse loudly. + VT_CHECK(sent - corrected >= 0 && sent - corrected <= prev_it->second, + "async spec computed-token correction out of range for '" + + req_id + "': scheduler sent " + std::to_string(sent) + + ", structural value " + std::to_string(corrected) + + ", prev drafts " + std::to_string(prev_it->second)); + input_batch_.num_computed_tokens_cpu[static_cast(req_index)] = + corrected; + } + } + + const std::map>* sched_spec = + &scheduler_output.scheduled_spec_decode_tokens; + std::map> filled_spec; + if (use_async_scheduling_ && !sched_spec->empty()) { + std::map*> own; + if (pending_drafts_.has_value()) { + const std::size_t n = std::min(pending_drafts_->req_ids.size(), + pending_drafts_->draft_token_ids.size()); + for (std::size_t i = 0; i < n; ++i) { + own[pending_drafts_->req_ids[i]] = &pending_drafts_->draft_token_ids[i]; + } + } + for (const auto& [req_id, placeholders] : *sched_spec) { + const auto own_it = own.find(req_id); + // Placeholders are only ever assigned to requests this runner sampled + // AND proposed for on the previous step (update_after_schedule skips + // prefill chunks; preemption clears them), so a miss is a defect and + // must say so rather than embed a -1. + VT_CHECK(own_it != own.end(), + "async draft fill: no drafts proposed for request '" + req_id + + "' (placeholders scheduled without a matching propose)"); + VT_CHECK(own_it->second->size() >= placeholders.size(), + "async draft fill: request '" + req_id + "' proposed " + + std::to_string(own_it->second->size()) + + " drafts but the scheduler placed " + + std::to_string(placeholders.size()) + " placeholders"); + filled_spec[req_id] = std::vector( + own_it->second->begin(), + own_it->second->begin() + + static_cast(placeholders.size())); + } + sched_spec = &filled_spec; + } + const int nr = input_batch_.num_reqs(); for (int i = 0; i < nr; ++i) { const std::string& req_id = *input_batch_.req_ids[static_cast(i)]; - input_batch_.update_req_spec_token_ids( - i, req_id, scheduler_output.scheduled_spec_decode_tokens); + input_batch_.update_req_spec_token_ids(i, req_id, *sched_spec); + } + + if (use_async_scheduling_) { + // Record THIS step's scheduled draft counts for (a) next step. + prev_sched_draft_counts_.clear(); + for (const auto& [rid, toks] : *sched_spec) { + prev_sched_draft_counts_[rid] = static_cast(toks.size()); + } } } @@ -1353,8 +1472,10 @@ std::optional GPUModelRunner::execute_model( // Ordering, all on the MAIN queue and therefore exact: replay -> uploads -> // combine -> forward. The forward is handed `device_input_ids` below, so the // host copy of step.input_token_ids is deliberately left stale for decode - // rows; nothing on this path reads it (the rejection-sampler path that does is - // spec-only, and spec forces the sync runner). + // rows; nothing on this path reads it (the rejection-sampler path that does + // is spec-only, and a speculator keeps async_input_combine_ OFF — since W7 + // that is the runner-INPUT lever alone, async SCHEDULING staying on; the + // spec host arrays stay fresh because the spec sampler is the sync one). if (AsyncDeviceInputs* dev = get_or_create_async_device_inputs(); dev != nullptr) { replay_last_sampled_ops(*dev); diff --git a/tests/vllm/config/test_speculative_mtp_depth.cpp b/tests/vllm/config/test_speculative_mtp_depth.cpp index dac716eaa..f4aae6b48 100644 --- a/tests/vllm/config/test_speculative_mtp_depth.cpp +++ b/tests/vllm/config/test_speculative_mtp_depth.cpp @@ -95,3 +95,25 @@ TEST_CASE("the other proposers keep their own depth resolution") { REQUIRE(dflash.num_speculative_tokens.has_value()); CHECK(*dflash.num_speculative_tokens == 3); } + +TEST_CASE("async_scheduling_compatible mirrors the pin's method families " + "(#1824)") { + // vllm/config/vllm.py:1076-1087 @ 555967922: async scheduling is disabled + // only for a method OUTSIDE EagleModelTypes ∪ NgramGPUTypes ∪ {"dspark"}. + // Over OUR methods: mtp/dflash/dspark compatible; host ngram (NOT the + // upstream ngram_gpu lane) and draft_model (allowed only past the pin, at + // b389ac2946) refused. + auto with_method = [](const char* m) { + SpeculativeConfig c; + c.method = m; + c.num_speculative_tokens = 2; + return c; + }; + CHECK(with_method("mtp").async_scheduling_compatible()); + CHECK(with_method("dflash").async_scheduling_compatible()); + CHECK(with_method("dspark").async_scheduling_compatible()); + CHECK(with_method("eagle").async_scheduling_compatible()); + CHECK(with_method("eagle3").async_scheduling_compatible()); + CHECK_FALSE(with_method("ngram").async_scheduling_compatible()); + CHECK_FALSE(with_method("draft_model").async_scheduling_compatible()); +} diff --git a/tests/vllm/v1/spec_decode/test_mtp_depth.cpp b/tests/vllm/v1/spec_decode/test_mtp_depth.cpp index 1fadb788f..700ec3e5d 100644 --- a/tests/vllm/v1/spec_decode/test_mtp_depth.cpp +++ b/tests/vllm/v1/spec_decode/test_mtp_depth.cpp @@ -781,3 +781,99 @@ TEST_CASE("W6: a multi-token PREFILL is not a verify shape") { // satisfied by a step that was never counted at all. CHECK(st.ragged_steps >= 1); } + +// ─────────────────────────────────────────────────────────────────────────── +// SPEC-DFLASH2 W7 (#1824): async scheduling for the Eagle-type speculative +// family. Before this wave, LoadedEngine forced synchronous scheduling for ANY +// configured speculator (model_loader.cpp async_scheduling_enabled_ carried +// `!resolved_spec_config_.has_value() &&`). Upstream disables async only for a +// method OUTSIDE EagleModelTypes ∪ NgramGPUTypes ∪ {"dspark"} +// (vllm/config/vllm.py:1064-1112 @ 555967922), and "mtp" / "dflash" / "dspark" +// are all inside that set. These cases are the production reach for the flip +// (the enable line IS the call site the reachability mutation reverts) and the +// token-identity gate the port is held to: spec decode is lossless, so the +// SAME tokens must come out under sync and async scheduling, through BOTH +// production fronts (LLMEngine::step depth-1 and AsyncLLM's depth-2 +// step_with_batch_queue). +// +// RED-first: on the pre-W7 tree the first CHECK below failed +// (`async_scheduling_enabled()` read false with an MTP speculator configured) +// in both cases; the identity case additionally never entered its async arms. +// ─────────────────────────────────────────────────────────────────────────── +TEST_CASE("W7 (#1824): an MTP spec engine resolves async scheduling ON") { + const HfConfig c = MakeDenseConfig(); + LoadedEngine eng(c, MakeDenseWeights(c), BuildFixture(), SpecParams(2), + MakeMtpHead(c)); + // The upstream polarity: "mtp" is an EagleModelType, so the default-ON + // resolution survives a configured speculator (vllm.py:1064-1112). + CHECK(eng.async_scheduling_enabled()); + // Depth-2 batch queue under async scheduling (max_concurrent_batches + // resolution unchanged from the non-spec path). + CHECK(eng.max_concurrent_batches() == 2); + + // The methods upstream refuses stay refused: host "ngram" is NOT upstream's + // async-capable "ngram_gpu", so an ngram engine keeps the synchronous + // scheduler (vllm.py:1076-1087 lists ngram_gpu, not ngram). + EngineParams pn; + pn.speculative_config = vllm::ParseSpeculativeConfigJson( + R"({"method":"ngram","num_speculative_tokens":2})"); + LoadedEngine ngram_eng(c, MakeDenseWeights(c), BuildFixture(), pn); + CHECK_FALSE(ngram_eng.async_scheduling_enabled()); + CHECK(ngram_eng.max_concurrent_batches() == 1); +} + +TEST_CASE("W7 (#1824): sync and async scheduling emit IDENTICAL tokens, " + "through BOTH engine fronts, with the drafts verified in every arm") { + const HfConfig c = MakeDenseConfig(); + const std::string prompt = "hello world"; + const int kN = 10; + const int kK = 2; + + // Arm 1 — the synchronous scheduler, forced by the same-binary rollback env + // (VT_ASYNC_SCHED=0). This is the pre-W7 production behavior and the anchor + // sequence. + std::vector sync_ids; + { + setenv("VT_ASYNC_SCHED", "0", /*overwrite=*/1); + LoadedEngine eng(c, MakeDenseWeights(c), BuildFixture(), SpecParams(kK), + MakeMtpHead(c)); + unsetenv("VT_ASYNC_SCHED"); + REQUIRE_FALSE(eng.async_scheduling_enabled()); + sync_ids = eng.engine().generate(prompt, Greedy(kN), "req") + .outputs[0].token_ids; + // The drafter ran to depth in this arm (identity alone passes on a + // speculator that never proposes — the witness pair is required, exactly + // as the k-identity case above). + CheckDraftDecodeForwards(eng, kK); + } + REQUIRE(static_cast(sync_ids.size()) == kN); + + // Arm 2 — async scheduling ON (the W7 default), depth-1 front + // (LLMEngine::step drives EngineCore::step with the AsyncScheduler: + // placeholders assigned, drafts filled worker-side, post_step skipped). + { + LoadedEngine eng(c, MakeDenseWeights(c), BuildFixture(), SpecParams(kK), + MakeMtpHead(c)); + REQUIRE(eng.async_scheduling_enabled()); + const std::vector ids = + eng.engine().generate(prompt, Greedy(kN), "req").outputs[0].token_ids; + CHECK(ids == sync_ids); + CheckDraftDecodeForwards(eng, kK); + } + + // Arm 3 — async scheduling ON, depth-2 front (AsyncLLM -> + // EngineCoreProc::step_with_batch_queue: schedule N+1 BEFORE step N's output + // is consumed — the ordering the placeholder arithmetic exists for). + { + LoadedEngine eng(c, MakeDenseWeights(c), BuildFixture(), SpecParams(kK), + MakeMtpHead(c)); + REQUIRE(eng.async_scheduling_enabled()); + REQUIRE(eng.max_concurrent_batches() == 2); + const RequestOutput out = + eng.async_engine().generate(prompt, Greedy(kN), "req"); + REQUIRE(out.finished); + REQUIRE(out.outputs.size() == 1); + CHECK(out.outputs[0].token_ids == sync_ids); + CheckDraftDecodeForwards(eng, kK); + } +} diff --git a/tests/vllm/v1/test_async_scheduler.cpp b/tests/vllm/v1/test_async_scheduler.cpp index 9f457bb31..93d84fafe 100644 --- a/tests/vllm/v1/test_async_scheduler.cpp +++ b/tests/vllm/v1/test_async_scheduler.cpp @@ -31,6 +31,7 @@ #include #include "vllm/config/scheduler.h" +#include "vllm/config/speculative.h" #include "vllm/sampling_params.h" #include "vllm/v1/core/kv_cache_utils.h" #include "vllm/v1/core/sched/async_scheduler.h" @@ -523,3 +524,152 @@ TEST_CASE("AsyncScheduler: c8 short-output chunked-prefill + preemption stays ba CHECK(out_counts[std::to_string(i)] == max_tokens); } } + +// ═══════════════════════════════════════════════════════════════════════════ +// SPEC-DFLASH2 W7 (#1824): async scheduling × speculative decoding — the +// draft-in-output scheduler half. Ported/derived from +// vllm/v1/core/sched/async_scheduler.py:14-45 (placeholder assignment) and +// tests/v1/core/test_async_scheduler.py:332 +// (test_no_placeholder_underflow_on_discarded_spec_frame) @ 555967922. +// +// RED-first: before W7 the AsyncScheduler had no speculative_config parameter +// (the placeholder-assignment half was on the header's own DEFERRED list), so +// this block did not COMPILE; the discard-guard case additionally fails +// behaviorally on a tree that compiles it without the async_tokens_to_discard +// guard in update_from_output (scheduler.py:1670-1675). +// ═══════════════════════════════════════════════════════════════════════════ +namespace { + +// create_scheduler(async_scheduling=True, num_speculative_tokens=k). +std::unique_ptr CreateAsyncSpecScheduler(int k) { + SchedulerConfig cfg; + cfg.max_num_seqs = 16; + cfg.max_num_batched_tokens = 8192; + cfg.enable_chunked_prefill = true; + cfg.max_model_len = 8192; + cfg.watermark = 0.0; + cfg.async_scheduling = true; + + KVCacheConfig kv_cfg; + kv_cfg.num_blocks = 10000; + kv_cfg.kv_cache_groups.emplace_back( + std::vector{"layer"}, + std::make_shared(16, /*num_kv_heads=*/1, + /*head_size=*/1, DType::kF32)); + vllm::SpeculativeConfig spec = + vllm::SpeculativeConfig::ResolveMtp(/*mtp_num_hidden_layers=*/1, k); + return std::make_unique(cfg, kv_cfg, /*block_size=*/16, + /*enable_caching=*/true, + /*structured_output_manager=*/nullptr, + std::move(spec)); +} + +// Feed a verify step's output: `tokens` are the emitted ids (accepted drafts + +// the bonus/replacement token). +void FeedOutput(AsyncScheduler& sched, const SchedulerOutput& so, + const std::vector& tokens) { + ModelRunnerOutput mro; + mro.req_ids.push_back("0"); + mro.req_id_to_index["0"] = 0; + mro.sampled_token_ids.push_back(tokens); + (void)sched.update_from_output(so, mro); +} + +} // namespace + +TEST_CASE("AsyncScheduler spec (#1824): -1 placeholders are assigned, verified " + "at 1+k, and the rejection rollback keeps the budget balanced") { + const int k = 2; + auto scheduler = CreateAsyncSpecScheduler(k); + auto requests = CreateRequests(/*num_requests=*/1, /*num_tokens=*/10, + /*max_tokens=*/20); + scheduler->add_request(std::move(requests[0])); + Request* r = scheduler->requests.at("0").get(); + const std::vector placeholders(static_cast(k), -1); + + // Step 1 — prefill. update_after_schedule assigns the NEXT step's + // placeholder drafts (async_scheduler.py:43-45) and reserves 1 output + // placeholder (no spec tokens were scheduled THIS step). + SchedulerOutput so1 = scheduler->schedule(); + CHECK(so1.num_scheduled_tokens.at("0") == 10); + CHECK(so1.num_spec_tokens_to_schedule == k); + CHECK(r->spec_token_ids == placeholders); + CHECK(r->num_output_placeholders == 1); + + // Step 2 — scheduled BEFORE step 1's output is processed (the depth-2 + // ordering). The placeholders are scheduled as 1+k tokens and ride the + // output as values the WORKER will replace. + SchedulerOutput so2 = scheduler->schedule(); + CHECK(so2.num_scheduled_tokens.at("0") == 1 + k); + REQUIRE(so2.scheduled_spec_decode_tokens.count("0") == 1); + CHECK(so2.scheduled_spec_decode_tokens.at("0") == placeholders); + CHECK(r->spec_token_ids == placeholders); // re-assigned for step 3 + CHECK(r->num_output_placeholders == 1 + (1 + k)); + + // Step 1's output lands: one prompt-sampled token. + FeedOutput(*scheduler, so1, {7}); + CHECK(r->num_output_placeholders == 3); + + // Step 3 — while step 2 is in flight. Steady state: exactly 1+k again. + SchedulerOutput so3 = scheduler->schedule(); + CHECK(so3.num_scheduled_tokens.at("0") == 1 + k); + CHECK(r->num_output_placeholders == 6); + + // Step 2's output: 1 of 2 drafts accepted (2 tokens emitted). The rollback + // rewinds BOTH num_computed_tokens and num_output_placeholders by + // num_rejected == 1 (scheduler.py:1683-1690), then the placeholder drain + // consumes the 2 emitted tokens. + const int computed_before = r->num_computed_tokens; + FeedOutput(*scheduler, so2, {8, 9}); + CHECK(r->num_computed_tokens == computed_before - 1); + CHECK(r->num_output_placeholders == 3); + + // Steady state holds across every acceptance count in {0, 1, k}. + SchedulerOutput so4 = scheduler->schedule(); + CHECK(so4.num_scheduled_tokens.at("0") == 1 + k); + FeedOutput(*scheduler, so3, {10}); // 0 accepted -> 2 rejected + CHECK(r->num_output_placeholders == 3); + SchedulerOutput so5 = scheduler->schedule(); + CHECK(so5.num_scheduled_tokens.at("0") == 1 + k); + FeedOutput(*scheduler, so4, {11, 12, 13}); // k accepted -> 0 rejected + CHECK(r->num_output_placeholders == 3); + CHECK(r->num_output_placeholders >= 0); + // The budget formula stayed exact: every step after the prefill scheduled + // exactly 1+k tokens, never a partial draft and never a stall. +} + +TEST_CASE("AsyncScheduler spec (#1824): a discarded in-flight spec frame does " + "not underflow the placeholder count") { + // Ported from tests/v1/core/test_async_scheduler.py:332 + // (test_no_placeholder_underflow_on_discarded_spec_frame @ 555967922), + // upstream's ngram_gpu swapped for our MTP config (the guard under test is + // method-independent). + const int num_spec = 5; + auto scheduler = CreateAsyncSpecScheduler(num_spec); + auto requests = CreateRequests(/*num_requests=*/1, /*num_tokens=*/10, + /*max_tokens=*/20); + Request* r = requests[0].get(); + r->num_computed_tokens = r->NumTokens(); + r->status = RequestStatus::kRunning; + r->num_output_placeholders = 1; + r->async_tokens_to_discard = num_spec; + scheduler->requests["0"] = std::move(requests[0]); + scheduler->running.push_back(r); + const int computed_before = r->num_computed_tokens; + + SchedulerOutput so; + so.num_scheduled_tokens["0"] = num_spec + 1; + so.total_num_scheduled_tokens = num_spec + 1; + so.scheduled_spec_decode_tokens["0"] = + std::vector(static_cast(num_spec), 10); + + FeedOutput(*scheduler, so, {999}); + + // The stale frame was dropped whole: no rollback (its pre-reset rejection + // count would underflow the counters), no placeholder drain, one discard + // frame consumed. + CHECK(r->num_output_placeholders == 1); + CHECK(r->num_computed_tokens == computed_before); + CHECK(r->async_tokens_to_discard == num_spec - 1); + CHECK(r->status == RequestStatus::kRunning); +} diff --git a/tests/vllm/v1/test_engine_core_proc.cpp b/tests/vllm/v1/test_engine_core_proc.cpp index 8bd687480..6d1f8083c 100644 --- a/tests/vllm/v1/test_engine_core_proc.cpp +++ b/tests/vllm/v1/test_engine_core_proc.cpp @@ -43,6 +43,7 @@ #include #include "vllm/config/scheduler.h" +#include "vllm/config/speculative.h" #include "vllm/sampling_params.h" #include "vllm/v1/core/kv_cache_utils.h" #include "vllm/v1/core/sched/output.h" @@ -655,3 +656,274 @@ TEST_CASE("EngineCoreProc: depth-2 async overlap cycle finishes every request ex } client.shutdown(); } + +// ═══════════════════════════════════════════════════════════════════════════ +// SPEC-DFLASH2 W7 (#1824): the async draft-in-output ENGINE contract, at the +// ModelRunnerBase seam. Under async scheduling the engine must NOT pull drafts +// out-of-band (post_step is guarded, core.py:617 @ 555967922 — "we update +// draft token ids in the worker process"); the scheduler ships -1 placeholders +// and the WORKER substitutes the real values it kept. Under the synchronous +// scheduler the out-of-band take_draft_token_ids -> update_draft_token_ids +// flow stays exactly as SPEC-MTP I2 landed it. Both flows must emit the SAME +// token stream — spec decode is lossless. +// +// RED-first: before W7 the async run FAILED three ways — take_calls read >0 +// (core_proc.cpp called post_step unguarded, installing real drafts under +// async), saw_placeholder_values read false (the placeholders were therefore +// never scheduled), and the AsyncScheduler could not even be constructed with +// a SpeculativeConfig (no such ctor parameter). +// ═══════════════════════════════════════════════════════════════════════════ +namespace { + +// A deterministic drafting model double. Request `id`'s true output at +// position p is Tok(id, p); the drafter proposes the true token except at +// every third position (one wrong draft), so per-step acceptance cycles +// through 0..k and the rejection-rollback arithmetic is exercised at every +// value. Verification is accept-iff-equal, exactly the greedy contract. +class SpecRunnerStub : public ModelRunnerBase { + public: + explicit SpecRunnerStub(int k) : k_(k) {} + + static int32_t Tok(const std::string& id, int p) { + return 100 + (std::stoi(id) * 31 + p) % 23; + } + int32_t Draft(const std::string& id, int p) const { + const int32_t t = Tok(id, p); + return (p % 3 == 2) ? t + 1 : t; + } + + // Read by the TEST THREAD after client.shutdown() (doctest assertions are + // not thread-safe, so the engine thread only records). + int take_calls = 0; + bool saw_real_draft_values = false; + bool saw_placeholder_values = false; + bool fill_miss = false; // placeholder row with no drafts of our own + bool value_mismatch = false; // sync-installed values != what we proposed + int drafted_verify_steps = 0; + + std::optional execute_model( + const SchedulerOutput& scheduler_output) override { + stashed_ = scheduler_output; + return std::nullopt; + } + + ModelRunnerOutput sample_tokens( + const std::optional& /*grammar_output*/) override { + ModelRunnerOutput mro; + int idx = 0; + std::map> proposed_now; + for (const auto& [req_id, n] : stashed_.num_scheduled_tokens) { + (void)n; + mro.req_ids.push_back(req_id); + mro.req_id_to_index[req_id] = idx++; + int& pos = out_pos_[req_id]; // output tokens emitted so far + + // The drafts this step verifies: REAL values under the sync contract + // (post_step installed them), -1 placeholders under async (the worker — + // this stub — holds the real ones and substitutes, mirroring + // gpu_input_batch.py:520-523 "placeholders ... overwritten"). + std::vector drafts; + const auto it = stashed_.scheduled_spec_decode_tokens.find(req_id); + if (it != stashed_.scheduled_spec_decode_tokens.end()) { + drafts = it->second; + bool has_placeholder = false; + for (const int32_t d : drafts) has_placeholder |= (d == -1); + const auto own = own_drafts_.find(req_id); + if (has_placeholder) { + saw_placeholder_values = true; + if (own == own_drafts_.end() || own->second.size() < drafts.size()) { + fill_miss = true; + drafts.clear(); + } else { + std::copy(own->second.begin(), + own->second.begin() + + static_cast(drafts.size()), + drafts.begin()); + } + } else if (!drafts.empty()) { + saw_real_draft_values = true; + if (own == own_drafts_.end() || own->second.size() < drafts.size()) { + value_mismatch = true; + } else { + for (std::size_t j = 0; j < drafts.size(); ++j) { + value_mismatch |= (drafts[j] != own->second[j]); + } + } + } + } + + std::vector emitted; + if (drafts.empty()) { + emitted.push_back(Tok(req_id, pos)); // prefill / no-draft decode + } else { + ++drafted_verify_steps; + int accepted = 0; + for (std::size_t j = 0; j < drafts.size(); ++j) { + if (drafts[j] == Tok(req_id, pos + static_cast(j))) { + ++accepted; + } else { + break; + } + } + for (int i = 0; i <= accepted; ++i) { + emitted.push_back(Tok(req_id, pos + i)); + } + } + pos += static_cast(emitted.size()); + mro.sampled_token_ids.push_back(std::move(emitted)); + + // Propose the next k drafts (positions pos..pos+k-1). + std::vector next; + next.reserve(static_cast(k_)); + for (int i = 0; i < k_; ++i) { + next.push_back(Draft(req_id, pos + i)); + } + proposed_now[req_id] = next; + } + // Stash for take_draft_token_ids (sync) AND for our own async fill. + vllm::v1::DraftTokenIds fresh; + for (auto& [rid, toks] : proposed_now) { + fresh.req_ids.push_back(rid); + fresh.draft_token_ids.push_back(toks); + own_drafts_[rid] = std::move(toks); + } + pending_ = std::move(fresh); + return mro; + } + + std::optional take_draft_token_ids() override { + ++take_calls; + std::optional out = std::move(pending_); + pending_.reset(); + return out; + } + + private: + const int k_; + SchedulerOutput stashed_; + std::map out_pos_; + std::map> own_drafts_; + std::optional pending_; +}; + +// The sync sibling of the async spec scheduler below. +std::unique_ptr CreateSyncSpecScheduler(int k) { + SchedulerConfig cfg; + cfg.max_num_seqs = 16; + cfg.max_num_batched_tokens = 8192; + cfg.enable_chunked_prefill = true; + cfg.max_model_len = 8192; + cfg.watermark = 0.0; + cfg.async_scheduling = false; + + KVCacheConfig kv_cfg; + kv_cfg.num_blocks = 10000; + kv_cfg.kv_cache_groups.emplace_back( + std::vector{"layer"}, + std::make_shared(16, /*num_kv_heads=*/1, + /*head_size=*/1, DType::kF32)); + return std::make_unique( + cfg, kv_cfg, /*block_size=*/16, /*enable_caching=*/true, + /*structured_output_manager=*/nullptr, + vllm::SpeculativeConfig::ResolveMtp(/*mtp_num_hidden_layers=*/1, k)); +} + +std::unique_ptr CreateAsyncSpecScheduler(int k) { + SchedulerConfig cfg; + cfg.max_num_seqs = 16; + cfg.max_num_batched_tokens = 8192; + cfg.enable_chunked_prefill = true; + cfg.max_model_len = 8192; + cfg.watermark = 0.0; + cfg.async_scheduling = true; + + KVCacheConfig kv_cfg; + kv_cfg.num_blocks = 10000; + kv_cfg.kv_cache_groups.emplace_back( + std::vector{"layer"}, + std::make_shared(16, /*num_kv_heads=*/1, + /*head_size=*/1, DType::kF32)); + return std::make_unique( + cfg, kv_cfg, /*block_size=*/16, /*enable_caching=*/true, + /*structured_output_manager=*/nullptr, + vllm::SpeculativeConfig::ResolveMtp(/*mtp_num_hidden_layers=*/1, k)); +} + +// One full run over `sched`: kNumReqs requests to completion; returns the +// per-request token streams. +std::map> RunSpecCycle( + Scheduler& sched, SpecRunnerStub& runner, int max_concurrent_batches, + int num_reqs, int max_tokens) { + Executor executor(runner); + InprocClient client(sched, executor, /*structured_output_manager=*/nullptr, + max_concurrent_batches, /*shutdown_timeout_s=*/0, + /*check_for_draft_tokens=*/true); + std::set ids; + for (int i = 0; i < num_reqs; ++i) { + std::string id = std::to_string(i); + client.add_request_async(MakeRequest(id, max_tokens)); + ids.insert(id); + } + std::map> outputs; + LoopUntilDone(client, ids, outputs); + client.shutdown(); + + std::map> streams; + for (const auto& [id, frames] : outputs) { + for (const auto& f : frames) { + streams[id].insert(streams[id].end(), f.new_token_ids.begin(), + f.new_token_ids.end()); + } + } + return streams; +} + +} // namespace + +TEST_CASE("W7 (#1824): async scheduling ships -1 placeholders, never pulls " + "drafts out-of-band, and emits the sync flow's exact tokens") { + constexpr int kK = 2; + constexpr int kNumReqs = 3; + constexpr int kMaxTokens = 13; + + // The expected stream is the stub's own true continuation — spec decode is + // lossless, so BOTH flows must emit exactly this. + std::map> expected; + for (int i = 0; i < kNumReqs; ++i) { + const std::string id = std::to_string(i); + for (int p = 0; p < kMaxTokens; ++p) { + expected[id].push_back(SpecRunnerStub::Tok(id, p)); + } + } + + // Arm 1 — the synchronous flow (SPEC-MTP I2): post_step pulls the drafts + // out-of-band and the scheduler carries REAL values. + SpecRunnerStub sync_runner(kK); + auto sync_sched = CreateSyncSpecScheduler(kK); + const auto sync_streams = RunSpecCycle(*sync_sched, sync_runner, + /*max_concurrent_batches=*/1, + kNumReqs, kMaxTokens); + CHECK(sync_streams == expected); + CHECK(sync_runner.take_calls > 0); + CHECK(sync_runner.saw_real_draft_values); + CHECK_FALSE(sync_runner.saw_placeholder_values); + CHECK_FALSE(sync_runner.value_mismatch); + CHECK(sync_runner.drafted_verify_steps > 0); + + // Arm 2 — the async draft-in-output flow (this wave): the scheduler ships + // placeholders, the worker fills, and the engine NEVER pulls out-of-band + // (no structured output is scheduled, so the deferred-grammar pull — the + // one legitimate async take site, core.py:718-731 — never runs either). + SpecRunnerStub async_runner(kK); + auto async_sched = CreateAsyncSpecScheduler(kK); + const auto async_streams = RunSpecCycle(*async_sched, async_runner, + /*max_concurrent_batches=*/2, + kNumReqs, kMaxTokens); + CHECK(async_streams == expected); + CHECK(async_streams == sync_streams); + CHECK(async_runner.take_calls == 0); + CHECK(async_runner.saw_placeholder_values); + CHECK_FALSE(async_runner.saw_real_draft_values); + CHECK_FALSE(async_runner.fill_miss); + CHECK(async_runner.drafted_verify_steps > 0); +} diff --git a/tests/vllm/v1/test_scheduler.cpp b/tests/vllm/v1/test_scheduler.cpp index 844ad96c9..c384afff6 100644 --- a/tests/vllm/v1/test_scheduler.cpp +++ b/tests/vllm/v1/test_scheduler.cpp @@ -1370,6 +1370,65 @@ TEST_CASE( CHECK(out.scheduled_spec_decode_tokens.at(req_id).size() == 3); } +TEST_CASE( + "Scheduler.update_draft_token_ids_in_output (#1824): trims to the " + "scheduled count, pads a short row with -1, and skips unknown requests") { + // Ported from scheduler.py:2072-2107 @ 555967922 (SPEC-DFLASH2 W7). The + // async draft-in-output path rewrites the drafts INSIDE a SchedulerOutput + // (the scheduler-side request state holds only placeholders under async + // scheduling); the grammar validate_tokens arm stays deferred exactly as in + // the sync update_draft_token_ids above (no per-request validate seam yet). + auto scheduler = CreateSpecScheduler(/*num_speculative_tokens=*/3); + auto requests = CreateRequests(1, /*num_tokens=*/4); + const std::string req_id = requests[0]->request_id; + (void)AddRequest(*scheduler, std::move(requests[0])); + + auto out = scheduler->schedule(); // prefill + FeedModelOutput(*scheduler, out, {{0}}); + DraftTokenIds drafts; + drafts.req_ids = {req_id}; + drafts.draft_token_ids = {{1, 2, 3}}; + scheduler->update_draft_token_ids(drafts); + + out = scheduler->schedule(); // schedules the 3 drafts + REQUIRE(out.scheduled_spec_decode_tokens.count(req_id) == 1); + REQUIRE(out.scheduled_spec_decode_tokens.at(req_id).size() == 3); + + // (a) The worker returned MORE drafts than were scheduled (chunked-prefill + // trim, scheduler.py:2091-2094): trimmed to the scheduled count. + DraftTokenIds more; + more.req_ids = {req_id}; + more.draft_token_ids = {{7, 8, 9, 10, 11}}; + scheduler->update_draft_token_ids_in_output(more, out); + CHECK(out.scheduled_spec_decode_tokens.at(req_id) == + std::vector{7, 8, 9}); + CHECK(out.num_invalid_spec_tokens.empty()); + + // (b) FEWER drafts than scheduled: padded to the scheduled count with -1 + // and the invalid tail recorded (scheduler.py:2099-2103). + DraftTokenIds fewer; + fewer.req_ids = {req_id}; + fewer.draft_token_ids = {{7}}; + scheduler->update_draft_token_ids_in_output(fewer, out); + CHECK(out.scheduled_spec_decode_tokens.at(req_id) == + std::vector{7, -1, -1}); + REQUIRE(out.num_invalid_spec_tokens.count(req_id) == 1); + CHECK(out.num_invalid_spec_tokens.at(req_id) == 2); + + // (c) An unknown request id is skipped (scheduler.py:2082-2085), and a known + // request with NO scheduled spec entry is skipped too (:2087-2089). + DraftTokenIds ghost; + ghost.req_ids = {"ghost", req_id}; + ghost.draft_token_ids = {{1, 2, 3}, {4, 5, 6}}; + CHECK_NOTHROW(scheduler->update_draft_token_ids_in_output(ghost, out)); + CHECK(out.scheduled_spec_decode_tokens.count("ghost") == 0); + CHECK(out.scheduled_spec_decode_tokens.at(req_id) == + std::vector{4, 5, 6}); + // Each call REPLACES num_invalid_spec_tokens (upstream builds a fresh dict + // and assigns it, scheduler.py:2075,2107) — (b)'s entry does not survive (c). + CHECK(out.num_invalid_spec_tokens.empty()); +} + TEST_CASE( "Scheduler: no speculator -> spec-decode paths inert (default-off)") { // Default-off proof: a scheduler built WITHOUT a SpeculativeConfig never sees a diff --git a/tests/vllm/v1/worker/test_runner.cpp b/tests/vllm/v1/worker/test_runner.cpp index 63cb7eea3..f33dfda5b 100644 --- a/tests/vllm/v1/worker/test_runner.cpp +++ b/tests/vllm/v1/worker/test_runner.cpp @@ -1428,8 +1428,11 @@ TEST_CASE("runner: sample_tokens_async decode is token-identical to sync") { GPUModelRunner runner(c, w, MakeKvConfig(c, DType::kBF16, DType::kF32), Q(), 8, kMaxModelLen, 64); runner.set_async_input_combine(async_output); - // The runner advertises async support exactly when the async path is on. - CHECK(runner.runner_supports_async() == async_output); + // SPEC-DFLASH2 W7 (#1824): runner_supports_async() is the env/backend + // capability predicate and no longer tracks the input-combine lever (async + // SCHEDULING must survive a spec engine whose combine is vetoed — I5e). + // The lever this case toggles is the combine itself: + CHECK(runner.async_input_combine() == async_output); std::vector tokens; auto sample = [&]() -> int32_t {