[Feature] Vanilla speculative decoding (standalone, opt-in offline interface)

[javierlimt6]

Motivation

Add vanilla speculative decoding (single linear chain, greedy verification — Leviathan/Chen 2023) to mini-sglang as a standalone, opt-in offline interface. This v1 lands the algorithm and correctness infrastructure with a minimal blast radius: a single new package whose only mini-sglang touchpoint is a pure verification function. Deep Engine/Scheduler integration (which is where the wall-clock win lives) is an explicit follow-up; see Known limitations.

The default serving path is untouched — speculation is reachable only by explicitly constructing SpeculativeEngine or running the offline CLI, so existing latency is unaffected.

Modifications

New package python/minisgl/speculative/ (additive; nothing else imports it eagerly):

• verify.py — verify_drafts(draft_tokens, target_logits) -> (accept_tokens, num_correct_drafts). Pure, framework-agnostic greedy verification: accept the matching prefix of K drafts against the target's K+1 verify-pass logits, then append the target's bonus token. This is the one reusable touchpoint for the future engine integration.
• engine.py — SpeculativeEngine: single-request, greedy speculative loop on a HuggingFace transformers + DynamicCache backend. Exposes cumulative accept_length (τ) and accept_rate (α) counters matching sglang's tokenizer_manager definitions.
• draft.py — StandaloneDraft: autoregressive draft from a smaller same-family model. Uses a K+1 trailing feed so the draft KV cache stays aligned with the target's K+1 verify pass, making rollback() a plain cache truncation.
• __main__.py — one-shot offline CLI: python -m minisgl.speculative --target-model X --draft-model Y --prompt "..." runs greedy speculative decoding and prints the decoded text plus accept_length/accept_rate.
• tests/core/test_speculative.py — upstream unit test for verify_drafts (pure-CPU; collected by pytest as test_verify_drafts and runnable directly via call_if_main, matching test_scheduler.py).
• docs/structures.md — one-line module entry.

Naming throughout follows sglang's spec-decoding conventions (accept/correct/bonus, num_/_ct/_rate).

Accuracy Tests

• verify_drafts unit scenarios (CI, pure-CPU): all-correct + bonus, all-reject, every interior partial-accept position, and the K=1 boundary.
• End-to-end byte-equality (local, GPU): SpeculativeEngine output is byte-for-byte identical to HF assisted_generation (which shares the same K-batch verify path) on Qwen3-1.7B target + Qwen3-0.6B draft, 4/4 prompts including the bf16 near-tie case ("Once upon a time,"). Note: equality is against the same K-batch compute path; in bf16 it can differ from per-token greedy at logit near-ties — this is a floating-point reordering artifact, not a logic bug.
• Measured per-round acceptance (out of K=4): 3.55 factual, 3.68 code, 1.89 creative. Target forward calls for 100 new tokens dropped to ~23 (factual/code) / ~36 (creative) vs ~101 non-speculative.

Speed Tests and Profiling

Reported honestly: at batch=1 on an RTX 4060 Laptop (8 GB), this v1 is a wall-clock regression, not a speedup — 0.55–1.13× depending on prompt and model pair (Qwen2.5-3B/0.5B ≈ break-even; Qwen3-1.7B/0.6B a clear loss). Although accepted drafts cut target forwards substantially, each round adds K+1 draft forwards, so the total forward count rises; at batch=1 every forward is launch/bandwidth-bound and a small-draft forward is not proportionally cheaper than a target forward.

Speedup is gated on the deferred work: CUDA-graph capture of the draft loop (lands with engine integration), a larger target/draft cost ratio (8B+ target), and/or batching. Because the feature is opt-in, no existing path regresses.

Known limitations / follow-ups

• Not wired into the serving Engine/Scheduler (offline SpeculativeEngine + CLI only) — the integrated worker is the planned follow-up and the prerequisite for a wall-clock win.
• Greedy only (v1); sampling unsupported.
• Single request; no continuous batching.
• Full-attention models only — sliding-window draft caches cannot be cropped on rollback (DynamicCache.crop raises past the window). Validated on full-attention Qwen3.
• Stops on a single EOS id; models with a multi-token stop set (e.g. Llama-3 <|eot_id|>) may differ from model.generate() in the tail.

Checklist

• Code formatted (pre-commit).
• Unit tests added (tests/core/test_speculative.py).
• Documentation updated (docs/structures.md).
• Accuracy + speed results provided (above; speed reported honestly as a v1 regression with the path to a win).
• Naming/code style follows sglang conventions.

[javierlimt6]

@DarkSharpness for your review if the proposed changes are good, i wanted a minimal implementation of speculative decoding for educational understanding

[YzXiao101]

It seems like a standalone offline demo, not integrated with minisgl runtime. Why should we land it in the main repo instead of a separate one? 🤔

[javierlimt6]

It seems like a standalone offline demo, not integrated with minisgl runtime. Why should we land it in the main repo instead of a separate one? 🤔

@YzXiao101 @DarkSharpness actually yes this standalone was intended so that the algorithm is self contained and since mini-sglang is made to be small thus i feel decoupling the logic between the algorithms made sense and would reduce the code volume / blast radius of the PR

however, if you believe that integrating this with the serving engine is more feasible to be merged ill be more than happy to work on that! this is technically the cleanest implementation and probably the end goal as well

[javierlimt6]

@YzXiao101 i made this PR as speculative decoding is such a big talking point but there are no issues or PRs about it, i thought working on a minimal implementation of speculative decoding and analysing benchmarks would be meaningful