mini-vllm

A from-scratch, pure-PyTorch inference engine for Qwen2.5-0.5B-Instruct, built to understand what vLLM actually does under the hood: hand-written attention, a paged KV cache, a continuous-batching scheduler, and an OpenAI-compatible server. Design rationale and the tradeoffs considered at each step are in docs/decisions.md.

Milestones

1. Forward pass matching transformers' greedy decode token-for-token (CPU, fp32).
2. Paged KV cache — fixed-size blocks from a shared pool instead of one slab per sequence.
3. Continuous-batching scheduler — admits/retires/preempts every decode step.
4. OpenAI-compatible server (/v1/completions, streaming) without losing batching.
5. Benchmark harness comparing mini-vllm against transformers (naive floor) and vLLM (production ceiling) on identical hardware.
6. Batched decode attention — one padded SDPA call over the whole ragged batch, replacing the per-sequence Python attention loop that the Milestone 5 benchmark exposed as the bottleneck.
7. Compiled decode step — torch.compile (dynamic shapes) over the whole per-token decode pipeline, plus GPU-side slot derivation and batched sampling, cutting the per-step Python/dispatch overhead that remained after Milestone 6.

Benchmark: mini-vllm vs. transformers vs. vLLM

Single RTX 4070 (12 GB), Qwen2.5-0.5B-Instruct, bf16, fixed output length (ignore_eos, 128 tokens/request) so throughput is directly comparable across engines. Full runbook in bench/README.md.

Numbers below are from the post-Milestone-7 run (compiled decode step). Earlier before/after deltas are in the Milestone 6 and 7 sections of docs/decisions.md.

Engine	Concurrency	Throughput (tok/s)	Lat p50 (s)	Lat p99 (s)	TTFT p50 (s)	TTFT p99 (s)	Peak mem (GB)	Max conc. before OOM
mini-vllm	1	65.90	1.91	2.87	0.06	0.13	8.10	n/a
mini-vllm	8	475.26	2.11	2.71	0.26	0.48	8.05	n/a
mini-vllm	32	1142.88	3.49	3.88	0.72	1.40	8.15	256
vllm	1	315.30	0.40	0.41	0.01	0.02	8.88	n/a
vllm	8	2419.20	0.42	0.43	0.02	0.04	8.88	n/a
vllm	32	7060.55	0.57	0.59	0.05	0.07	8.88	256
transformers	1	38.57	3.30	4.43	3.30	4.43	10.06	n/a
transformers	8	267.33	4.03	4.96	4.03	4.96	10.09	n/a
transformers	32	1050.11	3.84	4.17	3.84	4.17	10.25	n/a

mini-vllm throughput as % of vLLM: 20.9% at concurrency 1, 19.6% at concurrency 8, 16.2% at concurrency 32.

Caveats:

• The "max conc. before OOM" ramp (concurrency 8, 16, 24, ... until a request fails) is capped at 256 in this run rather than left uncapped. Both mini-vllm and vLLM reached 256 without a single failure, so 256 is a test ceiling, not either engine's real limit — treat the matching numbers as "at least 256," not "equally memory-efficient." transformers has no server in this harness, so it isn't measured for this column at all.
• mini-vllm now beats the naive transformers baseline at every concurrency: 1.71x / 1.78x / 1.09x of its throughput at concurrency 1 / 8 / 32 (it was 0.73-0.77x after Milestone 6, and 11x slower at concurrency 32 before that). The margin is thinnest at concurrency 32 partly because this benchmark shape flatters static batching: with short fixed-length requests (ignore_eos, 128 tokens each) no sequence finishes early, so transformers' static batches waste nothing. The remaining gap to vLLM is dominated by the full-history KV re-gather every decode step and per-sequence prefill — see the Milestone 7 section of docs/decisions.md.
• The compiled decode path adds ~5-6 minutes of one-time torch.compile warmup at server startup (four dynamic-shape graphs). Set MINIVLLM_COMPILE=0 to skip it and run eager.