CPU-Optimized vLLM: Easy, Fast LLM Inference Without a GPU

Unified CPU wheel with automatic ISA detection at runtime (AVX2, AVX-512, VNNI, BF16, AMX, NEON, FP16, DOTPROD)

This is an independent, community-maintained package — not affiliated with or funded by the vLLM project, its sister concerns, or any hardware vendors. The first successful unification of different CPU ISAs (AVX2, AVX-512, VNNI, BF16, AMX) into a single wheel was done by Mekayel Anik, for the benefit of the community.

---

---

Why vllm-cpu?

The upstream vLLM project publishes CPU wheels only on GitHub Releases with a +cpu local version suffix, which cannot be uploaded to PyPI. Users must manually copy long URLs to install. This project solves that:

Feature	Upstream (vllm)	This package (vllm-cpu)
Install	Manual URL from GitHub Releases	pip3 install vllm-cpu
PyPI	Not available (PEP 440 blocks +cpu)	Available
glibc	manylinux_2_35 (Ubuntu 22.04+)	manylinux_2_28 (Debian 10+, Ubuntu 18.04+)
Docker images	CUDA-only (vllm/vllm-openai)	CPU-optimized, multi-arch
ISA detection	Runtime auto-detect	Runtime auto-detect (same)

Quick Start

Install from PyPI

pip3 install vllm-cpu

Start an OpenAI-compatible API server

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-0.6B", device="cpu")
output = llm.generate("The future of AI is", SamplingParams(temperature=0.8, max_tokens=128))
print(output[0].outputs[0].text)

Or use the CLI

vllm serve Qwen/Qwen3-0.6B --device cpu --dtype auto

Then query it:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The future of AI is", "max_tokens": 128}'

Requirements

• Python: 3.10+ (stable ABI, one wheel for all versions)
• OS: Linux (glibc 2.28+) — Debian 10+, Ubuntu 18.04+, RHEL 8+, Amazon Linux 2023+
• CPU: x86_64 with AVX2 (minimum) or AVX-512 (optimal), or aarch64 with NEON (BF16 recommended)

Supported CPU Instructions

The unified wheel automatically detects and uses the best available instruction set:

CPU Feature	Support	Detected At
AVX2	Baseline (all x86_64)	Import time
AVX512	Optimal performance	Import time
AVX512-VNNI	INT8 acceleration	Import time
AVX512-BF16	BFloat16 native ops	Import time
AMX-BF16	Matrix acceleration (Sapphire Rapids+)	Import time
aarch64 NEON	ARM SIMD baseline	Import time
aarch64 FP16	Half-precision float	Import time
aarch64 DOTPROD	INT8 dot product acceleration	Import time
aarch64 BF16	Native BFloat16 (Graviton 3+, Ampere Altra+)	Import time

No configuration needed — the correct .so is loaded automatically at import vllm.

Install

PyPI

# Latest
pip3 install vllm-cpu

# Specific version
pip3 install vllm-cpu==0.25.1

Docker

# Docker Hub
docker pull mekayelanik/vllm-cpu:latest

# GHCR
docker pull ghcr.io/mekayelanik/vllm-cpu:latest

# Specific version
docker pull mekayelanik/vllm-cpu:0.25.1

# ARM64 without BF16 (for Graviton 2, Pi 5, older Altra)
docker pull mekayelanik/vllm-cpu:arm64-no-bf16-latest

Docker Usage

Quick start

docker run -d \
  --name vllm-cpu \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  mekayelanik/vllm-cpu:latest \
  --model Qwen/Qwen3-0.6B \
  --dtype auto

Docker Compose

services:
  vllm:
    image: mekayelanik/vllm-cpu:latest
    ports:
      - "8000:8000"
    volumes:
      - huggingface-cache:/root/.cache/huggingface
    command: ["--model", "Qwen/Qwen3-0.6B", "--dtype", "auto"]
    deploy:
      resources:
        limits:
          memory: 16g
    restart: unless-stopped

volumes:
  huggingface-cache:

Available Tags

Tag	Description
latest	Most recent stable release (multi-arch: amd64 + arm64)
X.Y.Z	Specific version (e.g., 0.19.0)
arm64-no-bf16-latest	Latest ARM64 build without BF16 instructions
arm64-no-bf16-X.Y.Z	ARM64 no-BF16 specific version (e.g., arm64-no-bf16-0.19.0)

ARM64 users: The default latest / X.Y.Z images include BF16 instructions for Graviton 3+, Ampere Altra Max, and Apple Silicon. If your ARM64 CPU lacks BF16 support (Graviton 2, Raspberry Pi 5, older Ampere Altra), use the arm64-no-bf16-* tags instead.

Supported Platforms

Platform	Wheel	Docker
x86_64 (amd64)	manylinux_2_28_x86_64	linux/amd64
aarch64 (arm64)	manylinux_2_28_aarch64	linux/arm64
aarch64 no-BF16	manylinux_2_28_aarch64 (no-bf16)	linux/arm64 (arm64-no-bf16-* tags)

How It Works

Starting with v0.17.0, vLLM ships a unified CPU wheel containing both AVX2 and AVX512 code paths:

1. The wheel includes _C.so (AVX512+BF16+VNNI+AMX) and _C_AVX2.so (AVX2 fallback)
2. At import time, vllm/platforms/cpu.py checks torch._C._cpu._is_avx512_supported()
3. The correct .so is loaded once — zero runtime dispatch overhead

Stable ABI (cp38-abi3)

The wheels use Python's stable ABI, meaning one wheel works with Python 3.10+. No per-Python-version builds needed.

Build Process

Wheels are built from source inside manylinux_2_28 containers with GCC 14, ensuring broad glibc compatibility while using modern compiler optimizations.

Registries

Registry	Image	URL
PyPI	vllm-cpu	pypi.org/project/vllm-cpu
GHCR	ghcr.io/mekayelanik/vllm-cpu	GitHub Packages
Docker Hub	mekayelanik/vllm-cpu	hub.docker.com
GitHub Releases	Wheel assets	Releases

Version Support

Version Range	Strategy	Status
v0.17.0+	Unified CPU wheel	Active
v0.8.5 -- v0.15.x	Legacy 5-variant wheels	Archived on PyPI

Deprecated Variant Packages

The following variant packages have been deprecated as of v0.16.0 (last release). Starting with v0.17.0, the unified vllm-cpu package replaces all of them with automatic ISA detection at runtime.

Package	Status	Migration
vllm-cpu-avx512	Deprecated (last: v0.16.0)	pip3 install vllm-cpu
vllm-cpu-avx512vnni	Deprecated (last: v0.16.0)	pip3 install vllm-cpu
vllm-cpu-avx512bf16	Deprecated (last: v0.16.0)	pip3 install vllm-cpu
vllm-cpu-amxbf16	Deprecated (last: v0.16.0)	pip3 install vllm-cpu

These packages remain available on PyPI for older vLLM versions but will not receive further updates.

Pipeline

Upstream vLLM release (v0.17.0+)
  --> Build unified CPU wheels in manylinux_2_28 (x86_64 + aarch64 + aarch64-no-bf16)
  --> Publish to PyPI + GitHub Releases
  --> Build multi-arch Docker images (linux/amd64 + linux/arm64)
  --> Build ARM64 no-BF16 Docker images (for CPUs without BF16 ISA)
  --> Push to GHCR + Docker Hub
  --> Promote :latest and :arm64-no-bf16-latest

Links

• vLLM Documentation
• vLLM GitHub
• Report Issues
• Changelog

License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0).

Note: The upstream vLLM project is licensed under Apache 2.0. This project (build infrastructure, Docker images, and distribution tooling) uses GPL-3.0. The vLLM library itself retains its original Apache 2.0 license.

---

Your support encourages me to keep creating/supporting my open-source projects.