Skip to content

About

A dynamic, context-aware LLM routing gateway

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Phase 3 — Inference Stack

Four backends (vLLM FP16, vLLM INT4, SGLang, llama.cpp CPU) with adaptive routing, prefix caching, quantization, and speculative decoding.

Headline results: INT4 fast path 3.5× lower ITL (9.3 ms/tok), LMCache MP-mode cross-restart KV persistence, n-gram spec decode 98% acceptance on repetition-heavy tasks.

📊 Full backend comparison table

About

A dynamic, context-aware LLM routing gateway

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages