diff --git a/README.md b/README.md index fdd123e..b9a881a 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ ______________________________________________________________________ ## πŸ“° News -- πŸ”€ **2026-07-14 Β· [v0.1.5](https://github.com/tile-ai/TileRT/releases/tag/v0.1.5) Released**. Introduce [**PD (prefill–decode) disaggregation**](https://www.tilert.ai/blog/tilert-vllm-disaggregation.html) β€” vLLM prefill + TileRT decode, behind an OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2. +- πŸ”€ **2026-07-14 Β· [v0.1.5](https://github.com/tile-ai/TileRT/releases/tag/v0.1.5.post2) Released**. Introduce [**PD (prefill–decode) disaggregation**](https://www.tilert.ai/blog/tilert-vllm-disaggregation.html) β€” vLLM prefill + TileRT decode, behind an OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2. - πŸ’₯ **2026-06-08 Β· [Breaking 1000 TPS on a 1T Model](https://www.tilert.ai/blog/breaking-1000-tps.html)**. In collaboration with [Xiaomi MiMo](https://mimo.xiaomi.com/blog/mimo-tilert-1000tps), TileRT pushes [**MiMo-V2.5-Pro-UltraSpeed**](https://platform.xiaomimimo.com/docs/en-US/model-intro/mimo-v2.5-pro-ultraspeed) past **1000 tokens/s** on a **1-trillion-parameter** model through extreme model–system co-design β€” a first without custom silicon, all on a single 8-GPU node. @@ -70,7 +70,7 @@ ______________________________________________________________________ ### Build environment of the v0.1.5 wheel -The official `tilert==0.1.5.post1` wheel on PyPI was compiled against the following stack. Treat these as **hard requirements**, not lower bounds. +The official `tilert==0.1.5.post2` wheel on PyPI was compiled against the following stack. Treat these as **hard requirements**, not lower bounds (`transformers` / `tokenizers` are lower bounds since v0.1.5.post2). | Component | Pinned version | | ---------------- | --------------------------------------------------- | @@ -79,8 +79,8 @@ The official `tilert==0.1.5.post1` wheel on PyPI was compiled against the follow | Operating System | Linux **x86_64**, glibc **β‰₯ 2.28** (manylinux_2_28) | | Python | **3.12** | | PyTorch | **`torch==2.11.0+cu130`** | -| `transformers` | **`4.46.3`** | -| `tokenizers` | **`0.20.3`** | +| `transformers` | **`>= 4.46.3`** | +| `tokenizers` | **`>= 0.20.3`** | ### Recommended: pre-built Docker image @@ -106,18 +106,18 @@ docker run --rm -it --gpus all --ipc=host \ ghcr.io/tile-ai/tilert:cu132-latest # Inside the container β€” install from PyPI: -pip install tilert==0.1.5.post1 +pip install tilert==0.1.5.post2 # Or pin the exact wheel from the GitHub Release page directly # (same artifact, useful when PyPI is unreachable): -pip install https://github.com/tile-ai/TileRT/releases/download/v0.1.5/tilert-0.1.5.post1-cp312-cp312-manylinux_2_28_x86_64.whl +pip install https://github.com/tile-ai/TileRT/releases/download/v0.1.5.post2/tilert-0.1.5.post2-cp312-cp312-manylinux_2_28_x86_64.whl ``` Verify the install: ```bash python -c "import tilert, torch; print('tilert', tilert.__version__, '/ torch', torch.__version__, '/ cuda', torch.version.cuda)" -# Expected: tilert 0.1.5.post1 / torch 2.11.0+cu130 / cuda 13.0 +# Expected: tilert 0.1.5.post2 / torch 2.11.0+cu130 / cuda 13.0 ``` Proceed to [Getting Started](#getting-started) to download and convert model weights. @@ -412,6 +412,58 @@ python -m tilert.pd_vllm.pd_router --vllm-url http://:8000 \ **Note.** Running NIXL end to end (both the native and TileRT connectors in NIXL mode) lets the shared prefill use a single transfer library. Only the prefill's `--kv-transfer-config` differs from Topology A; the TileRT decode node is unchanged, and the native decode instance plus its proxy follow vLLM's usual `NixlConnector` disaggregation setup. +### Benchmarking a PD deployment(glm5.1,vllm+tilert) + +Streaming is supported since v0.1.5.post2, so `vllm bench serve` can drive the +router directly through the standard OpenAI chat backend: + +```bash +vllm bench serve \ + --backend openai-chat \ + --base-url http://:23333 \ + --model /path/to/GLM-5.1-FP8 \ + --served-model-name glm5 \ + --dataset-name random \ + --random-input-len 64000 --random-output-len 3000 \ + --random-range-ratio 0.0 \ + --num-prompts 10 --request-rate inf \ + --max-concurrency 1 --ignore-eos +``` + +#### Reference numbers: single-node co-located deployment + +One 8Γ—B300 node running all three processes together β€” vLLM prefill, TileRT +decode and the router share the same 8 GPUs (the router itself is CPU-only). +GLM-5.1-FP8 with MTP, 64K input / 3K output*10, one request at a time, driven by +the `vllm bench serve` command above: + +| Latency | Mean | Median | P99 | +| ------------------------- | -------: | -------: | -------: | +| TTFT (ms) | 5153.37 | 5466.57 | 5488.29 | +| TPOT (ms) | 3.44 | 3.33 | 3.99 | +| ITL, per SSE chunk (ms) | 8.44 | 10.11 | 10.79 | +| End-to-end latency (ms) | 15471.96 | 15430.76 | 17458.84 | + +| Throughput | Value | +| --------------------------- | -------- | +| Output tokens | 193.90 tok/s | +| Input + output tokens | 4330.34 tok/s | +| Requests | 0.06 req/s | +| Duration, 10 requests | 154.72 s | + +Reading the numbers: a mean TPOT of 3.44 ms is **~291 total tokens/s & 441 output tokens/s for the +single in-flight request**, and the 5.15 s TTFT covers the full 64K-token vLLM +prefill (~12.4K tokens/s) plus the KV handoff. Aggregate output throughput +(193.9 tok/s) is lower than the per-request decode rate because prefill and +decode are serialized at concurrency 1. The mean ITL of 8.44 ms is per SSE +chunk, not per token β€” `8.44 / 3.44 β‰ˆ 2.5` accepted tokens per MTP step. + +Because prefill and decode are co-located, they contend for the same SMs and +the KV handoff never leaves the host; a two-node split trades that contention +for a real RDMA hop. + + + ## Status & Future Work TileRT is currently offered as a preview release, and we’re just getting started. diff --git a/pyproject.toml b/pyproject.toml index 407c497..1d1600e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,8 +20,8 @@ dependencies = [ # https://download.pytorch.org/whl/cu130``); installing from PyPI yields a # CUDA build that does not match the cu130-linked tilert binary. "torch==2.11.0", - "transformers==4.46.3", - "tokenizers==0.20.3", + "transformers>=4.46.3", + "tokenizers>=0.20.3", "numpy", "scipy", "einops", diff --git a/requirements.txt b/requirements.txt index c22551d..ade8509 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8,8 +8,8 @@ # # The recommended path remains the prebuilt Docker image (see README). torch==2.11.0 -transformers==4.46.3 -tokenizers==0.20.3 +transformers>=4.46.3 +tokenizers>=0.20.3 numpy scipy einops diff --git a/tilert/pd_vllm/pd_router.py b/tilert/pd_vllm/pd_router.py index 87a61e5..e599827 100644 --- a/tilert/pd_vllm/pd_router.py +++ b/tilert/pd_vllm/pd_router.py @@ -97,6 +97,37 @@ def _thinking_enabled(body: dict) -> bool: return bool(ctk.get("enable_thinking", True)) +# Client fields that must not survive into the prefill request, which is +# forwarded verbatim apart from the fields we set: stream_options contradicts +# the stream=False we force (vLLM rejects the pair with a 400 during body +# parsing), and max_completion_tokens takes precedence over max_tokens, so it +# would override our max_tokens=1. Streaming clients send both. +_PREFILL_DROP_FIELDS = ("stream_options", "max_completion_tokens") + + +def build_prefill_body(path: str, body: dict, node: DecodeNode) -> dict: + """The vLLM request that prefills only and hands the KV state to ``node``. + + Lives outside ``build_app`` so the rewrite can be exercised without a + router process, a vLLM instance or a decode node. + """ + prefill_body = dict(body) + prefill_body["max_tokens"] = 1 + prefill_body["stream"] = False + for field in _PREFILL_DROP_FIELDS: + prefill_body.pop(field, None) + if path.endswith("chat/completions"): + prefill_body["logprobs"] = True + prefill_body["top_logprobs"] = 1 + else: + prefill_body["logprobs"] = 1 + prefill_body["kv_transfer_params"] = { + "tilert_host": node.host, + "tilert_ctrl_port": node.ctrl_port, + } + return prefill_body + + class RouterCtx: """Immutable per-process context (tokenizer, parser factory, config).""" @@ -133,24 +164,13 @@ def pool_status(): # ── shared prefill step ────────────────────────────────────────────── def _prefill(path, body, node): - prefill_body = dict(body) - prefill_body["max_tokens"] = 1 - prefill_body["stream"] = False - if path.endswith("chat/completions"): - prefill_body["logprobs"] = True - prefill_body["top_logprobs"] = 1 - else: - prefill_body["logprobs"] = 1 - prefill_body["kv_transfer_params"] = { - "tilert_host": node.host, - "tilert_ctrl_port": node.ctrl_port, - } + prefill_body = build_prefill_body(path, body, node) r = requests.post(f"{ctx.vllm_url}{path}", json=prefill_body, timeout=600) r.raise_for_status() return r.json() def _sampling_of(body): - return {k: body[k] for k in ("temperature", "top_p", "top_k") if k in body} + return {k: body[k] for k in ("temperature", "top_p", "top_k", "ignore_eos") if k in body} def _max_tokens_of(body): return int(body.get("max_tokens") or body.get("max_completion_tokens") or 256) @@ -267,6 +287,17 @@ def _chunk(delta: dict, finish=None, usage=None) -> str: payload["usage"] = usage return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n" + def _usage_chunk(usage: dict) -> str: + payload = { + "id": chunk_id, + "object": "chat.completion.chunk", + "created": int(time.time()), + "model": model, + "choices": [], + "usage": usage, + } + return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n" + def _event_delta(ev: dict) -> dict: if ev["kind"] == "reasoning": return {"reasoning_content": ev["text"]} @@ -357,13 +388,13 @@ async def _gen(): yield _chunk(_event_delta(ev)) if saw_tool: finish_reason = "tool_calls" - yield _chunk( - {}, - finish=finish_reason, - usage={ + yield _chunk({}, finish=finish_reason) + yield _usage_chunk( + { "prompt_tokens": prompt_tokens, "completion_tokens": n_tokens, - }, + "total_tokens": (prompt_tokens or 0) + n_tokens, + } ) yield "data: [DONE]\n\n" completed_ok = True diff --git a/tilert/pd_vllm/profiles/mla_nsa.py b/tilert/pd_vllm/profiles/mla_nsa.py index a270f7c..ed3c079 100644 --- a/tilert/pd_vllm/profiles/mla_nsa.py +++ b/tilert/pd_vllm/profiles/mla_nsa.py @@ -364,6 +364,7 @@ def __init__(self, generator, with_mtp: bool): self.max_seq_len = getattr(generator.decode_layer, "max_seq_len", 200000) self.last_stats: dict = {} self.stop_ids = self._resolve_stop_ids(generator) + self._ignore_eos = False @staticmethod def _resolve_stop_ids(generator) -> set: @@ -392,6 +393,7 @@ def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_eve top_k=int(sampling.get("top_k", 256)), use_topp=True, ) + self._ignore_eos = bool(sampling.get("ignore_eos")) budget = min(int(max_tokens), self.max_seq_len - self._seq_len - 1) if budget <= 0: self.last_stats = {"finish_reason": "length"} @@ -403,7 +405,7 @@ def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_eve def _decode_mtp(self, first_token_id, budget, on_token, cancel_event): dl = self.gen.decode_layer T = self.mtp_seq_len - stop_ids = self.stop_ids + stop_ids = set() if self._ignore_eos else self.stop_ids torch = self._torch tokens = [int(first_token_id)] if on_token: @@ -453,7 +455,7 @@ def _decode_standard(self, first_token_id, budget, on_token, cancel_event): from tilert.models.deepseek_v3_2.temp_var_indices import Idx dl = self.gen.decode_layer - stop_ids = self.stop_ids + stop_ids = set() if self._ignore_eos else self.stop_ids torch = self._torch tokens = [int(first_token_id)] if on_token: