Document benchmarking for PD deployment with GLM-5.1

README.md

@@ -20,7 +20,7 @@ ______________________________________________________________________
 
 ## 📰 News
 
-- 🔀 **2026-07-14 · [v0.1.5](https://github.com/tile-ai/TileRT/releases/tag/v0.1.5) Released**. Introduce [**PD (prefill–decode) disaggregation**](https://www.tilert.ai/blog/tilert-vllm-disaggregation.html) — vLLM prefill + TileRT decode, behind an OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2.
+- 🔀 **2026-07-14 · [v0.1.5](https://github.com/tile-ai/TileRT/releases/tag/v0.1.5.post2) Released**. Introduce [**PD (prefill–decode) disaggregation**](https://www.tilert.ai/blog/tilert-vllm-disaggregation.html) — vLLM prefill + TileRT decode, behind an OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2.
 
 - 💥 **2026-06-08 · [Breaking 1000 TPS on a 1T Model](https://www.tilert.ai/blog/breaking-1000-tps.html)**. In collaboration with [Xiaomi MiMo](https://mimo.xiaomi.com/blog/mimo-tilert-1000tps), TileRT pushes [**MiMo-V2.5-Pro-UltraSpeed**](https://platform.xiaomimimo.com/docs/en-US/model-intro/mimo-v2.5-pro-ultraspeed) past **1000 tokens/s** on a **1-trillion-parameter** model through extreme model–system co-design — a first without custom silicon, all on a single 8-GPU node.
 
@@ -70,7 +70,7 @@ ______________________________________________________________________
 
 ### Build environment of the v0.1.5 wheel
 
-The official `tilert==0.1.5.post1` wheel on PyPI was compiled against the following stack. Treat these as **hard requirements**, not lower bounds.
+The official `tilert==0.1.5.post2` wheel on PyPI was compiled against the following stack. Treat these as **hard requirements**, not lower bounds (`transformers` / `tokenizers` are lower bounds since v0.1.5.post2).
 
 | Component        | Pinned version                                      |
 | ---------------- | --------------------------------------------------- |
@@ -79,8 +79,8 @@ The official `tilert==0.1.5.post1` wheel on PyPI was compiled against the follow
 | Operating System | Linux **x86_64**, glibc **≥ 2.28** (manylinux_2_28) |
 | Python           | **3.12**                                            |
 | PyTorch          | **`torch==2.11.0+cu130`**                           |
-| `transformers`   | **`4.46.3`**                                        |
-| `tokenizers`     | **`0.20.3`**                                        |
+| `transformers`   | **`>= 4.46.3`**                                     |
+| `tokenizers`     | **`>= 0.20.3`**                                     |
 
 ### Recommended: pre-built Docker image
 
@@ -106,18 +106,18 @@ docker run --rm -it --gpus all --ipc=host \
     ghcr.io/tile-ai/tilert:cu132-latest
 
 # Inside the container — install from PyPI:
-pip install tilert==0.1.5.post1
+pip install tilert==0.1.5.post2
 
 # Or pin the exact wheel from the GitHub Release page directly
 # (same artifact, useful when PyPI is unreachable):
-pip install https://github.com/tile-ai/TileRT/releases/download/v0.1.5/tilert-0.1.5.post1-cp312-cp312-manylinux_2_28_x86_64.whl
+pip install https://github.com/tile-ai/TileRT/releases/download/v0.1.5.post2/tilert-0.1.5.post2-cp312-cp312-manylinux_2_28_x86_64.whl
 ```
 
 Verify the install:
 
 ```bash
 python -c "import tilert, torch; print('tilert', tilert.__version__, '/ torch', torch.__version__, '/ cuda', torch.version.cuda)"
-# Expected: tilert 0.1.5.post1 / torch 2.11.0+cu130 / cuda 13.0
+# Expected: tilert 0.1.5.post2 / torch 2.11.0+cu130 / cuda 13.0
 ```
 
 Proceed to [Getting Started](#getting-started) to download and convert model weights.
@@ -412,6 +412,58 @@ python -m tilert.pd_vllm.pd_router --vllm-url http://<PREFILL_IP>:8000 \
 
 **Note.** Running NIXL end to end (both the native and TileRT connectors in NIXL mode) lets the shared prefill use a single transfer library. Only the prefill's `--kv-transfer-config` differs from Topology A; the TileRT decode node is unchanged, and the native decode instance plus its proxy follow vLLM's usual `NixlConnector` disaggregation setup.
 
+### Benchmarking a PD deployment(glm5.1,vllm+tilert)
+
+Streaming is supported since v0.1.5.post2, so `vllm bench serve` can drive the
+router directly through the standard OpenAI chat backend:
+
+```bash
+vllm bench serve \
+    --backend openai-chat \
+    --base-url http://<ROUTER_IP>:23333 \
+    --model /path/to/GLM-5.1-FP8 \
+    --served-model-name glm5 \
+    --dataset-name random \
+    --random-input-len 64000 --random-output-len 3000 \
+    --random-range-ratio 0.0 \
+    --num-prompts 10 --request-rate inf \
+    --max-concurrency 1 --ignore-eos
+```
+
+#### Reference numbers: single-node co-located deployment
+
+One 8×B300 node running all three processes together — vLLM prefill, TileRT
+decode and the router share the same 8 GPUs (the router itself is CPU-only).
+GLM-5.1-FP8 with MTP, 64K input / 3K output*10, one request at a time, driven by
+the `vllm bench serve` command above:
+
+| Latency | Mean | Median | P99 |
+| ------------------------- | -------: | -------: | -------: |
+| TTFT (ms) | 5153.37 | 5466.57 | 5488.29 |
+| TPOT (ms) | 3.44 | 3.33 | 3.99 |
+| ITL, per SSE chunk (ms) | 8.44 | 10.11 | 10.79 |
+| End-to-end latency (ms) | 15471.96 | 15430.76 | 17458.84 |
+
+| Throughput | Value |
+| --------------------------- | -------- |
+| Output tokens | 193.90 tok/s |
+| Input + output tokens | 4330.34 tok/s |
+| Requests | 0.06 req/s |
+| Duration, 10 requests | 154.72 s |
+
+Reading the numbers: a mean TPOT of 3.44 ms is **~291 total tokens/s & 441 output tokens/s for the
+single in-flight request**, and the 5.15 s TTFT covers the full 64K-token vLLM
+prefill (~12.4K tokens/s) plus the KV handoff. Aggregate output throughput
+(193.9 tok/s) is lower than the per-request decode rate because prefill and
+decode are serialized at concurrency 1. The mean ITL of 8.44 ms is per SSE
+chunk, not per token — `8.44 / 3.44 ≈ 2.5` accepted tokens per MTP step.
+
+Because prefill and decode are co-located, they contend for the same SMs and
+the KV handoff never leaves the host; a two-node split trades that contention
+for a real RDMA hop.
+
+
+
 ## Status & Future Work
 
 TileRT is currently offered as a preview release, and we’re just getting started.

pyproject.toml

@@ -20,8 +20,8 @@ dependencies = [
     # https://download.pytorch.org/whl/cu130``); installing from PyPI yields a
     # CUDA build that does not match the cu130-linked tilert binary.
     "torch==2.11.0",
-    "transformers==4.46.3",
-    "tokenizers==0.20.3",
+    "transformers>=4.46.3",
+    "tokenizers>=0.20.3",
     "numpy",
     "scipy",
     "einops",

requirements.txt

@@ -8,8 +8,8 @@
 #
 # The recommended path remains the prebuilt Docker image (see README).
 torch==2.11.0
-transformers==4.46.3
-tokenizers==0.20.3
+transformers>=4.46.3
+tokenizers>=0.20.3
 numpy
 scipy
 einops

tilert/pd_vllm/pd_router.py

@@ -97,6 +97,37 @@ def _thinking_enabled(body: dict) -> bool:
     return bool(ctk.get("enable_thinking", True))
 
 
+# Client fields that must not survive into the prefill request, which is
+# forwarded verbatim apart from the fields we set: stream_options contradicts
+# the stream=False we force (vLLM rejects the pair with a 400 during body
+# parsing), and max_completion_tokens takes precedence over max_tokens, so it
+# would override our max_tokens=1. Streaming clients send both.
+_PREFILL_DROP_FIELDS = ("stream_options", "max_completion_tokens")
+
+
+def build_prefill_body(path: str, body: dict, node: DecodeNode) -> dict:
+    """The vLLM request that prefills only and hands the KV state to ``node``.
+
+    Lives outside ``build_app`` so the rewrite can be exercised without a
+    router process, a vLLM instance or a decode node.
+    """
+    prefill_body = dict(body)
+    prefill_body["max_tokens"] = 1
+    prefill_body["stream"] = False
+    for field in _PREFILL_DROP_FIELDS:
+        prefill_body.pop(field, None)
+    if path.endswith("chat/completions"):
+        prefill_body["logprobs"] = True
+        prefill_body["top_logprobs"] = 1
+    else:
+        prefill_body["logprobs"] = 1
+    prefill_body["kv_transfer_params"] = {
+        "tilert_host": node.host,
+        "tilert_ctrl_port": node.ctrl_port,
+    }
+    return prefill_body
+
+
 class RouterCtx:
     """Immutable per-process context (tokenizer, parser factory, config)."""
 
@@ -133,24 +164,13 @@ def pool_status():
 
     # ── shared prefill step ──────────────────────────────────────────────
     def _prefill(path, body, node):
-        prefill_body = dict(body)
-        prefill_body["max_tokens"] = 1
-        prefill_body["stream"] = False
-        if path.endswith("chat/completions"):
-            prefill_body["logprobs"] = True
-            prefill_body["top_logprobs"] = 1
-        else:
-            prefill_body["logprobs"] = 1
-        prefill_body["kv_transfer_params"] = {
-            "tilert_host": node.host,
-            "tilert_ctrl_port": node.ctrl_port,
-        }
+        prefill_body = build_prefill_body(path, body, node)
         r = requests.post(f"{ctx.vllm_url}{path}", json=prefill_body, timeout=600)
         r.raise_for_status()
         return r.json()
 
     def _sampling_of(body):
-        return {k: body[k] for k in ("temperature", "top_p", "top_k") if k in body}
+        return {k: body[k] for k in ("temperature", "top_p", "top_k", "ignore_eos") if k in body}
 
     def _max_tokens_of(body):
         return int(body.get("max_tokens") or body.get("max_completion_tokens") or 256)
@@ -267,6 +287,17 @@ def _chunk(delta: dict, finish=None, usage=None) -> str:
                 payload["usage"] = usage
             return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"
 
+        def _usage_chunk(usage: dict) -> str:
+            payload = {
+                "id": chunk_id,
+                "object": "chat.completion.chunk",
+                "created": int(time.time()),
+                "model": model,
+                "choices": [],
+                "usage": usage,
+            }
+            return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"
+
         def _event_delta(ev: dict) -> dict:
             if ev["kind"] == "reasoning":
                 return {"reasoning_content": ev["text"]}
@@ -357,13 +388,13 @@ async def _gen():
                         yield _chunk(_event_delta(ev))
                 if saw_tool:
                     finish_reason = "tool_calls"
-                yield _chunk(
-                    {},
-                    finish=finish_reason,
-                    usage={
+                yield _chunk({}, finish=finish_reason)
+                yield _usage_chunk(
+                    {
                         "prompt_tokens": prompt_tokens,
                         "completion_tokens": n_tokens,
-                    },
+                        "total_tokens": (prompt_tokens or 0) + n_tokens,
+                    }
                 )
                 yield "data: [DONE]\n\n"
                 completed_ok = True

tilert/pd_vllm/profiles/mla_nsa.py

@@ -364,6 +364,7 @@ def __init__(self, generator, with_mtp: bool):
         self.max_seq_len = getattr(generator.decode_layer, "max_seq_len", 200000)
         self.last_stats: dict = {}
         self.stop_ids = self._resolve_stop_ids(generator)
+        self._ignore_eos = False
 
     @staticmethod
     def _resolve_stop_ids(generator) -> set:
@@ -392,6 +393,7 @@ def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_eve
                 top_k=int(sampling.get("top_k", 256)),
                 use_topp=True,
             )
+        self._ignore_eos = bool(sampling.get("ignore_eos"))
         budget = min(int(max_tokens), self.max_seq_len - self._seq_len - 1)
         if budget <= 0:
             self.last_stats = {"finish_reason": "length"}
@@ -403,7 +405,7 @@ def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_eve
     def _decode_mtp(self, first_token_id, budget, on_token, cancel_event):
         dl = self.gen.decode_layer
         T = self.mtp_seq_len
-        stop_ids = self.stop_ids
+        stop_ids = set() if self._ignore_eos else self.stop_ids
         torch = self._torch
         tokens = [int(first_token_id)]
         if on_token:
@@ -453,7 +455,7 @@ def _decode_standard(self, first_token_id, budget, on_token, cancel_event):
         from tilert.models.deepseek_v3_2.temp_var_indices import Idx
 
         dl = self.gen.decode_layer
-        stop_ids = self.stop_ids
+        stop_ids = set() if self._ignore_eos else self.stop_ids
         torch = self._torch
         tokens = [int(first_token_id)]
         if on_token: