v0.1.5.post3 release

README.md

@@ -20,7 +20,7 @@ ______________________________________________________________________
 
 ## 📰 News
 
-- 🔀 **2026-07-14 · [v0.1.5](https://github.com/tile-ai/TileRT/releases/tag/v0.1.5) Released**. Introduce [**PD (prefill–decode) disaggregation**](https://www.tilert.ai/blog/tilert-vllm-disaggregation.html) — vLLM prefill + TileRT decode, behind an OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2.
+- 🔀 **2026-07-14 · [v0.1.5](https://github.com/tile-ai/TileRT/releases/tag/v0.1.5.post3) Released**. Introduce [**PD (prefill–decode) disaggregation**](https://www.tilert.ai/blog/tilert-vllm-disaggregation.html) — vLLM prefill + TileRT decode, behind an OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2.
 
 - 💥 **2026-06-08 · [Breaking 1000 TPS on a 1T Model](https://www.tilert.ai/blog/breaking-1000-tps.html)**. In collaboration with [Xiaomi MiMo](https://mimo.xiaomi.com/blog/mimo-tilert-1000tps), TileRT pushes [**MiMo-V2.5-Pro-UltraSpeed**](https://platform.xiaomimimo.com/docs/en-US/model-intro/mimo-v2.5-pro-ultraspeed) past **1000 tokens/s** on a **1-trillion-parameter** model through extreme model–system co-design — a first without custom silicon, all on a single 8-GPU node.
 
@@ -70,7 +70,7 @@ ______________________________________________________________________
 
 ### Build environment of the v0.1.5 wheel
 
-The official `tilert==0.1.5.post1` wheel on PyPI was compiled against the following stack. Treat these as **hard requirements**, not lower bounds.
+The official `tilert==0.1.5.post3` wheel on PyPI was compiled against the following stack. Treat these as **hard requirements**, not lower bounds (`transformers` / `tokenizers` are lower bounds since v0.1.5.post2).
 
 | Component        | Pinned version                                      |
 | ---------------- | --------------------------------------------------- |
@@ -79,8 +79,8 @@ The official `tilert==0.1.5.post1` wheel on PyPI was compiled against the follow
 | Operating System | Linux **x86_64**, glibc **≥ 2.28** (manylinux_2_28) |
 | Python           | **3.12**                                            |
 | PyTorch          | **`torch==2.11.0+cu130`**                           |
-| `transformers`   | **`4.46.3`**                                        |
-| `tokenizers`     | **`0.20.3`**                                        |
+| `transformers`   | **`>= 4.46.3`**                                     |
+| `tokenizers`     | **`>= 0.20.3`**                                     |
 
 ### Recommended: pre-built Docker image
 
@@ -106,18 +106,18 @@ docker run --rm -it --gpus all --ipc=host \
     ghcr.io/tile-ai/tilert:cu132-latest
 
 # Inside the container — install from PyPI:
-pip install tilert==0.1.5.post1
+pip install tilert==0.1.5.post3
 
 # Or pin the exact wheel from the GitHub Release page directly
 # (same artifact, useful when PyPI is unreachable):
-pip install https://github.com/tile-ai/TileRT/releases/download/v0.1.5/tilert-0.1.5.post1-cp312-cp312-manylinux_2_28_x86_64.whl
+pip install https://github.com/tile-ai/TileRT/releases/download/v0.1.5.post3/tilert-0.1.5.post3-cp312-cp312-manylinux_2_28_x86_64.whl
 ```
 
 Verify the install:
 
 ```bash
 python -c "import tilert, torch; print('tilert', tilert.__version__, '/ torch', torch.__version__, '/ cuda', torch.version.cuda)"
-# Expected: tilert 0.1.5.post1 / torch 2.11.0+cu130 / cuda 13.0
+# Expected: tilert 0.1.5.post3 / torch 2.11.0+cu130 / cuda 13.0
 ```
 
 Proceed to [Getting Started](#getting-started) to download and convert model weights.
@@ -363,6 +363,8 @@ python -m tilert.pd_vllm.pd_router \
 
 Send OpenAI requests to `http://<router>:23333/v1/chat/completions`. The router runs the prefill on vLLM (first token), hands the attention state to the TileRT decode node over RDMA, and streams the completion back.
 
+A decode engine serves one sequence at a time, so the router reserves a node per request and answers `429` while they are all busy. Add `--queue-timeout <seconds>` to make a request wait for a free node instead of failing: useful when a single client fans out into concurrent sub-conversations — an agentic session spawning sub-agents, say — and the burst is wider than the pool but short-lived. Waits longer than 0.1 s are logged. The default, `0`, keeps the fail-fast behaviour.
+
 ### Topology B: shared prefill → TileRT decode **and** native vLLM decode
 
 One prefill pool feeds two decode pools side by side, composed under vLLM's `MultiConnector`. Each request is claimed by exactly one connector — the TileRT connector claims requests marked with `tilert_host`, and vLLM's native connector handles the rest — so latency-critical traffic goes to TileRT while general traffic stays on native vLLM decode, behind the same OpenAI surface.

pyproject.toml

@@ -20,8 +20,8 @@ dependencies = [
     # https://download.pytorch.org/whl/cu130``); installing from PyPI yields a
     # CUDA build that does not match the cu130-linked tilert binary.
     "torch==2.11.0",
-    "transformers==4.46.3",
-    "tokenizers==0.20.3",
+    "transformers>=4.46.3",
+    "tokenizers>=0.20.3",
     "numpy",
     "scipy",
     "einops",

requirements.txt

@@ -8,8 +8,8 @@
 #
 # The recommended path remains the prebuilt Docker image (see README).
 torch==2.11.0
-transformers==4.46.3
-tokenizers==0.20.3
+transformers>=4.46.3
+tokenizers>=0.20.3
 numpy
 scipy
 einops

tilert/pd_vllm/decode_server.py

@@ -17,6 +17,7 @@
 import contextlib
 import json
 import logging
+import os
 import queue as queue_mod
 import socket
 import threading
@@ -33,6 +34,9 @@
 logger = logging.getLogger("pd_vllm.decode_server")
 
 
+DECODE_POLL_S = max(0.0, float(os.environ.get("TILERT_DECODE_POLL_MS") or "200")) / 1000.0
+
+
 class DecodeBody(BaseModel):
     rid: str
     first_token_id: int
@@ -176,6 +180,12 @@ def pd_decode(body: DecodeBody):
         # streaming: ndjson lines {"t":[ids...]}* then {"done":true,...};
         # lock/engine ownership transfers to the generator.
         q: queue_mod.Queue = queue_mod.Queue()
+        fin: dict = {"loop": None, "ev": None}
+
+        def _signal_done() -> None:
+            loop, ev = fin["loop"], fin["ev"]
+            if loop is not None and ev is not None:
+                loop.call_soon_threadsafe(ev.set)
 
         def _run():
             try:
@@ -187,9 +197,11 @@ def _run():
                     cancel_event=cancel,
                 )
                 q.put(("done", tokens))
+                _signal_done()
             except Exception as e:  # pragma: no cover
                 logger.exception("stream decode failed for %s", body.rid)
                 q.put(("error", str(e)))
+                _signal_done()
 
         worker = threading.Thread(target=_run, name="pd-decode", daemon=True)
 
@@ -204,11 +216,28 @@ async def _gen():
             import anyio
             from starlette.concurrency import run_in_threadpool
 
+            fin["loop"] = asyncio.get_running_loop()
+            fin["ev"] = asyncio.Event()
             worker.start()
             try:
                 batch: list[int] = []
                 done_msg = None
                 last_activity = time.time()
+                while done_msg is None:
+                    try:
+                        first = q.get_nowait()
+                    except queue_mod.Empty:
+                        if time.time() - last_activity > 600:
+                            yield json.dumps({"error": "decode stalled"}) + "\n"
+                            return
+                        await asyncio.sleep(0.001)
+                        continue
+                    if isinstance(first, int):
+                        yield json.dumps({"t": [first]}) + "\n"
+                    else:
+                        done_msg = first
+                    last_activity = time.time()
+                    break
                 while done_msg is None:
                     drained = False
                     while True:
@@ -232,7 +261,8 @@ async def _gen():
                             yield json.dumps({"error": "decode stalled"}) + "\n"
                             return
                         else:
-                            await asyncio.sleep(0.005)
+                            with contextlib.suppress(TimeoutError):
+                                await asyncio.wait_for(fin["ev"].wait(), timeout=DECODE_POLL_S)
                 kind, payload = done_msg
                 if kind == "done":
                     timing = {