feat(pd_router): queue for a free decode node instead of failing fast - #1
Closed
CrimsonDump wants to merge 2 commits into
Closed
feat(pd_router): queue for a free decode node instead of failing fast#1CrimsonDump wants to merge 2 commits into
CrimsonDump wants to merge 2 commits into
Conversation
# 摘要 一个 decode engine 一次只服务一条序列，所以 router 给每个请求预留一个节点，池子占满 时直接返回 429。对**单客户端扇出**的负载这个语义太硬：一个 agentic 会话派生出并发的 子对话时，突发宽度大于池子但持续很短，本可以稍等即得的负载被判了 429。 `--queue-timeout <秒>` 让请求等待而不是失败；默认 `0` 保持原来的 fail-fast 行为， 所以这项改动在默认配置下是惰性的。 # 改了什么 ## `Pool` —— `Lock` 换 `Condition` `acquire` 在 `queue_timeout > 0` 时阻塞等待，`release` 归还后 `notify` 一个等待者。 等待者在 `while` 里重扫全部节点，所以不会丢唤醒。两个调用点本来就在线程池里 （非流式的 `_handle` 整个被 `run_in_threadpool` 调起；流式那条显式包了一层）， 阻塞不会卡住事件循环。 ## 修一处会让节点永久 busy 的取消泄漏 客户端断连会取消这个任务，而 `CancelledError` 继承 `BaseException`、 不被原来的 `except Exception` 捕获；流式尚未开始，生成器的 `finally` （平时的释放路径）也还不存在。于是节点的 `busy` 永不复位 —— 单节点部署下 router 之后会永远返回 429。 两条路径：**prefill 期间断连**（这条在加排队之前就存在）、**排队期间断连** （这条是排队新引入的取消点，窗口宽度就是 `--queue-timeout`）。 修法是把预留放进 `try`、`node` 先置 `None`、两个 except 都释放； `acquire` 那次 await 另包一层 `anyio.CancelScope(shield=True)`， 使断连不能打断等待本身、留下一个谁也拿不到的预留。⚠️ 只加 shield 不够：anyio 把挂起的取消投递在 shield 作用域的**出口**， 赋值已完成但 `if node is None` 那行还没执行到，所以预留必须落在 `try` 里面。 ## 可观测性与 429 语义 * 排队超过 0.1 s 打一行 `queued %.1fs for decode node %s`，否则这段等待在响应里 完全看不出来； * 429 的 body 区分两种情况：`all decode nodes busy`（fail-fast）与 `no decode node free after waiting %.1fs`（等到超时）。 ## README Topology A 的 router 段落补一段说明 `--queue-timeout` 的适用场景与默认值。 # 测试 * `Pool` 语义 16 条断言：fail-fast 兼容、超时触发、6 路 fan-out 全部成功且串行化、 多节点并行度 = 节点数、反复争用无饥饿； * 取消泄漏 6 条断言（真 anyio + 真 `Pool`，三种结构 × 两种断连时机）： 旧结构两种都漏、只加 shield 仍漏、本改动都不漏； * `pre-commit run` 全绿（isort / black / flake8+插件 / mypy / bandit / pyupgrade / codespell）。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(pd_router): queue for a free decode node instead of failing fast