Our c1 arm has a reproducible ~4 s TTFT outlier on request 3 of every leg that the pinned oracle does not have.
Found 2026-08-19 in the raw vllm bench serve --save-detailed files of the Qwen3.8-27B bf16 c1/c8 re-measure (#915, #979), out/bench-20260819T035148Z/.
The measurement
Concurrency is 1, so the six requests are strictly serialized. Read straight out of the ttfts array of each leg (seconds):
| leg |
r1 |
r2 |
r3 |
r4 |
r5 |
r6 |
| warmup |
77.005 |
0.727 |
3.981 |
0.850 |
0.845 |
0.839 |
| c1-r1 |
0.738 |
0.826 |
3.924 |
0.844 |
0.905 |
0.848 |
| c1-r2 |
0.729 |
0.808 |
4.006 |
0.915 |
0.925 |
0.852 |
| c1-r3 |
0.729 |
0.836 |
3.955 |
0.890 |
0.892 |
0.890 |
Four legs of four, always the same index. Every other request in the same legs is 0.727-0.925 s.
The pinned oracle 0.1.dev1+g555967922, on the byte-identical client invocation, same box and same lease (out/vllm-20260819T095758Z/), has no such point:
| leg |
r1 |
r2 |
r3 |
r4 |
r5 |
r6 |
| warmup |
43.767 |
0.838 |
0.842 |
0.852 |
0.846 |
0.878 |
| c1-r1 |
0.838 |
0.876 |
0.851 |
0.902 |
0.897 |
0.877 |
| c1-r2 |
0.834 |
0.941 |
0.868 |
0.874 |
0.865 |
0.917 |
| c1-r3 |
0.843 |
1.015 |
0.905 |
0.897 |
0.858 |
0.883 |
18 requests across three legs, every TTFT between 0.834 and 1.015 s.
What does not explain it
- Not prompt length. Request 3 carries a 1024-token prompt exactly as requests 4, 5 and 6 do.
- Not first inference. The 77.005 s first value in our warmup leg is the separate, already-known first-inference cost behind a liveness-only
/health; the oracle pays its own (43.767 s) at the same index. The outlier is at index 2, three requests later, and recurs in every subsequent leg after the server is fully warm.
- Not noise. A fixed request index reproducing in four legs of four is a behaviour.
What is and is not affected
Nothing published is wrong. The repository quotes the MEDIAN and labels it, and the median of six averages ranks three and four, which the outlier never occupies.
It moves the MEAN — ours 1347.6 / 1372.6 / 1365.4 ms against vLLM's 873.3 / 883.4 / 900.2 ms, while the medians read 883.78 against 876.4 — and it costs wall time: request 4 starts 31.63 s after request 3 where every other gap is ~28.4 s, so roughly 3.1 s of the 174.39 s c1 wall.
Why this is filed rather than left in the record
A reproducible outlier at a fixed request index is a behaviour rather than noise, and the record that mentions it is a benchmark row that writes no product code and holds no GPU. The cause is deliberately not chased here.
Record defect this also corrects
The finding above was appended to .agents/issue-index.md under the link #1365, in 9e1a5e573 (#1369). #1365 is a different issue — the O(n^2) BPE merge loop — so that landed row anchors this finding to unrelated work, and the index carries no row for the BPE issue's own number. The row landed on origin/main, so the append-only rule forbids editing it in place (the #979 precedent permits in-place correction only before the row lands). The correction is therefore an appended row naming this issue as the finding's real anchor.
Our c1 arm has a reproducible ~4 s TTFT outlier on request 3 of every leg that the pinned oracle does not have.
Found 2026-08-19 in the raw
vllm bench serve --save-detailedfiles of the Qwen3.8-27B bf16 c1/c8 re-measure (#915, #979),out/bench-20260819T035148Z/.The measurement
Concurrency is 1, so the six requests are strictly serialized. Read straight out of the
ttftsarray of each leg (seconds):Four legs of four, always the same index. Every other request in the same legs is 0.727-0.925 s.
The pinned oracle
0.1.dev1+g555967922, on the byte-identical client invocation, same box and same lease (out/vllm-20260819T095758Z/), has no such point:18 requests across three legs, every TTFT between 0.834 and 1.015 s.
What does not explain it
/health; the oracle pays its own (43.767 s) at the same index. The outlier is at index 2, three requests later, and recurs in every subsequent leg after the server is fully warm.What is and is not affected
Nothing published is wrong. The repository quotes the MEDIAN and labels it, and the median of six averages ranks three and four, which the outlier never occupies.
It moves the MEAN — ours 1347.6 / 1372.6 / 1365.4 ms against vLLM's 873.3 / 883.4 / 900.2 ms, while the medians read 883.78 against 876.4 — and it costs wall time: request 4 starts 31.63 s after request 3 where every other gap is ~28.4 s, so roughly 3.1 s of the 174.39 s c1 wall.
Why this is filed rather than left in the record
A reproducible outlier at a fixed request index is a behaviour rather than noise, and the record that mentions it is a benchmark row that writes no product code and holds no GPU. The cause is deliberately not chased here.
Record defect this also corrects
The finding above was appended to
.agents/issue-index.mdunder the link#1365, in9e1a5e573(#1369).#1365is a different issue — the O(n^2) BPE merge loop — so that landed row anchors this finding to unrelated work, and the index carries no row for the BPE issue's own number. The row landed onorigin/main, so the append-only rule forbids editing it in place (the #979 precedent permits in-place correction only before the row lands). The correction is therefore an appended row naming this issue as the finding's real anchor.