From 7fa7894b60060088af0431b1ca80d489161c2935 Mon Sep 17 00:00:00 2001
From: Ettore Di Giacinto <mudler@localai.io>
Date: Wed, 19 Aug 2026 12:13:42 +0000
Subject: [PATCH 1/2] record(BENCH-QWEN38-27B-BF16): our arm completes 162/162,
 and the two cells stay open for two DIFFERENT reasons

#915's c1/c8 debt was withheld because our server dropped requests. #931 landed
that fix, and this records the re-measure that discharges OUR half of it: three
reps at each concurrency on an idle leased box, 162 of 162 requests completed,
`failed=0` and zero non-empty `errors` on every leg. c1 output throughput
4.4040 tok/s (CV 0.039%), c8 22.6402 tok/s (CV 0.205%).

Neither cell became a ratio, and the reason differs per cell, which is why they
are recorded separately rather than as one open gap.

At c1 both arms completed everything and `gpu_clock_state compare` returned
PAIRING_VERDICT=DISCARD on all three pairings. The shape of that refusal is
recorded precisely because reading it as "the arms disagree" inverts it: the
cross-arm rule passed perfectly, same boot and both arms at a 2489 MHz median
with a 0.0% offset, and the WITHIN-RUN rule failed on both, 13.58/26.36/14.34%
for us and 10.16/17.48/18.52% for vLLM against a 5% ceiling, with SwThermal
throttling in every window. All six of our legs and all three of vLLM's breached
that ceiling and the record says so beside the numbers, because stable medians
do not launder a breach. No ours-over-vLLM ratio is derived anywhere. The ratio
is OWED, not withheld for being unflattering: the instrument that decides
whether a pair may be divided refused the pair.

At c8 the vLLM denominator is NOT MEASURABLE on this box at the recorded
configuration, and that is the ANSWER to the cell rather than a gap in it. The
KV reservation took 48,715 MB in a single 4-second window, the last observed
value was 6,261 MB, and the worker died inside one 2-second sampling interval.
So this box leaves 6-7 GB of headroom at the recorded knobs and a sampling
watchdog cannot guard it at any floor that still lets the configuration run.
Every way to create that headroom is an engine knob that would change the
denominator, so none was attempted. This is a statement about headroom and
guard granularity here, not a claim that vLLM is defective.

Two findings outlive the campaign and are recorded where a future reader hits
them rather than only in this entry. Clock pinning is unavailable inside an `rc`
lease -- `nvidia-smi -lgc` returns LGC_RC=4 as root -- and every clock-pinned
figure in these records was taken over the retired host+ssh+flock path, so the
migration to leases removed the capability and no record said so. That is the
root cause of the discarded pairing, it is filed as #1354, and it is written
into `.agents/environment.md` and beside the instruction it contradicts in
`.agents/benchmarking.md`. And a guard set inside a configuration's own
operating point manufactures the finding it was meant to detect: a 12,000 MB
watchdog killed a healthy server whose own arithmetic predicts an 11,917 MB
floor, a 306 MB match, and reporting that as "the denominator collapses in a
lease" would have been a fabricated finding about vLLM.

One divergence was found in the raw files that nobody was looking for and it is
filed as #1355: our server reports 5,942 prompt tokens where vLLM reports 6,144
for the identical generated prompts. `output_lens` is [128]xN on both arms, so
the throughput and per-token figures here stand; total-token throughput does
not, and whether we under-report usage or truncate the prompt is not decidable
from these artifacts.

Whether the HOST rebooted or only the k3s pod was lost when the c8 worker died
is UNDETERMINED and recorded as owed, with the one command that settles it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
---
 .agents/benchmark-record.md                | 324 +++++++++++++++++++++
 .agents/benchmarking.md                    |   8 +
 .agents/environment.md                     |  57 ++++
 .agents/issue-index.md                     |   2 +
 .agents/specs/bench-qwen38-27b-four-way.md |  41 +++
 .agents/specs/qwen38-27b-bf16-gate.md      |  68 ++++-
 docs/BENCHMARKS.md                         |  20 +-
 docs/STATUS.md                             |  52 +++-
 8 files changed, 561 insertions(+), 11 deletions(-)

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
index 3ce9df22d..e9e3d1e2c 100644
--- a/.agents/benchmark-record.md
+++ b/.agents/benchmark-record.md
@@ -24128,3 +24128,327 @@ registry still counts probe refusals only and a replay-time refusal would abort
 `mem/mem_<tag>.csv` (the per-call `cudaMemGetInfo` trace with node counts),
 `mem/smp_<tag>.csv` (the `nvidia-smi` and RSS sampler) and `out-bytes/ids_<tag>.json`
 (the token artifacts).
+
+## BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE — our arm lands complete, the c1 pairing is DISCARDED on clock spread, and the c8 vLLM denominator is NOT MEASURABLE on this box (2026-08-19, `row/BENCH-QWEN38-27B-BF16-RESULT`, `dgx:gpu0` GB10 sm_121a via `rc` leases, #915 / #979, cause #931)
+
+Subject `Qwen/Qwen3.8-27B` @ `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`, bf16,
+`config.json` `architectures = ['Qwen3_5ForConditionalGeneration']`,
+`model_type = qwen3_5`, 18 shards. Ours built at `1dac4f9a70195b282d16c536f319e8b171c925f8`
+(`origin/main` at run time), source tarball sha256
+`c74c45d1dc910087c313f5f29b81b4cbc6f8d649886618f2399d7988b0bd2fc0`, binary sha256
+`7d0c3cafb224f66ef6789f40df3caeb2b22987605922911fffcaa3e188344b0a` asserted
+`WANT == GOT` at launch. Oracle `0.1.dev1+g555967922`, the pin, used as the
+CLIENT for both arms and as the server for the vLLM arm. Client identity read
+from `vllm.__file__` in the lease and printed per series.
+
+Workload, identical string on both arms: `vllm bench serve --backend openai
+--endpoint /v1/completions --dataset-name random --random-input-len 1024
+--random-output-len 128 --random-range-ratio 0 --num-prompts <6 x C>
+--max-concurrency C --request-rate inf --ignore-eos --temperature 0 --seed 0`.
+Fresh server per concurrency, one untimed discarded warmup, then 3 reps.
+Ours: `--device cuda --max-model-len 2048 --max-num-seqs 32
+--max-num-batched-tokens 8192 --no-enable-prefix-caching --language-model-only`,
+with `VT_SERVER_SSE_PING_S=0` exported. vLLM: the recorded #915 configuration
+UNMODIFIED — `--gpu-memory-utilization 0.85 --max-num-seqs 32
+--max-num-batched-tokens 8192 --max-model-len 2048 --no-enable-prefix-caching
+--language-model-only --mamba-ssm-cache-dtype float32 --seed 0`.
+
+Box state asserted rather than assumed. Both series printed
+`COTENANTS_AT_SERIES_START=0` and `IDLE_BOX=YES` from a
+`nvidia-smi --query-compute-apps` count, and exit 38 instead of measuring when
+that count is non-zero. Both teardowns printed `COMPUTE_APPS_AFTER_TEARDOWN=0`
+and `TEARDOWN_VERDICT=CLEAN`.
+
+### OUR ARM — three clean legs at each concurrency, every request completed
+
+`out/bench-20260819T035148Z/`. Medians over 3 reps; CV is the population
+coefficient of variation over the same three values.
+
+| Axis | c1 | c8 |
+|---|---:|---:|
+| completed / num_prompts, per rep | 6, 6, 6 of 6 | 48, 48, 48 of 48 |
+| `failed`, per rep | 0, 0, 0 | 0, 0, 0 |
+| non-empty `errors` entries | 0 | 0 |
+| output token throughput, median | **4.4040 tok/s** | **22.6402 tok/s** |
+| output token throughput, CV over 3 reps | 0.039% | 0.205% |
+| total token throughput, median | 38.4776 tok/s | 196.0967 tok/s |
+| median TPOT | 218.11 ms | 250.57 ms |
+| median ITL | 216.56 ms | 232.83 ms |
+| median TTFT | 883.78 ms | 3623.5 ms |
+| median E2EL | 28,586.6 ms | 35,098.0 ms |
+| cold start to first `/health` | 30 s | 54 s |
+| SM clock median over the timed window | 2489 MHz, all three legs | 2515 MHz, all three legs |
+| within-run SM-clock spread | 13.58% / 26.36% / 14.34% | 14.99% / 13.44% / 12.92% |
+
+162 of 162 requests completed across the whole series, zero failures. The floor
+of `MemAvailable` over 1,077 samples spanning both legs was 21,100 MB.
+
+### vLLM ARM c1 — three clean legs, on the production graphed configuration
+
+`out/vllm-20260819T095758Z/`. The configuration is read back from the engine's
+own startup line rather than from the command line: `enforce_eager=False`,
+`cudagraph_mode: FULL_AND_PIECEWISE`, `cudagraph_capture_sizes [1, 2, 4, 8, 16,
+24, 32, 40, 48, 56, 64]`, `FLASH_ATTN` (FlashAttention version 2),
+`dtype=torch.bfloat16`, `enable_chunked_prefill=True`. That is vLLM's production
+shape and not `--enforce-eager`.
+
+| Axis | c1 |
+|---|---:|
+| completed / num_prompts, per rep | 6, 6, 6 of 6 |
+| `failed`, per rep | 0, 0, 0 |
+| output token throughput, median | **4.2835 tok/s** |
+| output token throughput, CV over 3 reps | 0.033% |
+| total token throughput, median | 38.5516 tok/s |
+| median TPOT | 228.36 ms |
+| median ITL | 226.86 ms |
+| median TTFT | 876.4 ms |
+| cold start to first `/health` | 426 s |
+| SM clock median over the timed window | 2489 MHz, all three legs |
+| within-run SM-clock spread | 10.16% / 17.48% / 18.52% |
+
+### NO RATIO IS DERIVED AT c1, AND THE REASON IS THE WITHIN-RUN RULE
+
+`tools/bench/gpu_clock_state.py compare` returned rc=1 with a non-empty
+`reasons` list on all three c1 pairings. `PAIRING_VERDICT=DISCARD`. The shape of
+the refusal is unusual and is recorded precisely, because reading it as "the two
+arms disagree" inverts it:
+
+- **The cross-arm rule passed perfectly.** Same boot id
+  `3fd9745a-d25a-426c-ba3c-97c958a85515` on both arms. Median SM clock 2489 MHz
+  on BOTH arms in all three reps, a median offset of **0.0%** against a 1%
+  ceiling. Same GPU (`NVIDIA GB10`), same driver `580.173.02`, same
+  `clocks.max.sm` 3003 MHz, same `clocks.applications.graphics` 2418 MHz, same
+  persistence mode `Enabled`.
+- **The within-run rule failed on BOTH arms.** Ours 13.58% / 26.36% / 14.34%,
+  vLLM 10.16% / 17.48% / 18.52%, against the 5% ceiling
+  `.agents/benchmarking.md` sets. `SwThermalSlowdown` (`0x20`) is active in
+  EVERY one of the nine windows across both arms and both concurrencies; our c1
+  rep 2 additionally carries `HwSlowdown + SwThermal + HwThermal` (`0x68`).
+
+So both c1 absolutes are recorded above as facts with their own clock blocks,
+and **no ours-over-vLLM ratio is derived from them, here or anywhere else**.
+The ratio is OWED, not withheld because it is unflattering: the two absolutes
+happen to sit within a few percent of each other, and the reason there is no
+number is that the instrument that decides whether a pair may be divided
+refused the pair. A number the clock gate discarded is not a number.
+
+The window itself was observed rather than nominal: 155-163 retained busy
+samples per c1 window and 244-246 per c8 window, against the 30-sample floor,
+with 15-26 idle samples excluded per window.
+
+An observation that is an ARGUMENT and not a licence: three c1 legs with
+spreads of 13.58%, 26.36% and 14.34%, one of them hardware-throttled, produced
+output throughput within 0.039% and median TPOT within 0.007% of each other.
+That is evidence this batch-1 path is not clock-limited. It says nothing about
+c8, nothing about a different kernel mix, and nothing about any other boot, and
+it does not convert a DISCARD into a pairing.
+
+### THE c8 vLLM DENOMINATOR IS NOT MEASURABLE ON THIS BOX AT THE RECORDED CONFIGURATION
+
+This is the ANSWER to the c8 denominator question, not a gap in it, and it is a
+statement about headroom and guard granularity on this box. **It is not a claim
+that vLLM is defective**, and no reader may take it as one.
+
+The c8 vLLM server loaded, reached `/health` after 373 s (launched 10:18:54 UTC,
+first `GET /health 200 OK` logged at 10:25:07 UTC), and the worker was then lost
+during the untimed warmup, before any timed leg ran. The memory trajectory is
+the finding, from the series' own 2-second `MemAvailable` sampler
+(`out/vllm-20260819T095758Z/mem.samples`, epoch seconds and MB):
+
+| Sample time (UTC) | MemAvailable | What it is |
+|---|---:|---|
+| 10:18:54 | 116,869 MB | before launch |
+| 10:24:41 | 58,453 MB | weights loaded, compile and capture done |
+| 10:24:43 | 38,708 MB | the KV reservation, first step |
+| 10:24:45 | 9,738 MB | the KV reservation, second step |
+| 10:25:07 | ~9,950 MB | first `GET /health 200 OK` |
+| 10:25:26 | 6,261 MB | last observed value; no sample after it |
+
+**48,715 MB left in a single 4-second window.** The last observed value was
+6,261 MB and the worker died inside one 2-second sampling interval.
+
+CONSEQUENCE, stated narrowly. At `--gpu-memory-utilization 0.85
+--max-num-batched-tokens 8192`, graphed, this box leaves roughly **6-7 GB of
+headroom**, and the fall from healthy to dead is faster than a 2-second sampler
+can resolve. **A sampling watchdog is therefore not a viable guard for the c8
+denominator here at ANY floor that still lets the configuration run**: 12,000 MB
+kills a healthy server (below), and 5,000 MB was never reached before the worker
+was lost — `watchdog.log` is empty, zero bytes, because the last value the
+sampler saw was above it.
+
+Every way to create that headroom is an ENGINE KNOB. Lowering
+`gpu_memory_utilization` or `max_num_batched_tokens` produces a
+surviving-but-different engine whose number is not the denominator #915 and #979
+ask for. That is why none was attempted, and why the honest result is
+NOT MEASURABLE rather than a number taken at a configuration nobody recorded.
+
+For scale on the same box and the same workload, ours held a `MemAvailable`
+floor of 21,100 MB across both of its legs.
+
+**UNDETERMINED, and recorded as owed rather than guessed.** Whether the HOST
+rebooted or only the k3s pod was lost when the worker died is not decidable from
+these artifacts. One command settles it: read
+`/proc/sys/kernel/random/boot_id` inside any later `dgx:gpu0` job and compare
+against `3fd9745a-d25a-426c-ba3c-97c958a85515`. A different value means the host
+rebooted.
+
+### FINDING THAT OUTLIVES THIS CAMPAIGN 1 — CLOCK PINNING IS UNAVAILABLE INSIDE AN `rc` LEASE
+
+Measured 2026-08-19 on `dgx:gpu0`, in a job running as root in the `rc` worker
+pod:
+
+```text
+$ nvidia-smi -lgc 2190
+The current user does not have permission to change clocks for GPU 0000000F:01:00.0.
+LGC_RC=4
+```
+
+Reproduced in both attempts of the vLLM arm and in our arm, three jobs.
+
+`.agents/benchmarking.md` instructs "Pin the clocks before measuring, under the
+lock", and **every clock-pinned figure in this repository's records was taken
+over the retired host + `ssh` + `flock` path**. The migration to `rc` leases
+silently removed clock pinning and no record said so. Same class as
+[#1265](https://github.com/mudler/vllm.cpp/issues/1265): a capability the
+records assume, which the current access path does not provide. Inside a lease
+the SM clock can only be SAMPLED. `tools/bench/gpu_clock_state.py` remains
+usable and is the only attribution these numbers carry — and it is what turned
+this campaign's c1 pairing into a DISCARD.
+
+Not the first sighting: this record already carries the refusal once, as a
+per-run note under `ENG-CUDAGRAPH-DEDUP W6`. What is new is that the refusal is
+a property of the ACCESS PATH rather than of one run, and that a task guide
+still tells the next reader to do the impossible thing. Recorded in
+`.agents/environment.md` beside the other measured lease capabilities, and
+`.agents/benchmarking.md` now names the exception where it instructs the pin.
+
+### FINDING THAT OUTLIVES THIS CAMPAIGN 2 — A GUARD SET INSIDE A CONFIGURATION'S OWN OPERATING POINT MANUFACTURES THE FINDING IT WAS MEANT TO DETECT
+
+`out/vllm-20260819T073125Z/`, the FIRST vLLM attempt. Its `MemAvailable`
+watchdog fired 18 s after the server reached `/health` and killed it:
+
+```text
+WATCHDOG: MemAvailable 11917MB < 12000MB -- KILLING vllm pgid=110154 to save the box
+```
+
+The obvious reading — "the recorded denominator configuration collapses inside a
+lease" — was WRONG, and wrong in the most dangerous way, because it matched the
+hypothesis already held and would have been believed. The arithmetic refutes it:
+
+| Quantity | Value |
+|---|---:|
+| `MemTotal` | 122,502 MB |
+| `0.85 x MemTotal`, reserved BY DESIGN | 104,127 MB |
+| `MemAvailable_MB_before_server`, from the job log | 116,350 MB |
+| predicted free after the reservation | 12,223 MB |
+| OBSERVED floor at which the guard fired | 11,917 MB |
+| difference | **306 MB** |
+
+The 46 GB "collapse" is the KV reservation doing exactly what
+`gpu_memory_utilization` configures, which on GB10's unified pool comes out of
+host RAM. The watchdog floor of 12,000 MB sat ABOVE the configuration's own
+steady-state free memory, so it fired on a healthy server. Attempt 2 confirmed
+it directly: with the floor moved to 5,000 MB, the c1 legs ran to completion at
+a steady state around 7,500 MB — BELOW the floor of attempt 1.
+
+**THE REUSABLE RULE.** A tripped guard is evidence about the GUARD until its
+threshold is shown to sit outside the guarded thing's operating point. Predict
+the number from the configuration and compare; a 306 MB match settled in one
+line what no amount of log-reading would have. And distinguish an ENGINE KNOB
+from an INSTRUMENT THRESHOLD: `gpu_memory_utilization` and
+`max_num_batched_tokens` define the denominator and may never be tuned to make a
+run survive, while the watchdog floor appears nowhere in the engine
+configuration and changes nothing about what is measured. Between attempt 1 and
+attempt 2 the floor moved 12,000 -> 5,000 MB and **no engine knob moved**.
+
+Two further defects in the same harness, both repaired between the attempts: the
+watchdog `break`s after firing, which would have left every later leg
+unprotected; and the reps kept driving a dead server, recording
+`completed=0 / failed=6 / duration 0.013 s` three times.
+
+### WHAT #915's TWO WITHHELD CELLS LOOK LIKE NOW
+
+Against the previously recorded cells (2026-08-15, a DIFFERENT boot
+`03717c9d-63c8-4652-a8fe-a63d012c5718`, clocks PINNED flat at 2184 MHz):
+
+| Axis | prior | this series | completion, prior -> now |
+|---|---:|---:|---|
+| c1 output tok/s, ours | 2.37 | 4.4040 | 5, 5, 5 of 6 -> 6, 6, 6 of 6 |
+| c1 median TPOT ms, ours | 220.6 | 218.11 | as above |
+| c8 output tok/s, ours | 15.96 | 22.6402 | 36, 37, 36 of 48 -> 48, 48, 48 of 48 |
+| c8 median TPOT ms, ours | 261.1 | 250.57 | as above |
+
+**The SHAPE of that difference is the diagnosis, and the MAGNITUDE is not
+established by these two rows.** The throughput axis moves a great deal while
+the per-token axis moves 1-4%. That is exactly what
+[#931](https://github.com/mudler/vllm.cpp/issues/931) predicts: the per-token
+axis was always measuring the engine, because it is computed over completed
+requests, while `output_throughput` was dividing live tokens by a wall duration
+that still contained the dead requests. Withholding rather than publishing
+0.677x was correct, and this is the evidence for that judgement.
+
+The magnitude is NOT established because the two rows come from different boots
+with different clock attribution — 2184 MHz pinned then, 2489/2515 MHz sampled
+and thermally throttled now, which is 14-15% apart on the median alone. Per
+`.agents/benchmarking.md`, a ratio is valid only between two arms measured in
+the same window, and these are not. Read the direction and the shape; do not
+read the percentage.
+
+**One caveat on the #931 fix.** The measured arm exported
+`VT_SERVER_SSE_PING_S=0` explicitly. So this series demonstrates that the
+keepalive frame was the cause and that disabling it removes the drops. It does
+NOT independently demonstrate that the shipped DEFAULT is `0`, because the
+harness never depended on the default.
+
+### A DIVERGENCE THE CAMPAIGN DID NOT SET OUT TO FIND — THE TWO ARMS REPORTED DIFFERENT PROMPT-TOKEN COUNTS
+
+Read out of the raw result files rather than out of any summary. On the
+IDENTICAL client invocation, seed and dataset, `input_lens` (which
+`vllm/benchmarks/lib/endpoint_request_func.py:247` overwrites from the SERVER's
+`usage.prompt_tokens`) differs between the arms:
+
+| Leg | ours | vLLM |
+|---|---|---|
+| c1, per request | `[915, 931, 1024, 1024, 1024, 1024]`, sum **5,942** | `[1024] x 6`, sum **6,144** |
+| c8, distribution | 29 of 48 at 1024, 19 short (877-941), sum **47,072** | not measured |
+
+Deterministic: byte-identical `input_lens` across all three c1 reps and all
+three c8 reps. `vllm bench serve` re-aligns prompts against the server's own
+`/tokenize` when the first prompt disagrees
+(`vllm/benchmarks/serve.py:2041-2044`), and it printed no `tokenizer mismatch`
+warning on either arm, so our `/tokenize` agreed on 1024 while our
+`usage.prompt_tokens` reported 915 for the same request.
+
+**What this does and does not touch.** `output_throughput` is
+`total_output_tokens / duration`, and `output_lens` is `[128] x N` on BOTH arms
+in every leg, so the output-throughput, TPOT and ITL figures above are
+unaffected. `total_token_throughput` has input tokens in its numerator and IS
+affected: our c8 figure of 196.10 tok/s is computed over 47,072 input tokens
+where the intended workload is 49,152.
+
+**Two causes remain open and the artifacts cannot separate them**: our server
+under-reports `usage.prompt_tokens`, or our server actually processed a
+truncated prompt. Filed, not fixed in flow — this row writes no product code
+and holds no GPU.
+
+### EVIDENCE
+
+`/mnt/nas_share/rc/q38bf16/` — `NOTES.txt` (the campaign's own 253-line
+provenance and reasoning file, the primary source for this entry), `build.sh`,
+`bench.sh`, `job.sh`, `vllm-arm.sh`, `reap-orphans.sh`, `STAGED-SHA256.txt`
+(hashes of all five), `src-1dac4f9a7.tar.gz` with `src.sha256`, and under `out/`:
+`RESULT.txt`, `build.log`, `cfg.log`, `bench-20260819T035148Z/` (our arm: six
+result JSONs, six `clock-*.json` windows with their raw samples, `CLOCKS.txt`,
+`SUMMARY.txt`, `mem.samples`, `job.log`, `ckpt-sha256.txt`,
+`ckpt-resolved.txt`, `client-identity.txt`, per-leg server logs),
+`vllm-20260819T073125Z/` (attempt 1, including the non-empty `watchdog.log`),
+and `vllm-20260819T095758Z/` (attempt 2: three c1 JSONs and clock windows, both
+server logs, `mem.samples`, the empty `watchdog.log`).
+
+**Two inconsistencies inside the evidence directory, resolved in favour of the
+executing artifact.** `NOTES.txt` states a binary sha256 of `ab0b9a1e...`; the
+job log asserted and printed `7d0c3caf...` as both `WANT` and `GOT` at launch,
+and `out/RESULT.txt` agrees, so `7d0c3caf...` is the binary that ran and
+`NOTES.txt` carries a stale value. `NOTES.txt` gives the checkpoint as
+55,586,040,114 bytes where `.agents/specs/qwen38-27b-bf16-gate.md` records
+55,586,114,863; neither was re-derived here and the difference is not adjudicated.
diff --git a/.agents/benchmarking.md b/.agents/benchmarking.md
index 8d3d957b2..b46c25c6d 100644
--- a/.agents/benchmarking.md
+++ b/.agents/benchmarking.md
@@ -81,6 +81,14 @@ sudo nvidia-smi -lgc 2100        # pin, before the first leg
 sudo nvidia-smi -rgc             # release, after the last one
 ```
 
+**This works on the host path only.** Inside an `rc` lease `nvidia-smi -lgc`
+returns `LGC_RC=4`, "The current user does not have permission to change
+clocks", even as root, measured 2026-08-19 on `dgx:gpu0` in three jobs. Fleet
+devices are reachable by lease only, so for them the SM clock can be SAMPLED and
+not pinned, and a pairing may be refused on within-run spread with no lever to
+fix it. Read
+[`environment.md`](environment.md) before you plan a paired series.
+
 Pinning is a **shared-host mutation**. Never run `-lgc` or `-rgc` while another
 session holds `$HOME/gpu.lock` — it silently reprices their in-flight
 measurement, which is the very defect this section exists for. Take the lock,
diff --git a/.agents/environment.md b/.agents/environment.md
index 00923f5af..8f04ee26b 100644
--- a/.agents/environment.md
+++ b/.agents/environment.md
@@ -145,6 +145,63 @@ already has one. This row measured `dgx`'s worker and adds the part that turns
 an open gap into a blocker for the parity gates, which is that the ORACLE VENV
 is also unreachable from a lease.
 
+### Clock pinning does NOT work inside an `rc` lease, measured 2026-08-19
+
+[#1354](https://github.com/mudler/vllm.cpp/issues/1354) owns this.
+
+`nvidia-smi -lgc` is refused inside the leased worker, in a job running as
+**root**:
+
+```text
+$ nvidia-smi -lgc 2190
+The current user does not have permission to change clocks for GPU 0000000F:01:00.0.
+LGC_RC=4
+```
+
+Reproduced in three separate `rc run` jobs on `dgx:gpu0` on 2026-08-19
+(`.agents/benchmark-record.md`, `BENCH-QWEN38-27B-BF16` c1/c8). The container is
+root but lacks the capability the driver requires.
+
+**This outlives one campaign, and it is a records defect as much as a capability
+one.** [`benchmarking.md`](benchmarking.md) instructs "Pin the clocks before
+measuring, under the lock", and **every clock-pinned figure in this
+repository's records was taken over the retired host + `ssh` + `flock` path**.
+The migration to `rc` leases silently removed clock pinning and no record said
+so. Same class as [#1265](https://github.com/mudler/vllm.cpp/issues/1265): a
+capability the records assume, which the current access path does not provide.
+
+**Consequence for anyone measuring from a lease.** The SM clock can only be
+SAMPLED, never pinned. `tools/bench/gpu_clock_state.py` still works and is the
+only attribution such a number carries. Plan for its within-run rule to fail:
+the 2026-08-19 series recorded within-run spreads of 12.92% to 26.36% across
+nine windows on a thermally throttling GB10, against the 5% ceiling, and
+`gpu_clock_state compare` therefore returned `PAIRING_VERDICT=DISCARD` on every
+c1 pairing even though the cross-arm rule passed perfectly. Two arms measured in
+a lease may not be dividable, so budget for absolutes and say so in advance
+rather than discovering it after the GPU time is spent.
+
+**A model DOES run inside a lease.** The same series ran the pinned oracle
+`0.1.dev1+g555967922` as a server on a 52 GiB bf16 checkpoint from a lease, no
+`ssh` and no container image, and it served three clean benchmark legs. That
+retires the "nobody has run a model that way" reading of
+[#1185](https://github.com/mudler/vllm.cpp/issues/1185) recorded above. What is
+still unreachable is the image-based path SGLang needs
+([#1265](https://github.com/mudler/vllm.cpp/issues/1265)).
+
+**And one instrument rule, paid for in the same series.** A guard whose
+threshold sits inside the guarded configuration's own operating point
+manufactures the finding it was meant to detect. A 12,000 MB `MemAvailable`
+watchdog killed a healthy `vllm serve` 18 s after it reached `/health`, which
+read as "the denominator collapses in a lease" — and the arithmetic refuted it:
+`0.85 x 122,502 MB` reserved by design predicts 12,223 MB free against an
+observed floor of 11,917 MB, a 306 MB match, and the same server later ran three
+clean legs at a ~7,500 MB steady state. **A tripped guard is evidence about the
+GUARD until its threshold is shown to sit outside the guarded thing's operating
+point.** Predict the number from the configuration before believing the verdict,
+and keep instrument thresholds strictly apart from engine knobs: moving the
+floor changes nothing about what is measured, and moving
+`gpu_memory_utilization` changes what the number means.
+
 ### The lease carries bytes, and the exec bit is a mount option, measured 2026-08-17
 
 Probed with two `rc run -d dgx:gpu0 --max-runtime 3m` jobs,
diff --git a/.agents/issue-index.md b/.agents/issue-index.md
index d556cd0af..a227bff27 100644
--- a/.agents/issue-index.md
+++ b/.agents/issue-index.md
@@ -427,3 +427,5 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1325](https://github.com/mudler/vllm.cpp/issues/1325) | `ENG-RECORD-ANCHOR-RATCHET` | `scripts/record-anchor-baseline.json` stores a `"total"` that no code reads. `load_record_anchor_baseline` in `scripts/check-agent-record.py` returns `{bucket: int(data["buckets"][bucket]) for bucket in RECORD_ANCHOR_BUCKETS}` with `RECORD_ANCHOR_BUCKETS = ("stale", "broken")`, and it is the file's only reader; `check_record_anchors` iterates those two buckets, and `write_record_anchor_baseline`'s refusal compares `result.total > sum(previous.values())` — the buckets, not the stored `total`. Measured at `af87251c5`: mutating `"total": 38` to `39` (a file whose total disagrees with `32 + 6`) leaves `check-agent-record.py` exit 0, tree restored byte-for-byte by sha256. So this row's own budget file carries the exact shape the row exists to name: a recorded figure no gate reads, sitting beside the figures that are read and presenting as if it were checked. `--write-baseline` compounds it by printing `-> 38`, which reads as the value it stored and is the one no later run consults. TWO candidate resolutions, deliberately not chosen here because choosing belongs to the fixing row: read it and assert `total == stale + broken` on load, or drop the field and derive it at read time, which is the shape AGENTS.md §Records prefers. Either is a semantic change to `check-agent-record.py` owing a spec and a red-before case in `tests/scripts/test_agent_record.py` `RecordAnchorRatchet`. Distinct from [#1287](https://github.com/mudler/vllm.cpp/issues/1287) and [#1270](https://github.com/mudler/vllm.cpp/issues/1270), neither of which reaches the unread field. Also under `## Owed` in [`record-anchor-ratchet.md`](specs/record-anchor-ratchet.md) | bug |
 | [#1316](https://github.com/mudler/vllm.cpp/issues/1316) | — | `scripts/main-baseline.py` renders a scheduled run that executed ZERO jobs as `RED` with all 11 covered jobs `missing`, so `NEWEST BASELINE: RED at <sha>` names a tree the run never checked out. Measured at `origin/main` `250db75a2`: runs `32206456661` and `32140419182` both return `startedAt: null` for every job, because GitHub cancelled them while they were pending in the single `ci-schedule-refs/heads/main-mudler/vllm.cpp` group, whose queue holds one run ([#274](https://github.com/mudler/vllm.cpp/issues/274)). Fail-closed, and the `missing (expected, never ran)` line is accurate about the jobs; the defect is the verdict word, because a run that executed nothing is NOT RUN rather than RED, and the newest verdict should fall through to the newest run that actually ran. NOT fixed in flow: `test_an_expected_job_the_payload_never_mentions_is_red` and `test_a_narrowed_run_reports_red_and_names_what_never_ran` deliberately assert missing-is-red so a narrowed run cannot pass, and separating "narrowed" from "never started" changes what the verdict means, which owes its own spec, red-before evidence and a fresh reviewer. Owed under `## Owed` of [baseline-lane-eviction.md](specs/baseline-lane-eviction.md), which removes the only observed producer of a zero-job run | bug |
 | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature |
+| [#1354](https://github.com/mudler/vllm.cpp/issues/1354) | — | Clock pinning is UNAVAILABLE inside an `rc` lease. `nvidia-smi -lgc 2190` returns `LGC_RC=4`, "The current user does not have permission to change clocks for GPU 0000000F:01:00.0", in three separate `rc run` jobs on `dgx:gpu0` on 2026-08-19, each running as **root** in the worker pod. `.agents/benchmarking.md` instructs "Pin the clocks before measuring, under the lock" and ships `sudo nvidia-smi -lgc 2100` as the recipe, and **every clock-pinned figure in this repository was taken over the host + `ssh` + `flock` path** that AGENTS.md now forbids for a fleet device — so the migration to leases silently removed clock pinning and no record said so. Same class as [#1265](https://github.com/mudler/vllm.cpp/issues/1265): a capability the records assume, which the current access path does not provide. Measured cost: nine timed windows across two arms recorded within-run SM-clock spreads of 12.92% to 26.36% against the 5% ceiling, `SwThermalSlowdown` active in every one and `HwSlowdown+HwThermal` in one, so `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three Qwen3.8-27B c1 pairings **even though the cross-arm rule passed perfectly** (same boot, both arms 2489 MHz median, 0.0% offset). The cell therefore has two clean complete absolutes and no ratio. NOT fixed in flow: the fix is either an `rc` worker capability this row has no authority over, or a demonstrated settle-and-hold procedure, or a ratified different clock rule for lease-measured pairs — each its own spec, and none of them a widening of the assertion to turn a red green. Records updated meanwhile in `.agents/environment.md`, `.agents/benchmarking.md` and `.agents/benchmark-record.md`. Owed under `## Owed` in [bench-qwen38-27b-four-way.md](specs/bench-qwen38-27b-four-way.md) | bug |
+| [#1355](https://github.com/mudler/vllm.cpp/issues/1355) | — | Our server reports 5,942 prompt tokens where vLLM reports 6,144 for the IDENTICAL generated prompts. Found 2026-08-19 in the raw `vllm bench serve --save-detailed` files of the Qwen3.8-27B bf16 re-measure ([#915](https://github.com/mudler/vllm.cpp/issues/915)), both arms driven by the byte-identical client invocation from the same pinned wheel `0.1.dev1+g555967922`, same dataset, same seed. `input_lens` is the SERVER-reported length — `vllm/benchmarks/lib/endpoint_request_func.py:247` overwrites `output.prompt_len` from the streamed `usage.prompt_tokens` — and reads `[915, 931, 1024, 1024, 1024, 1024]` for us against `[1024] x 6` for vLLM at c1, with 19 of 48 short (877-941) at c8, byte-identical across all three reps of each leg. Not the client: `_align_prompts_to_server_tokenizer` (`vllm/benchmarks/serve.py:74,2041-2044`) re-aligns against the server's own `/tokenize` and prints `WARNING: tokenizer mismatch` when it disagrees, and NEITHER arm printed it, so our `/tokenize` agreed on 1024 while our `usage.prompt_tokens` reported 915 for the same request. `output_lens` is `[128]xN` on both arms in every leg, so the campaign's output-throughput, TPOT and ITL figures are unaffected; `total_token_throughput` is affected, our c8 196.10 tok/s being computed over 47,072 input tokens where the intended workload is 49,152. TWO causes and the artifacts cannot separate them: under-reported usage, or a genuinely truncated prompt — and the second would mean the two arms did not run the same workload. A greedy token gate cannot see either, which is why it survived the gate on this checkpoint. NOT fixed in flow: the finding row writes no product code and holds no GPU. Owed under `## Owed` in [qwen38-27b-bf16-gate.md](specs/qwen38-27b-bf16-gate.md) | bug |
diff --git a/.agents/specs/bench-qwen38-27b-four-way.md b/.agents/specs/bench-qwen38-27b-four-way.md
index 30ffbf29f..ccdc130d2 100644
--- a/.agents/specs/bench-qwen38-27b-four-way.md
+++ b/.agents/specs/bench-qwen38-27b-four-way.md
@@ -719,6 +719,36 @@ without a rewrite this branch is forbidden to perform. Under
 on a squash in any case, so the pull request body is the correction of record for
 those, and sections 2.2 and 3 here are the correction of record for the substance.
 
+**FIRST MEASURED CELL, 2026-08-19, and one pair that cannot exist.** The
+`ours vs vLLM` bf16 pair of section 2.4's matrix has now been attempted on
+`Qwen/Qwen3.8-27B` @`1d4bf0f2`, both arms in an `rc` lease on `dgx:gpu0`, and
+the two concurrencies resolve differently:
+
+- **c1 produced both absolutes and no ratio.** Ours 4.4040 tok/s, vLLM 4.2835
+  tok/s, three reps each, every request completed on both arms.
+  `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three
+  pairings because the within-run SM-clock spread breached the 5% ceiling on
+  both arms, so the ratio is OWED rather than derived.
+- **c8's vLLM denominator is NOT MEASURABLE on this box at the recorded
+  configuration**, and that is the cell's answer rather than a gap in it. The
+  server reached `/health`, then the KV reservation took 48,715 MB in one
+  4-second window and the worker died with 6,261 MB left. Every way to create
+  the missing headroom is an engine knob that would change the denominator.
+  This is a statement about this box, not about vLLM.
+
+The measurement, the clock blocks, the memory trajectory and the two findings
+that outlive the campaign are in
+[`../benchmark-record.md`](../benchmark-record.md).
+
+**A campaign premise is falsified in passing.** `.agents/environment.md`
+recorded that the pinned oracle builds inside a lease but that "nobody has run a
+model that way, so no oracle-side MEASUREMENT is unblocked yet"
+([#1185](https://github.com/mudler/vllm.cpp/issues/1185)). It ran. Three clean
+c1 legs of `vllm serve` on a 52 GiB checkpoint, from a lease, with no `ssh` and
+no container image. What is still blocked is `sglang`, which needs the image
+path [#1265](https://github.com/mudler/vllm.cpp/issues/1265) forbids, and the
+c8 point on this hardware.
+
 ## Owed
 
 - [#979](https://github.com/mudler/vllm.cpp/issues/979) owns this campaign and is
@@ -742,3 +772,14 @@ those, and sections 2.2 and 3 here are the correction of record for the substanc
   assertions named in section 2.5: no e4m3 scale byte with its sign bit set, and
   the GDN head permutation at `conversion/qwen.py:378-386` shown to be a
   permutation of the same codes.
+- [#1354](https://github.com/mudler/vllm.cpp/issues/1354): **clock pinning is
+  unavailable inside an `rc` lease**, so every remaining pair in section 2.4 has
+  the same exposure this one hit — a within-run spread the clock gate refuses,
+  with no lever to reduce it. Recorded in
+  [`../environment.md`](../environment.md).
+- Whether the HOST rebooted or only the k3s pod was lost when the c8 vLLM worker
+  died. Read `/proc/sys/kernel/random/boot_id` inside a later `dgx:gpu0` job and
+  compare against `3fd9745a-d25a-426c-ba3c-97c958a85515`.
+- A c8 vLLM denominator for this checkpoint from a box with more than 6-7 GB of
+  headroom at `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192`.
+  Unowned, and deliberately not obtained by tuning either knob.
diff --git a/.agents/specs/qwen38-27b-bf16-gate.md b/.agents/specs/qwen38-27b-bf16-gate.md
index fcb3d6f8d..bc4f014df 100644
--- a/.agents/specs/qwen38-27b-bf16-gate.md
+++ b/.agents/specs/qwen38-27b-bf16-gate.md
@@ -101,10 +101,49 @@ is the claim. What it adds is evidence, listed under Evidence required.
 
 ## Now
 
-`PARTIAL`. The token axis is closed and passing. The speed axis is closed only
-at **c4** (0.963x throughput, 1.008x ITL); c1 and c8 throughput are withheld until
-[#931](https://github.com/mudler/vllm.cpp/issues/931) closes, because our server
-drops requests there and vLLM does not. Nothing else advances this row.
+`PARTIAL`, and it stays `PARTIAL` after the 2026-08-19 re-measure. The token
+axis is closed and passing. The speed axis is closed only at **c4** (0.963x
+throughput, 1.008x ITL).
+
+**Our half of the c1/c8 debt is DISCHARGED.**
+[#931](https://github.com/mudler/vllm.cpp/issues/931) is closed and the fix
+holds on hardware: with `VT_SERVER_SSE_PING_S=0`, three reps at each
+concurrency on an idle leased box completed **162 of 162** requests with
+`failed=0` and zero non-empty `errors` entries on every leg. c1 output
+throughput **4.4040 tok/s** (CV 0.039%), median TPOT 218.11 ms, median ITL
+216.56 ms, median TTFT 883.78 ms. c8 output throughput **22.6402 tok/s**
+(CV 0.205%), total token throughput 196.10 tok/s, median TPOT 250.57 ms, median
+ITL 232.83 ms, median TTFT 3623.5 ms. Both teardowns `TEARDOWN_VERDICT=CLEAN`.
+
+**Neither cell became a ratio, and the two halves are blocked differently.** At
+c1 the pinned oracle `0.1.dev1+g555967922` also completed every request on its
+production graphed configuration (`enforce_eager=False`,
+`cudagraph_mode: FULL_AND_PIECEWISE`, capture sizes `[1..64]`, `FLASH_ATTN`,
+read back from its own startup line): **4.2835 tok/s** (CV 0.033%), median TPOT
+228.36 ms, median ITL 226.86 ms, median TTFT 876.4 ms, cold start 426 s. Both
+absolutes stand as facts with their own clock blocks, and
+`gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three
+pairings, so **no ratio is derived — the c1 ratio is OWED, not withheld for
+being unflattering**.
+
+The refusal is about spread, not disagreement. Cross-arm: same boot
+`3fd9745a-d25a-426c-ba3c-97c958a85515`, both arms at a 2489 MHz median, offset
+**0.0%**. Within-run: ours 13.58/26.36/14.34%, vLLM 10.16/17.48/18.52%, against
+the 5% ceiling, with `SwThermalSlowdown` in every window and
+`HwSlowdown+HwThermal` in one of ours. **All six of our legs and all three of
+vLLM's breached that ceiling**, c8's included (12.92-14.99%).
+
+**At c8 the vLLM denominator is NOT MEASURABLE on this box at the recorded
+configuration.** That is the c8 answer, not a gap in it, and it is a statement
+about headroom and guard granularity here rather than a claim that vLLM is
+defective. Detail, trajectory and arithmetic in
+[`../benchmark-record.md`](../benchmark-record.md).
+
+**What advances this row next** is a c1 pairing taken in a window the clock gate
+will accept, which needs either clock pinning (unavailable in a lease, below) or
+a thermally quiet window, and a c8 denominator taken somewhere with more than
+6-7 GB of headroom at the recorded configuration. Neither is reachable from
+`dgx:gpu0` today.
 
 ## Outcome
 
@@ -206,3 +245,24 @@ reproduction recording HTTP status and exception class is the next step.
 nothing about its NVFP4 or Q4_K_M arms (#821), no claim that #910 is fixed (this
 is the second checkpoint to be costed by it), and no throughput number at c1 or
 c8 until #931 closes. Concurrencies above 8 were not run.
+
+## Owed
+
+- [#915](https://github.com/mudler/vllm.cpp/issues/915) stays OPEN. Our arm's
+  c1/c8 debt is discharged; the vLLM half is not. The c1 ratio is refused by the
+  clock gate and the c8 denominator does not exist, so the row's own question —
+  what our speed is against vLLM's production configuration at c1 and c8 — is
+  still unanswered.
+- Whether the HOST rebooted or only the k3s pod was lost when the c8 vLLM worker
+  died on 2026-08-19. The artifacts cannot distinguish them. Settle it by
+  reading `/proc/sys/kernel/random/boot_id` inside any later `dgx:gpu0` job and
+  comparing it against `3fd9745a-d25a-426c-ba3c-97c958a85515`.
+- [#1355](https://github.com/mudler/vllm.cpp/issues/1355), the prompt-token
+  divergence found in this campaign's raw result files: our server reported
+  5,942 prompt tokens where vLLM reported 6,144 for the identical
+  client-generated prompts, 19 of 48 short at c8. Whether we under-report
+  `usage.prompt_tokens` or actually truncate the prompt is not decidable from
+  these artifacts.
+- The checkpoint size disagrees between records: this spec's `## Outcome` says
+  55,586,114,863 bytes and the campaign's `NOTES.txt` says 55,586,040,114.
+  Neither was re-derived on 2026-08-19.
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
index 6ea4e3a9a..80a24a5b4 100644
--- a/docs/BENCHMARKS.md
+++ b/docs/BENCHMARKS.md
@@ -57,7 +57,7 @@ The first series free of both, at the pin, graphed, and at a pinned clock is in
 | Qwen3.6-35B-A3B | NVFP4 `modelopt_mixed` | 0.25.0 ROLLBACK, SUPERSEDED | 2/18 | 3-rep grid 2026-08-05 @`1ea26427`: 0.93-1.03x, VOID as ratios (#520, #414). At the pin, clocks pinned: **0.995x c1 / 0.946x c4 TPOT**. ★ probe found a prod async batch-1 greedy DEGENERATION bug the mirror fixes |
 | DeepSeek-V2-Lite | bf16 MLA | 0.25.0 ROLLBACK, SUPERSEDED | 4/25 | Attributed miss, row stays `ACTIVE` |
 | Qwen3.5-4B | bf16 direct-load | 0.26.0.dev0 | **1.0283x tput, `PENDING`** | OPEN: TTFT/TPOT/E2E 1.085/1.017/1.029x, VRAM +118.7 MiB ([data](bench-evidence/qwen35-4b-sm120-main-20260807.md)) |
-| Qwen3.8-27B | bf16 (@`1d4bf0f2`) | 0.26.0.dev0 at the pin, graphed, clocks 2184 MHz | **1 of 3 concurrency cells** | Token gate PASSES. Only c4 is like-for-like: tput **0.963x**, ITL **1.008x**. c1/c8 tput NOT ESTABLISHED: we failed 1/6 and ~11/48 requests, vLLM none ([#931](https://github.com/mudler/vllm.cpp/issues/931)) |
+| Qwen3.8-27B | bf16 (@`1d4bf0f2`) | 0.26.0.dev0 at the pin, graphed; c4 at a pinned 2184 MHz, the 2026-08-19 re-measure SAMPLED only | **1 of 3 concurrency cells** | Token gate PASSES. c4 like-for-like: tput **0.963x**, ITL **1.008x**. c1 re-measured COMPLETE, pairing DISCARDED on clock spread; c8 vLLM denominator NOT MEASURABLE ([#915](https://github.com/mudler/vllm.cpp/issues/915)) |
 
 ### GDN prefill kernels by GPU
 
@@ -197,9 +197,9 @@ record. The same P0 hit classic dense `Qwen3ForCausalLM` (quant-independent), fi
 | Axis | c1 | c4 | c8 |
 |---|---:|---:|---:|
 | Requests completed, ours / vLLM | 5,5,5 / 6,6,6 of 6 | 24,24,24 / 24,24,24 of 24 | 36,37,36 / 48,48,48 of 48 |
-| Output token throughput | NOT ESTABLISHED, we dropped requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | **0.963x** | NOT ESTABLISHED, we dropped requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) |
+| Output token throughput | NOT ESTABLISHED as a ratio; both arms' absolutes are re-measured below | **0.963x** | NOT ESTABLISHED as a ratio; ours is re-measured below and vLLM has no leg |
 | Total token throughput | NOT ESTABLISHED | **0.918x** | NOT ESTABLISHED |
-| Status of the two withheld cells | cause LANDED: the drop was our own SSE keepalive frame, and `VT_SERVER_SSE_PING_S` now defaults to 0 ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | (cell stands) | still withheld, now awaiting the paired 3-rep re-run owed by [#915](https://github.com/mudler/vllm.cpp/issues/915), not a diagnosis |
+| Status of the two withheld cells | SUPERSEDED 2026-08-19 by the re-measure rows below: the cause was our SSE keepalive frame and our arm now completes every request ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | (cell stands) | SUPERSEDED: our arm is complete, and the missing half is now vLLM's, for a different reason ([#915](https://github.com/mudler/vllm.cpp/issues/915)) |
 | Median ITL, over completed only | 1.013x | **1.008x** | 1.021x |
 | Median TPOT, over completed only | 1.014x | 0.980x | 0.925x |
 | Median TTFT, over completed only | 0.733x | 0.881x | 1.268x |
@@ -210,6 +210,20 @@ record. The same P0 hit classic dense `Qwen3ForCausalLM` (quant-independent), fi
 | Why that ratio is not like-for-like | ours answers `/health` on process liveness only (`api_server.cpp:286-294`); no dummy run, no kernel warmup, decode CUDA graph captures lazily on first use | vLLM warms up and captures before it serves (`gpu_worker.py:697-708`, `api_server.py:780-785`) | 53 s is "weights loaded", 780 s is "warmed and graph-captured" |
 | What our readiness signal defers | first request TTFT **91.613 s**, the same with the SSE keepalive on and off, so it is genuine first-inference cost, not the [#931](https://github.com/mudler/vllm.cpp/issues/931) defect | cell stands as measured; a like-for-like comparison has not been taken | forensics in `.agents/benchmark-record.md` |
 | Host memory after warmup | **42.5 vs 110.1 GiB = 2.59x**, but vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV, so it is what the configured engine holds, not what the model needs | | |
+| **RE-MEASURED 2026-08-19, absolutes only** ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979)) | ours **4.4040 tok/s** (CV 0.039%), vLLM **4.2835 tok/s** (CV 0.033%) | (cell stands) | ours **22.6402 tok/s** (CV 0.205%); vLLM **NOT MEASURABLE**, below |
+| Requests completed on the re-measure | ours 6,6,6 of 6; vLLM 6,6,6 of 6; `failed=0` every leg | (cell stands) | ours 48,48,48 of 48, `failed=0`; 162/162 over the whole series |
+| Median TPOT / ITL on the re-measure | ours 218.11 / 216.56 ms; vLLM 228.36 / 226.86 ms | (cell stands) | ours 250.57 / 232.83 ms |
+| **Why no c1 ratio is quoted** | `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings. The ratio is OWED, not withheld for being unflattering | (cell stands) | no vLLM leg exists to pair |
+| What the clock gate actually refused | cross-arm rule PASSED: same boot, both arms 2489 MHz, median offset **0.0%**. The within-run rule FAILED on both arms | (cell stands) | ours 12.92-14.99% spread, same breach |
+| Within-run SM-clock spread, 5% ceiling | ours 13.58 / 26.36 / 14.34%, vLLM 10.16 / 17.48 / 18.52%; `SwThermal` in every window, `HwSlowdown+HwThermal` in one | (cell stands) | **all six of our legs and all three vLLM legs breached it** |
+| Clocks were SAMPLED, never pinned | `nvidia-smi -lgc` returns `LGC_RC=4`, permission denied, as root inside an `rc` lease. Every prior pinned figure used the retired `ssh`+`flock` path | (cell stands) | same lease, same refusal |
+| **vLLM c8 denominator** | (c1 landed) | (cell stands) | **NOT MEASURABLE on this box at the recorded config**, which is the answer and not a gap. Not a claim that vLLM is defective |
+| Evidence for that verdict | `/health` reached at 373 s, then the worker was lost during warmup | (cell stands) | the KV reservation took **48,715 MB in one 4 s window** (38,708 -> 9,738); last value 6,261 MB; death inside one 2 s sample |
+| Why no watchdog can guard it | ~6-7 GB headroom at `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192` | (cell stands) | 12,000 MB kills a healthy server, 5,000 MB is never reached in time. Every other fix is an engine knob that changes the denominator |
+| Undetermined, recorded as owed | whether the HOST rebooted or only the k3s pod was lost. These artifacts cannot tell | (cell stands) | settle it by reading `/proc/sys/kernel/random/boot_id` against `3fd9745a-...85515` inside a later job |
+| #915's own-arm debt, DISCHARGED | prior 2.37 -> 4.4040 tok/s while median TPOT moved 220.6 -> 218.11 | (cell stands) | prior 15.96 -> 22.6402 while TPOT moved 261.1 -> 250.57. Different boots, so read the SHAPE, not the percentage |
+| What that shape proves | throughput moved a lot and per-token 1-4%: the throughput axis was dividing live tokens by a wall still holding dead requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | (cell stands) | withholding 0.677x rather than publishing it was correct, and this is the evidence for that |
+| A divergence found in the raw files | our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts; 19 of 48 short at c8 | (cell stands) | affects total-token throughput only: `output_lens` is `[128]xN` on both arms, so tput/TPOT/ITL stand |
 | Why only c4 counts | `output_throughput` divides tokens by a wall duration that still contains the dead request, so a cell where one arm dropped requests is withheld, not quoted | 3 paired reps, clocks 2184 MHz | token gate: 4/7 strict, 3 exact fp32 ties ([#915](https://github.com/mudler/vllm.cpp/issues/915)) |
 
 ### Qwen3.8-27B quantized arms, both gates PENDING and no number quoted
diff --git a/docs/STATUS.md b/docs/STATUS.md
index d9625ec06..ba24a6338 100644
--- a/docs/STATUS.md
+++ b/docs/STATUS.md
@@ -976,10 +976,54 @@ those throughput cells are **withheld, not quoted**
 tokens by a duration that still contains the dead request, which is why c1 reads
 0.677x while median TPOT in the same file reads 1.014x in our favour.
 
-**That cause has landed, and the two cells are now waiting on a re-run rather
-than on a diagnosis.** The dropped requests were our own SSE keepalive comment
-frame, and `VT_SERVER_SSE_PING_S` now defaults to `0`. Both cells stay withheld
-until [#915](https://github.com/mudler/vllm.cpp/issues/915) re-runs them paired.
+**That cause has landed, and the re-run happened on 2026-08-19.** The dropped
+requests were our own SSE keepalive comment frame, and `VT_SERVER_SSE_PING_S`
+now defaults to `0`. With it off, our arm completed **162 of 162** requests,
+three reps at each concurrency, `failed=0` on every leg: c1 output throughput
+**4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%).
+
+**Our half of the debt is discharged. Neither cell is a ratio, and each is
+blocked for its own reason.** At c1 vLLM also completed every request
+(**4.2835 tok/s**, CV 0.033%) and both absolutes stand, but
+`gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three
+pairings. **The c1 ratio is OWED, not withheld for being unflattering.**
+
+The refusal is about clock spread, not about the arms disagreeing. The
+cross-arm rule passed perfectly — same boot, both arms at a 2489 MHz median,
+offset **0.0%** — while the within-run rule failed on both, at
+13.58/26.36/14.34% for us and 10.16/17.48/18.52% for vLLM against a 5% ceiling,
+with `SwThermalSlowdown` active in every window and one of ours also carrying
+`HwSlowdown+HwThermal`. All six of our legs and all three of vLLM's breached
+that ceiling, and stable medians do not launder it.
+
+**At c8 the vLLM denominator is NOT MEASURABLE on this box at the recorded
+configuration, and that is the answer rather than a gap in it.** The server
+reached `/health` at 373 s and the worker was then lost during warmup. The KV
+reservation took **48,715 MB in a single 4-second window** (38,708 -> 9,738 MB),
+the last observed value was 6,261 MB, and the death fell inside one 2-second
+sampling interval.
+
+So `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192` leaves roughly
+6-7 GB of headroom here, and a sampling watchdog cannot guard it at any floor
+that still lets the configuration run: 12,000 MB kills a healthy server and
+5,000 MB is never reached in time. Every way to create that headroom is an
+engine knob that would change the denominator, so none was attempted. **This is
+a statement about headroom and guard granularity on this box, not a claim that
+vLLM is defective.**
+
+**Clocks were SAMPLED, never pinned, and that is new.** `nvidia-smi -lgc`
+returns `LGC_RC=4`, "The current user does not have permission to change
+clocks", running as root inside an `rc` lease. Every clock-pinned figure in
+these records was taken over the retired host + `ssh` + `flock` path, so the
+migration to leases removed clock pinning and no record said so until now. Same
+class as [#1265](https://github.com/mudler/vllm.cpp/issues/1265), and the root
+cause of the discarded pairing.
+
+**Undetermined, and recorded as owed rather than guessed.** Whether the host
+rebooted or only the k3s pod was lost when the worker died cannot be decided
+from these artifacts. One command settles it: compare
+`/proc/sys/kernel/random/boot_id` against `3fd9745a-d25a-426c-ba3c-97c958a85515`
+inside any later `dgx:gpu0` job.
 
 Resource axes on the same series: cold start to first `/health` **53 s vs
 780 s = 14.7x**, and host memory after warmup **42.5 vs 110.1 GiB = 2.59x** —

From a6d83d670f3b0237d19d5ae2169392071390e102 Mon Sep 17 00:00:00 2001
From: Ettore Di Giacinto <mudler@localai.io>
Date: Wed, 19 Aug 2026 12:57:47 +0000
Subject: [PATCH 2/2] record(BENCH-QWEN38-27B-BF16): the KV parenthetical names
 the half-length step, and a c8 fact sat in the c1 column

Every headline number in this entry reproduces from the raw JSONs. What did not
survive review is the compression into the public documents, and every repair
here is to the wording rather than to a measurement.

`docs/BENCHMARKS.md` and `docs/STATUS.md` both wrote the KV reservation as
"48,715 MB in a 4-second window (38,708 -> 9,738)". Subtraction falsifies it:
from `vllm-20260819T095758Z/mem.samples` the window is 10:24:41 58,453 ->
10:24:43 38,708 -> 10:24:45 9,738, so 58,453 - 9,738 = 48,715 over 4 s while the
pair that was printed is 28,970 over 2 s. The parenthetical named the endpoints
of the SECOND, half-length step while carrying the magnitude and duration of the
full one. `NOTES.txt` and the record both render it correctly, so the defect was
introduced by the compression and nowhere else.

The `docs/BENCHMARKS.md` c1 cell of "Evidence for that verdict" carried two
c8-vLLM facts. vLLM's c1 cold start was 426 s (`job.log:27
VLLM_COLD_START_TO_HEALTH_S=426`) and all three c1 legs completed, and the row
directly above already reads `(c1 landed)`. The two rows below used the c1 column
as overflow prose in the same way. A keyed table's columns are a contract, and
`check-public-doc-tables` measures shape rather than attribution, so nothing
could have caught this.

The row's own spec quoted `total token throughput 196.10 tok/s` as a clean fact
while its `## Owed` described #1355 without ever connecting it to that axis.
`.agents/benchmark-record.md` carried 196.0967, 38.4776 and 38.5516 unmarked with
the caveat two hundred lines below. The c1 pair is the dangerous one: our 38.4776
sat directly beside the oracle's 38.5516 on the exact axis the divergence
corrupts, with ours deflated by 202 missing prompt tokens, so the oracle read
ahead on an axis that is not comparable. All four figures are now marked where
they appear.

"Affects total-token throughput only" was stronger than the evidence, and the
bias it dismissed is larger than the precision published beside the figures.
Under genuine truncation a shorter prompt means less prefill, a shorter wall and
therefore an INFLATED `output_throughput`. Re-derived here: at c1 the wall equals
the sum of the per-request E2ELs to within 1 ms, the marginal prefill cost over
the 15 non-outlier points is 1.10 ms/token, and the two short prompts sit
0.21-0.23 s below the 1024-token TTFT mean, so ~0.23 s of 174.39 s, about 0.13%,
against a published CV of 0.039%. At c8 the first wave prefills at 1,300-1,800
tok/s, so 2,080 missing tokens are ~1.1-1.6 s of 271.0 s, about 0.4-0.6%, against
a published CV of 0.205%. Both are lower bounds, since a shorter context also
cheapens decode.

The two published TTFT medians (883.78 ms and 876.4 ms) are comparable, and no
record said why. They are: our two short prompts produce the two LOWEST TTFTs in
all three reps, so the median of six, which averages ranks three and four, falls
on a 1024-token request on both arms. The conclusion was right by luck, and it is
now right by evidence.

Four records contradicted themselves. `.agents/environment.md` still left
standing, in bold, that the lease-compliant path "still holds for a MODEL RUN"
and that "nobody has run a model that way", twenty lines above the section this
same campaign added recording that the pinned oracle SERVED a 52 GiB checkpoint
from a lease; corrected in place, as that paragraph's own pattern does.
`.agents/model-matrix.md`, the OWNING matrix row, still stated the withholding in
the present tense with no correction and no pointer. The checkpoint-size
disagreement was recorded as "neither was re-derived", which the run's own
artifact contradicts: `bench-20260819T035148Z/job.log:47,49` print
`CKPT_SRC_BYTES=55586040114` and `CKPT_DST_BYTES=55586040114`, agreeing with
`NOTES.txt` and differing from the spec by 74,749 bytes, so what is unresolved is
WHY they differ and not whether anyone measured. And "the per-token axis moves
1-4%" is a cross-boot percentage taken across the very 2184 MHz-pinned /
2489 MHz-sampled boundary the same paragraph declares non-dividable; it is now
qualitative, in the record and in the scoreboard cell that repeated it.

Two sourcing corrections. The memory-trajectory table's `before launch` row is
presented as coming from `mem.samples` and is not there (`grep -c 116869` returns
0); it comes from `job.log:214 MemAvailable_MB_before_server`. And `NOTES.txt:208`
gives the c8 cold start as 374 s where the record derives 373 s from the log's own
timestamps -- a fourth disagreement with that file that went unflagged while the
others were flagged.

One new finding is filed rather than left in the record. #1365: our c1 arm has a
reproducible ~4 s TTFT outlier on request 3 of EVERY leg -- 3.981 / 3.924 / 4.006
/ 3.955 s across the warmup leg and all three reps, against 0.73-0.93 s for every
other request in the same leg, on a 1024-token prompt exactly like requests 4, 5
and 6. The oracle has no such point in 18 requests. Nothing published is wrong,
because this repository quotes the labelled MEDIAN and the median of six averages
ranks three and four, which the outlier never occupies; it moves the MEAN (ours
1347.6-1372.6 ms against 873.3-900.2 ms) and costs roughly 3.1 s of the 174.39 s
wall. A reproducible outlier at a FIXED request index in four legs of four is a
behaviour, not noise. The cause is deliberately not chased.

`docs/FEATURES.md` carried the same stale claim as the matrix row, "c1/c8
withheld (#931)", and it is corrected in the same commit. That is also what
`check-doc-checkpoint --base` requires per commit: a `.agents/model-matrix.md`
edit is a support-surface claim owing `docs/FEATURES.md`, and a
`.agents/benchmark-record.md` measurement owes `docs/STATUS.md` and
`docs/BENCHMARKS.md`. This work first landed as three commits, two of which paid
only part of that, so it is one commit instead. The one-shot
`check-doc-checkpoint.py` cannot see the difference, because it validates the
committed HEAD rather than each commit in a range.

The c1 prefill deficit is quoted at ~0.22 s rather than at the top of its range.
The three reps give 0.20, 0.23 and 0.20 s between the two short prompts' TTFTs
and the 1024-token mean of 884.6 ms, and 202 tokens at the 1.1042 ms/token
regression slope is 0.22 s. A repair whose subject is a figure quoted more
precisely than its evidence supports cannot itself do that.

No measured value changed, and no ratio is derived anywhere.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
---
 .agents/benchmark-record.md           | 93 +++++++++++++++++++++++----
 .agents/environment.md                | 11 +++-
 .agents/issue-index.md                |  1 +
 .agents/model-matrix.md               |  2 +-
 .agents/specs/qwen38-27b-bf16-gate.md | 42 ++++++++++--
 docs/BENCHMARKS.md                    | 12 ++--
 docs/FEATURES.md                      |  2 +-
 docs/STATUS.md                        | 10 ++-
 8 files changed, 145 insertions(+), 28 deletions(-)

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
index e9e3d1e2c..2d1b6e0c3 100644
--- a/.agents/benchmark-record.md
+++ b/.agents/benchmark-record.md
@@ -24171,7 +24171,7 @@ coefficient of variation over the same three values.
 | non-empty `errors` entries | 0 | 0 |
 | output token throughput, median | **4.4040 tok/s** | **22.6402 tok/s** |
 | output token throughput, CV over 3 reps | 0.039% | 0.205% |
-| total token throughput, median | 38.4776 tok/s | 196.0967 tok/s |
+| total token throughput, median — **CORRUPTED, [#1355](https://github.com/mudler/vllm.cpp/issues/1355); NOT comparable to vLLM's** | 38.4776 tok/s, over 5,942 input tokens where the workload intends 6,144 | 196.0967 tok/s, over 47,072 where it intends 49,152 |
 | median TPOT | 218.11 ms | 250.57 ms |
 | median ITL | 216.56 ms | 232.83 ms |
 | median TTFT | 883.78 ms | 3623.5 ms |
@@ -24198,7 +24198,7 @@ shape and not `--enforce-eager`.
 | `failed`, per rep | 0, 0, 0 |
 | output token throughput, median | **4.2835 tok/s** |
 | output token throughput, CV over 3 reps | 0.033% |
-| total token throughput, median | 38.5516 tok/s |
+| total token throughput, median — **do not set this beside ours**; ours is deflated by 202 missing prompt tokens ([#1355](https://github.com/mudler/vllm.cpp/issues/1355)), so the axis is not comparable | 38.5516 tok/s, over the full 6,144 input tokens |
 | median TPOT | 228.36 ms |
 | median ITL | 226.86 ms |
 | median TTFT | 876.4 ms |
@@ -24253,7 +24253,10 @@ The c8 vLLM server loaded, reached `/health` after 373 s (launched 10:18:54 UTC,
 first `GET /health 200 OK` logged at 10:25:07 UTC), and the worker was then lost
 during the untimed warmup, before any timed leg ran. The memory trajectory is
 the finding, from the series' own 2-second `MemAvailable` sampler
-(`out/vllm-20260819T095758Z/mem.samples`, epoch seconds and MB):
+(`out/vllm-20260819T095758Z/mem.samples`, epoch seconds and MB). One row is NOT
+from the sampler: the `before launch` value is `MemAvailable_MB_before_server`
+from `job.log:214`, read once at launch, and `116869` appears nowhere in
+`mem.samples`.
 
 | Sample time (UTC) | MemAvailable | What it is |
 |---|---:|---|
@@ -24380,7 +24383,11 @@ Against the previously recorded cells (2026-08-15, a DIFFERENT boot
 
 **The SHAPE of that difference is the diagnosis, and the MAGNITUDE is not
 established by these two rows.** The throughput axis moves a great deal while
-the per-token axis moves 1-4%. That is exactly what
+the per-token axis barely moves at all — and that comparison is qualitative on
+purpose, because the two rows straddle the 2184 MHz-pinned / 2489 MHz-sampled
+boundary this paragraph goes on to declare non-dividable, so quoting a percentage
+here (220.6 -> 218.11 and 261.1 -> 250.57) would break the rule stated three
+sentences later. That is exactly what
 [#931](https://github.com/mudler/vllm.cpp/issues/931) predicts: the per-token
 axis was always measuring the engine, because it is computed over completed
 requests, while `output_throughput` was dividing live tokens by a wall duration
@@ -24419,18 +24426,73 @@ three c8 reps. `vllm bench serve` re-aligns prompts against the server's own
 warning on either arm, so our `/tokenize` agreed on 1024 while our
 `usage.prompt_tokens` reported 915 for the same request.
 
-**What this does and does not touch.** `output_throughput` is
-`total_output_tokens / duration`, and `output_lens` is `[128] x N` on BOTH arms
-in every leg, so the output-throughput, TPOT and ITL figures above are
-unaffected. `total_token_throughput` has input tokens in its numerator and IS
-affected: our c8 figure of 196.10 tok/s is computed over 47,072 input tokens
-where the intended workload is 49,152.
+**What this does and does not touch, BOUNDED rather than asserted.**
+`total_token_throughput` has input tokens in its numerator and is corrupted
+outright: our c1 figure of 38.4776 tok/s is computed over 5,942 input tokens and
+our c8 figure of 196.10 tok/s over 47,072, where the intended workload is 6,144
+and 49,152. `output_lens` is `[128] x N` on BOTH arms in every leg, so TPOT and
+ITL — both computed per completed request — stand.
+
+**`output_throughput` does NOT stand unconditionally, and the earlier wording
+"affects total-token throughput only" was too strong.** It is
+`total_output_tokens / duration`, and under GENUINE truncation the missing
+prompt tokens mean less prefill and therefore a shorter wall, which biases it
+UP. The size is derivable from these files. At c1 the wall equals the sum of the
+per-request E2ELs to within 1 ms (174.4387 s against a 174.4397 s `duration`),
+so the whole effect is in TTFT: over the 15 non-outlier c1 points the marginal
+prefill cost is 1.10 ms/token, so the missing 202 tokens are 0.22 s, and the two
+short prompts' TTFTs sit 0.20, 0.23 and 0.20 s below the 1024-token mean of
+884.6 ms across the three reps — **~0.22 s of 174.39 s, about 0.13%, against a
+published CV of 0.039%**. At c8 the first wave prefills at 1,300-1,800
+tok/s (six of eight first tokens by 4.579 s in rep 3), so 2,080 missing tokens
+are **~1.1-1.6 s of 271.0 s, about 0.4-0.6%, against a published CV of 0.205%**.
+Both are LOWER bounds, because a shorter context also cheapens every decode step.
+So the systematic bias on our headline throughput figures is larger than the
+precision printed beside them, and a reader must see that.
+
+**The two median TTFTs ARE comparable, and by evidence rather than by luck.**
+883.78 ms ours against 876.4 ms vLLM: every vLLM prompt is 1024 tokens, and on
+our side the two short prompts (915 and 931) produce the two LOWEST TTFTs in all
+three reps — sorted rep 1 `0.7381, 0.8264, 0.8436, 0.8484, 0.9047, 3.9243`, rep 2
+`0.7295, 0.8078, 0.8523, 0.9152, 0.9248, 4.0061`, rep 3 `0.7294, 0.8363, 0.8899,
+0.8900, 0.8921, 3.9548`. The median of six averages the third and fourth, both of
+which are 1024-token requests in every rep, so both arms' medians fall on the same
+prompt length. Had a short prompt landed at rank three or four, this comparison
+would have been wrong and nothing in the record would have said so.
 
 **Two causes remain open and the artifacts cannot separate them**: our server
 under-reports `usage.prompt_tokens`, or our server actually processed a
 truncated prompt. Filed, not fixed in flow — this row writes no product code
 and holds no GPU.
 
+### A SECOND THING THE CAMPAIGN DID NOT SET OUT TO FIND — A REPRODUCIBLE TTFT OUTLIER AT A FIXED REQUEST INDEX
+
+[#1365](https://github.com/mudler/vllm.cpp/issues/1365). At c1 the six requests
+are strictly serialized, and OUR arm puts a ~4 s TTFT on request 3 of every leg:
+
+| leg | `ttfts`, seconds |
+|---|---|
+| `warmup-c1.json` | 77.005, 0.727, **3.981**, 0.850, 0.845, 0.839 |
+| `c1-r1.json` | 0.738, 0.826, **3.924**, 0.844, 0.905, 0.848 |
+| `c1-r2.json` | 0.729, 0.808, **4.006**, 0.915, 0.925, 0.852 |
+| `c1-r3.json` | 0.729, 0.836, **3.955**, 0.890, 0.892, 0.890 |
+
+Four legs of four, always index 2, and request 3 carries a 1024-token prompt
+exactly as requests 4, 5 and 6 do, so prompt length does not separate it. The
+77.005 s first value is the separate, already-recorded first-inference cost
+behind a liveness-only `/health`. The oracle has no such point: 18 requests
+across its three c1 legs, every TTFT between 0.834 and 1.015 s.
+
+**Nothing recorded above is wrong, and that is exactly why this is filed.** The
+figure this repository quotes is the MEDIAN, labelled as such, and the median of
+six averages ranks three and four, which the outlier never occupies. What it does
+move is the MEAN — ours 1347.6 / 1372.6 / 1365.4 ms against the oracle's 873.3 /
+883.4 / 900.2 ms, while the medians read 883.78 ms and 876.4 ms — and the wall:
+request 4 starts 31.63 s after request 3 where every other gap is ~28.4 s, so
+roughly 3.1 s of the 174.39 s c1 wall. A reproducible outlier at a FIXED request
+index is a behaviour rather than noise. The cause is deliberately NOT chased
+here; this row writes no product code and holds no GPU.
+
 ### EVIDENCE
 
 `/mnt/nas_share/rc/q38bf16/` — `NOTES.txt` (the campaign's own 253-line
@@ -24445,10 +24507,17 @@ result JSONs, six `clock-*.json` windows with their raw samples, `CLOCKS.txt`,
 and `vllm-20260819T095758Z/` (attempt 2: three c1 JSONs and clock windows, both
 server logs, `mem.samples`, the empty `watchdog.log`).
 
-**Two inconsistencies inside the evidence directory, resolved in favour of the
+**Three inconsistencies inside the evidence directory, resolved in favour of the
 executing artifact.** `NOTES.txt` states a binary sha256 of `ab0b9a1e...`; the
 job log asserted and printed `7d0c3caf...` as both `WANT` and `GOT` at launch,
 and `out/RESULT.txt` agrees, so `7d0c3caf...` is the binary that ran and
 `NOTES.txt` carries a stale value. `NOTES.txt` gives the checkpoint as
 55,586,040,114 bytes where `.agents/specs/qwen38-27b-bf16-gate.md` records
-55,586,114,863; neither was re-derived here and the difference is not adjudicated.
+55,586,114,863, a difference of 74,749 bytes; the run DID re-derive it and it
+agrees with `NOTES.txt` — `out/bench-20260819T035148Z/job.log:47,49` print
+`CKPT_SRC_BYTES=55586040114` and `CKPT_DST_BYTES=55586040114` over the staged tree
+that served every leg — so what is unadjudicated is why the spec's figure differs,
+not whether anybody measured. And `NOTES.txt:208` gives the c8 cold start as 374 s
+where this entry derives 373 s from the log's own timestamps (10:18:54 launch,
+10:25:07 first `GET /health 200 OK`); the derived value is the better one and the
+disagreement is recorded here rather than left silent, since the other two were.
diff --git a/.agents/environment.md b/.agents/environment.md
index 8f04ee26b..77ccd8fd0 100644
--- a/.agents/environment.md
+++ b/.agents/environment.md
@@ -119,12 +119,17 @@ reimage, so host-side oracle work needs `sudo -n docker run` against
 `vllmcpp-build:gb10` or `nvidia/cuda:13.0.1-devel-ubuntu24.04`, reached over
 `ssh`, which is the bypass. **The sentence this paragraph used to carry, "no
 vLLM leg of any row can currently run on `dgx.casa` by a lease-compliant path",
-is FALSIFIED for the BUILD step and still holds for a MODEL RUN.** On 2026-08-18
+is FALSIFIED, for the BUILD step and for a MODEL RUN alike.** On 2026-08-18
 two `rc run` jobs built the pin from source inside a lease, installed the wheel,
 imported it, and reported `cuda True NVIDIA GB10`
 ([#1185](https://github.com/mudler/vllm.cpp/issues/1185), and "The pinned oracle
-builds inside a lease on `dgx:gpu0`" further down). Nobody has run a model that
-way, so no oracle-side MEASUREMENT is unblocked yet. Read the old reason
+builds inside a lease on `dgx:gpu0`" further down). On 2026-08-19 the same pin
+then SERVED: `vllm serve` on a 52 GiB bf16 checkpoint, from a lease, no `ssh` and
+no container image, three clean benchmark legs (`.agents/benchmark-record.md`,
+`BENCH-QWEN38-27B-BF16` c1, and "A model DOES run inside a lease" below). So
+oracle-side MEASUREMENT from a lease is no longer blocked; what is still
+unreachable is the image-based path SGLang needs
+([#1265](https://github.com/mudler/vllm.cpp/issues/1265)). Read the old reason
 carefully before you quote it, because it was never the worker's missing
 toolchain. "The lease carries bytes, and the exec bit is a mount option" below
 measures staged content starting under the dynamic loader and after a copy to
diff --git a/.agents/issue-index.md b/.agents/issue-index.md
index a227bff27..2aacf2149 100644
--- a/.agents/issue-index.md
+++ b/.agents/issue-index.md
@@ -429,3 +429,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature |
 | [#1354](https://github.com/mudler/vllm.cpp/issues/1354) | — | Clock pinning is UNAVAILABLE inside an `rc` lease. `nvidia-smi -lgc 2190` returns `LGC_RC=4`, "The current user does not have permission to change clocks for GPU 0000000F:01:00.0", in three separate `rc run` jobs on `dgx:gpu0` on 2026-08-19, each running as **root** in the worker pod. `.agents/benchmarking.md` instructs "Pin the clocks before measuring, under the lock" and ships `sudo nvidia-smi -lgc 2100` as the recipe, and **every clock-pinned figure in this repository was taken over the host + `ssh` + `flock` path** that AGENTS.md now forbids for a fleet device — so the migration to leases silently removed clock pinning and no record said so. Same class as [#1265](https://github.com/mudler/vllm.cpp/issues/1265): a capability the records assume, which the current access path does not provide. Measured cost: nine timed windows across two arms recorded within-run SM-clock spreads of 12.92% to 26.36% against the 5% ceiling, `SwThermalSlowdown` active in every one and `HwSlowdown+HwThermal` in one, so `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three Qwen3.8-27B c1 pairings **even though the cross-arm rule passed perfectly** (same boot, both arms 2489 MHz median, 0.0% offset). The cell therefore has two clean complete absolutes and no ratio. NOT fixed in flow: the fix is either an `rc` worker capability this row has no authority over, or a demonstrated settle-and-hold procedure, or a ratified different clock rule for lease-measured pairs — each its own spec, and none of them a widening of the assertion to turn a red green. Records updated meanwhile in `.agents/environment.md`, `.agents/benchmarking.md` and `.agents/benchmark-record.md`. Owed under `## Owed` in [bench-qwen38-27b-four-way.md](specs/bench-qwen38-27b-four-way.md) | bug |
 | [#1355](https://github.com/mudler/vllm.cpp/issues/1355) | — | Our server reports 5,942 prompt tokens where vLLM reports 6,144 for the IDENTICAL generated prompts. Found 2026-08-19 in the raw `vllm bench serve --save-detailed` files of the Qwen3.8-27B bf16 re-measure ([#915](https://github.com/mudler/vllm.cpp/issues/915)), both arms driven by the byte-identical client invocation from the same pinned wheel `0.1.dev1+g555967922`, same dataset, same seed. `input_lens` is the SERVER-reported length — `vllm/benchmarks/lib/endpoint_request_func.py:247` overwrites `output.prompt_len` from the streamed `usage.prompt_tokens` — and reads `[915, 931, 1024, 1024, 1024, 1024]` for us against `[1024] x 6` for vLLM at c1, with 19 of 48 short (877-941) at c8, byte-identical across all three reps of each leg. Not the client: `_align_prompts_to_server_tokenizer` (`vllm/benchmarks/serve.py:74,2041-2044`) re-aligns against the server's own `/tokenize` and prints `WARNING: tokenizer mismatch` when it disagrees, and NEITHER arm printed it, so our `/tokenize` agreed on 1024 while our `usage.prompt_tokens` reported 915 for the same request. `output_lens` is `[128]xN` on both arms in every leg, so the campaign's output-throughput, TPOT and ITL figures are unaffected; `total_token_throughput` is affected, our c8 196.10 tok/s being computed over 47,072 input tokens where the intended workload is 49,152. TWO causes and the artifacts cannot separate them: under-reported usage, or a genuinely truncated prompt — and the second would mean the two arms did not run the same workload. A greedy token gate cannot see either, which is why it survived the gate on this checkpoint. NOT fixed in flow: the finding row writes no product code and holds no GPU. Owed under `## Owed` in [qwen38-27b-bf16-gate.md](specs/qwen38-27b-bf16-gate.md) | bug |
+| [#1365](https://github.com/mudler/vllm.cpp/issues/1365) | — | Our c1 arm has a **reproducible ~4 s TTFT outlier on request 3 of every leg** that the pinned oracle does not have. Found 2026-08-19 in the raw `vllm bench serve --save-detailed` files of the Qwen3.8-27B bf16 c1/c8 re-measure ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979)), `out/bench-20260819T035148Z/`. Concurrency 1, so the six requests are strictly serialized, and index 2 reads 3.981 / 3.924 / 4.006 / 3.955 s across the warmup leg and all three reps against 0.73-0.93 s for every other request in the same leg — FOUR legs of four, always the same index. Request 3 carries a 1024-token prompt exactly as requests 4, 5 and 6 do, so prompt length does not separate it, and the 77.005 s first value in the warmup leg is the separate known first-inference cost behind a liveness-only `/health`. The oracle `0.1.dev1+g555967922` on the byte-identical client invocation, same box and same lease (`out/vllm-20260819T095758Z/`), has no such point: 18 requests across three legs, every TTFT between 0.834 and 1.015 s. **Nothing published is wrong**: the repository quotes the MEDIAN and labels it, and the median of six averages ranks three and four, which the outlier never occupies. It moves the MEAN — ours 1347.6 / 1372.6 / 1365.4 ms against vLLM's 873.3 / 883.4 / 900.2 ms while the medians read 883.78 against 876.4 — and it costs wall time, request 4 starting 31.63 s after request 3 where every other gap is ~28.4 s, so roughly 3.1 s of the 174.39 s c1 wall. A reproducible outlier at a FIXED request index in four legs of four is a behaviour rather than noise, which is why it is filed rather than left in the record. NOT fixed in flow: the finding row writes no product code and holds no GPU, and the cause is deliberately not chased. Owed under `## Owed` in [qwen38-27b-bf16-gate.md](specs/qwen38-27b-bf16-gate.md) | bug |
diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md
index dc01be5be..3c8bd3ae4 100644
--- a/.agents/model-matrix.md
+++ b/.agents/model-matrix.md
@@ -91,7 +91,7 @@ Engaged architectures (the 53 non-`INVENTORIED` rows):
 |---|---|---|---|---|
 | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` |
 | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` |
-| ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput are **WITHHELD, not quoted**: our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 reads 0.677x while median TPOT in the SAME file reads 1.014x in our favour. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` |
+| ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput WERE withheld on 2026-08-15 (superseded, below): our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 read 0.677x while median TPOT in the SAME file read 1.014x in our favour. **SUPERSEDED 2026-08-19 by the c1/c8 RE-MEASURE ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979), `.agents/benchmark-record.md` `BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE`):** #931 landed, and with `VT_SERVER_SSE_PING_S=0` our arm completed **162 of 162** requests, `failed=0` on every leg — c1 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%). **Our half of the withholding is discharged; NEITHER cell became a ratio and the two halves are blocked differently.** At c1 vLLM also completed everything (**4.2835 tok/s**, CV 0.033%) and `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings — the cross-arm rule PASSED (same boot, both arms 2489 MHz median, 0.0% offset) and the WITHIN-RUN rule failed on both against the 5% ceiling ([#1354](https://github.com/mudler/vllm.cpp/issues/1354): clocks cannot be pinned inside an `rc` lease), so the c1 ratio is OWED, not withheld for being unflattering. At c8 the vLLM denominator is **NOT MEASURABLE on this box at the recorded configuration** — that is the answer, not a gap, and not a claim that vLLM is defective. Read the two output-throughput absolutes with [#1355](https://github.com/mudler/vllm.cpp/issues/1355): our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts, which corrupts total-token throughput outright and biases output throughput up by more than its own CV. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` |
 | ✅ | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4 text path + PUBLISHED BF16 text path; vision implemented, NOT gated) | text-gen STRICT token-exact 315/315 vs vLLM 0.25.0 on the NVFP4 requant; **the PUBLISHED bf16 repo now generates too and is oracle-gated (2026-08-15, #740 + #864): greedy 7 prompts x 3 repeats x 16 tokens @`995ad96e` vs the pinned oracle, 6/7 prompts STRICT 16/16, the seventh one exact logit tie (`top2_gap_mnats = 0.0`) our on-device argmax breaks toward the higher id where `torch.argmax` takes the lower (#910) — PASS under the ratified near-tie doctrine; only the FIRST divergence per prompt is adjudicable, so the raw 108/112 position count is NOT a quality score. SACRED inertness 3/3 with real counts, goldens byte-identical (27B 235/235, 35B 315/315, Coder 138/138, 688 assertions). NO throughput, latency or memory number exists for this checkpoint**; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; vision tower LOADED and the forked VL forward LANDED (#891, M2/M3), but the binding image/video token-exact gates vs the pinned oracle are OWED, so the row stays `PARTIAL` | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` |
 | 🚧 | `Qwen3_5ForCausalLM` | Qwen3.5 text-only dense arm (no checkpoint fits this box) | **REGISTERED, NOT RUN-GATED (2026-08-12, issue #490).** Ahead-of-pin forward port of upstream PR vllm#50210 (`ad5d29db7`): additive registration against the EXISTING dense factory + one backbone weight-namespace decision per checkpoint (`model.` vs `model.language_model.`, mixed index REFUSED). Dispatch, flat-config and namespace tests green; 27B/35B/Coder inert (VL prefix is still the seam default, goldens md5 unchanged). **NO token-exact oracle run and NO speed number — the run gate is OWED**, and no text-only Qwen3.5 checkpoint fits GB10. Unlike the MoE arm, this loader DOES route BF16/FP8/NVFP4 per projection by tensor presence, so a flat bf16 checkpoint is expected to load | `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm` |
 | 🚧 | `Qwen3_5MoeForCausalLM` | `Qwen/Qwen3.8-2.4T-A95B` (2.4T MoE — DOES NOT FIT this hardware) | **REGISTERED, NOT RUN-GATED (2026-08-12, issue #490).** Same ahead-of-pin port against the EXISTING MoE factory: 3.8 is the token-exact Qwen3.6-35B-A3B GDN-hybrid MoE backbone at larger scale, every knob config-driven, with the BACKBONE weight names identical modulo the backbone prefix. **CORRECTED 2026-08-12 — REGISTRATION IS NOT LOADABILITY.** This loader reads ONLY per-expert NVFP4 routed experts; the published `Qwen/Qwen3.8-2.4T-A95B` and `Qwen/Qwen3.6-35B-A3B` indices ship 3-D STACKED experts and ZERO `weight_scale`/`input_scale` (read live 2026-08-12), and our gated 35B row reads the REQUANTIZED `nvidia/Qwen3.6-35B-A3B-NVFP4` — so at that date **the bf16/stacked MoE routed-expert arm was NOT implemented and such a checkpoint was REFUSED by a message naming it**. **SUPERSEDED 2026-08-15 — THE LOADER HALF IS CLOSED, WITH A TOKEN.** #740 landed the 3-D stacked bf16 routed-expert reader and #864 the bf16 GDN tower, attention tower, shared expert and `lm_head`, all selected by tensor presence; both published indices now satisfy the load plan completely, and the binding token-exact greedy gate RAN on `Qwen/Qwen3.6-35B-A3B` bf16 @`995ad96e` vs the pinned oracle at 6/7 prompts STRICT 16/16 (the seventh an exact logit tie, #910), with SACRED 27B/35B/Coder inertness 3/3 and goldens byte-identical. **THE RUN GATE FOR THIS CLASS IS STILL OWED, and the reason is now purely hardware:** that gate ran through `Qwen3_5MoeForConditionalGeneration`, not through `Qwen3_5MoeForCausalLM`, whose only published checkpoint is 2.4T bf16 ≈ 4.8 TB / FP8 ≈ 2.4 TB against 128 GB unified. NO token and NO speed number exist for this class; its load plan resolves against the published index, which is name/shape/dtype resolution and not a token. Also owed: MTP and GGUF arms for 3.8 | `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` |
diff --git a/.agents/specs/qwen38-27b-bf16-gate.md b/.agents/specs/qwen38-27b-bf16-gate.md
index bc4f014df..e3f0f815e 100644
--- a/.agents/specs/qwen38-27b-bf16-gate.md
+++ b/.agents/specs/qwen38-27b-bf16-gate.md
@@ -112,8 +112,22 @@ concurrency on an idle leased box completed **162 of 162** requests with
 `failed=0` and zero non-empty `errors` entries on every leg. c1 output
 throughput **4.4040 tok/s** (CV 0.039%), median TPOT 218.11 ms, median ITL
 216.56 ms, median TTFT 883.78 ms. c8 output throughput **22.6402 tok/s**
-(CV 0.205%), total token throughput 196.10 tok/s, median TPOT 250.57 ms, median
-ITL 232.83 ms, median TTFT 3623.5 ms. Both teardowns `TEARDOWN_VERDICT=CLEAN`.
+(CV 0.205%), total token throughput **196.10 tok/s, CORRUPTED — do not quote it**
+([#1355](https://github.com/mudler/vllm.cpp/issues/1355), `## Owed`), median TPOT
+250.57 ms, median ITL 232.83 ms, median TTFT 3623.5 ms. Both teardowns
+`TEARDOWN_VERDICT=CLEAN`.
+
+**And read the two output-throughput figures with the same caveat, bounded rather
+than dismissed.** `output_lens` is `[128]xN` on both arms, so TPOT and ITL stand.
+`output_throughput` divides output tokens by a WALL, and a genuinely truncated
+prompt shortens that wall, so it is biased UP: at the c1 marginal prefill cost the
+missing 202 tokens are ~0.22 s of a 174.39 s wall (~0.13%), and the missing 2,080
+at c8 are ~1.1-1.6 s of 271.0 s (~0.4-0.6%). Both exceed the 0.039% and 0.205% CVs
+published beside them, so "unaffected" is wrong and the bias sits outside our own
+stated precision. Shorter context also cheapens decode, so these are lower bounds.
+The median TTFTs (883.78 ms ours, 876.4 ms vLLM) ARE comparable: our two short
+prompts are the two LOWEST TTFTs in every rep, so on both arms the median falls on
+a 1024-token request.
 
 **Neither cell became a ratio, and the two halves are blocked differently.** At
 c1 the pinned oracle `0.1.dev1+g555967922` also completed every request on its
@@ -262,7 +276,25 @@ c8 until #931 closes. Concurrencies above 8 were not run.
   5,942 prompt tokens where vLLM reported 6,144 for the identical
   client-generated prompts, 19 of 48 short at c8. Whether we under-report
   `usage.prompt_tokens` or actually truncate the prompt is not decidable from
-  these artifacts.
+  these artifacts. **It corrupts `total_token_throughput` on both legs** — c1
+  38.4776 tok/s and c8 196.10 tok/s are computed over 5,942 and 47,072 input
+  tokens where the workload intends 6,144 and 49,152 — and it biases
+  `output_throughput` up by more than that figure's own CV; `## Now` carries the
+  bound. Quoting either total-token figure, or setting our 38.4776 beside vLLM's
+  38.5516, compares two different workloads.
+- [#1365](https://github.com/mudler/vllm.cpp/issues/1365), a reproducible ~4 s
+  TTFT outlier on request 3 of every c1 leg of ours, which the oracle does not
+  have: index 2 reads 3.981 / 3.924 / 4.006 / 3.955 s across the warmup leg and
+  all three reps against 0.73-0.93 s for every other request, four legs of four,
+  on a 1024-token prompt like requests 4, 5 and 6. Nothing published is wrong,
+  because the median of six averages ranks three and four and the outlier never
+  occupies either; it moves the MEAN (ours 1347.6-1372.6 ms against the oracle's
+  873.3-900.2 ms) and costs ~3.1 s of the 174.39 s c1 wall. The cause is not
+  chased here.
 - The checkpoint size disagrees between records: this spec's `## Outcome` says
-  55,586,114,863 bytes and the campaign's `NOTES.txt` says 55,586,040,114.
-  Neither was re-derived on 2026-08-19.
+  55,586,114,863 bytes and the campaign's `NOTES.txt` says 55,586,040,114, a
+  difference of 74,749 bytes. The 2026-08-19 run DID re-derive it, and it agrees
+  with `NOTES.txt`: `out/bench-20260819T035148Z/job.log:47,49` print
+  `CKPT_SRC_BYTES=55586040114` and `CKPT_DST_BYTES=55586040114` over the staged
+  tree that then served every leg. What is unresolved is WHY the `## Outcome`
+  figure differs, which the artifacts cannot settle.
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
index 80a24a5b4..b001057ba 100644
--- a/docs/BENCHMARKS.md
+++ b/docs/BENCHMARKS.md
@@ -218,12 +218,14 @@ record. The same P0 hit classic dense `Qwen3ForCausalLM` (quant-independent), fi
 | Within-run SM-clock spread, 5% ceiling | ours 13.58 / 26.36 / 14.34%, vLLM 10.16 / 17.48 / 18.52%; `SwThermal` in every window, `HwSlowdown+HwThermal` in one | (cell stands) | **all six of our legs and all three vLLM legs breached it** |
 | Clocks were SAMPLED, never pinned | `nvidia-smi -lgc` returns `LGC_RC=4`, permission denied, as root inside an `rc` lease. Every prior pinned figure used the retired `ssh`+`flock` path | (cell stands) | same lease, same refusal |
 | **vLLM c8 denominator** | (c1 landed) | (cell stands) | **NOT MEASURABLE on this box at the recorded config**, which is the answer and not a gap. Not a claim that vLLM is defective |
-| Evidence for that verdict | `/health` reached at 373 s, then the worker was lost during warmup | (cell stands) | the KV reservation took **48,715 MB in one 4 s window** (38,708 -> 9,738); last value 6,261 MB; death inside one 2 s sample |
-| Why no watchdog can guard it | ~6-7 GB headroom at `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192` | (cell stands) | 12,000 MB kills a healthy server, 5,000 MB is never reached in time. Every other fix is an engine knob that changes the denominator |
-| Undetermined, recorded as owed | whether the HOST rebooted or only the k3s pod was lost. These artifacts cannot tell | (cell stands) | settle it by reading `/proc/sys/kernel/random/boot_id` against `3fd9745a-...85515` inside a later job |
+| Evidence for that verdict | (c1 landed: vLLM's own c1 cold start was 426 s and all three legs ran) | (cell stands) | `/health` at 373 s, then the worker was lost during warmup; the KV reservation took **48,715 MB in one 4 s window** (58,453 -> 9,738); last value 6,261 MB; death inside one 2 s sample |
+| Why no watchdog can guard it | (c1 landed) | (cell stands) | ~6-7 GB headroom at `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192`; 12,000 MB kills a healthy server and 5,000 MB is never reached in time. Every other fix is an engine knob |
+| Undetermined, recorded as owed | (c1 landed) | (cell stands) | whether the HOST rebooted or only the k3s pod was lost; these artifacts cannot tell. Settle it by reading `/proc/sys/kernel/random/boot_id` against `3fd9745a-...85515` in a later job |
 | #915's own-arm debt, DISCHARGED | prior 2.37 -> 4.4040 tok/s while median TPOT moved 220.6 -> 218.11 | (cell stands) | prior 15.96 -> 22.6402 while TPOT moved 261.1 -> 250.57. Different boots, so read the SHAPE, not the percentage |
-| What that shape proves | throughput moved a lot and per-token 1-4%: the throughput axis was dividing live tokens by a wall still holding dead requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | (cell stands) | withholding 0.677x rather than publishing it was correct, and this is the evidence for that |
-| A divergence found in the raw files | our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts; 19 of 48 short at c8 | (cell stands) | affects total-token throughput only: `output_lens` is `[128]xN` on both arms, so tput/TPOT/ITL stand |
+| What that shape proves | throughput moved a lot, per-token barely at all; qualitative, the boots differ: the throughput axis was dividing live tokens by a wall still holding dead requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | (cell stands) | withholding 0.677x rather than publishing it was correct, and this is the evidence for that |
+| A divergence found in the raw files | our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts; 19 of 48 short at c8 | (cell stands) | `output_lens` is `[128]xN` on both arms, so TPOT and ITL stand. Total-token throughput is CORRUPTED, and output throughput is BIASED UP, next row |
+| Size of that bias, which EXCEEDS our own stated precision | if the prompts were truly truncated the wall is short by ~0.22 s of 174.39 s, about **0.13%**, beside a published CV of 0.039% | (cell stands) | ~1.1-1.6 s of 271.0 s, about **0.4-0.6%**, beside a published CV of 0.205%. A lower bound: shorter context also cheapens decode |
+| Are the two TTFT medians still comparable? | YES, by evidence and not by luck: our two short prompts are the two LOWEST TTFTs in all three reps, so both arms' medians fall on 1024-token requests | (cell stands) | no vLLM leg exists to compare against |
 | Why only c4 counts | `output_throughput` divides tokens by a wall duration that still contains the dead request, so a cell where one arm dropped requests is withheld, not quoted | 3 paired reps, clocks 2184 MHz | token gate: 4/7 strict, 3 exact fp32 ties ([#915](https://github.com/mudler/vllm.cpp/issues/915)) |
 
 ### Qwen3.8-27B quantized arms, both gates PENDING and no number quoted
diff --git a/docs/FEATURES.md b/docs/FEATURES.md
index 42c2d1748..3c83b3e1b 100644
--- a/docs/FEATURES.md
+++ b/docs/FEATURES.md
@@ -115,7 +115,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 <!-- supported-arch-table:begin -->
 | Architecture | Tested checkpoint(s) | Correctness gate | Speed vs reference |
 |---|---|---|---|
-| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16; **Qwen3.8-27B BF16** @`1d4bf0f2` | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3; Qwen3.8-27B 4/7 strict, 3 exact fp32 ties in band (#915) | `unsloth` 27B at/above vLLM, ModelOpt 0.85x; 4B 1.021x; 3.8-27B c4 **0.963x**, c1/c8 withheld (#931). Loads BF16/per-tensor FP8/NVFP4 (CT + ModelOpt); `modelopt_mixed` FP8 tower NATIVE (#164), GDN qkvz merged. CUDA/CPU |
+| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16; **Qwen3.8-27B BF16** @`1d4bf0f2` | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3; Qwen3.8-27B 4/7 strict, 3 exact fp32 ties in band (#915) | `unsloth` 27B at/above vLLM, ModelOpt 0.85x; 4B 1.021x; 3.8-27B c4 **0.963x**, c1/c8 absolutes (#915). Loads BF16/per-tensor FP8/NVFP4 (CT+ModelOpt); `modelopt_mixed` FP8 tower NATIVE (#164), GDN qkvz merged. CUDA/CPU |
 | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4 text; published BF16 text + vision tower) | NVFP4 strict 315/315 vs vLLM 0.25.0; published BF16 6/7 prompts strict 16/16 vs the pin, 7th an exact tie (#910). Image/video IMPLEMENTED, NOT GATED (#891): the tower loads and runs, mm gate OWED | gate model: 0.93x to 1.03x grid; NO BF16 or mm speed claim |
 | `Qwen3_5ForCausalLM`, `Qwen3_5MoeForCausalLM` | none: no text-only Qwen3.5 checkpoint fits this hardware | **NO RUN GATE, OWED.** Gated on `test_qwen3_8_text_only.cpp`; NO token claim. Loader reads stacked BF16 experts (#740) plus BF16 towers, shared expert and `lm_head` (#864), so both published indices satisfy the load plan | not measured |
 | `Qwen3ForCausalLM` | Qwen3 dense 0.6B/1.7B/4B/32B, NVFP4A16 | near-tie strict 16/16 vs vLLM 0.25.0 | c1 every-axis parity, c8 decode residual |
diff --git a/docs/STATUS.md b/docs/STATUS.md
index ba24a6338..dc0cb0154 100644
--- a/docs/STATUS.md
+++ b/docs/STATUS.md
@@ -982,6 +982,14 @@ now defaults to `0`. With it off, our arm completed **162 of 162** requests,
 three reps at each concurrency, `failed=0` on every leg: c1 output throughput
 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%).
 
+**Read both of those absolutes with one caveat.** Our server reported 5,942
+prompt tokens at c1 where vLLM reported 6,144 on the identical prompts, and 19 of
+48 were short at c8 ([#1355](https://github.com/mudler/vllm.cpp/issues/1355)).
+`output_lens` is `[128]xN` on both arms, so TPOT and ITL stand and total-token
+throughput does not. Output throughput is biased UP if the prompts were truly
+truncated, by roughly 0.13% at c1 and 0.4-0.6% at c8 — larger than the 0.039% and
+0.205% CVs published beside them, so the bias is not inside the stated precision.
+
 **Our half of the debt is discharged. Neither cell is a ratio, and each is
 blocked for its own reason.** At c1 vLLM also completed every request
 (**4.2835 tok/s**, CV 0.033%) and both absolutes stand, but
@@ -999,7 +1007,7 @@ that ceiling, and stable medians do not launder it.
 **At c8 the vLLM denominator is NOT MEASURABLE on this box at the recorded
 configuration, and that is the answer rather than a gap in it.** The server
 reached `/health` at 373 s and the worker was then lost during warmup. The KV
-reservation took **48,715 MB in a single 4-second window** (38,708 -> 9,738 MB),
+reservation took **48,715 MB in a single 4-second window** (58,453 -> 9,738 MB),
 the last observed value was 6,261 MB, and the death fell inside one 2-second
 sampling interval.