measure(BENCH-QWEN38-27B): both withheld cells are clean, the c1 pairing is discarded on clock spread, and c8 has no denominator on this box - #1369
Merged
Conversation
… cells stay open for two DIFFERENT reasons #915's c1/c8 debt was withheld because our server dropped requests. #931 landed that fix, and this records the re-measure that discharges OUR half of it: three reps at each concurrency on an idle leased box, 162 of 162 requests completed, `failed=0` and zero non-empty `errors` on every leg. c1 output throughput 4.4040 tok/s (CV 0.039%), c8 22.6402 tok/s (CV 0.205%). Neither cell became a ratio, and the reason differs per cell, which is why they are recorded separately rather than as one open gap. At c1 both arms completed everything and `gpu_clock_state compare` returned PAIRING_VERDICT=DISCARD on all three pairings. The shape of that refusal is recorded precisely because reading it as "the arms disagree" inverts it: the cross-arm rule passed perfectly, same boot and both arms at a 2489 MHz median with a 0.0% offset, and the WITHIN-RUN rule failed on both, 13.58/26.36/14.34% for us and 10.16/17.48/18.52% for vLLM against a 5% ceiling, with SwThermal throttling in every window. All six of our legs and all three of vLLM's breached that ceiling and the record says so beside the numbers, because stable medians do not launder a breach. No ours-over-vLLM ratio is derived anywhere. The ratio is OWED, not withheld for being unflattering: the instrument that decides whether a pair may be divided refused the pair. At c8 the vLLM denominator is NOT MEASURABLE on this box at the recorded configuration, and that is the ANSWER to the cell rather than a gap in it. The KV reservation took 48,715 MB in a single 4-second window, the last observed value was 6,261 MB, and the worker died inside one 2-second sampling interval. So this box leaves 6-7 GB of headroom at the recorded knobs and a sampling watchdog cannot guard it at any floor that still lets the configuration run. Every way to create that headroom is an engine knob that would change the denominator, so none was attempted. This is a statement about headroom and guard granularity here, not a claim that vLLM is defective. Two findings outlive the campaign and are recorded where a future reader hits them rather than only in this entry. Clock pinning is unavailable inside an `rc` lease -- `nvidia-smi -lgc` returns LGC_RC=4 as root -- and every clock-pinned figure in these records was taken over the retired host+ssh+flock path, so the migration to leases removed the capability and no record said so. That is the root cause of the discarded pairing, it is filed as #1354, and it is written into `.agents/environment.md` and beside the instruction it contradicts in `.agents/benchmarking.md`. And a guard set inside a configuration's own operating point manufactures the finding it was meant to detect: a 12,000 MB watchdog killed a healthy server whose own arithmetic predicts an 11,917 MB floor, a 306 MB match, and reporting that as "the denominator collapses in a lease" would have been a fabricated finding about vLLM. One divergence was found in the raw files that nobody was looking for and it is filed as #1355: our server reports 5,942 prompt tokens where vLLM reports 6,144 for the identical generated prompts. `output_lens` is [128]xN on both arms, so the throughput and per-token figures here stand; total-token throughput does not, and whether we under-report usage or truncate the prompt is not decidable from these artifacts. Whether the HOST rebooted or only the k3s pod was lost when the c8 worker died is UNDETERMINED and recorded as owed, with the one command that settles it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ngth step, and a c8 fact sat in the c1 column Every headline number in this entry reproduces from the raw JSONs. What did not survive review is the compression into the public documents, and every repair here is to the wording rather than to a measurement. `docs/BENCHMARKS.md` and `docs/STATUS.md` both wrote the KV reservation as "48,715 MB in a 4-second window (38,708 -> 9,738)". Subtraction falsifies it: from `vllm-20260819T095758Z/mem.samples` the window is 10:24:41 58,453 -> 10:24:43 38,708 -> 10:24:45 9,738, so 58,453 - 9,738 = 48,715 over 4 s while the pair that was printed is 28,970 over 2 s. The parenthetical named the endpoints of the SECOND, half-length step while carrying the magnitude and duration of the full one. `NOTES.txt` and the record both render it correctly, so the defect was introduced by the compression and nowhere else. The `docs/BENCHMARKS.md` c1 cell of "Evidence for that verdict" carried two c8-vLLM facts. vLLM's c1 cold start was 426 s (`job.log:27 VLLM_COLD_START_TO_HEALTH_S=426`) and all three c1 legs completed, and the row directly above already reads `(c1 landed)`. The two rows below used the c1 column as overflow prose in the same way. A keyed table's columns are a contract, and `check-public-doc-tables` measures shape rather than attribution, so nothing could have caught this. The row's own spec quoted `total token throughput 196.10 tok/s` as a clean fact while its `## Owed` described #1355 without ever connecting it to that axis. `.agents/benchmark-record.md` carried 196.0967, 38.4776 and 38.5516 unmarked with the caveat two hundred lines below. The c1 pair is the dangerous one: our 38.4776 sat directly beside the oracle's 38.5516 on the exact axis the divergence corrupts, with ours deflated by 202 missing prompt tokens, so the oracle read ahead on an axis that is not comparable. All four figures are now marked where they appear. "Affects total-token throughput only" was stronger than the evidence, and the bias it dismissed is larger than the precision published beside the figures. Under genuine truncation a shorter prompt means less prefill, a shorter wall and therefore an INFLATED `output_throughput`. Re-derived here: at c1 the wall equals the sum of the per-request E2ELs to within 1 ms, the marginal prefill cost over the 15 non-outlier points is 1.10 ms/token, and the two short prompts sit 0.21-0.23 s below the 1024-token TTFT mean, so ~0.23 s of 174.39 s, about 0.13%, against a published CV of 0.039%. At c8 the first wave prefills at 1,300-1,800 tok/s, so 2,080 missing tokens are ~1.1-1.6 s of 271.0 s, about 0.4-0.6%, against a published CV of 0.205%. Both are lower bounds, since a shorter context also cheapens decode. The two published TTFT medians (883.78 ms and 876.4 ms) are comparable, and no record said why. They are: our two short prompts produce the two LOWEST TTFTs in all three reps, so the median of six, which averages ranks three and four, falls on a 1024-token request on both arms. The conclusion was right by luck, and it is now right by evidence. Four records contradicted themselves. `.agents/environment.md` still left standing, in bold, that the lease-compliant path "still holds for a MODEL RUN" and that "nobody has run a model that way", twenty lines above the section this same campaign added recording that the pinned oracle SERVED a 52 GiB checkpoint from a lease; corrected in place, as that paragraph's own pattern does. `.agents/model-matrix.md`, the OWNING matrix row, still stated the withholding in the present tense with no correction and no pointer. The checkpoint-size disagreement was recorded as "neither was re-derived", which the run's own artifact contradicts: `bench-20260819T035148Z/job.log:47,49` print `CKPT_SRC_BYTES=55586040114` and `CKPT_DST_BYTES=55586040114`, agreeing with `NOTES.txt` and differing from the spec by 74,749 bytes, so what is unresolved is WHY they differ and not whether anyone measured. And "the per-token axis moves 1-4%" is a cross-boot percentage taken across the very 2184 MHz-pinned / 2489 MHz-sampled boundary the same paragraph declares non-dividable; it is now qualitative, in the record and in the scoreboard cell that repeated it. Two sourcing corrections. The memory-trajectory table's `before launch` row is presented as coming from `mem.samples` and is not there (`grep -c 116869` returns 0); it comes from `job.log:214 MemAvailable_MB_before_server`. And `NOTES.txt:208` gives the c8 cold start as 374 s where the record derives 373 s from the log's own timestamps -- a fourth disagreement with that file that went unflagged while the others were flagged. One new finding is filed rather than left in the record. #1365: our c1 arm has a reproducible ~4 s TTFT outlier on request 3 of EVERY leg -- 3.981 / 3.924 / 4.006 / 3.955 s across the warmup leg and all three reps, against 0.73-0.93 s for every other request in the same leg, on a 1024-token prompt exactly like requests 4, 5 and 6. The oracle has no such point in 18 requests. Nothing published is wrong, because this repository quotes the labelled MEDIAN and the median of six averages ranks three and four, which the outlier never occupies; it moves the MEAN (ours 1347.6-1372.6 ms against 873.3-900.2 ms) and costs roughly 3.1 s of the 174.39 s wall. A reproducible outlier at a FIXED request index in four legs of four is a behaviour, not noise. The cause is deliberately not chased. `docs/FEATURES.md` carried the same stale claim as the matrix row, "c1/c8 withheld (#931)", and it is corrected in the same commit. That is also what `check-doc-checkpoint --base` requires per commit: a `.agents/model-matrix.md` edit is a support-surface claim owing `docs/FEATURES.md`, and a `.agents/benchmark-record.md` measurement owes `docs/STATUS.md` and `docs/BENCHMARKS.md`. This work first landed as three commits, two of which paid only part of that, so it is one commit instead. The one-shot `check-doc-checkpoint.py` cannot see the difference, because it validates the committed HEAD rather than each commit in a range. The c1 prefill deficit is quoted at ~0.22 s rather than at the top of its range. The three reps give 0.20, 0.23 and 0.20 s between the two short prompts' TTFTs and the 1024-token mean of 884.6 ms, and 202 tokens at the 1.1042 ms/token regression slope is 0.22 s. A repair whose subject is a figure quoted more precisely than its evidence supports cannot itself do that. No measured value changed, and no ratio is derived anywhere. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Taken so the trailer and style gates run in place rather than skipping on a base that is no longer an ancestor. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:Opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 19, 2026
…onflicts `origin/main` moved to 9e1a5e5 while this branch was under fresh review. The conflict is real rather than the union artefact: #1369 rewrote the Qwen3.8-27B row of `docs/BENCHMARKS.md`, which sits on the line directly BELOW the Qwen3.5-4B row this branch restores, and both branches appended a block to the end of `.agents/benchmark-record.md`. Resolved as AGENTS.md Records requires, and not by accepting the three-way merge. For each file the complete `origin/main` version was taken first, then this branch's scoped edit was re-applied on top, then the untouched keys were proved equal rather than assumed: - `docs/BENCHMARKS.md`: main's 572 lines taken wholesale, then the single Qwen3.5-4B row re-applied at line 59. 571 of 572 lines are byte-for-byte main's, and the one differing line is this branch's own key. #1369's row 60, its "NOT ESTABLISHED as a ratio" cells and its SUPERSEDED withheld-cells row are main's text unaltered. - `.agents/benchmark-record.md`: append-only, and both sides were verified to be pure tail appends over the merge base before anything was combined. Main's 24,523 lines were taken wholesale and this branch's 49-line block appended after them, so main's file is a byte-for-byte prefix of the result. No existing entry was rewritten. Every figure was re-verified at THIS head rather than carried over from the pre-merge one: `grep -c 118.7` is 1 in `docs/BENCHMARKS.md` and 2 in `docs/STATUS.md`; the restored cells measure 210 and 216 characters against the 220-char per-entry cap; and `docs/STATUS.md` holds its ratchet unmoved at oversized_cells 44, long_paragraphs 75, h2_sections 11. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two Qwen3.8-27B bf16 cells this row has owed since 2026-08-15 are measured,
on a box verified idle rather than assumed idle, with every request completed.
Our arm: c1 4.4040 tok/s and c8 22.6402 tok/s at the median of three reps,
162 of 162 requests completed, zero failed, both teardowns returning the GPU.
The pinned oracle's c1 arm ran in the same window on the same boot at 4.2835
tok/s in its production graphed configuration.
No ratio is derived, and the reason is spread rather than disagreement. The
cross-arm clock rule passed perfectly -- same boot, both arms at a 2489 MHz
median, offset 0.0 percent -- while the within-run rule failed on both, at
13.58/26.36/14.34 percent for us and 10.16/17.48/18.52 against a 5 percent
ceiling, with thermal throttling in every window. All nine legs are recorded as
breaching it. The c1 ratio is owed, not withheld for being unflattering.
At c8 the denominator is not measurable on this box at the recorded
configuration, and that is the answer to that cell rather than a gap in it. The
server reached health, then the reservation took 48,715 MB in a single
four-second window and the worker was lost inside one two-second sampling
interval, leaving roughly 6 to 7 GB of headroom. No sampling watchdog can guard
that at any floor which still lets the configuration run: 12,000 MB kills a
healthy server and 5,000 MB is never reached in time. Every way to create the
headroom is an engine knob that would change what is being measured, so none was
attempted. This is a statement about headroom and guard granularity on this
hardware, not a claim that vLLM is defective.
What the completion counts settle is the keepalive defect. Completion went from
five of six and thirty-six of forty-eight to six of six and forty-eight of
forty-eight, and the throughput axis moved while the per-token axis barely did.
That shape is the diagnosis: the per-token axis was always measuring the engine,
and the throughput axis was dividing live tokens by a wall that still contained a
dead request. The magnitude of the change is not established, because the earlier
figures were taken on a different boot at a pinned clock and these were sampled.
Two figures carry a caveat at the point of use rather than in a distant
paragraph. The arms did not present identical prompts -- 5,942 tokens against
6,144 at c1 and 47,072 against 49,152 at c8 -- so total-token throughput is
corrupted and must not be set beside the oracle's. The effect on output
throughput is bounded rather than dismissed: about 0.13 percent at c1 and 0.4 to
0.6 percent at c8, both larger than the coefficients of variation published
beside them, and both lower bounds.
Two findings outlive the campaign. Clock pinning is unavailable inside a lease,
which no record said, so every earlier pinned figure came by a path the protocol
no longer permits. And a guard set inside a configuration's own operating point
manufactures the finding it was meant to detect -- the first attempt was killed
by our own watchdog, settled by a 306 MB arithmetic match rather than by
argument.
Refs #915, #979, #1265, #1354, #1355, #1365.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:Opus-5 [Claude Code]