diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 31dc52c58..4822cc7ba 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -24322,7 +24322,15 @@ can resolve. **A sampling watchdog is therefore not a viable guard for the c8 denominator here at ANY floor that still lets the configuration run**: 12,000 MB kills a healthy server (below), and 5,000 MB was never reached before the worker was lost — `watchdog.log` is empty, zero bytes, because the last value the -sampler saw was above it. +sampler saw was above it. **A separate and NARROWER statement holds for the +reboot settled below, and it is not this conclusion made stronger.** A watchdog +is a userspace process on the box, so a kernel reboot ends it with everything it +was guarding: against the REBOOT CLASS of failure there is no floor and no +cadence at which a sampler survives the event it is supposed to report. That is +a general property of samplers and kernels and it stands on its own. It does not +strengthen the c8 sentence above, because strengthening THAT would require the +reboot to be what killed THIS worker, which is exactly what is NOT established +below. Every way to create that headroom is an ENGINE KNOB. Lowering `gpu_memory_utilization` or `max_num_batched_tokens` produces a @@ -24333,12 +24341,123 @@ NOT MEASURABLE rather than a number taken at a configuration nobody recorded. For scale on the same box and the same workload, ours held a `MemAvailable` floor of 21,100 MB across both of its legs. -**UNDETERMINED, and recorded as owed rather than guessed.** Whether the HOST -rebooted or only the k3s pod was lost when the worker died is not decidable from -these artifacts. One command settles it: read -`/proc/sys/kernel/random/boot_id` inside any later `dgx:gpu0` job and compare -against `3fd9745a-d25a-426c-ba3c-97c958a85515`. A different value means the host -rebooted. +**SETTLED THE SAME DAY — THE HOST REBOOTED, AND THE TWO HALVES OF THAT ARE NOT +EQUALLY STRONG.** This entry carried the pod-versus-host question as owed. An +`rc run` job on `dgx:gpu0`, job id `97cf3e63-e4a4-4506-bde7-f19f19be3bbf`, +answered it. This is the probe log in full and verbatim, 312 bytes, +sha256 `25b88023d85dbd7c751389be6427547ceb75f58992be18ba6d7f421a0418fd94`: + +```text +rc: queued at position 1 for dgx:gpu0 +rc: job 97cf3e63-e4a4-4506-bde7-f19f19be3bbf on dgx:gpu0 +BOOT_ID_NOW=64c495a3-8c9c-4b20-8496-a97efda0e332 +BOOT_ID_AT_BENCH=3fd9745a-d25a-426c-ba3c-97c958a85515 +VERDICT=REBOOTED -- the host boot_id changed since the 2026-08-19 benchmark +UPTIME_S=38868 +MemAvailable_MB=117436 +``` + +**OBSERVED: the machine rebooted.** `/proc/sys/kernel/random/boot_id` is +kernel-wide and regenerated once per boot, so a changed value is a reboot and +nothing else. A pod restart, a container teardown and a `k3s` restart all leave +it unchanged. The disjunction is retired. + +**DERIVED, conditional, and ONE-SIDED.** `UPTIME_S=38868` is `/proc/uptime` +read from INSIDE the `rc` worker, so it is the HOST's uptime only if that worker +does not virtualize `/proc`. `lxcfs` and its equivalents do virtualize +`/proc/uptime` and none of them can touch `boot_id`, which is exactly why the +observed half above does not inherit this caveat. The worker is a k3s pod, where +an unvirtualized `/proc` is the default, and no artifact here asserts that +`lxcfs` is absent. The VALUE argues against virtualization on its own: a +container-scoped `/proc/uptime` reports the POD's age, and 38,868 s is 10.8 h, +which is not a plausible age for a job submitted minutes before the read. That +is an argument and not the missing assertion, so the caveat stands. + +**The arithmetic is one-sided, and the interval recorded here earlier was not.** +That earlier text also said no tighter figure was supportable, which was wrong +in both directions: a tighter UPPER bound exists and the LOWER bound is not +established at all. 38,868 s is 10:47:48. The probe log's last write is +21:29:35.606816Z, and that mtime is when the FINAL line (`MemAvailable_MB`) +landed, so `UPTIME_S` was read strictly BEFORE it. The mtime is therefore an +UPPER bound on the read instant rather than a midpoint, and 21:30:02Z, when the +run was reported, is later still and bounds nothing usefully. Both endpoints +that were recorded here are at or after the true read instant, so the interval +they formed excluded the whole region the value occupies: + +```text +21:29:35.606816Z - 10:47:48 = 10:41:47.606816Z +``` + +So the derived boot is **at or before 2026-08-19T10:41:47.6Z**, half a second +later still if `UPTIME_S` rounds `/proc/uptime` rather than truncating it — the +probe script was not retained, so which one is unknown. No LOWER bound is +stated. One would have to come from when the probe STARTED, which is not in the +log's content, and it would cross the same unpinned clock boundary as the upper +one. + +**The two clocks are not pinned to each other and nothing here quantifies the +skew.** 21:29:35.606816Z is the mtime of a file on the LOCAL host, written there +by the `rc` client. `UPTIME_S` was read on `dgx`, and the timeline below is +`dgx`-side content plus mtimes on the shared CIFS mount. Neither is the local +host's clock, so the derived instant is compared across an offset no artifact +here measures. + +| Time (UTC) | Strength | Event | +|---|---|---| +| 10:18:51 | observed | c1 legs complete, `TEARDOWN_VERDICT=CLEAN`, `MemAvailable` 116,869 MB | +| 10:18:54 | observed | c8 vLLM server launched, `SERVER_PID=123868`; also `job.log`'s last CONTENT line | +| 10:25:07 | observed | c8 server answers `GET /health 200 OK`; `vllm-server-c8.log`'s last CONTENT line | +| 10:25:26 | observed | last `MemAvailable` sample, 6,261 MB; nothing after it. `mem.samples` mtime is 10:25:28.570640 | +| **at or before 10:41:47.6** | **derived** | **upper bound on the boot of the kernel now running** | +| 11:26:32.079 | mtime only | `job.log` and `vllm-server-c8.log` both re-stamped. NOT a content write and NOT established as liveness | + +**The 11:26 row is a file mtime, and this entry earlier recorded it as +`11:26:00`, "the job's last write".** Both halves are wrong: the actual mtimes +are `11:26:32`, and nothing establishes that the job wrote them. No CONTENT +anywhere in `out/vllm-20260819T095758Z/` is later than 10:25:07 +(`vllm-server-c8.log`) or +10:25:26 (`mem.samples`), and `job.log`'s last content line is the 10:18:54 +launch banner. The two 11:26:32 mtimes — `job.log` at `.079517600` and +`vllm-server-c8.log` at `.079047100` — are **0.47 ms apart**, which is one bulk +event touching both files, not a job writing to either. + +**Two readings of that row are possible, they are incompatible, and NEITHER is +established.** If a process flushed at 11:26:32 the box was alive then, which +contradicts a boot at or before 10:41:47.6Z that ended it. If it was a reaper or +a CIFS flush, then the last evidence of anything alive is `mem.samples` at +10:25:28.570640 and the derived boot is about 16 minutes AFTER it. The row is +recorded at mtime strength and the tension is left open, rather than resolved by +preferring whichever reading fits the rest. + +**A second open discrepancy comes from this box's own prior reboot.** +[`environment.md`](environment.md) records that one with an exact +`journalctl --list-boots` pair: boot `-1` ending 09:10:15Z against boot `0` +beginning 09:13:55Z, about **3m40s** of downtime. Apply that shape here and a +boot at 10:41:47 puts the box GOING DOWN around 10:38, thirteen minutes after +every writer in the evidence directory had already stopped, which does not fit a +simple crash at 10:25:28. Recorded as an open discrepancy and not resolved: the +3m40s is one sample from one reboot, the bound above is one-sided so the true +boot may be much earlier, and the two clocks are unpinned. + +**The WARMUP clause belongs to the worker LOSS, not to the boot.** The loss was +during the untimed warmup before any timed leg ran, which is observed and is +what this entry records above. At the derived boot instant nothing had written +for about 16 minutes, so that instant is not "during the warmup" in any useful +sense. Against the observed marks the bound sits at most 16m41s after the +`/health` 200 and at most 16m22s after the last memory sample. + +**NOT ESTABLISHED: that the reboot killed the worker.** Three facts are +consistent with it — the bound above, the descent this entry already records +(9,738 -> 6,261 MB, then a loss inside one 2-second sample), and this box's +documented habit of rebooting instead of OOM-killing. Consistency is not a +trace, nothing here ties the reboot to the worker's death, and an apparent +explanation is a hypothesis until it is traced. Recorded as an observed reboot +with a one-sided derived bound, never as a cause. + +**NO NUMBER MOVES.** This settles provenance and creates nothing. The c8 vLLM +denominator is still NOT MEASURABLE at the recorded configuration, the c1 +pairing is still `PAIRING_VERDICT=DISCARD`, and no ratio is derived or restored +from either. What the finding does is explain the shape of the absence. ### FINDING THAT OUTLIVES THIS CAMPAIGN 1 — CLOCK PINNING IS UNAVAILABLE INSIDE AN `rc` LEASE diff --git a/.agents/environment.md b/.agents/environment.md index 49ec638de..48754f8f2 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -529,6 +529,57 @@ environment: parallel-flake advice in the Apple/Metal profile below does not transfer here. Serialising also means every other probe queues behind the suite, so run attribution arms BEFORE a full suite, never during one. + - **★ TREAT A SERVING LEG AS AT RISK OF A REBOOT TOO, NOT ONLY A BUILD OR A + LOAD — THOUGH ONLY THE REBOOT IS OBSERVED AND ITS PLACEMENT NEXT TO THE LEG + IS DERIVED — AND FROM INSIDE A LEASE `boot_id` IS THE ONLY INSTRUMENT THAT + SEES A REBOOT AT ALL + (measured 2026-08-19, [#915](https://github.com/mudler/vllm.cpp/issues/915)).** + **Read the heading at the strength of its parts.** What `boot_id` observes is + that a reboot happened somewhere between two readings about **11.2 hours** + apart, and across the same unpinned clock boundary: the old value is last + recorded in `clock-vllm-c1-r3.json`, written 10:18:51.7Z on `dgx`, and the + new one landed in a log written 21:29:35.6Z on the local host. The serving + leg occupied about + **6.5 minutes** of that span. Putting the reboot next to the leg needs the + DERIVED `/proc/uptime` bound below, and tying it to the worker's death is not + claimed at all. So "under a serving load" is the working assumption this + bullet is written for, not a measured fact. + The pinned oracle's c8 denominator leg for `Qwen/Qwen3.8-27B` bf16 — vLLM's + production graphed shape at `--gpu-memory-utilization 0.85 + --max-num-batched-tokens 8192` — answered `GET /health 200 OK` at 10:25:07Z + with about 9,950 MB of `MemAvailable`, read 6,261 MB at 10:25:26Z, and the + worker was then lost inside one 2-second sample, during the untimed warmup. + That is OBSERVED, and it is the loss rather than the reboot. That + configuration leaves roughly **6-7 GB of headroom** on this box. The + earlier reboots this file records for this machine are a `ctest -j 4` and an + oracle LOAD; this one is only ASSOCIATED with a server that was already + healthy and serving-ready, and on that association "survived startup" is not + a safe state. + **Read `boot_id` in every leased job that loads anything large.** A lease + gives you a pod, not the box's history: `uptime` resetting and + `journalctl --list-boots` are host instruments a pod does not have, and a + lost worker looks identical whether the pod died or the machine did. That + ambiguity stood unresolved in the campaign record for a day. + `/proc/sys/kernel/random/boot_id` is kernel-wide and regenerated per boot, + so a changed value is a reboot and nothing else can forge it: a later job + (`97cf3e63-e4a4-4506-bde7-f19f19be3bbf`) read + `64c495a3-8c9c-4b20-8496-a97efda0e332` against the benchmark's + `3fd9745a-d25a-426c-ba3c-97c958a85515`. **Do not promote a boot TIME derived + from `/proc/uptime` to the same strength.** Read inside a pod, `/proc/uptime` + is the host's only if the worker does not virtualize `/proc` — `lxcfs` does, + and it cannot touch `boot_id` — so the identity change is observed and any + derived timestamp carries that assumption. State it beside the number. The + bound is also ONE-SIDED: the log mtime you subtract from is when the LAST + line landed, so it is an upper bound on the read instant and never a + midpoint. Here that gives a boot at or before 10:41:47.6Z with no lower + bound, and the mtime is on a different host's clock from the uptime, with the + offset unmeasured. + **And a sampling watchdog cannot guard the REBOOT CLASS of failure at all.** + A userspace sampler dies with the kernel, so there is no floor and no cadence + at which it reports a reboot; that holds on its own and needs no link to any + particular worker loss. Here the 2-second sampler never even saw a value + below its own 5,000 MB floor. Detail in + [`specs/qwen38-27b-bf16-gate.md`](specs/qwen38-27b-bf16-gate.md). - **GPU mutex:** this runs INSIDE an `rc` lease, never instead of one. The lease decides who gets the box. The mutex serialises the work of whoever holds it. Every CUDA test/model/serve/benchmark/profile holds the diff --git a/.agents/specs/qwen38-27b-bf16-gate.md b/.agents/specs/qwen38-27b-bf16-gate.md index e3f0f815e..02e9033b5 100644 --- a/.agents/specs/qwen38-27b-bf16-gate.md +++ b/.agents/specs/qwen38-27b-bf16-gate.md @@ -150,7 +150,13 @@ vLLM's breached that ceiling**, c8's included (12.92-14.99%). **At c8 the vLLM denominator is NOT MEASURABLE on this box at the recorded configuration.** That is the c8 answer, not a gap in it, and it is a statement about headroom and guard granularity here rather than a claim that vLLM is -defective. Detail, trajectory and arithmetic in +defective. A HOST REBOOT is now OBSERVED on this box on the benchmark day — +`boot_id` changed, so it was not merely a lost k3s pod — but nothing ties that +reboot to the worker loss behind this verdict, and `## Outcome` does not claim +it does. What the reboot establishes on its own is narrower: against the reboot +class of failure a sampling watchdog is not a guard at any cadence, because a +userspace sampler dies with the kernel. Neither half creates a denominator. +Detail, trajectory and arithmetic in [`../benchmark-record.md`](../benchmark-record.md). **What advances this row next** is a c1 pairing taken in a window the clock gate @@ -255,6 +261,137 @@ log live during a failing leg and it is 27 lines, all startup, no error, no request line, no rejection. So the cause is not named yet, and a controlled reproduction recording HTTP status and exception class is the next step. +**SETTLED, and the two halves of it are not equally strong.** This spec carried +one question as owed: whether the HOST rebooted or only the k3s pod was lost +when the c8 vLLM worker died. An `rc run` job on `dgx:gpu0`, job id +`97cf3e63-e4a4-4506-bde7-f19f19be3bbf`, read the answer back. This is the probe +log in full and verbatim, 312 bytes, sha256 +`25b88023d85dbd7c751389be6427547ceb75f58992be18ba6d7f421a0418fd94`: + +```text +rc: queued at position 1 for dgx:gpu0 +rc: job 97cf3e63-e4a4-4506-bde7-f19f19be3bbf on dgx:gpu0 +BOOT_ID_NOW=64c495a3-8c9c-4b20-8496-a97efda0e332 +BOOT_ID_AT_BENCH=3fd9745a-d25a-426c-ba3c-97c958a85515 +VERDICT=REBOOTED -- the host boot_id changed since the 2026-08-19 benchmark +UPTIME_S=38868 +MemAvailable_MB=117436 +``` + +**THE HOST REBOOTED. That half is OBSERVED and it is certain.** +`/proc/sys/kernel/random/boot_id` is generated once per kernel boot and is +kernel-wide, so a changed value is a reboot and cannot be anything else. A pod +restart, a container teardown and a `k3s` restart all leave it alone. The +disjunction is retired: it was not merely a lost pod. + +**WHEN it rebooted is DERIVED, and it rests on an assumption this measurement +does not test.** `UPTIME_S=38868` is `/proc/uptime` as read from INSIDE the `rc` +worker, so it is the HOST's uptime only if that worker does not virtualize +`/proc`. `lxcfs` and its equivalents do virtualize `/proc/uptime`; none of them +can touch `boot_id`, which is why the observed half does not inherit this +caveat. The worker is a k3s pod, where an unvirtualized `/proc` is the default, +but nothing in these artifacts asserts that `lxcfs` is absent. The VALUE argues +against virtualization by itself — a container-scoped `/proc/uptime` reports the +POD's age, and 38,868 s is 10.8 h, which is not a plausible age for a job +submitted minutes before the read — but an argument is not the assertion, so the +caveat stands. Establishing it costs one line in a later job and no lease was +taken for it here. + +**The bound is ONE-SIDED, and the interval an earlier draft of this section +recorded was not.** That draft also said no tighter figure was honest, which was +wrong in both directions: a tighter UPPER bound exists and the LOWER bound is +not established at all. 38,868 s is 10:47:48. The probe +log's last write is 21:29:35.606816Z, and that mtime is when the FINAL line +(`MemAvailable_MB`) landed, so `UPTIME_S` was read strictly BEFORE it. The mtime +is an UPPER bound on the read instant, not a midpoint, and 21:30:02Z — the run's +reported completion — is later still. Both of the endpoints recorded earlier sit +at or after the true read instant, so the interval they formed excluded the +whole region the value occupies. The arithmetic: + +```text +21:29:35.606816Z - 10:47:48 = 10:41:47.606816Z +``` + +So the derived boot is **at or before 2026-08-19T10:41:47.6Z**, half a second +later still if `UPTIME_S` rounds `/proc/uptime` rather than truncating it; the +probe script was not retained, so which one is unknown. No LOWER bound is +stated. One would have to come from when the probe STARTED rather than from +anything in the log's content, and it would cross the same unpinned clock +boundary as the upper one. + +**The two clocks are not pinned to each other and this measurement does not +quantify the skew.** 21:29:35.606816Z is the mtime of a file on the LOCAL host, +written there by the `rc` client. `UPTIME_S` was read on `dgx`, and the table +below is `dgx`-side content plus mtimes on the shared CIFS mount. Neither is the +local host's clock, so the derived instant is being compared across an offset +nothing here measures. + +| Time (UTC) | Strength | Event | +|---|---|---| +| 10:18:51 | observed | c1 legs complete, `TEARDOWN_VERDICT=CLEAN`, `MemAvailable` 116,869 MB | +| 10:18:54 | observed | c8 vLLM server launched, `SERVER_PID=123868`; also `job.log`'s last CONTENT line | +| 10:25:07 | observed | c8 server answers `GET /health 200 OK`; `vllm-server-c8.log`'s last CONTENT line | +| 10:25:26 | observed | last `MemAvailable` sample, 6,261 MB; nothing after it. `mem.samples` mtime is 10:25:28.570640 | +| **at or before 10:41:47.6** | **derived** | **upper bound on the boot of the kernel now running** | +| 11:26:32.079 | mtime only | `job.log` and `vllm-server-c8.log` both re-stamped. NOT a content write and NOT established as liveness | + +**The 11:26 row is a file mtime, and an earlier draft recorded it as `11:26:00`, +"the job's last write".** It is neither. The actual mtimes are 11:26:32.079517600 +(`job.log`) and 11:26:32.079047100 (`vllm-server-c8.log`), **0.47 ms apart**, +which is one bulk event touching both files rather than a job writing to either. +No CONTENT anywhere in `out/vllm-20260819T095758Z/` is later than 10:25:07 +(`vllm-server-c8.log`) or 10:25:26 (`mem.samples`), and `job.log`'s last content +line is the 10:18:54 launch banner. + +**Two readings of that row are possible, they are incompatible, and NEITHER is +established.** If a process flushed at 11:26:32 then the box was alive at +11:26:32, which contradicts a boot at or before 10:41:47.6Z that ended it. If it +was a reaper or a CIFS flush, the last evidence of anything alive is +`mem.samples` at 10:25:28.570640 and the derived boot is about 16 minutes AFTER +it. The row is recorded at mtime strength and the tension is left open, not +resolved by preferring whichever reading fits. + +**A second open discrepancy, from this box's own prior reboot.** +[`../environment.md`](../environment.md) records that one with an exact +`journalctl --list-boots` pair: boot `-1` ending 09:10:15Z against boot `0` +beginning 09:13:55Z, about **3m40s** of downtime. Apply that shape here and a +boot at 10:41:47 puts the box GOING DOWN around 10:38, thirteen minutes after +every writer in the evidence directory had already stopped, which does not fit a +simple crash at 10:25:28. Recorded as an open discrepancy and left unresolved: +the 3m40s is one sample from one reboot, the bound above is one-sided so the +true boot may be much earlier, and the two clocks are unpinned. + +**The WARMUP clause belongs to the worker LOSS, not to the boot.** The loss was +during the untimed warmup before any timed leg ran; that is observed. At the +derived boot instant nothing had written for about 16 minutes, so that instant is +not "during the warmup". Measured against the observed marks, the bound sits at +most 16m41s after the `/health` 200 and at most 16m22s after the last memory +sample. + +**NOT established: that the reboot killed the worker.** Three things are +consistent with it — the bound above, the descent `NOTES.txt` FINDING 4 records +(9,738 -> 6,261 MB over ~40 s, then the worker lost between two 2-second +samples), and this box's documented habit of rebooting instead of OOM-killing +(`.agents/environment.md`). Consistency is not a trace. Nothing here ties the +reboot to the worker's death, so it stays a hypothesis with a one-sided derived +bound, exactly as this repository requires, and no record may read it as a cause. + +**No number moves.** This settles a provenance question and creates no +denominator. The c8 vLLM denominator is still NOT MEASURABLE at the recorded +configuration, the c1 pairing is still `PAIRING_VERDICT=DISCARD`, and neither +cell becomes a ratio because of this finding. What the finding does is explain +the shape of the absence. + +**What the reboot DOES establish about watchdogs, stated at its own scope.** A +watchdog is a userspace process on the box, and a kernel reboot ends it with +everything else, so against the REBOOT CLASS of failure there is no floor and no +cadence at which a sampler lives long enough to report the event. That is a +general property of samplers and kernels and it needs no causal link. It does +NOT strengthen the campaign's separate conclusion about the c8 denominator, +which would require the reboot to be what killed this worker — the thing the +paragraph above declines to claim. `boot_id`, read after the fact, is the only +instrument that saw the reboot at all. + **What was NOT established.** Nothing about the vision path on this checkpoint, nothing about its NVFP4 or Q4_K_M arms (#821), no claim that #910 is fixed (this is the second checkpoint to be costed by it), and no throughput number at c1 or @@ -267,10 +404,34 @@ c8 until #931 closes. Concurrencies above 8 were not run. clock gate and the c8 denominator does not exist, so the row's own question — what our speed is against vLLM's production configuration at c1 and c8 — is still unanswered. -- Whether the HOST rebooted or only the k3s pod was lost when the c8 vLLM worker - died on 2026-08-19. The artifacts cannot distinguish them. Settle it by - reading `/proc/sys/kernel/random/boot_id` inside any later `dgx:gpu0` job and - comparing it against `3fd9745a-d25a-426c-ba3c-97c958a85515`. +- **The boot TIME, which is derived and one-sided.** The 2026-08-19 HOST reboot + is OBSERVED from `boot_id` and owes nothing to this item. Only the derived + bound — at or before 10:41:47.6Z — depends on the `dgx:gpu0` `rc` worker not + virtualizing `/proc/uptime`, and on a local-to-`dgx` clock offset that nothing + measures. `journalctl --list-boots` retires both at once, because it prints the + boot's own down and up timestamps in the host's clock and needs neither + `/proc/uptime` nor an independently known boot. **Whether a LEASED job can reach + it is itself unestablished, and this item must not assume that it can.** The + `boot_id` bullet in [`../environment.md`](../environment.md) states that the + boot list is a HOST instrument a pod does not have, which is exactly why + `boot_id` was the instrument here; a leased worker runs as root and may or may + not have the host journal mounted, and no artifact settles that. So the job is: + run `journalctl --list-boots` from inside a lease and record whether it answers + at all, and print `/proc/uptime` beside `date -u` in the same job so the + virtualization question falls out of the comparison either way. If the lease + cannot read the boot list, this item needs host access and stays owed. The + `thor:gpu0` build reboot that `../environment.md` records under + [#1380](https://github.com/mudler/vllm.cpp/issues/1380) reaches the same + instrument from the other side and leaves the same access question open, so one + job that answers it discharges part of both. No lease was taken here. +- **Two open discrepancies around that bound, recorded rather than resolved.** + The `11:26:32` mtimes on `job.log` and `vllm-server-c8.log` are 0.47 ms apart + and admit two incompatible readings, one of which puts the box alive after the + derived boot. And the only `journalctl` downtime this repository has measured + on this box is ~3m40s, which applied here would put it going down around 10:38, + thirteen minutes after every writer had stopped. `## Outcome` states both. The + job above settles the second and bounds the first, if it can read the boot + list at all. - [#1355](https://github.com/mudler/vllm.cpp/issues/1355), the prompt-token divergence found in this campaign's raw result files: our server reported 5,942 prompt tokens where vLLM reported 6,144 for the identical diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 60aefb83a..abdf636a3 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -221,7 +221,7 @@ record. The same P0 hit classic dense `Qwen3ForCausalLM` (quant-independent), fi | **vLLM c8 denominator** | (c1 landed) | (cell stands) | **NOT MEASURABLE on this box at the recorded config**, which is the answer and not a gap. Not a claim that vLLM is defective | | Evidence for that verdict | (c1 landed: vLLM's own c1 cold start was 426 s and all three legs ran) | (cell stands) | `/health` at 373 s, then the worker was lost during warmup; the KV reservation took **48,715 MB in one 4 s window** (58,453 -> 9,738); last value 6,261 MB; death inside one 2 s sample | | Why no watchdog can guard it | (c1 landed) | (cell stands) | ~6-7 GB headroom at `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192`; 12,000 MB kills a healthy server and 5,000 MB is never reached in time. Every other fix is an engine knob | -| Undetermined, recorded as owed | (c1 landed) | (cell stands) | whether the HOST rebooted or only the k3s pod was lost; these artifacts cannot tell. Settle it by reading `/proc/sys/kernel/random/boot_id` against `3fd9745a-...85515` in a later job | +| Settled: the HOST rebooted | (c1 landed) | (cell stands) | OBSERVED: `boot_id` moved `3fd9745a-...` -> `64c495a3-...` in a later leased job, so not a lost pod. DERIVED, only if it does not virtualize `/proc`: boot at or before 10:41:47.6Z, one-sided. Causation NOT traced | | #915's own-arm debt, DISCHARGED | prior 2.37 -> 4.4040 tok/s while median TPOT moved 220.6 -> 218.11 | (cell stands) | prior 15.96 -> 22.6402 while TPOT moved 261.1 -> 250.57. Different boots, so read the SHAPE, not the percentage | | What that shape proves | throughput moved a lot, per-token barely at all; qualitative, the boots differ: the throughput axis was dividing live tokens by a wall still holding dead requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | (cell stands) | withholding 0.677x rather than publishing it was correct, and this is the evidence for that | | A divergence found in the raw files | our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts; 19 of 48 short at c8 | (cell stands) | `output_lens` is `[128]xN` on both arms, so TPOT and ITL stand. Total-token throughput is CORRUPTED, and output throughput is BIASED UP, next row | diff --git a/docs/STATUS.md b/docs/STATUS.md index d00ee4131..bd13334c2 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1100,11 +1100,15 @@ migration to leases removed clock pinning and no record said so until now. Same class as [#1265](https://github.com/mudler/vllm.cpp/issues/1265), and the root cause of the discarded pairing. -**Undetermined, and recorded as owed rather than guessed.** Whether the host -rebooted or only the k3s pod was lost when the worker died cannot be decided -from these artifacts. One command settles it: compare -`/proc/sys/kernel/random/boot_id` against `3fd9745a-d25a-426c-ba3c-97c958a85515` -inside any later `dgx:gpu0` job. +**Settled, and only one half of it is observed.** A later `dgx:gpu0` job read +`/proc/sys/kernel/random/boot_id` as `64c495a3-...` against the benchmark's +`3fd9745a-...`, so the HOST rebooted and it was not a lost k3s pod. The boot +TIME is DERIVED from `/proc/uptime` inside the worker, is ONE-SIDED — at or +before 10:41:47.6Z, with no lower bound — and is conditional on that worker not +virtualizing `/proc`. That the reboot KILLED the worker is consistent but +untraced, so it is not claimed and no number changes. What the reboot does +establish is narrower and stands on its own: no sampling watchdog can guard the +REBOOT CLASS of failure, because a userspace sampler dies with the kernel. Resource axes on the same series: cold start to first `/health` **53 s vs 780 s = 14.7x**, and host memory after warmup **42.5 vs 110.1 GiB = 2.59x** —