fix(ENG-EXPERT-STREAM): refuse a GGUF the device cannot hold, at load, by name (#1123)

CMakeLists.txt

@@ -717,6 +717,7 @@ add_library(vllm STATIC
   src/vllm/model_executor/model_loader/gguf_reader.cpp
   src/vllm/model_executor/model_loader/gguf_dequant.cpp
   src/vllm/model_executor/model_loader/gguf_keep_quant.cpp
+  src/vllm/model_executor/model_loader/gguf_device_fit.cpp
   src/vllm/model_executor/model_loader/nvfp4_dequant.cpp
   src/vllm/model_executor/model_loader/awq_gptq_dequant.cpp
   src/vllm/model_executor/model_loader/mxfp4_dequant.cpp

docs/ENVIRONMENT.md

@@ -81,6 +81,7 @@ allocated up front and never grown — the engine prints the resolved values as
 | `VT_MOE_EXPERT_STREAM_SLOTS` | `64` | How many expert slices stay resident. Parsed as a decimal integer; unset, empty, zero, negative and unparseable values all keep `64`. Every slot acquired during a step is protected from eviction until the step ends, so a budget smaller than one step's working set exhausts the cache: those slices fall back to reading the tower directly, which is correct but slow, and is counted. Sized against a real model this wants to be large — the measured run used `8000` |
 | `VT_MOE_EXPERT_STREAM_SLOT_BYTES` | the LARGEST of the gate/up/down slices of the first MoE layer reached | Bytes reserved per slot, fixed for the process's life. Parsed as a decimal integer; unset, empty, zero, negative and unparseable values all keep the default. The default is the largest of the three slices rather than the first one taken, because a dynamic (UD) quant keeps `down_proj` at a higher precision than the gate/up pair and sizing from a gate slice then refuses the first down slice mid-decode. A slice that still does not fit is refused BY NAME (`vt: expert stream: a slice of N bytes exceeds the slot budget of M; raise VT_MOE_EXPERT_STREAM_SLOT_BYTES`) rather than truncated or silently routed back to the mmap path, so a streaming benchmark cannot quietly measure the mmap path instead |
 | `VT_MOE_EXPERT_STREAM_STATS_EVERY` | `16` | How many decode steps between the PERIODIC expert-stream statistics line on stderr; `0` silences the periodic line only. Parsed as a decimal integer; unset, empty, negative and unparseable values all keep `16`. The line is `[expert-stream] steps=N hits=H misses=M evictions=E fills=F bytes=B exhausted=X advised=A`. **Exactly one FINAL line is printed when the process ends**, whatever this is set to and whatever the run did, including `steps=0`, for as long as the lane built a store. That is the line to read, and it exists because the row's first published decode figure was measured on a cache that had switched itself off partway through the third token while nothing in the run could say so. **`steps == 0` or `exhausted > 0` means the lane is not streaming**, whatever the startup line claimed. Absence of the final line means either that no store was ever built — in which case the `[expert-stream] ON ...` banner is absent too, and the lane was never reached — or that the process did not run its static destructors (a crash, a signal, `_exit`). A fourth shape exists but no shipped command can produce it: the line is printed once per process, and an internal test seam that flushes it mid-run takes that one print. `docs/USAGE.md` tabulates all four |
+| `VT_DEVICE_WEIGHT_BUDGET_BYTES` | the device's own probe (`cudaMemGetInfo` total on CUDA; UNKNOWN, i.e. no check, everywhere else) | Overrides the device memory pool that a GGUF's staged weight bytes are compared against at LOAD time (issue #1123). A GGUF whose weights cannot fit is refused by name during the load instead of dying on the first forward with `vt cuda: cudaMalloc: out of memory` — `Qwen3.8-2.4T-A95B UD-Q1_0` (369.96 GiB) reached a serving state on a 119.631 GiB GB10 after 26 minutes and then died mid-stream, because the larger-than-memory lane that makes it fit is HOST-ONLY. Set this LOWER when something else lives in the pool, or HIGHER (or `0`) to suppress the refusal and get the late failure back — it does not make the model fit. Parsed as decimal digits only: a value with a sign, a space or trailing garbage is IGNORED and the probe stands, because reading a typo as `0` would silently disable the guard. Compared against the pool TOTAL, not the free bytes, so the verdict does not move with contention. The bound counts WEIGHTS only, never the KV cache, activations or the driver context, so a checkpoint just under the pool still passes and can still fail later. It can also count a little too MUCH: a tensor present in the file that this load will not stage — the MTP / `nextn` block on a load with no speculator, 8.33 GiB of the measured 369.96 GiB checkpoint — is still in the sum, so a budget in that narrow window refuses a weight set that would have fitted; raise this value if you land in it ([#1136](https://github.com/mudler/vllm.cpp/issues/1136)). Inert on every platform that does not stage weights, which today means everything except CUDA — including every `--device cpu` load, and including ROCm, Vulkan and Metal, which read the mapping where it lies and have no staging allocation to fail |
 
 ## Rollback and bisect switches
 

docs/USAGE.md

@@ -3629,6 +3629,69 @@ A run whose `steps` is 0, or whose `exhausted` is large, is not a measurement of
 streaming, whatever the startup line said. See
 [`docs/ENVIRONMENT.md`](ENVIRONMENT.md) for every knob and its parsing rules.
 
+### `--device cuda` refuses a checkpoint it cannot hold
+
+Streaming is a **host** capability. The GGUF mapping is borrowed in place on the
+CPU path, so a routed-expert tower costs no resident bytes, which is the whole
+reason a 369.96 GiB checkpoint serves on a 119.631 GiB box. A weight-staging
+device has no such lane: it copies every tower into device memory, one
+`cudaMalloc` per stacked `[E*N,K]` tower.
+
+For `Qwen3.8-2.4T-A95B UD-Q1_0` that is 276 towers of 1,275,068,416 bytes plus
+three of 2,818,572,288, so 335.62 GiB in total, against a pool `cudaMemGetInfo`
+reports as
+128,452,956,160 bytes (119.631 GiB). Until that lane exists
+([#1124](https://github.com/mudler/vllm.cpp/issues/1124)), the engine **refuses
+at load** and names what is missing:
+
+```text
+device 'cuda' cannot serve this GGUF: staging its weights needs at least N bytes
+(X GiB) of device memory across T tensors, the largest single allocation being M
+bytes (Y GiB, '<tensor>'), and this device's memory pool is B bytes (Z GiB).
+THE MISSING PART: ... there is no device-side expert slot store and no device
+streaming lane ... Use device=cpu, which serves this checkpoint today, or a
+checkpoint that fits the pool.
+```
+
+It used to load for 26 minutes, report ready, and then die on the first request
+with `vt cuda: cudaMalloc: out of memory` from inside the engine's busy loop
+([#1123](https://github.com/mudler/vllm.cpp/issues/1123)).
+
+The refusal is keyed on the measured condition and not on the device or the file
+format, so **a GGUF that fits the pool still loads on `--device cuda`**. Three
+things it deliberately does not do:
+
+- it never fires on a platform that does not stage weights, so every
+  `--device cpu` load is unchanged;
+- it never fires when no budget is known. Today exactly one platform stages
+  weights (CUDA) and exactly one probes a budget (CUDA, with `cudaMemGetInfo`),
+  so **every NVIDIA GPU this build runs on — discrete or GB10 — gets both the
+  probe and the refusal**, while ROCm, Vulkan and Metal answer
+  `needs_weight_staging() == false`: they read the GGUF mapping where it already
+  lies, so there is no staging allocation to fail and nothing for this check to
+  decide. What is owed there is the `Backend::DeviceMemoryInfo` probe CUDA does
+  not implement ([#1126](https://github.com/mudler/vllm.cpp/issues/1126)), which
+  is a different capability;
+- it counts **weights only**. The KV cache, activations, scratch pools and the
+  driver context are not in the bound, so a checkpoint just under the pool
+  passes this check and can still fail later;
+- it can also count a little **too much**: a tensor present in the file that this
+  load will not stage — the MTP / `nextn` block on a load with no speculator, 8.33
+  GiB of the measured 369.96 GiB checkpoint — is still in the sum, so a budget in
+  that narrow window refuses a weight set that would have fitted. Raise
+  `VT_DEVICE_WEIGHT_BUDGET_BYTES` if you land in it
+  ([#1136](https://github.com/mudler/vllm.cpp/issues/1136)).
+
+`VT_DEVICE_WEIGHT_BUDGET_BYTES` moves the budget: lower it when something else
+lives in the pool, or raise it (or set `0`) to suppress the refusal and get the
+late failure back. It does not make the model fit.
+
+**The instrument matters here.** `nvidia-smi
+--query-gpu=memory.total,memory.free,memory.used` answers `[N/A], [N/A], [N/A]`
+on a GB10, because host and device share one pool. `cudaMemGetInfo` answers
+honestly, and its `total` is EXACTLY `/proc/meminfo MemTotal`
+(125442340 kB) times 1024. Do not size this from `nvidia-smi`.
+
 ## SSE keepalives on long prefill
 
 Async chat/completion streams can emit SSE **comment** frames (`:\n\n`) while

include/vllm/entrypoints/model_loader.h

@@ -199,6 +199,31 @@ struct EngineParams {
 vt::Queue SelectQueueForModel(std::string_view architecture,
                               vllm::Device device);
 
+// The device type `SelectQueueForModel` will pick.
+//
+// It exists because the load-time GGUF device-fit refusal (issue #1123) has to
+// know the target device BEFORE any weight I/O, and the load's own queue is not
+// created until after the weights are loaded. Throws for an explicitly named
+// device that this build/process cannot serve, exactly as the queue selector
+// does; the auto arm falls back to `kCPU` instead of throwing, also exactly as it
+// does.
+//
+// The two agree because both arms run one implementation, and on the AUTO arm
+// that implementation CREATES A QUEUE and destroys it. #1136 measured why the
+// cheaper version was wrong: `SelectQueueForModel`'s auto arm falls back to CPU
+// when `CreateQueue()` throws, so a resolver that only asked `CurrentPlatform()`
+// answered `'cuda'` on a box where the load would run on CPU, and the fit refusal
+// then rejected a checkpoint by naming a device nothing was going to run on.
+//
+// The cost of agreeing is one extra stream created and destroyed, and it is bounded
+// by where this function is called: the load-time GGUF fit check is the only caller
+// outside `SelectQueueForModel` itself, so a safetensors load pays nothing, an
+// explicitly named device pays nothing (that arm creates no queue here), and an
+// auto-arm GGUF load pays one `CreateQueue`/`DestroyQueue` pair. That is not free,
+// and it is smaller than removing a working load.
+vt::DeviceType ResolveModelDeviceType(std::string_view architecture,
+                                      vllm::Device device);
+
 // Owns the full V1 engine stack (config + weights + tokenizer + Scheduler +
 // runner -> Executor -> EngineCore; Input/OutputProcessor -> LLMEngine) for a
 // registered model. The concrete weights/forward are held behind LoadedModel;

include/vllm/model_executor/model_loader/gguf_device_fit.h

@@ -0,0 +1,141 @@
+// vllm.cpp original. Pinned vLLM (555967922) has no GGUF load format — the whole
+// tree carries two incidental mentions of the word and no loader — so there is no
+// upstream counterpart to mirror.
+//
+// The closest upstream idea is the startup memory profile, and it answers a
+// different question. `GPUWorker.determine_available_memory`
+// (`vllm/v1/worker/gpu_worker.py:451-495`) runs `memory_profiling` around
+// `profile_run` (`vllm/v1/worker/gpu/model_runner.py:682`) and passes the weight
+// bytes IN as a known quantity, `weights_memory=model_memory_usage`
+// (`gpu_worker.py:493`). That quantity is recorded AFTER the load has finished
+// (`gpu/model_runner.py:315`). So upstream measures what is left once the weights
+// are resident; it never asks whether they will fit, because by then it has paid
+// for them. This file asks the question upstream does not.
+//
+// ENG-EXPERT-STREAM, issue #1123: a load-time answer to "can this device
+// actually hold this checkpoint's weights?".
+//
+// The failure this exists to remove: `Qwen3.8-2.4T-A95B UD-Q1_0` (369.96 GiB)
+// reached a serving state on `--device cuda` on a 119.631 GiB GB10 after 26
+// minutes and then died on the FIRST forward with `vt cuda: cudaMalloc: out of
+// memory`. The load succeeds because a keep-quant expert tower is BORROWED from
+// the GGUF mapping and costs zero anonymous bytes; the forward dies because a
+// weight-staging platform copies each borrowed tower into device memory
+// (`ResidentWeight`, `qwen3_5.cpp:1011`, `d.b.Alloc(w.bytes.size())` on a
+// STACKED `[E*N,K]` tower: 276 towers of 1,275,068,416 bytes plus 3 of
+// 2,818,572,288, so 360,374,599,680 bytes = 335.62 GiB in total).
+//
+// See `.agents/specs/expert-streaming.md`, section "`--device cuda` loads for 26
+// minutes and then dies", for the measurement this is keyed on.
+#pragma once
+
+#include <cstddef>
+#include <string>
+#include <string_view>
+
+#include "vllm/model_executor/model_loader/gguf_reader.h"
+
+namespace vllm {
+
+// A lower bound on the device-resident bytes a weight-staging platform must
+// allocate to stage EVERY TENSOR IN THIS FILE.
+//
+// Read that scope literally, because it is where the one over-count comes from: it
+// is NOT a lower bound on what a particular load stages, since a load may stage a
+// subset of the file. The per-tensor term is exact-or-low; the SET is exact-or-high.
+//
+// Per tensor the bound is `min(gguf_bytes, elems * model_dtype_bytes)`: a weight
+// the loader keeps quantized is staged verbatim (`gguf_bytes`), a weight it
+// expands is staged at the model dtype (`elems * 2` for bf16), and which of the
+// two happens is a per-tensor loader policy this header deliberately does not
+// try to predict. Taking the minimum makes the per-tensor term a true lower
+// bound on that tensor's staged size.
+//
+// Both directions of error are named here rather than claimed away, because a
+// bound whose error direction is unstated is not a bound.
+//
+//   * It can OVER-count, by including a tensor the loader never stages. The one
+//     such class in this tree is the MTP / `nextn` block, which is attached only
+//     when a speculator is configured
+//     (`model_loader.cpp`, the `speculative_config->method == "mtp"` guard).
+//     Measured on the target checkpoint, that is block 92: 20 tensors,
+//     8,940,488,704 of 397,245,341,184 bytes, so 2.2506 %.
+//   * It UNDER-counts by everything that is NOT a weight: KV cache,
+//     activations, the device scratch pool and the driver context. That term is
+//     far larger than 2.2506 %.
+//
+// The two errors are on DIFFERENT quantities and do not cancel, so neither
+// rescues the other. In particular the refusal CAN over-refuse: a budget in
+// [what a default load stages, what this counts) rejects a weight set that fits.
+// On the target checkpoint that window is 8.33 GiB wide on a 369.96 GiB
+// checkpoint, `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the operator's way out of it,
+// and `test_gguf_device_fit` pins the direction rather than leaving it described
+// (issue #1136). Closing it means teaching the bound which tensors THIS load will
+// stage, which is load policy and not a property of the file, so it is owed and
+// not invented here. The under-count is owed to the startup memory profile
+// (`KV-WARMUP-PROFILE`) for the same reason: an invented headroom fraction here
+// would be the guess this bound exists to avoid.
+struct GgufStagedFootprint {
+  // The bound, in bytes. The name is accurate for what it measures — the sum of
+  // per-tensor lower bounds over the file's whole tensor table — and it is NOT a
+  // lower bound on one load's staging, for the reason above.
+  size_t lower_bound_bytes = 0;
+  // How many tensor records went into it. A caller that reports a footprint
+  // without this cannot say how many things it examined.
+  size_t tensor_count = 0;
+  // The largest single tensor, which is the largest single contiguous
+  // allocation the staging path will ask the driver for. An aggregate that fits
+  // is not the same as a contiguous block that fits.
+  size_t largest_tensor_bytes = 0;
+  std::string largest_tensor_name;
+};
+
+// `model_dtype_bytes` is the resolved model dtype's size (2 for bf16, which is
+// what every GGUF path here loads at). vLLM resolves ONE model dtype and every
+// layer inherits it, so one value is the faithful shape.
+GgufStagedFootprint GgufStagedWeightFootprint(const GgufFile& gguf,
+                                              size_t model_dtype_bytes = 2);
+
+// The budget to compare a footprint against, in bytes, or 0 for UNKNOWN.
+//
+// `device_memory_total_bytes` is the platform's own probe
+// (`ResidencyPolicy::device_memory_total_bytes`), which is 0 on every platform
+// that does not probe one. `VT_DEVICE_WEIGHT_BUDGET_BYTES` overrides it, for an
+// operator whose pool is smaller than the probe reports because something else
+// lives in it, and for an operator who wants to attempt the load anyway. A
+// value of 0 in the environment means "unknown", i.e. disables the check, and
+// an unparseable value is ignored rather than treated as 0, because silently
+// disabling a guard on a typo is the failure shape this tree refuses.
+//
+// TOTAL rather than FREE on purpose: `free` at load time carries the page cache
+// and whatever else the box is doing, which would make the verdict a function
+// of contention. `total` is a device property.
+size_t DeviceWeightBudgetBytes(size_t device_memory_total_bytes);
+
+// The verdict. `refuse == false` with a zero budget means "not decided",
+// which is NOT the same as "it fits" — see the comment on the budget above.
+struct DeviceWeightFit {
+  bool refuse = false;
+  size_t needed_bytes = 0;
+  size_t budget_bytes = 0;
+  // Empty unless `refuse`. Names the device, both numbers, the missing
+  // capability and what to do instead.
+  std::string message;
+};
+
+// The predicate, in one place so one description exists:
+//
+//   refuse  <=>  needs_weight_staging  AND  budget != 0  AND  needed > budget
+//
+// Keyed on the MEASURED condition, never on "CUDA + GGUF" and never on an
+// architecture name, so a GGUF that genuinely fits the pool still loads.
+// Strictly greater than: a checkpoint whose footprint exactly equals the budget is
+// not refused here. The footprint is approximate in both directions, so equality is
+// not evidence of anything, and the tie goes to attempting the load.
+DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf,
+                                     std::string_view device_name,
+                                     bool needs_weight_staging,
+                                     size_t budget_bytes,
+                                     size_t model_dtype_bytes = 2);
+
+}  // namespace vllm