Follow-up to #1123, which added the knob as an environment variable on purpose.
#1123 lands a load-time refusal for a GGUF whose device-resident weight bytes exceed the device memory pool. Its budget comes from ResidencyPolicy::device_memory_total_bytes (probed with cudaMemGetInfo), and VT_DEVICE_WEIGHT_BUDGET_BYTES overrides it. The override has two real uses: an operator whose pool is smaller than the probe reports because something else lives in it, and an operator who wants to try the load anyway.
It should be a config key rather than an environment variable, for the same reason ENG-RESIDENCY-CONFIG gives for the five VT_GGUF_* / VT_MOE_EXPERT_STREAM* knobs it is converting: an operator reaches for --offload-config to control weight placement and finds nothing there for the tier the big-model case needs.
It was NOT done in the same flow because #1110 / PR #1119 is in flight and adds exactly the vllm_cpp namespace inside --offload-config that this key belongs in, together with include/vllm/config/weight_residency.h. Landing a second, competing config surface while that one is unmerged would create the conflict both changes then have to resolve, in the file (src/vllm/entrypoints/model_loader.cpp) they both already touch.
What closing this looks like, once #1119 has landed:
- a
device_weight_budget_bytes key under the vllm_cpp object, parsed by the same parser and refused by name when unknown, like its siblings;
- the loader reads
EngineParams::weight_residency instead of std::getenv;
- the environment variable kept as the documented A/B override or retired, whichever
ENG-RESIDENCY-CONFIG chose for the other five;
docs/ENVIRONMENT.md and docs/USAGE.md follow.
Owning row: ENG-EXPERT-STREAM, listed under ## Owed in expert-streaming.md.
Follow-up to #1123, which added the knob as an environment variable on purpose.
#1123 lands a load-time refusal for a GGUF whose device-resident weight bytes exceed the device memory pool. Its budget comes from
ResidencyPolicy::device_memory_total_bytes(probed withcudaMemGetInfo), andVT_DEVICE_WEIGHT_BUDGET_BYTESoverrides it. The override has two real uses: an operator whose pool is smaller than the probe reports because something else lives in it, and an operator who wants to try the load anyway.It should be a config key rather than an environment variable, for the same reason
ENG-RESIDENCY-CONFIGgives for the fiveVT_GGUF_*/VT_MOE_EXPERT_STREAM*knobs it is converting: an operator reaches for--offload-configto control weight placement and finds nothing there for the tier the big-model case needs.It was NOT done in the same flow because #1110 / PR #1119 is in flight and adds exactly the
vllm_cppnamespace inside--offload-configthat this key belongs in, together withinclude/vllm/config/weight_residency.h. Landing a second, competing config surface while that one is unmerged would create the conflict both changes then have to resolve, in the file (src/vllm/entrypoints/model_loader.cpp) they both already touch.What closing this looks like, once #1119 has landed:
device_weight_budget_byteskey under thevllm_cppobject, parsed by the same parser and refused by name when unknown, like its siblings;EngineParams::weight_residencyinstead ofstd::getenv;ENG-RESIDENCY-CONFIGchose for the other five;docs/ENVIRONMENT.mdanddocs/USAGE.mdfollow.Owning row:
ENG-EXPERT-STREAM, listed under## Owedinexpert-streaming.md.