Skip to content

VT_DEVICE_WEIGHT_BUDGET_BYTES should be a weight-residency config key, not an environment variable #1127

Description

@localai-bot

Follow-up to #1123, which added the knob as an environment variable on purpose.

#1123 lands a load-time refusal for a GGUF whose device-resident weight bytes exceed the device memory pool. Its budget comes from ResidencyPolicy::device_memory_total_bytes (probed with cudaMemGetInfo), and VT_DEVICE_WEIGHT_BUDGET_BYTES overrides it. The override has two real uses: an operator whose pool is smaller than the probe reports because something else lives in it, and an operator who wants to try the load anyway.

It should be a config key rather than an environment variable, for the same reason ENG-RESIDENCY-CONFIG gives for the five VT_GGUF_* / VT_MOE_EXPERT_STREAM* knobs it is converting: an operator reaches for --offload-config to control weight placement and finds nothing there for the tier the big-model case needs.

It was NOT done in the same flow because #1110 / PR #1119 is in flight and adds exactly the vllm_cpp namespace inside --offload-config that this key belongs in, together with include/vllm/config/weight_residency.h. Landing a second, competing config surface while that one is unmerged would create the conflict both changes then have to resolve, in the file (src/vllm/entrypoints/model_loader.cpp) they both already touch.

What closing this looks like, once #1119 has landed:

  • a device_weight_budget_bytes key under the vllm_cpp object, parsed by the same parser and refused by name when unknown, like its siblings;
  • the loader reads EngineParams::weight_residency instead of std::getenv;
  • the environment variable kept as the documented A/B override or retired, whichever ENG-RESIDENCY-CONFIG chose for the other five;
  • docs/ENVIRONMENT.md and docs/USAGE.md follow.

Owning row: ENG-EXPERT-STREAM, listed under ## Owed in expert-streaming.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions