Skip to content

brutus/vllm-xpu: co-resident instances under-budget KV on cold boot (ordering) #34

Description

@jasonboukheir

Symptom

On brutus, when all three vllm-xpu instances boot together, vllm-xpu-chat can fail engine init with:

```
ValueError: To serve at least one request with the model's max seq len (131072),
3.36 GiB KV cache is needed, which is larger than the available KV cache memory (2.13 GiB).
Based on the available memory, the estimated maximum model length is 46528.
```

Stopping `vllm-xpu-stt` and letting chat start alone clears the error. Confirmed against config in hosts/brutus/services/vllm-xpu.nix at `maxNumSeqs=3`, `maxModelLen=131072`, `gpuMemoryUtilization=0.75` (chat), `0.07` (embedding), `0.07` (stt).

Why

vLLM's `--gpu-memory-utilization` is a fraction of free VRAM measured at engine init, not total. Whichever instance starts last sees a shrunken pool — chat's `0.75 × free` undershoots the planned ~3.25 GiB KV budget once the other two have claimed theirs. Today the three units have no `Before`/`After` ordering between them so the loser is non-deterministic.

Options

  1. Serial start ordering. Add `Before`/`After` between the three units so chat measures free VRAM first. Cheap but a chat restart blocks embedding/stt.
  2. Absolute KV budgets. Switch to vLLM's `kv-cache-memory-bytes` (≥0.10) per instance — no coupling to free-VRAM at init. Requires plumbing in vllm-xpu-nix (`kvCacheMemoryBytes` option) and reworking budgets here.
  3. Hardcode chat to start first only. Looser variant of (1): only chat declares `Before=vllm-xpu-embedding`/`Before=vllm-xpu-stt`, the others stay independent of each other.

Related

  • Surfaced while shipping `cudagraphMode` — the cudagraph crash is fixed; this is the next blocker that shows up cold-booting all three.
  • Commit `767a3d3` sized the KV budget assuming chat sees a roughly empty XPU at init; the assumption doesn't hold post-reboot.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions