Symptom
On brutus, when all three vllm-xpu instances boot together, vllm-xpu-chat can fail engine init with:
```
ValueError: To serve at least one request with the model's max seq len (131072),
3.36 GiB KV cache is needed, which is larger than the available KV cache memory (2.13 GiB).
Based on the available memory, the estimated maximum model length is 46528.
```
Stopping `vllm-xpu-stt` and letting chat start alone clears the error. Confirmed against config in hosts/brutus/services/vllm-xpu.nix at `maxNumSeqs=3`, `maxModelLen=131072`, `gpuMemoryUtilization=0.75` (chat), `0.07` (embedding), `0.07` (stt).
Why
vLLM's `--gpu-memory-utilization` is a fraction of free VRAM measured at engine init, not total. Whichever instance starts last sees a shrunken pool — chat's `0.75 × free` undershoots the planned ~3.25 GiB KV budget once the other two have claimed theirs. Today the three units have no `Before`/`After` ordering between them so the loser is non-deterministic.
Options
- Serial start ordering. Add `Before`/`After` between the three units so chat measures free VRAM first. Cheap but a chat restart blocks embedding/stt.
- Absolute KV budgets. Switch to vLLM's `kv-cache-memory-bytes` (≥0.10) per instance — no coupling to free-VRAM at init. Requires plumbing in vllm-xpu-nix (`kvCacheMemoryBytes` option) and reworking budgets here.
- Hardcode chat to start first only. Looser variant of (1): only chat declares `Before=vllm-xpu-embedding`/`Before=vllm-xpu-stt`, the others stay independent of each other.
Related
- Surfaced while shipping `cudagraphMode` — the cudagraph crash is fixed; this is the next blocker that shows up cold-booting all three.
- Commit `767a3d3` sized the KV budget assuming chat sees a roughly empty XPU at init; the assumption doesn't hold post-reboot.
Symptom
On brutus, when all three vllm-xpu instances boot together,
vllm-xpu-chatcan fail engine init with:```
ValueError: To serve at least one request with the model's max seq len (131072),
3.36 GiB KV cache is needed, which is larger than the available KV cache memory (2.13 GiB).
Based on the available memory, the estimated maximum model length is 46528.
```
Stopping `vllm-xpu-stt` and letting chat start alone clears the error. Confirmed against config in hosts/brutus/services/vllm-xpu.nix at `maxNumSeqs=3`, `maxModelLen=131072`, `gpuMemoryUtilization=0.75` (chat), `0.07` (embedding), `0.07` (stt).
Why
vLLM's `--gpu-memory-utilization` is a fraction of free VRAM measured at engine init, not total. Whichever instance starts last sees a shrunken pool — chat's `0.75 × free` undershoots the planned ~3.25 GiB KV budget once the other two have claimed theirs. Today the three units have no `Before`/`After` ordering between them so the loser is non-deterministic.
Options
Related