Add multi-engine KV-aware routing to embedded Dynamo vLLM handler - #1
Draft
akshayjadiyanv wants to merge 4 commits into
Draft
Add multi-engine KV-aware routing to embedded Dynamo vLLM handler#1akshayjadiyanv wants to merge 4 commits into
akshayjadiyanv wants to merge 4 commits into
Conversation
Extends the embedded NVIDIA Dynamo mode from a single vLLM engine to one or more GPU-pinned aggregated engines behind a KV-aware dynamo.frontend, so the router has a real placement choice to exploit prefix-cache locality. This is the value-bearing follow-on to the single-engine embedded mode (which showed no routing benefit because there was only one cache to route to). What changed: - New private _dynamo_runtime.py: typed DynamoRuntimeConfig / DynamoVLLMEngineSpec public config plus _DynamoLocalRuntime owning the etcd + frontend + N-engine topology. Per-engine CUDA_VISIBLE_DEVICES, unique DYN_SYSTEM_PORT and ZMQ KV-event ports (all allocated in one pick_port call for guaranteed uniqueness), PYTHONHASHSEED pinned only under KV routing, and per-child etcd endpoint injection instead of mutating global os.environ. - Conditional routing defaults: --no-router-kv-events is dropped only for real ZMQ KV events; --enable-prefix-caching is added in KV mode. Round-robin and approximate modes stay event-free. - Phased readiness waits for every engine's system health endpoint (not just the first registered model, which could let a two-engine run proceed at half capacity) and fail-fast config validation (disjoint GPUs, TP*PP == device count, event/router-mode compatibility) runs at handler construction. - Cleanup terminates the whole child process group so vLLM/CUDA descendants do not leak GPU memory across a restart. - VLLMCompletionsModelHandler / VLLMChatModelHandler gain dynamo_runtime_config; legacy use_dynamo=True (no config) is unchanged and mutually exclusive with dynamo_frontend_kwargs. - vllm_text_completion.py gains --dynamo_engines / --dynamo_router_mode / --dynamo_kv_event_mode so a multi-GPU KV smoke is runnable. - Opt-in vllmDynamoKvTests (2x L4) Dataflow task, intentionally not wired into the default postcommit suite. - New A/B/C benchmark (native-dp / dynamo-rr / dynamo-kv) with a prefix-reuse dataset generator for measuring KV-routing benefit. Unit tests cover topology construction, per-engine env/ports, validation, readiness, health, and cleanup with a fully mocked process launcher (no GPU). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dynamo parses DYN_SYSTEM_PORT as a signed 16-bit integer, but per-engine system ports were allocated via subprocess_server.pick_port, which returns OS ephemeral ports (>= 32768 on Linux). Every dynamo.vllm engine crashed at startup with "invalid value unsigned int `55587`, expected i16 for key SYSTEM_PORT" before reaching model load. Draw engine system ports from a low free range (<= 32767) via _pick_low_ports; frontend HTTP, etcd, and ZMQ KV-event ports remain ordinary ephemeral u16 ports (disjoint from the low range). Also drop DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS, which this Dynamo version has deprecated and ignores (health is now determined by endpoints that register health-check payloads). Adds unit tests asserting engine system ports fit in i16 and the deprecated env var is no longer set.
The A/B/C arms were being compared on inference_batch_latency_micro_secs (a latency-under-concurrency metric) and job wall time (polluted by provisioning and model load). Neither isolates steady-state records/s, which is what the success criteria are defined on. Emit an inference_wall_clock_ms Distribution (epoch ms per completed inference); its min/max/count bound the inference window excluding model load, so throughput = count / ((max - min) / 1000) is a fair per-arm records/s.
Under multi-engine KV-aware routing, a burst of same-prefix requests can concentrate load on the engine that owns that prefix, and the frontend then returns 503 "ResourceExhausted: All workers are busy". With the OpenAI client default of 2 retries this exhausts in ~1-2s and fails the whole Beam bundle / Dataflow job (observed on arm C at 75% reuse; 0/25/95% succeeded). Raise max_retries to 6 on both the sync and async vLLM clients. The client already applies exponential backoff with jitter and honors Retry-After, so this rides out the admission-control window instead of hard-failing. Applies to every arm through the shared handler path, so A/B/C comparisons stay fair.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Extends embedded NVIDIA Dynamo mode from a single vLLM engine to one or more GPU-pinned aggregated engines behind a KV-aware
dynamo.frontend, giving the router a real placement choice so it can exploit prefix-cache locality. This is the value-bearing follow-on to the single-engine embedded mode, which showed no routing benefit (only one cache to route to). ImplementsDYNAMO_DATAFLOW_NEXT_PR_POC_PLAN.md.Changes
_dynamo_runtime.py— typed publicDynamoRuntimeConfig/DynamoVLLMEngineSpecplus private_DynamoLocalRuntimeowning the etcd + frontend + N-engine topology:CUDA_VISIBLE_DEVICES, uniqueDYN_SYSTEM_PORTand ZMQ KV-event ports (all allocated in onepick_portcall for guaranteed uniqueness);PYTHONHASHSEEDpinned only under KV routing (prefix-hash agreement); per-child etcd endpoint injection instead of mutating globalos.environ;--no-router-kv-eventsdropped only for real ZMQ events,--enable-prefix-cachingadded in KV mode;VLLMCompletionsModelHandler/VLLMChatModelHandlergaindynamo_runtime_config=. Legacyuse_dynamo=True(no config) is unchanged and mutually exclusive withdynamo_frontend_kwargs.vllm_text_completion.pygains--dynamo_engines/--dynamo_router_mode/--dynamo_kv_event_mode.vllmDynamoKvTests(2× L4), intentionally not in the default postcommit suite to avoid doubling GPU cost.vllm_dynamo_kv_benchmark.py(armsnative-dp/dynamo-rr/dynamo-kv) + prefix-reuse dataset generator + tests, for measuring KV-routing benefit.Testing
62 unit tests pass (29 runtime + 8 handler + 17 benchmark + 8 pre-existing), all yapf-clean against
setup.cfg. Topology construction, per-engine env/ports, validation, readiness, health, and cleanup are covered with a fully mocked process launcher — no GPU exercised here. Real 2-engine behavior must be validated in the Dataflow 2-GPU smoke gate before any measured run.Reviewer notes
use_dynamo=Truepath are unchanged.native-dp) needs no handler change — it'suse_dynamo=False+data-parallel-size; confirm that flag on the pinned vLLM image via the version gate.