feat(recipe): Add Qwen2.5-7B ACK agentic recipe support - #1
Open
zlaazlaa wants to merge 2 commits into
Open
Conversation
KunWuLuan
pushed a commit
that referenced
this pull request
Aug 13, 2026
Author: @sophiayyya @hou2lin ## Summary Add a recipe-side **Dynamo** rollout backend for verl training (`actor_rollout_ref.rollout.name=dynamo`). The backend runs [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) as an async rollout engine: it brings up the Dynamo control plane (etcd + NATS), launches one `dynamo.vllm` worker per rollout GPU, and serves generation through a single Dynamo **frontend** whose **KV-aware router** (`router_mode=kv`) picks the worker with the best prefix-cache overlap. Everything is confined to `recipe/dynamo` and is opt-in via config — no changes to verl core semantics. Generation, sleep/wake, KV-cache clearing, global-step propagation, and bucketed weight updates from the trainer all work in the hybrid (colocated) topology. ## Architecture ``` verl trainer (hybrid, colocated on the rollout GPUs) │ DynamoAgentLoopManager ──▶ DynamoLLMServerManager ──▶ DynamoServerManager │ (one shared frontend, no verl-side LB) ▼ Dynamo frontend (KV-aware router) │ HTTP /v1/completions ┌─────────────────────┼─────────────────────┐ ▼ ▼ ▼ dynamo.vllm #0 dynamo.vllm #1 ... dynamo.vllm #N + ZMQ control + ZMQ control + ZMQ control (weight update / sleep / kv-clear) └──────── etcd + NATS (control plane, master node) ──────┘ ``` Routing lives inside Dynamo's KV router, **not** in verl's `GlobalRequestLoadBalancer`; verl only ever talks to the single shared frontend actor. ## Changes (all under `recipe/dynamo/`) **Entry point & registration** - `main_dynamo.py` — training entry point; runs `main_ppo.run_ppo` with the dynamo Hydra config. - `config/dynamo_trainer.yaml` — inherits `ppo_trainer`; only deltas are `rollout.name=dynamo` and `rollout.mode=async`. All dynamo-specific knobs live under `rollout.engine_kwargs.dynamo.*`. - `register.py` — for `VERL_USE_EXTERNAL_MODULES`: registers `DynamoReplica` in `RolloutReplicaRegistry`, wires `(dynamo, async) → dynamo_rollout.ServerAdapter`, and monkeypatches `LLMServerManager → DynamoLLMServerManager`. **Serving / lifecycle (`dynamo_async_server.py`)** - `DynamoReplica` (`RolloutReplica`) — owns one logical serving replica across N nodes. `init_hybrid_worker_pool()` takes over all rollout GPUs, maps workers → nodes/GPU ids with per-shard rank offsets, and launches one `DynamoHttpServer` actor per node (master starts etcd/NATS/frontend, slaves wait on the master address, then all wait for frontend readiness). Enforces `TP ≤ gpus_per_node` (CUDA-IPC constraint). - `DynamoHttpServer` — Ray actor acting as GPU placeholder + Dynamo subprocess watchdog. Boots subprocesses in order (etcd → NATS → `dynamo.vllm` workers → frontend), serves `generate()` over HTTP to the frontend (aiohttp), and bridges control RPCs (`collective_rpc`, `wake_up`, `sleep`, `clear_kv_cache`, `set_global_steps`) to the per-worker ZMQ control sidecars. Optionally publishes per-worker `/metrics` endpoints for KV telemetry. **Agent loop / server manager (`dynamo_agent_loop.py`)** - `DynamoServerManager` — talks to the **single** shared frontend actor (asserts exactly one server; intentionally bypasses verl's request load balancer since Dynamo owns routing). - `DynamoLLMServerManager` (`LLMServerManager`) — builds the `DynamoReplica` in hybrid mode and, when `rollout.prometheus.enable=true`, wires the frontend into the prometheus target (requires `disable_log_stats=False`). - `DynamoAgentLoopManager` / `DynamoAgentLoopWorker` — thin `AgentLoopManager` subclasses so the standard agent-loop execution model runs against the Dynamo server manager. **Trainer-side rollout adapter (`dynamo_rollout.py`)** - `ServerAdapter` (extends the vLLM async `ServerAdapter`) — HTTP generation path is unchanged; overrides only the Ray actor naming so control RPCs land on `dynamo_server_*`. Gates control RPCs to **one trainer rank per physical node** (`_is_node_control_rank`), and implements `update_weights()` via `BucketedWeightSender` racing the engine-side IPC receiver, followed by per-node `clear_kv_cache` + `set_global_steps`. **vLLM worker integration** - `dynamo_worker_extension.py` — `vLLMDynamoColocateWorkerExtension` overrides `_get_zmq_handle()` to add `VERL_DYNAMO_RANK_OFFSET` to the TP-local rank, so trainer-side sender and engine-side receiver agree on the IPC socket path in the multi-DP-per-node topology. Also wraps `update_weights_from_ipc` in `set_current_vllm_config` for vLLM 0.20 compatibility. - `_dynamo_vllm_with_control.py` — wrapper around `python -m dynamo.vllm` that injects a verl-private **ZMQ control sidecar**: it captures the `AsyncLLM` engine and serves `collective_rpc` / `engine_method` / `generate_direct` requests (endpoint from `VERL_DYNAMO_CONTROL_ZMQ`), enabling weight updates and a direct-generate fallback. **Telemetry** - `metrics_sidecar.py` — stdlib-only scraper that appends per-target `/metrics` snapshots to JSONL over the whole job lifetime. `--targets-glob` (prometheus targets, vLLM arm) / `--endpoints-glob` (dynamo worker `/metrics`, written when `enable_worker_system_metrics=true`). Captures engine-level `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total` so Dynamo and vLLM KV hit rates are computed from the **same underlying vLLM engine counters**. ### Key `engine_kwargs.dynamo.*` knobs | key | default | purpose | |---|---|---| | `namespace` | `verl_dynamo` | Dynamo cluster namespace | | `router_mode` | `kv` | routing strategy (`kv` = KV-aware) | | `frontend_http_port` / `etcd_port` / `etcd_peer_port` / `nats_port` | `0` (auto) | control-plane / frontend ports | | `free_engine_on_train` | `false` | sleep/wake workers around training steps | | `enable_worker_system_metrics` | `true` | expose per-worker `/metrics` (engine-level prefix-cache counters) | | `frontend_connection_limit` | `0` (unlimited) | aiohttp connector limit to the frontend | | `request_timeout_s` | `600` | frontend HTTP request timeout | | `extra_args` / `frontend_extra_args` | `[]` | pass-through CLI args to worker / frontend | ## Validation - Experiment: retool - Model: Qwen30B - wandb report: https://wandb.ai/yangjingyi_algo/verl_Dynamo_compare?nw=nwuseryangjingyi_algo - Dynamo (kv) is ~7% faster than vLLM ; <img width="333" height="274" alt="image" src="https://github.com/user-attachments/assets/a0572271-9da8-4b0a-b556-f0541dc7fddc" /> <meta charset="utf-8"><b style="font-weight:normal;" id="docs-internal-guid-f97569ab-7fff-e09d-4aea-f22d880214aa"><p dir="ltr" style="line-height:1.38;background-color:#ffffff;margin-top:0pt;margin-bottom:5pt;"></p><div dir="ltr" style="margin-left:0pt;" align="left"> Config | ms/token | kv cache hit rate | Resp_len mean -- | -- | -- | -- Dynamo kv, interval=100 | 1.5956 | 89.21% | 876.1 vLLM baseline | 1.7220 | 76.51% | 872.3 </div></b> --------- Co-authored-by: Sophia Yang <sopyang@cw-dfw-cs-001-dc-02.cm.cluster> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Sophia Yang <sopyang@r6515-0154.ipp3a1.colossus.nvidia.com> Co-authored-by: Sophia Yang <sopyang@cw-dfw-cs-001-vscode-01.cm.cluster> Co-authored-by: OpenAI Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds an ACK-oriented Qwen2.5-7B agentic training script and exposes the Harbor ACK base image registry option for SWE-bench runs.
Changes
agentic-qwen2.5-7b.shfor running the SWE-bench agentic demo with Qwen2.5-7B.REGISTRYIMAGE_PULL_SECRETSERVICE_ACCOUNTUSE_BUILDKITBUILDKIT_ADDRESSBASE_IMAGE_REGISTRYand pass it to Harbor asbase_image_registry.Harbor dependency
This PR is intended to be used together with the Harbor PR:
The Harbor PR fixes the ACK Kubernetes stream client isolation issue and adds support for rewriting Dockerfile base image sources during BuildKit builds.
This recipe exposes that capability through:
For example, users can avoid Docker Hub pull limits by pre-syncing SWE-bench base images into ACR and then running:
In that setup, Harbor uses the ACK IMAGE_PULL_SECRET / ACR credentials and pulls base images through the internal ACR VPC endpoint during BuildKit builds, improving speed and reducing 429 Too Many Requests risk.
Example