Repository navigation
[PowerX] replace client GPU sampling with native telemetry / [PowerX] 用原生遥测替代客户端 GPU 采样 - #3684
edwingao28 wants to merge 17 commits into
Conversation
…collector Split out of the benchmark_lib.sh port (#3610) so the port diff is only the port. None of these paths is reachable from a checked-in config: - llm-d: Dockerfile, binaries, recipes and the multi-node job/submit/server scripts. - SWE-bench Lite and Modal: run_swebench_eval and its agentic generation, Modal credential handling, scorer, task YAML, runtime patches, workflow inputs, secrets, uploads, runtime settings and the threshold. - native_power_collect.sh and native_power_lifecycle.sh, used only by the retired AMD lanes. The adapter that reads LOGS/native_power stays.
…rics patches - SPEED-Bench: vLLM ships --chat-template-kwargs for client-rendered datasets since v0.23.0 (vllm-project/vllm#44244). Default collection to vllm/vllm-openai:v0.30.0, note the minimum on the workflow input, and remove apply_chat_template_kwargs_shim and the DSpark preflight. - MiniMax-M3 TRT-LLM AgentX: stop patching PerfMetricsManager off. return_perf_metrics, which these recipes need for trtllm-serve's Prometheus metrics, also enables per-step timing until TensorRT-LLM decouples the two upstream. Both recipes now share minimaxm3-trtllm-agentx.sh, which only applies the store=false patch.
…/SWE-bench leftovers Remove 18 benchmark_lib.sh functions with no callers (_background_process_descendants, _background_process_groups_alive, _background_process_is_running, agentic_kv_offload_enabled, agentic_pip_install, append_command, exit_after_background_process_cleanup, require_agentic_kv_offload_backend, require_agentic_kv_offload_none, rewrite_lm_eval_meta_env, run_amd_multinode_after_preflight, select_available_server_port, setup_eval_context, stop_background_process_groups, stop_background_process_tree, validate_required_agentic_server_metrics, write_agentic_result_json, write_command) along with the one test that exercised select_available_server_port, the llmd-vllm case in multi_node/runtime_settings.sh, and SWE-bench's "resolved" metric family in eval result collection.
Nothing reads metrics_plots.png or workload_distribution_plots.png. Delete generate_aiperf_plots.py, drop the histogram from analyze_benchmark_distributions (the ISL/OSL summary stays), and drop matplotlib from the AgentX client venv.
Delete benchmarks/benchmark_lib.sh and move its behavior into the container-side Python package infx/bench (stdlib only, Python 3.10): wait, fixed-seq, agentic and eval commands behind one `python3 -m infx.bench` entrypoint, with shared process, readiness and input helpers. srt entry scripts become thin shims; recipe paths are unchanged. benchmarks/check_env.sh keeps check_env_vars for workflows and recipe setup scripts. AgentX single-concurrency validation moves from the workflow into LaunchRequest. Also drops code only the retired AMD/TileRT lanes used (server_watch, agentic --server-pid, the monitor command, fixed-seq point mode), the PROFILE trace relay, and dead topology aliases from the meta_env.json mapping.
Follows the plot removal in the base PR: no generate_aiperf_plots call and no matplotlib in the AIPerf client venv.
移除客户端 GPU 功耗采样，改用 SRT 原生测量窗口。
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @edwingao28's task in 1m 11s —— View job Review summary
🔴 1 blocking issue (inline): What I checked and found OK: the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
将功耗采样清理的变更日志链接更新为 PR 3684。
There was a problem hiding this comment.
Beyond the inline finding, I also checked two related single-node power paths: fixed_seq.py's new unconditional SRT_MEASUREMENT_WINDOW_DIR requirement looks intentional (its own comment states removing the local sampler must not silently downgrade measured points to unmeasured), and the GPU_COUNT lookup in fixed_sequence.py's aggregate_power_result can't actually raise in practice, since srt_slurm/single_node.py already requires GPU_COUNT for any single-node launch before results processing ever runs.
Extended reasoning...
This PR removes client-side GPU power sampling from single- and multi-node benchmark/launch paths (run.py, fixed_seq.py, fixed_sequence.py, single_node.py, plus tests) in favor of SRT measurement windows, with no auth/crypto surface touched. A confirmed inline finding already flags that single-node AgentX jobs now hard-fail when SRT_MEASUREMENT_WINDOW_DIR is unset, which matches the author's own stated "blocker" about single-node lanes not yet being covered. I additionally traced two other candidate issues from the ruled-out list (fixed_seq.py's unconditional env.require, and a potential GPU_COUNT KeyError in aggregate_power_result) and confirmed both are either intentional or unreachable given the launcher's existing GPU_COUNT requirement.
|
|
||
| def _power_mode(enabled: str, multinode: bool, env: Mapping[str, str]) -> PowerMode: | ||
| def _power_mode(enabled: str, env: Mapping[str, str]) -> PowerMode: | ||
| if enabled not in TRUE_VALUES: | ||
| return "off" |
There was a problem hiding this comment.
🔴 Single-node AgentX jobs with ENABLE_AGENTX_POWER=1 and REQUIRE_POWER=1 now fail the whole run instead of measuring power locally, once a launcher hasn't set SRT_MEASUREMENT_WINDOW_DIR. _power_mode (run.py:118) returns "missing" for any job lacking that var regardless of node count; run.py:204's --multinode-contract-missing path then reaches power_adapter.py's _fail_multinode_adapter, which returns 1 when --require-power is set. On the base branch single-node always got "monitor" mode (local GpuMonitor sampling), which never depended on that var. Fix: don't let REQUIRE_POWER enforcement for single-node reach the hard-fail path until native telemetry/SRT_MEASUREMENT_WINDOW_DIR is actually provisioned for that lane (the PR's own stated blocker).
Why this was flagged
Trigger: a single-node AgentX point with ENABLE_AGENTX_POWER=1 and REQUIRE_POWER=1, launched before the operator's infra sets SRT_MEASUREMENT_WINDOW_DIR, entering via Plan.from_env -> _power_mode (run.py:115-118). That function now returns "missing" there, so _power_audit (run.py:199-204) sends --multinode-contract-missing --require-power to power_adapter.py's main(), whose _fail_multinode_adapter (power_adapter.py:227-229) returns 1, failing the benchmark run entirely. On the base branch this same job got power="monitor" and ran infx.bench.gpu_monitor.GpuMonitor locally, succeeding independent of SRT_MEASUREMENT_WINDOW_DIR. No check in run.py gates REQUIRE_POWER on whether the launcher has actually rolled out native telemetry for the single-node lane; the PR description calls this exact gap a merge blocker but the code enforces it unconditionally.
Verification: Base run.py _power_mode returned "monitor" for any non-multinode job, so single-node AgentX jobs sampled GPUs locally via GpuMonitor regardless of SRT_MEASUREMENT_WINDOW_DIR. The new _power_mode (run.py:116-119) drops the node-count branch and returns "missing" whenever SRT_MEASUREMENT_WINDOW_DIR is unset, which then reaches _fail_multinode_adapter, returning 1 when require_power (power_adapter.py:227-229), propagated as a full run failure.
为单节点 NVIDIA 与 AMD 作业接入原生功耗采集、产物保留和 AgentX 结果处理。
edwingao28
changed the title
中文：明确单节点结果收尾函数的名称。
- uv everywhere: shared infx/bench/uv.py; lm-eval, vendor evals, BFCL and the AIPerf venv install with uv (BFCL passes --excludes to reuse the image stack). - fixed-seq: drop the pip3 runtime install and SWEEP_FLAGS so single- and multi-node clients send the same policy. - AgentX: drop inputs the agentx scenario (#3646) owns or nobody varies; keep only the two 062126 trace loaders; move venv deps to requirements.txt. - Tests audited for regression value; perf-changelog entry for the loader pins.
中文：使原生功率遥测适配基准客户端重构。
精简原生功耗遥测的配置说明。
中文：将原生遥测接入当前主分支并解决合并冲突。
中文：同步主分支的 Klaud 更新。
中文：同步主分支后保留 llm-d 的脚本、配方、文档和客户端测试。
|
close as superseded by #3781 |
3 participants
Description
Replace client GPU sampling with native NVIDIA/AMD telemetry for single-node throughput and AgentX. Retain formal results and provenance, finalize AgentX power, and preserve legacy CSV reads.
Testing: CPU regressions, 21 native configuration renders, Ruff and workflow security pass; hardware qualification pending.
Dependency: #3610; AMD producer patch upstreaming remains pending.
中文
以原生 NVIDIA/AMD 遥测替代单节点吞吐量和 AgentX 的客户端 GPU 采样。保留正式结果和来源信息，完成 AgentX 功耗处理，并保留历史 CSV 读取。
测试： CPU 回归测试、21 项原生配置生成检查、Ruff 和 workflow 安全检查通过；硬件验收待完成。
依赖： #3610；AMD producer 补丁尚待提交上游。
GPT-6 用于实现及委派审查，精确变体和早期贡献者模型版本无法核实。
AI model disclosure
Related Issue
Stacked on #3610.
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesfull-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.