benchmark: integrate pprof analysis into reports - #1464
Conversation
…scenarios - Use ai-dynamo/aiperf for Load Gen and dynamo mocker for Mock Backend - Deploy mocker with kthena resource definitions - Provide s2 scenario: routing latency under different QPS; provide s7 scenario: latency variance on heterogeneous backends; provide s8 scenario: schedule strategy comparison Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
…others use `rate` Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
Signed-off-by: Leo Xie <stleox@qq.com>
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@StLeoX PTAL when you get a chance. added the pprof parser with scheduler/plugin filtering, CPU and heap support, and fixture-based tests |
|
I expect the Analyzer can be integrated into build_report, instead of a separate module. You can checkout this branch first. |
Signed-off-by: Avinash Kumar Deepak <avinash8655279@gmail.com>
a93ad6a to
b8bae40
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 29 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
benchmark/router-benchmark/scripts/router_ab_test/kubernetes.py:506
- _start_port_forward() always targets "{target_type}/{self.ROUTER_DEPLOYMENT}". This happens to work today because ROUTER_DEPLOYMENT and ROUTER_SVC_NAME are both "kthena-router", but it couples service and deployment naming and will break silently if they ever diverge. Consider selecting the target resource based on target_type.
local_endpoint = f"localhost:{local_port}"
print(f" Starting port-forward ({local_endpoint} → {description})")
target = f"{target_type}/{self.ROUTER_DEPLOYMENT}"
proc = subprocess.Popen(
| mode = str(schedule.get("mode", "")).lower() | ||
| if mode in {"rate", "constant_rate"}: | ||
| cmd.extend(["--request-rate", str(schedule.get("rate", 10))]) | ||
| self._append_traffic_args(cmd, traffic, benchmark_secs) | ||
| else: | ||
| cmd.extend(["--concurrency", str(concurrency.get("connections", 10))]) | ||
| self._append_concurrency_args(cmd, concurrency) |
|
lgtm, it's stacked now |
What type of PR is this?
/kind feature
What this PR does / why we need it:
Integrates pprof analysis into
ResultReporter.build_report().The reporter parses collected CPU and heap profiles, resolves samples to functions, and adds the hottest scheduler/plugin functions to each configuration's
pprof_analysisreport section. Individual profile errors are recorded without preventing the benchmark report from being generated.Which issue(s) this PR fixes:
Fixes #1462
Bug evidence (required for bug-related PRs):
N/A
Special notes for your reviewer:
Uses the checked-in protobuf binding generated from
profile.proto. The tests use captured CPU and heap profiles from the router pprof endpoint.Validation performed:
Does this PR introduce a user-facing change?: