feat: add --memory flag to measure RAM and VRAM per phase by DingmaomaoBJTU · Pull Request #861 · microsoft/winml-cli

DingmaomaoBJTU · 2026-06-10T08:13:54Z

Summary

Add --memory / --no-memory CLI flag to winml perf that measures process RAM (RSS) and VRAM (GPU/NPU dedicated + shared) at phase boundaries: baseline → compile → inference.

Output

Memory:
  RAM:  670.7 MB -> model load: +77.3 MB | inference: +1.4 MB | total: +78.7 MB
  VRAM: 17.2/17.6 MB (local/shared) -> model load: +17.2/+17.6 MB | inference: +0.0/+0.0 MB | total: +17.2/+17.6 MB

Design

RAM: psutil RSS snapshots at phase boundaries (baseline after _load_model(), after compile(), after benchmark loop)
VRAM: PDH \GPU Process Memory counters queried at same phase boundaries, reported as local/shared separately
Default: enabled (--no-memory to disable). Snapshots are taken between phases and add zero overhead to latency measurements
VRAM line only shown when device memory > 0 (hidden for CPU-only runs)

Display changes

Hardware section: Mem → RAM, Device Mem → VRAM
Memory section: single line per metric with -> showing deltas

Files changed

src/winml/modelkit/commands/perf.py — memory integration in _run_single(), console display, _resolve_adapter_luid()
src/winml/modelkit/session/monitor/memory_tracker.py — get_rss_mb() + get_vram_mb() (~50 lines)
src/winml/modelkit/commands/_live_chart.py — rename labels
src/winml/modelkit/session/monitor/live_display.py — rename labels
tests/unit/session/monitor/test_memory_tracker.py — unit tests

Implement per-phase memory tracking that captures Working Set, Private Bytes, and device (NPU/GPU) memory at each benchmark phase boundary: baseline, post-load, post-compile, and post-inference. Key design decisions: - Default enabled (--no-memory to disable) since snapshots are taken between phases and add zero overhead to latency measurements - Pure ctypes implementation (K32GetProcessMemoryInfo) with no new dependencies - Device memory via single-shot PDH query reusing existing adapter resolution logic - Console output shows a table with per-phase deltas and peak summary - JSON output includes full memory profile under 'memory' key New files: - session/monitor/memory_tracker.py: MemoryTracker, MemorySnapshot, MemoryProfile dataclasses and Windows/Linux memory query functions - tests/unit/session/monitor/test_memory_tracker.py: unit tests

…play Shows the actual memory footprint during inference (steady state) rather than the process lifetime peak which may include transient allocations from model loading or compilation.

xieofxie · 2026-06-10T08:41:54Z

I think we could just enabled it, it doesn't take much time

xieofxie · 2026-06-10T08:42:05Z

Also consider adding this to --monitor

…emory-measurement

The test from PR #855 mocks BenchmarkResult but did not set memory_profile=None, causing MagicMock comparison failure in display_console_report.

- Remove snapshot_post_load (load+compile combined as model_load) - Use psutil for RSS with ctypes/proc fallback - Simplify MemorySnapshot to rss_mb + peak_wset_mb + device fields - Add delta breakdown in console: model load / inference / total - Update tests for new 3-phase API (14 tests passing)

Call WinMLSession._init_winml_eps_once() before taking the baseline snapshot so one-time EP DLL loading costs are excluded from the model_load_delta metric.

- Reorder: load → compile → snapshot → generate_inputs → benchmark (matches reference: input alloc now in inference delta, not model load) - Add gc.collect() before baseline and post_compile snapshots - Fix format bug: use :+.1f for deltas (shows -24.8 not +-24.8)

- Remove ctypes/proc fallback (psutil is already a dependency) - Remove peak_wset_mb, peak_delta_mb (not meaningful for inference) - Remove device_shared_mb (unused in display) - Remove MemorySnapshot class (just store raw floats) - Flatten MemoryProfile fields for direct access

Inline RSS measurement directly in PerfBenchmark.run() — just 3 calls to get_rss_mb() and arithmetic. memory_tracker.py is now only two functions (~50 lines). No classes, no state management.

Aligns with mem_ov.py: baseline excludes Python library imports and model build pipeline. model_load_delta renamed to compile_delta since it now only measures ORT session compilation.

Baseline now taken after heavy Python imports (WinMLAutoModel etc.) and EP DLL loading, so model_load_delta measures only model-specific work: reading ONNX, graph optimization, ORT session compilation.

Takes RSS baseline after _load_model() completes but before compile(). This naturally excludes all Python imports, EP DLLs, and build pipeline overhead without fragile pre-import hacks. model_load_delta now measures only ORT session compilation (actual model weights in memory).

- Add comment explaining device memory query position (only grows) - Inline adapter_luid call to reduce intermediate variable

Shows device memory growth per phase so users can see how much the model load vs inference contributes to device allocation.

Device memory uses WDDM PDH counters which only work reliably for GPU (DML). For NPU/QNN they report tiny numbers that don't reflect actual device usage. Simplify to process RSS only.

The WDDM PDH device memory counters are GPU-specific and don't provide meaningful data for NPU workloads.

Remove Device Mem from live chart, hw_monitor properties, and to_dict() output. Update tests accordingly.

This reverts commit c9c9fec.

Query GPU Process Memory PDH counters at same phase boundaries (baseline, after compile, after inference) to report VRAM deltas for model load and inference separately.

Integrate composite model support (_run_sub_models/_run_single) with memory measurement. Memory snapshots now live in _run_single().

## Summary Adds end-to-end tests for the \--memory\/\--no-memory\ perf CLI flag introduced in #861. ## Changes - Added \memory\ parameter to \_build_perf_args()\ test helper - \ est_benchmark_cpu_memory\: Verifies \--memory\ produces JSON with \memory\ section containing RSS fields, and console output shows Memory/RAM sections - \ est_benchmark_cpu_no_memory\: Verifies \--no-memory\ excludes memory from both JSON and console output - \ est_benchmark_npu_memory\: Verifies VRAM local/shared fields are populated when running on NPU device ## Stacked on - #861 (perf memory measurement implementation) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

## Summary `winml perf` crashes on a clean install with `No module named 'psutil'` because `src/winml/modelkit/session/monitor/memory_tracker.py` imports `psutil` at module level, and the perf flow imports that module unconditionally. `psutil` was never declared in `[project].dependencies` — only the dev type stub `types-psutil` is present — so the published wheel's `Requires-Dist` omits it, breaking `winml perf` (and `--monitor` / `--memory`) out-of-the-box for every user. Regression from #861 (`feat: add --memory flag`); `memory_tracker.py` did not exist in v0.1.0, so `winml perf` was unaffected there. Installing `psutil` manually confirms perf/build/`--monitor` otherwise work correctly — the only defect is the missing dependency declaration. ## Change Add `psutil>=7` to `[project].dependencies` (aligns with the existing `types-psutil>=7.2.2` stub). Targeting `release/v0.2.0` directly as a release hotfix; `main` will pick it up via the post-release merge-back. Closes #936

DingmaomaoBJTU requested a review from a team as a code owner June 10, 2026 08:13

github-advanced-security AI found potential problems Jun 10, 2026

View reviewed changes

Comment thread tests/unit/session/monitor/test_memory_tracker.py Fixed

DingmaomaoBJTU added 3 commits June 10, 2026 16:23

Simplify memory output to single line with peak process + device memory

e50a39c

Remove unnecessary del statement (CodeQL)

a8bef09

Use post-inference working set instead of process peak for memory dis…

1b7d016

…play Shows the actual memory footprint during inference (steady state) rather than the process lifetime peak which may include transient allocations from model loading or compilation.

DingmaomaoBJTU and others added 10 commits June 10, 2026 18:32

Merge remote-tracking branch 'origin/main' into dingmaomaobjtu/perf-m…

753bec6

…emory-measurement

Fix CI: add missing memory_profile mock in TestPerfFormatJson

125c455

The test from PR #855 mocks BenchmarkResult but did not set memory_profile=None, causing MagicMock comparison failure in display_console_report.

feat: warmup EP before memory baseline

ed8fac1

Call WinMLSession._init_winml_eps_once() before taking the baseline snapshot so one-time EP DLL loading costs are excluded from the model_load_delta metric.

simplify: remove MemoryTracker/MemoryProfile classes

b4e8bd3

Inline RSS measurement directly in PerfBenchmark.run() — just 3 calls to get_rss_mb() and arithmetic. memory_tracker.py is now only two functions (~50 lines). No classes, no state management.

simplify: reduce _resolve_adapter_luid branching

b927dc2

fix: take baseline after model load, before compile

b1a9c08

Aligns with mem_ov.py: baseline excludes Python library imports and model build pipeline. model_load_delta renamed to compile_delta since it now only measures ORT session compilation.

fix: pre-import libs + warmup EP before baseline

e5b8408

Baseline now taken after heavy Python imports (WinMLAutoModel etc.) and EP DLL loading, so model_load_delta measures only model-specific work: reading ONNX, graph optimization, ORT session compilation.

github-advanced-security AI found potential problems Jun 15, 2026

View reviewed changes

Comment thread src/winml/modelkit/commands/perf.py Fixed

xieofxie reviewed Jun 16, 2026

View reviewed changes

Comment thread src/winml/modelkit/commands/perf.py Outdated

github-actions Bot added 2 commits June 16, 2026 10:39

fix: address PR review comments

b5795cb

- Add comment explaining device memory query position (only grows) - Inline adapter_luid call to reduce intermediate variable

feat: query device memory after compile and inference

e4e176a

Shows device memory growth per phase so users can see how much the model load vs inference contributes to device allocation.

github-advanced-security AI found potential problems Jun 16, 2026

View reviewed changes

Comment thread src/winml/modelkit/commands/perf.py Fixed

Comment thread src/winml/modelkit/commands/perf.py Fixed

Comment thread src/winml/modelkit/commands/perf.py Fixed

github-actions Bot added 6 commits June 16, 2026 13:07

Remove device memory measurement (unreliable for NPU)

155547f

Device memory uses WDDM PDH counters which only work reliably for GPU (DML). For NPU/QNN they report tiny numbers that don't reflect actual device usage. Simplify to process RSS only.

Remove Device Mem line from hardware monitor display

7e68fef

The WDDM PDH device memory counters are GPU-specific and don't provide meaningful data for NPU workloads.

Remove device memory from live display and hw_monitor

c9c9fec

Remove Device Mem from live chart, hw_monitor properties, and to_dict() output. Update tests accordingly.

Revert "Remove device memory from live display and hw_monitor"

66b0c99

This reverts commit c9c9fec.

Remove dim styling from memory breakdown line

fccd5e1

Rename Mem → RAM and Device Mem → VRAM for clarity

b4b3e9f

xieofxie reviewed Jun 16, 2026

View reviewed changes

Comment thread src/winml/modelkit/commands/perf.py

Add VRAM phase measurement alongside RAM

e677812

Query GPU Process Memory PDH counters at same phase boundaries (baseline, after compile, after inference) to report VRAM deltas for model load and inference separately.

github-advanced-security AI found potential problems Jun 16, 2026

View reviewed changes

Comment thread src/winml/modelkit/commands/perf.py Fixed

Comment thread src/winml/modelkit/commands/perf.py Fixed

github-actions Bot added 3 commits June 16, 2026 14:38

Compact memory display to single line per metric

4b5db1d

Split VRAM into local/shared in memory display

62a0064

Merge main and resolve conflict in perf.py

fce3cc3

Integrate composite model support (_run_sub_models/_run_single) with memory measurement. Memory snapshots now live in _run_single().

github-advanced-security AI found potential problems Jun 16, 2026

View reviewed changes

DingmaomaoBJTU changed the title ~~Add --memory flag to winml perf for per-phase memory measurement~~ feat: add --memory flag to measure RAM and VRAM per phase Jun 16, 2026

github-actions Bot added 2 commits June 16, 2026 15:03

Fix CodeQL: initialize memory variables before conditional blocks

de15995

Fix type error: use self._single in _resolve_adapter_luid

11e573c

xieofxie approved these changes Jun 16, 2026

View reviewed changes

DingmaomaoBJTU merged commit a8257d4 into main Jun 16, 2026
9 checks passed

DingmaomaoBJTU deleted the dingmaomaobjtu/perf-memory-measurement branch June 16, 2026 07:20

DingmaomaoBJTU mentioned this pull request Jun 16, 2026

Add e2e tests for perf memory measurement #898

Merged

This was referenced Jun 23, 2026

winml perf crashes on clean install — psutil imported but not a runtime dependency #936

Closed

fix: declare psutil as a runtime dependency #937

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add --memory flag to measure RAM and VRAM per phase#861

feat: add --memory flag to measure RAM and VRAM per phase#861
DingmaomaoBJTU merged 29 commits into
mainfrom
dingmaomaobjtu/perf-memory-measurement

DingmaomaoBJTU commented Jun 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

xieofxie commented Jun 10, 2026

Uh oh!

xieofxie commented Jun 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

DingmaomaoBJTU commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Output

Design

Display changes

Files changed

Uh oh!

Uh oh!

xieofxie commented Jun 10, 2026

Uh oh!

xieofxie commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DingmaomaoBJTU commented Jun 10, 2026 •

edited

Loading

xieofxie commented Jun 10, 2026 •

edited

Loading