[UR][Benchmarks] Support for Unitrace for all benchmarks #19320

mateuszpn · 2025-07-07T09:03:57Z

Run main.py script with --unitrace to get unitrace logs for every benchmark.
Use --unitrace-inclusive to get benchmark results and unitrace logs.

Signed-off-by: Mateusz P. Nowak <[email protected]>

pbalcer · 2025-07-07T09:07:11Z

devops/scripts/benchmarks/utils/unitrace.py

+                "-DBUILD_WITH_XPTI=1",
+                "-DBUILD_WITH_MPI=0",
+            ],
+            ld_library=get_oneapi().ld_libraries() + [f"{options.sycl}/lib"],


🤔 removed

pbalcer · 2025-07-07T09:08:11Z

devops/scripts/benchmarks/utils/unitrace.py

@@ -0,0 +1,182 @@
+# Copyright (C) 2024-2025 Intel Corporation


pbalcer · 2025-07-07T09:09:15Z

devops/scripts/benchmarks/utils/unitrace.py

+        print(f"Moved {pid_json_files[0]} to {dst}")
+
+    # Prune old unitrace directories
+    prune_unitrace_dirs(options.unitrace_res_dir, FILECNT=5)


You use this only here. Why bother with defining a default value of an argument that you are going to override anyway?

Debugging leftover, removed

pbalcer · 2025-07-07T09:12:34Z

devops/scripts/benchmarks/utils/unitrace.py

+
+    shutil.move(os.path.join(options.benchmark_cwd, pid_json_files[0]), dst)
+    if options.verbose:
+        print(f"Moved {pid_json_files[0]} to {dst}")


all the logs will need to be changed after #19158 is merged.

pbalcer · 2025-07-07T09:35:44Z

devops/scripts/benchmarks/utils/unitrace.py

+    return bench_dir, unitrace_output, unitrace_command
+
+
+def handle_unitrace_output(bench_dir, unitrace_output, timestamp):


I feel like this is all very complicated. Maybe we can use directory structure to help us?

Let's say we have two benchmarks: "SubmitKernel" and "Hashtable". We can organize the unitrace output such that each trace file is in its own directory:

.../results/traces/SubmitKernel/YYYYMMDD_HHMMSS_[name].PID.out .../results/traces/Hashtable/YYYYMMDD_HHMMSS_[name].PID.out

Removing old results would become simply removing all but 5 newest files in a directory. We wouldn't have to find, rename and move files, since we could make unitrace output them to the destination place upfront.

Done, various approaches are possible, no problem to follow yours.

pbalcer · 2025-07-07T09:37:30Z

devops/scripts/benchmarks/main.py

+    parser.add_argument(
+        "--unitrace",
+        action="store_true",
+        help="Unitrace tracing for sigle iteration of benchmarks",
+    )
+
+    parser.add_argument(
+        "--unitrace-inclusive",
+        action="store_true",
+        help="Regular run of benchmarks iterations and unitrace tracing in single additional run",
+    )


please do --unitrace inclusive and --unitrace as a single option, not two separate ones. See how --output-html is done.

pbalcer · 2025-07-07T09:43:25Z

devops/scripts/benchmarks/benches/compute.py

@@ -224,6 +224,71 @@ def parse_unit_type(compute_unit):


 class ComputeBenchmark(Benchmark):
+
+    # list of benchmarks to exclude from unitrace due to SIGSEGV, SIGABRT or timeouts
+    unitrace_exclusion_list = [


We should just let things fail. If unitrace is failing, it's an urgent bug that needs to be fixed.

Signed-off-by: Mateusz P. Nowak <[email protected]>

devops/scripts/benchmarks/utils/unitrace.py

devops/scripts/benchmarks/main.py

devops/scripts/benchmarks/history.py

devops/scripts/benchmarks/benches/base.py

devops/scripts/benchmarks/benches/benchdnn.py

devops/scripts/benchmarks/utils/unitrace.py

PatKamin · 2025-07-07T12:21:54Z

devops/scripts/benchmarks/utils/unitrace.py

+                    print(f"Renamed {src} to {dst}")
+                break
+
+    # Handle {name}.{pid}.json files in cwd: move and rename to {self.name()}_{timestamp}.json


Doesn't unitrace already have an option for changing the default name of jsons and .pid logs?

unitrace produces two kinds of output files - traces, configurable with --output (pid is added anyway), and jsons for visualization in Perfetto UI, created with no control over their names - just ..json in cwd

Bummer. Maybe we could change unitrace to use the output name for both outputs. The visualization json could be something like name.pid.vis.json.

devops/scripts/benchmarks/utils/unitrace.py

pbalcer

One general comment - when under tracing, we should reduce the number of iterations etc, to shorten the duration of the benchmark.
If we really need to disable a benchmark (because it's difficult to make it sufficiently short for example), the Benchmark class should have something like traceable() to filter out which benchmarks should / shouldn't be traced.

mateuszpn · 2025-07-08T12:22:41Z

the Benchmark class should have something like traceable() to filter out which benchmarks should / shouldn't be traced.

Added in Benchmark, and a stub in ComputeBenchmark

Signed-off-by: Mateusz P. Nowak <[email protected]>

Copilot

Pull Request Overview

This PR adds support for capturing Unitrace logs alongside benchmark runs.

Introduces a new Unitrace utility to clone, build, and manage Unitrace traces
Extends CLI (--unitrace) and options.py to configure trace capture
Updates benchmark runner (run_bench, main.py, and individual benches) and history saving to integrate Unitrace outputs

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
devops/scripts/benchmarks/utils/unitrace.py	Add Unitrace tool clone/build, setup/cleanup, and output handling
devops/scripts/benchmarks/options.py	Add `save_name` option and change `build_jobs` calculation
devops/scripts/benchmarks/main.py	Introduce `--unitrace` flag, create/get instance, and update run flow
devops/scripts/benchmarks/history.py	Incorporate Unitrace timestamp when saving run history
devops/scripts/benchmarks/benches/base.py	Extend `run_bench` and signature of `run` to pass `run_unitrace` flag
devops/scripts/benchmarks/benches/*	Update all benchmark `run` methods to accept `run_unitrace` and forward flag

Comments suppressed due to low confidence (4)

devops/scripts/benchmarks/main.py:550

[nitpick] The --unitrace flag overload with optional values is confusing; consider defining separate flags (e.g., --unitrace and --unitrace-inclusive) for clearer CLI design.

        choices=["inclusive", True],

devops/scripts/benchmarks/utils/unitrace.py:66

[nitpick] The parameter FILECNT uses uppercase naming that deviates from snake_case; consider renaming to file_count or max_files for consistency.

    def _prune_unitrace_dirs(self, dir: str, FILECNT: int = 10):

devops/scripts/benchmarks/utils/unitrace.py:14

[nitpick] Public Unitrace class and its methods lack docstrings; consider adding docstrings to explain their purpose, parameters, and behavior.

class Unitrace:

devops/scripts/benchmarks/utils/unitrace.py:1

This new unitrace.py module introduces complex clone/build/cleanup logic but no tests; consider adding unit tests or integration tests to verify its behavior.

# Copyright (C) 2025 Intel Corporation

Copilot · 2025-07-08T15:26:39Z

devops/scripts/benchmarks/options.py

@@ -68,7 +70,7 @@ class Options:
    build_igc: bool = False
    current_run_name: str = "This PR"
    preset: str = "Full"
-    build_jobs: int = multiprocessing.cpu_count()
+    build_jobs: int = len(os.sched_getaffinity(0))  # Cores available for the process.


Using os.sched_getaffinity may not be supported on non-Linux platforms; consider adding a fallback (e.g., os.cpu_count()) for portability.

Suggested change

build_jobs: int = len(os.sched_getaffinity(0)) # Cores available for the process.

try:

build_jobs: int = len(os.sched_getaffinity(0)) # Cores available for the process.

except (AttributeError, NotImplementedError):

build_jobs: int = os.cpu_count() # Fallback for non-Linux platforms.

Benchmarks are not intended to run on non-Linux platforms

Copilot · 2025-07-08T15:26:40Z

devops/scripts/benchmarks/utils/unitrace.py

+    def _prune_unitrace_dirs(self, dir: str, FILECNT: int = 10):
+        files = os.listdir(dir)
+        files.sort()  # Lexicographical sort matches timestamp order
+        if len(files) > 2 * FILECNT:
+            for f in files[: len(files) - 2 * FILECNT]:
+                full_path = os.path.join(dir, f)


[nitpick] Parameter name dir shadows the built-in function dir; rename it (e.g., dir_path) to avoid confusion.

Suggested change

def _prune_unitrace_dirs(self, dir: str, FILECNT: int = 10):

files = os.listdir(dir)

files.sort() # Lexicographical sort matches timestamp order

if len(files) > 2 * FILECNT:

for f in files[: len(files) - 2 * FILECNT]:

full_path = os.path.join(dir, f)

def _prune_unitrace_dirs(self, dir_path: str, FILECNT: int = 10):

files = os.listdir(dir_path)

files.sort() # Lexicographical sort matches timestamp order

if len(files) > 2 * FILECNT:

for f in files[: len(files) - 2 * FILECNT]:

full_path = os.path.join(dir_path, f)

Signed-off-by: Mateusz P. Nowak <[email protected]>

mateuszpn added 4 commits June 30, 2025 11:04

Add Unitrace support

4d32fe5

Signed-off-by: Mateusz P. Nowak <[email protected]>

Merge branch 'intel:sycl' into unitrace

72d8108

Add Unitrace support

d653883

Signed-off-by: Mateusz P. Nowak <[email protected]>

Merge branch 'unitrace' of github.com:mateuszpn/llvm into unitrace

5834aa9

mateuszpn requested a review from a team as a code owner July 7, 2025 09:03

mateuszpn temporarily deployed to WindowsCILock July 7, 2025 09:04 — with GitHub Actions Inactive

mateuszpn added 2 commits July 7, 2025 09:12

Add list of Unitrace exclusions

d8f6b2b

Signed-off-by: Mateusz P. Nowak <[email protected]>

Merge remote-tracking branch 'upstream/sycl' into unitrace

01c1c1a

mateuszpn requested a review from Copilot July 7, 2025 09:17

This comment was marked as outdated.

Sign in to view

mateuszpn temporarily deployed to WindowsCILock July 7, 2025 09:25 — with GitHub Actions Inactive

pbalcer reviewed Jul 7, 2025

View reviewed changes

Final fixes

be36447

Signed-off-by: Mateusz P. Nowak <[email protected]>

PatKamin reviewed Jul 7, 2025

View reviewed changes

devops/scripts/benchmarks/benches/benchdnn.py Outdated Show resolved Hide resolved

devops/scripts/benchmarks/utils/unitrace.py Outdated Show resolved Hide resolved

devops/scripts/benchmarks/utils/unitrace.py Outdated Show resolved Hide resolved

PatKamin reviewed Jul 7, 2025

View reviewed changes

pbalcer reviewed Jul 7, 2025

View reviewed changes

mateuszpn temporarily deployed to WindowsCILock July 8, 2025 11:47 — with GitHub Actions Inactive

mateuszpn temporarily deployed to WindowsCILock July 8, 2025 12:08 — with GitHub Actions Inactive

mateuszpn temporarily deployed to WindowsCILock July 8, 2025 12:22 — with GitHub Actions Inactive

mateuszpn temporarily deployed to WindowsCILock July 8, 2025 12:43 — with GitHub Actions Inactive

mateuszpn added 2 commits July 8, 2025 12:47

apply comments

deb9aaf

Signed-off-by: Mateusz P. Nowak <[email protected]>

add traceable() to Benchmark class

a5b1fd1

Signed-off-by: Mateusz P. Nowak <[email protected]>

mateuszpn requested a review from Copilot July 8, 2025 15:24

Copilot AI reviewed Jul 8, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/sycl' into unitrace

91db168

mateuszpn requested a review from PatKamin July 9, 2025 15:29

mateuszpn temporarily deployed to WindowsCILock July 9, 2025 15:29 — with GitHub Actions Inactive

mateuszpn had a problem deploying to WindowsCILock July 9, 2025 16:00 — with GitHub Actions Failure

mateuszpn temporarily deployed to WindowsCILock July 9, 2025 16:00 — with GitHub Actions Inactive

Add logs

30baba2

Signed-off-by: Mateusz P. Nowak <[email protected]>

mateuszpn temporarily deployed to WindowsCILock July 10, 2025 07:23 — with GitHub Actions Inactive

		@@ -0,0 +1,182 @@
		# Copyright (C) 2024-2025 Intel Corporation

		return bench_dir, unitrace_output, unitrace_command


		def handle_unitrace_output(bench_dir, unitrace_output, timestamp):

[UR][Benchmarks] Support for Unitrace for all benchmarks #19320

Are you sure you want to change the base?

[UR][Benchmarks] Support for Unitrace for all benchmarks #19320

Uh oh!

Conversation

mateuszpn commented Jul 7, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pbalcer left a comment

Choose a reason for hiding this comment

Uh oh!

mateuszpn commented Jul 8, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!