Skip to content

Add nsys command argument to profile cuda graph workload#138

Merged
hemildesai merged 3 commits intoNVIDIA-NeMo:mainfrom
guyueh1:fix_nsys_command
Jan 23, 2025
Merged

Add nsys command argument to profile cuda graph workload#138
hemildesai merged 3 commits intoNVIDIA-NeMo:mainfrom
guyueh1:fix_nsys_command

Conversation

@guyueh1
Copy link
Contributor

@guyueh1 guyueh1 commented Jan 21, 2025

This PR does two changes

  1. Add nsys command argument --cuda-graph-trace=node to allow nsys to see into cuda graph replays
  2. Fix nsys profile export path, to make it visible in the container to facilitate export

Guyue Huang added 2 commits January 21, 2025 10:01
…ofile export path

Signed-off-by: Guyue Huang <guyueh@login-preos02.a51.clusters.nvidia.com>
Signed-off-by: Guyue Huang <guyueh@login-preos02.a51.clusters.nvidia.com>
@guyueh1
Copy link
Contributor Author

guyueh1 commented Jan 21, 2025

I reverted change 2 since the issue is fixed in main; Only keep change 1.

hemildesai
hemildesai previously approved these changes Jan 22, 2025
@guyueh1
Copy link
Contributor Author

guyueh1 commented Jan 23, 2025

What else is needed for this PR, could you trigger workflows? @hemildesai

@hemildesai
Copy link
Contributor

What else is needed for this PR, could you trigger workflows? @hemildesai

Sorry didn't realize I had to approve. Looks like a test failed, but should be a one line fix. Can you push the fix? I'll reapprove and merge after.

Signed-off-by: Guyue Huang <guyueh@nvidia.com>
@guyueh1
Copy link
Contributor Author

guyueh1 commented Jan 23, 2025

I pushed a fix to the test.

@hemildesai hemildesai merged commit 5ed6128 into NVIDIA-NeMo:main Jan 23, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants