Skip to content

fix null gpuSlots in LocalTaskHandler #6189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

fix null gpuSlots in LocalTaskHandler #6189

wants to merge 5 commits into from
+139 −8

Conversation

thealanjason
Copy link

gpuSlots for the LocalTaskHandler need to be set before calling submit()

@thealanjason
fix null gpuSlots in LocalTaskHandler
0d9350d 
gpuSlots for the LocalTaskHandler need to be set before calling
`submit()`

Signed-off-by: Alan J Correa <[email protected]>
@thealanjason
Copy link
Author

Starting of from: #5850

I was able to compile after rebasing on the current master branch.

With the fix in this PR I was able to get the MWE at: https://github.com/thealanjason/showmehow/tree/gpu_test/usecase99 to run succesfully.

Here is the nextflow output:

(nextflow_gpu)$: ~/Work/gpu_scheduling/showmehow/usecase99$ ../../nextflow/build/releases/nextflow-25.05.0-edge-dist SHOWME.how.nf

 N E X T F L O W   ~  version 25.05.0-edge

Launching `SHOWME.how.nf` [soggy_mclean] DSL2 - revision: 91b25ac0cf

executor >  local (17)
[76/0a6eb1] GENERATE_PARAMS  [100%] 1 of 1 ✔
[84/e1ff2e] MATMUL_CUPY1 (6) [ 30%] 3 of 10
[0d/37993b] MATMUL_CUPY2 (5) [ 40%] 4 of 10
[e3/c271f9] MATMUL_CUPY3 (4) [ 50%] 5 of 10

Following is the output from nvidia-smi showing that the scheduler works as expected:

$: ~/Work/gpu_scheduling$ nvidia-smi
Tue Jun 17 02:30:15 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    Off |   00000000:3F:00.0 Off |                    0 |
| N/A   33C    P0             98W /  350W |    2741MiB /  46068MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L40S                    Off |   00000000:56:00.0 Off |                    0 |
| N/A   34C    P0             98W /  350W |    2741MiB /  46068MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L40S                    Off |   00000000:C3:00.0 Off |                    0 |
| N/A   33C    P0            102W /  350W |    2741MiB /  46068MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L40S                    Off |   00000000:DA:00.0 Off |                    0 |
| N/A   34C    P0            108W /  350W |    2741MiB /  46068MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A          440517      C   python                                 2728MiB |
|    1   N/A  N/A          440753      C   python                                 2728MiB |
|    2   N/A  N/A          440394      C   python                                 2728MiB |
|    3   N/A  N/A          440537      C   python                                 2728MiB |
+-----------------------------------------------------------------------------------------+

@thealanjason thealanjason mentioned this pull request Jun 17, 2025
Open
@bentsherman bentsherman requested a review from a team as a June 18, 2025 16:15
@bentsherman
Copy link
Member

Incorporated into #5850

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bentsherman bentsherman bentsherman left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@thealanjason @bentsherman @pditommaso