NVIDIA Programmatic Dependent Launch for Llama.cpp

[agray3]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

The NVIDIA Programmatic Dependent Launch (PDL) mechanism, available for Hopper, Blackwell (+ future) GPUs, allows for a dependent secondary kernel to launch before the primary kernel it depends on in the same CUDA stream has finished executing.

https://docs.nvidia.com/cuda/cuda-c-programming-guide/#programmatic-dependent-launch-and-synchronization

Benefits can be achieved though enabling PDL, even without attempting to overlap any kernel logic, since it allows e.g. the scheduling of blocks in the secondary kernel to start even before the primary kernel is completely finished. Further benefits may be available from more in-depth refactoring to enhance overlap.

#15480 introduces PDL to llama.cpp via:

1. Adjusting each freshly captured CUDA graph to set the appropriate PDL launch properties to each node/edge
2. Systematically adding cudaTriggerProgrammaticLaunchCompletion()/cudaGridDependencySynchronize() to the entry/exit points of all kernels, to guarantee appropriate synchronization.

For 2. these sync points are currently set as conservatively as possible, not attempting to overlap any kernel logic, so this change doesn't alter the ordering of (application-level) instructions in any way. But this opens up possibilities for further optimisation through refactoring, e.g. weights could be loaded before cudaGridDependencySynchronize() since they don't depend on any previous kernel. Note that cudaTriggerProgrammaticLaunchCompletion() is not strictly required in this current implementation, since the appropriate instruction is added at the end of all kernels anyway. But I've included it to open up scope for any further overlapping of work at the end of kernels, for which the subsequent kernel does not depend on the result.

Motivation

Here are the benefits I observe from #15480:

As above, this may open up possibilities for further gains through refactoring kernels to achieve more overlapping.
Note that the PR currently disables PDL in the presence of any library (e.g. BLAS) kernels, since further work is required to ensure appropriate synchronization for these (see comments in code).

[agray3]

@ggerganov @slaren @JohannesGaessler As you can see here and in the PR, introducing PDL offers significant benefits but requires quite a lot of extra lines of code, albeit all very systematic. Hence I've kept this as a "Draft" PR for now, hoping for your feedback on whether you want to proceed with this.

[JohannesGaessler]

My first impression is that these are a lot of lot of changes for a marginal speedup. Particularly, the speedup seems to be largest when running very small models on very fast GPUs. That is a common scenario for benchmarking but not necessarily for production use.

IIRC there was previously a request to refactor the code around CUDA graphs more generally. My suggestion would be to do this refactor and add features like this afterwards.