metal : increase GPU duty-cycle during inference

[ggerganov]

Apparently there is a significant GPU downtime between Metal compute encoders within a single ggml_metal_graph_compute():

See #6506 for instructions how to generate the trace from the picture.

My expectation was that enqueuing the command buffers in parallel would make them execute without any downtime. The goal of this issue is to understand where this overhead comes from and if there is a way to avoid it.

Obviously, using a single command buffer will avoid all the GPU downtime, but it is much slower to construct it in a single thread. Ideally, we want to continue queuing multiple encoders, but not have the gaps in-between during execution.

[ggerganov]

My analysis show that the gaps in these plots are some sort of XCode profiler artifact and they don't really exist. I guess they merely serve as a visual separator between the command buffers.

Anyway, the changes in #9698 should resolve most of the Metal encoding overhead, which was the goal of this issue.