What the meaning of 'CPU latency' in p2pBandwidthLatencyTest ?

[JinwoongKim]

Following is a snippet of my result with p2pBandwidthLatencyTest

  1 P2P=Disabled Latency Matrix (us)
  2    GPU     0      1      2      3
  3      0   2.61  14.95  19.61  14.42
  4      1  11.09   2.10  19.16  18.72
  5      2  13.34  16.44   2.06  16.80
  6      3  11.92  16.62  16.72   2.09
  7
  8    CPU     0      1      2      3
  9      0   2.67   6.78   6.18   6.14
 10      1   6.71   2.62   6.33   6.30
 11      2   6.35   6.27   2.50   5.78
 12      3   6.20   6.24   5.79   2.49
 13 P2P=Enabled Latency (P2P Writes) Matrix (us)
 14    GPU     0      1      2      3
 15      0   2.59   2.31   1.85   1.86
 16      1   2.26   2.10   1.88   1.83
 17      2   1.81   1.83   2.07   2.31
 18      3   1.84   1.81   2.23   2.09
 19
 20    CPU     0      1      2      3
 21      0   2.71   1.91   1.97   1.93
 22      1   2.11   2.77   2.16   2.00
 23      2   1.92   1.76   2.58   1.94
 24      3   1.84   1.91   1.88   2.55

When I enabled P2P, GPU latency decreases because they use NVLink (line 2 ~ 6, 14 ~ 18).
By the way, what does CPU latency represent for? I don't know why they show it and sometime I don't understand the result. For instance, line number 23, CPU latency (1.92) is bigger than GPU latency(1.81) even if GPU clock encloses CPU clock https://github.com/NVIDIA/cuda-samples/blob/master/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu#L527C1-L528C1. which means GPU clock is always behind CPU clock.

My hardware spec is A100 80GB x 4 with NVLink Bridge (GPU 0-1, 2-3 connected)