Confused with how bcopy threshold is calculated?

[haipeng31]

`size_t ucp_ep_config_get_zcopy_auto_thresh(size_t iovcnt,
const uct_linear_growth_t *reg_cost,
const ucp_context_h context,
double bandwidth)
{
double zcopy_thresh;
double bcopy_bw = context->config.ext.bcopy_bw;

zcopy_thresh = (iovcnt * reg_cost->overhead) /
               ((1.0 / bcopy_bw) - (1.0 / bandwidth) - (iovcnt * reg_cost->growth));

if (zcopy_thresh < 0.0) {
    return SIZE_MAX;
}

return zcopy_thresh;

}`

zcopy_thresh = (iovcnt * reg_cost->overhead) /
               ((1.0 / bcopy_bw) - (1.0 / bandwidth) - (iovcnt * reg_cost->growth));

do a transformation, we get:
zcopy_thressh / bcopy_bw = iovcnt * reg_cost->overhead + zcopy_thresh / bandwidth + zcopy_thresh * (iovcnt * reg_cost->growth)

there are two points confuse me:

1. the upper transformation means if we use zcopy we have extra zcopy_thresh / bandwith cost，why zcopy has this extra cost?
2. zcopy_thresh * (iovcnt * reg_cost->growth)， what does this cost means? where does it come from

[yosefe]

1. zcopy_thresh / bandwidth is the time to actually send the data over the network: size/bw=time
2. this seems a bug since need only multiply zcopy_thresh * reg_cost->growth. so the original formula should be

zcopy_thresh = (iovcnt * reg_cost->overhead) /
               ((1.0 / bcopy_bw) - (1.0 / bandwidth) - reg_cost->growth);

[haipeng31]

@yosefe sorry, maybe I don't decribe my questions clearly:

1. for zcopy_thread / bandwidth, is the time to send data over network。In my opinion，no matter we use bcopy or zcopy，we all need to send data over netword, so that zcopy_thresh / bandwith cost exist no matter bcopy or zcopy，however the formlula above seems only zcopy has zcopy_threashold / bandwith cost, it's very confusing。In my oponion, compare zcopy and zcopy，we should only compre buffer copy cost and memory register cost?
2. the second question is reg_cost->growth come from, can you give more detail description, what i know register memory is just register memory cost and already decribed by reg_cost->overhead, reg_cost->growth means what cost?

[yosefe]

1. the formula calculates the message size for which sending with bcopy is same time as sending with bcopy
   zcopy_thressh / bcopy_bw = iovcnt * reg_cost->overhead + zcopy_thresh / bandwidth + zcopy_thresh * reg_cost->growth
   in this case left side is bcopy send cost and right side is zcopy send cose (assuming bcopy_bw is slower than network)
2. reg_cost is a linear function of memory size: overhead+growth*size. since sometimes there is extra code depending on the number of pages, for example.

[haipeng31]

@yosefe thanks for your answer.
If the formula is compared for latency:
send one message with bcopy should do the following operations one by one:

1. copy the user data to preallocated memory already registered;
2. send the data;
   so that latency should be the sum of latency of these two ops

If the formula is compare for throughput, because copy is done by cpu, send data is done by NIC，they can be done pipeline, so as you said, we can calculate cost by max(size/bcopy_bw, size/network_bw), however, if compare for throughtput, zcopy cost should something like max(register_cost(size), size/network_bw) because register memory and send data can also be pipelined if multiple message send at the same time. however the formula just add register_cost and send data cost?

[haipeng31]

more on the second problem, if we use rcache, will the reg_cost->growth be zero？

[yosefe]

the formula calculates latency when the send and memory copy are pipelined by using multiple fragments (it does not take into account copy of first fragment)

[haipeng31]

thanks for your reply.