private ctx is more expensive?

[zhongchen530]

Hi,

I was in the middle of trying to measure the performance difference for the different ctx options. From my understanding of the specification, a private ctx is one that doesn't need to handle access from many threads and and is hence no more resource intensive than a regular ctx. But from my understanding of the SOS source code, a private CTX created with SHMEM_CTX_PRIVATE is actually more resource intensive as a dedicated STX is assigned to it. On another hand, STX can be shared by multiple regular ctxs. Is a private ctx supposed to provide performance isolation in addition to completion isolation?

[davidozog]

I think the specification does not go into specifics regarding the resource intensity of contexts. This offers some freedom to the OpenSHMEM library implementations. However, in practice, I think the intent was for SHMEM_CTX_PRIVATE to indicate that a transmit context (which often corresponds to an underlying hardware resource in multi-queue NICs) will not be accessed simultaneously by other threads. For example, this might mean the transport layer would be able to remove some costly locking around the context (as in SOS).

So yes, I'd say private contexts are more resource intensive in SOS, and in general. When it comes to performance measurement... not many libfabric providers have great support for the relevant feature (fi_stx_context), and you will need to set SHMEM_OFI_STX_MAX to a value greater than 1 to make use of the feature. You might also try the tests-sos/performance/shmem_perf_suite/shmem_bw_put_ctx_perf test using the flags -C MULTIPLE -T <number_of_threads>.

So yes, the intent is performance and completion isolation. but... completion isolation is another story. SOS might have room for improvement there because there is only 1 CQ on the endpoint and all STX's are associated with that endpoint. Scalable endpoints might be the better choice here if possible.

[zhongchen530]

Thank you for the response. I was just concerned that if a user create more private CTXs than shmem_transport_stx_max, there will be an uneven distribution of physical resources to the threads because all CTXs beside the first shmem_transport_stx_max - 1 created would be assigned the same STX due to fallback to using shared STX. In theory, the user would be better off using serialized CTXs if there are too many threads, although I haven't observed this yet with my benchmark.

As for “there is only 1 CQ on the endpoint and all STX's are associated with that endpoint”， as far as my understanding goes, SOS currently have one endpoint for receiving all RDMA operations named shmem_transport_ofi_target_ep that has one associated CQ. There are shmem_transport_stx_max STXs shared by all the endpoints and each endpoint has its own CQ. It doesn't seem to me that there is currently an entity that all STXs are associated with?

By "Scalable endpoints might be the better choice here if possible", do you mean we can have a single scalable endpoints with many TXs and we simply assign to each CTX an index into the scalable endpoint's TXs instead of assigning a separate endpoint to each CTX? This way, actual creation of TXs is offloaded to the provider? Although, a TX in a scalable endpoint seems rather similar to a regular endpoint as they both can be passed into fi_write directly while the scalable endpoint itself can't?

Thank you so much again for having this conversation with me!

[davidozog]

In theory, the user would be better off using serialized CTXs if there are too many threads

Yes, possibly. It is certainly possible for users to exhaust the available contexts using SHMEM_CTX_PRIVATE, after which they will be granted a shared context (I believe the spec permits this, albeit implicitly). Currently users need to set env var SHMEM_DEBUG=1 before seeing the warning, private STX unavailable, falling back to STX sharing.

... It doesn't seem to me that there is currently an entity that all STXs are associated with?

Yes thanks for the correction. There is a CQ per target endpoint and a CQ per ctx endpoint in SOS. I made a fix above to the original answer. I believe everything else you mentioned is correct.

"Scalable endpoints might be the better choice here if possible", do you mean we can have a single scalable endpoints with many TXs and we simply assign to each CTX an index into the scalable endpoint's TXs instead of assigning a separate endpoint to each CTX?

Yes, exactly; though I'm unsure it would matter much depending on the provider, like you suggest.

[zhongchen530]

Thanks for the verification!