[question]How many consumers or producers can be created on a pulsar client?

[xuesongxs]

How many consumers or producers can be created on a pulsar client?

[wmccarley]

In the Java implementation the PulsarClient class stores producer and consumer instances in a set backed by a ConcurrentHashMap so the count is theoretically unbounded I guess. In practice everything has a limit but it's dependent on your system resources.

I have seen a client application open 30K concurrent consumers to one single topic (wasn't by design, it was a poorly coded application.) Neither the client nor the broker actually crashed in that instance. The only ill effects I observed on the broker were the prometheus metrics collection stopping because it had exposeConsumerLevelMetricsInPrometheus=true and the broker could not render all the metrics before prometheus scrape timed out. Likewise the client swore up and down that their application was healthy and only agreed to change their code after I showed them the output of the topic stats command showing 1000s of consumer instances.

Since connection management/pooling is handled at the PulsarClient level there is not a 1-1 correlation between the number of producers/consumers and the number of TCP connections opened up to brokers. Based on my experience I suspect a well-designed application could open hundreds or thousands of producers/consumers and be fine.

Since Pulsar Producer and Consumer instances are thread-safe and highly tunable, in general you shouldn't have to create a lot of them to get high performance. Generally speaking you only need to create many instances if you need to interact with many different topics.

[codelipenghui]

The issue had no activity for 30 days, mark with Stale label.