How to tweak a Pulsar cluster to have a large number of topics with a reasonable throughput

[yehancha]

My requirement is to create 200K topics and get 100K msg/s throughput across all those topics. (Each topic gets 1 msg every 2 seconds and these are 1KB messages.)

I created 80K topics under 8 namespaces and got 40K msg/s. But when I try to go beyond that, the throughput drops.

I have 3 x 64GB bookkeepers. Currently, I have 2 x 120GB brokers but I also have tested the setup with number of different configurations for the broker (memory per broker and number of brokers) as well.

The above limitation seems to be with bookkeeper usage. Because, when I use non-persistent topics, I could reach up to 150K topics and got 75K msg/s only with 1 x 120GB broker. Did not test beyond that.

I appreciate any advice, suggestion on;

• how should I configure this setup (number of bookkeepers and brokers, there memory and CPU requirements, etc.)
• where should I look for the bottlenecks
• what configurations I should tweak in bookkeeper and broker to get more throughput
• anything else

[yehancha]

The issue seems to be with DispatchRate limit. In my case, I need 0.5 b/s throughput on a single topic. I have set the dispatch rate limit to 4 kb/s, well above my need. But even with that, the throughput slows down (which I will look into in the future). When I remove this limit, everything works fine.

I will keep the discussion open until I hit my initial target of 100K msg/s on 200K topics to discuss any other issues on the way.