Question: Consumer Configuration Recommendations for 1000+ Consumer Production System

[nhr-one]

Hi all,

I'm in the process of doing a KafkaJS -> @confluentinc/kafka-javascript migration for my company. We heavily use Kafka as our backbone, and we have > 1000 consumer groups across hundreds of topics.

Right now, we use KafkaJS with no special configuration for the consumer. When swapping out the two libraries in place, we run into OOM issues upon startup.

Are there recommendations on configuration options we should set to reach more or less equivalent behavior to KafkaJS? I've found the following options, and tried setting them a few different times, but still running into problems:

'fetch.max.bytes'
'queued.max.messages.kbytes'
'queued.min.messages'
'fetch.message.max.bytes'

What are reasonable values to consider here? Are there other parameters we should look to tweak?

Thanks in advance!

[emasab]

Hi @nhr-one , please take a look at the performance section in librdkafka introduction.

If it's related to the batch size that is too small in eachBatch, we're in the process of making it configurable.

What are your memory limits for this instance?

[EricMCornelius]

@nhr-one from what I have witnessed, this binding has a non-trivial per consumer overhead regardless of actual partition assignment status.

That means you'll be playing significantly higher memory costs than in kafkajs if you have a (relatively standard) approach of having a large number of redundant consumers in a pool for load distribution, even when the majority are not actually doing any work.