-
Notifications
You must be signed in to change notification settings - Fork 0
Yes, broker based balanced consumer groups are supported and requires Apache Kafka broker version >= 0.9.
-
C:
rd_kafka_subscribe()et.al -
C++:
KafkaConsumerclass.
The Consumer class is the legacy simple consumer that works with any broker >= 0.8, it does not feature any consumer group support but allows full flexibility in regards to partitions to consume and their offsets.
-
C:
rd_kafka_consume_start()et.al. -
C++:
Consumerclass
The KafkaConsumer class is the new high-level balanced consumer which requires broker version >= 0.9, the subscribed topics and partitions are assigned to the memmbers in a consumer group so that only one consumer may consume messages from a single partition at any time.
-
C:
rd_kafka_subscribe(),rd_kafka_assign(), et.al. -
C++:
KafkaConsumerclass
If there are no stored offsets for a partition (and group in case of the KafkaConsumer) the consumer will default its starting offset to the topic configuration setting auto.offset.reset which defaults to latest - that is, it will start consuming at the current end of a partition.
If you are using the KafkaConsumer you probably do not have a per-topic configuration object but should use the default topic config, see default_topic_conf.
Since you dont need to create any topic objects when using KafkaConsumer you might wonder how to set topic configuration.
The default_topic_conf configuration property solves this, it takes a topic configuration object and applies it to all internally created topics as their default configuration.
If you need special configuration for a sub-set of your topics you will need to create them prior to calling subscribe() or assign() and pass their explicit configuration objects at topic creation time.
Because synchronous producing is a performance killer and scales very poorly, it is effectively bound by network and broker latency. If the round-trip to produce a message is 2ms the maximum achievable rate is 500 messages per second. In scenarios where the broker or network is getting slower for whatever reason this rate decreases even more, possibly causing backpressure in the application affecting the upstream data source.
It is thus better to design the application to use asynchronous sends and use the delivery report callback to take action when the message is finally delivered or fails permanently.
If you still think you need a sync producer, see How to implement a sync producer
Make sure you use a delivery callback and set the produce.offset.report topic configuration property to true, this comes at a very minimal performance cost.
Yes, it is. But you need to read this: Broker version compatibility
Yes, librdkafka is completely thread-safe (unless otherwise noted in the API documentation).
Any API, short of the destructor functions, may be called at any time from any thread.
The common restrictions of object destruction still applies (e.g., you must not call rd_kafka_destroy() while another thread is calling rd_kafka_poll() or similar).
That is up to you, the library itself is implemented in C and the C++ interface is a thin layer on top of the C code. The C++ interface may lag behind the C interface functionally (but usually not by far).
For production use: use latest official release (currently 0.9.1)
For testing and development use: use latest master branch
This is covered in Proper termination sequence
No, there is currently no Kafka protocol support for managing topics, but work is being done in KIP-4.
librdkafka does support automatic topic creation, which needs to be enabled with auto.create.topics.enable=true on the broker.