Failover subscriptions across multiple topics and horizontal scaling #20989
Unanswered
AlecMMiller
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
For my use case, I have somewhere on the order of 100 entities, each with the same set of topics associated with them. E.g I'm going to have something like topic A-foo, A-bar, B-foo, B-bar, and so on. I want to have a horizontally scalable number of consumers processing data for these topics, while maintaining ordering consistency for a given entity.
From the documentation, it seems like this should be doable with a failover subscription with multiple topics, as described at
https://pulsar.apache.org/docs/3.1.x/concepts-messaging/#failover--non-partitioned-topics
and this image seems to be pretty much exactly what I want
https://pulsar.apache.org/assets/images/pulsar-failover-subscriptions-3-2b2e00ef5ee525ca7c1fd21dcd67e328.svg
However, the issue I'm running into is that it seems like the first client to subscribe is being assigned all of the topics, instead of having them be reallocated each time a new client subscribes to the subscription
I have tried subscribing with both a regex pattern and with a topic list, and have had no success with either. I've also tried setting the consumer name with a psuedo-random UUID. In all cases, all messages are being routed to the first client to start up. If it matters, I'm using the Node.js client
Am I understanding the way failover subscriptions are supposed to be working correctly, and this should be working? Or is a failover subscription tied to the entire partition that the topics are on, even if there are multiple topics?
All reactions