Topic consuming blocked on one failed message in Apache Pulsar

[ghost]

I have a problem with Key_Shared subscription.

If a consumer fails to process one message, processing of all other messages with other keys is also stalled. Including the other consumers of the subscription (there are 10-15 of them). On a processing fail, I don’t send a nack in order not to break the message order. The consumer just has a minute to try to process other keys, and then restarted.

I was able to reproduce similar behaviour in test. I produce 1000 messages with 4 different keys. Only one of the 1000 messages fails to be processed. An individual ack is performed successfully from the consumer point of view, even after the corrupted message fails. But, apparently, the acks are not “commited” and discarded by the broker. Interesting thing is that if I produce messages with some time gap (say, 1 second), the corrupted message doesn’t block the further messages. So, I suspect some batching be the cause. However, as I said, I acknowledge individually.

What could cause the blocking of the all the keys?

I have Pulsar 2.8.1, but the same behaviour is also observed on 3.0.0. You can find the fragment of topic-stats below.

"subscriptions" : {
    "subscriptionName" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "bytesOutCounter" : 1099336,
      "msgOutCounter" : 11480,
      "msgRateRedeliver" : 0.0,
      "messageAckRate" : 0.0,
      "chunkedMessageRate" : 0,
      "msgBacklog" : 4,
      "backlogSize" : 119640,
      "earliestMsgPublishTimeInBacklog" : 0,
      "msgBacklogNoDelayed" : 4,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "unackedMessages" : 820,
      "type" : "Key_Shared",
      "msgRateExpired" : 0.0,
      "totalMsgExpired" : 0,
      "lastExpireTimestamp" : 0,
      "lastConsumedFlowTimestamp" : 1691146017587,
      "lastConsumedTimestamp" : 1691146017565,
      "lastAckedTimestamp" : 0,
      "lastMarkDeleteAdvancedTimestamp" : 0,
      "consumers" : [ {
        ...
        "messageAckRate" : 0.0,
        ...
        "availablePermits" : 680,
        "unackedMessages" : 820,
        ...
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "16:3",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 1691146017565,
        "lastConsumedFlowTimestamp" : 1691146017582,
        ...
        "lastAckedTime" : "1970-01-01T00:00:00Z",
        "lastConsumedTime" : "2023-08-04T10:46:57.565Z"
      },

[ghost]

The issue was batch ack wasn't enabled on the consumer side and the broker.