Strange publish latency outliers coming from Pulsar Broker/Bookeeper

[kuskmen]

Introduction

Hi,
we were experiencing some intermittent latencies when it comes to publishing in pulsar. I've made simple console applications to produce and consume simple messages on our dev pulsar cluster and I managed to reproduce it fairly easily.

Setup

1. Pulsar cluster setup is 3 brokers with ensemble size 2, write quorum size 2, and ack quorum size 2.
2. Topic I am testing with is persistent.
3. Producer code

while (true)
{
    var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
    
    // no special settings besides EnableBatching = false, but this is a business requirement we cannot change yet.
    await publisher.PublishAsync(new PubSubMsg<long>(now), CancellationToken.None);
    Console.WriteLine($"Message published with latency of {DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() - now}ms");
    
    await Task.Delay(1000);
}

4. Consumer code

await using var client = DotPulsar.PulsarClient.Builder()
    .ServiceUrl(new Uri(pulsarSettings.ConnectionString))
    .Build();

await using var consumer = client.NewConsumer(DotPulsar.Schema.Int64)
    .SubscriptionName("local-pulsar-consumer-subscription")
    .Topic(pulsarSettings.Topic)
    .InitialPosition(DotPulsar.SubscriptionInitialPosition.Latest)
    .Create();

await foreach (var message in consumer.Messages())
{
    var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
    Console.WriteLine($@"{DateTime.UtcNow:O} Message consumed after: Producer publish time:{now - long.Parse(Encoding.UTF8.GetString(message.Data))}ms - Broker publish time: {now - (long)message.PublishTime}ms");
}

Observations

As you can see from the picture from time to time there are some strange outliers with such a small rate (1 msg/s), these outliers scale with the rate of messages as well while ping remains stable.

what we did to improve persistent topic latency is reduce ack quorum size - this made outliers appear less often and less spiky but nevertheless still appear.

when we switched from persistent to non-persistent results were the following:

as you can see, outliers were not found anymore, however, I was told that non-persistent topics should not be used in a production environment, and after some time of investigating there offer a very limited set of features that does not satisfy even the simples scenarios (like maintaining backlog).

Conclusion

1. Is it safe to have Ack quorum size downgraded to 1 while the write quorum size remains as it is? What would be the impact of such change besides the obvious?
2. Could there be something else besides ack quorum size that we can adjust in order to reduce(eliminate) the outliers we are experiencing because we observe outliers of 2 seconds sometimes for topics that are far from loaded (2msg/s)?

Well, that's it, thanks for your patience and your help in advance,
should you need more information about our setup don't hesitate to ask I will try to provide it in a timely manner.

[kuskmen]

Is there someone who can shed some light, we are building high-throughput streaming system and outliers with 1,2 seconds can be critical for the business :/