High memory allocation when using longer AckTimeout

[adrianotr]

There seems to be a correlation between the ack timeout and memory allocation. The longer I set it, the more memory gets allocated.

Here you can see the allocations if the timeout is set to 100 days, but the problem is visible for smaller ack timeouts as well.

You can use the program bellow to reproduce the issue:

var client = await new PulsarClientBuilder()
    .ServiceUrl("pulsar://localhost:6650")
    .BuildAsync();

long memoryBefore = GC.GetTotalMemory(true);

await using var consumer = await client.NewConsumer()
    .Topic("test-topic")
    .SubscriptionName("test-subscription")
    .AckTimeout(TimeSpan.FromDays(100))
    .SubscribeAsync();

long memoryAfter = GC.GetTotalMemory(true);
Console.WriteLine($"Memory used: {(memoryAfter - memoryBefore) / 1_000_000} mb");

[adrianotr]

Btw, the reason why we discovered this issue is that if we have the consumer set as the default (Zero) it seems that the client library effectively sets the ack timeout to 30s (which is a discrepancy to the Java client that keeps it as infinite).

[Lanayx]

@adrianotr good notice! It indeed correlates well with the code which totally correlates with Java code

we discovered this issue

Not sure I understand what is the issue though

[adrianotr]

Hi, thanks for the quick response. Sorry for not being very clear; let me explain that better.

Not sure I understand what is the issue though

It's a separate issue, but basically it's why we were using very long ack timeouts. If you don't set the ack timeout, then the lib sets the ack timeout to 30s instead of keeping it as zero (only when retries are enabled apparently). This is what I meant that is different from the Java client.

It indeed correlates well with the code

So the problem there is not the correlation but the immense amount of memory used.
For example, this is the memory allocation I get by using different ack timeouts:

AckTimeout	Memory Allocation
30 seconds	3 mb
24 hours	15 mb
2 days	27 mb
10 days	132 mb
100 days	750 mb

[Lanayx]

If you don't set the ack timeout, then the lib sets the ack timeout to 30s instead of keeping it as zero

That's incorrect, Pulsar.Client follows Java and sets timeout to zero

So the problem there is not the correlation but the immense amount of memory used gets increased.

I've provided the code reference so you could understand why the amount of memory used gets increased.

let stepsCount = Math.Ceiling(ackTimeout.TotalMilliseconds / tickDuration.TotalMilliseconds) |> int
for _ in [1..stepsCount] do
    timePartitions.Enqueue(RedeliverSet())

According to the Math, if you increase ackTimeout, the timePartitions queue will increase as well. You can reduce memory used by increasing AckTimeoutTickTime together with increasing AckTimeout

[Lanayx]

Having said that, I've just noticed, that you might see 30s because of this configuration. And it was removed in Java at some point, so you can create a PR with applying this improvement to Pulsar.Client as well.