[fix][cli] Restore the 64M default client memory limit in pulsar-perf

[AlvaroStream]

Fixes #26340

Motivation

pulsar-perf can exhaust direct memory and die with OutOfDirectMemoryError whenever the
producer outruns the brokers:

failed to allocate 4194304 byte(s) of direct memory (used: 4131389440, max: 4131389440)

The same run against a 3.0.x client completes normally, which makes this look like a client
regression. It is, but not in the allocator. It is a lost default.

Before #20663, PerfClientUtils.createClientBuilderFromArguments never called
ClientBuilder#memoryLimit, so pulsar-perf inherited the client default of 64M from
ClientConfigurationData#memoryLimitBytes.

That PR added the --memory-limit option and applied it unconditionally:

ClientBuilder clientBuilder = PulsarClient.builder()
        .memoryLimit(arguments.memoryLimit, SizeUnit.BYTES)

but the backing field has no initializer:

@Option(names = { "-ml", "--memory-limit", }, ...)
public long memoryLimit;

so when the option is not supplied, pulsar-perf passes 0. And 0 is not "no override", it is
"no limit":

public boolean isMemoryLimited() {
    return memoryLimit > 0;
}

So pulsar-perf went from bounded to unbounded client memory without anyone choosing that.

With the limit disabled there is no backpressure on the producer. When the brokers cannot keep up
with the offered rate, outbound buffers accumulate until direct memory is exhausted and the client
dies, rather than throttling and reporting the achievable rate, which is what a benchmarking tool
should do. Passing --memory-limit 64M restores the old behaviour and the same run completes.

This also affects PerformanceConsumer, PerformanceReader, PerformanceTransaction and
LoadSimulationClient, which share PerformanceBaseArguments, and on master it reaches both the
existing builder and the v5 PulsarClientBuilder call site.

Modifications

• Extract the existing literal into ClientConfigurationData.DEFAULT_MEMORY_LIMIT_BYTES and use it
  for memoryLimitBytes. This is a pure refactor of a value that is already 64M; it just gives the
  default a name so the CLI can reference it. It follows the existing convention in the sibling
  ProducerConfigurationData, which already exposes DEFAULT_BATCHING_MAX_MESSAGES and friends
  that PerformanceProducer imports.
• Initialise PerformanceBaseArguments#memoryLimit to that constant, so an unset --memory-limit
  reproduces the pre-[feat][cli] Add command line option for configuring the memory limit #20663 behaviour instead of silently disabling the limit. Referencing the
  constant rather than repeating 64 * 1024 * 1024 means the CLI default cannot drift from the
  client default.
• Update the option description to state the default and to document that 0 disables the limit.

--memory-limit 0 still disables the limit explicitly, so the capability added by #20663 is
retained. Only the unset behaviour changes.

Verifying this change

This change is already covered by existing tests, such as
PerformanceBaseArgumentsTest#testMemoryLimitCliArgument, which continues to verify that explicit
values (-ml 1, -ml 1K, --memory-limit 1G) are parsed correctly.

PerformanceBaseArgumentsTest#testMemoryLimitCliArgumentDefault asserted the previous 0 default
and is updated to assert the restored 64M default.

Added PerformanceBaseArgumentsTest#testMemoryLimitCanBeDisabled to pin the escape hatch, so a
future change cannot quietly remove the ability to run unbounded via -ml 0.

./gradlew :pulsar-testclient:test --tests '*PerformanceBaseArgumentsTest*'
BUILD SUCCESSFUL
8 tests, 0 failures, 0 skipped

The end-to-end behaviour was verified against a 3-node cluster: pulsar-perf produce -r 100000 -time 300 -s 1024 fails with OutOfDirectMemoryError on 4.0.9 and 4.0.13, and completes once the
64M limit is in effect.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

The default of the pulsar-perf --memory-limit option changes from 0 (unlimited) to 64M.
This restores the behaviour that pulsar-perf had before #20663 rather than introducing a new
one, and it aligns the tool with the documented client default. Existing runs that relied on
unbounded client memory need --memory-limit 0 to keep it.

ClientConfigurationData#memoryLimitBytes itself is unchanged at 64M; only the literal moves to a
named constant.

[lhotari]

Before changing this, we'd need to find the actual root cause since this change will cause different pulsar-perf results in certain cases. This has happened in the past with changes in #13344. #15723 and #15748 were made at that time to address the performance regression, #15748 has some context.

Although setting the memory limit will bound the memory usage, it changes the results. For users using specific parameters with an older version of the tool will get different results where the backpressure is expected to be applied by maxPendingMessages and maxPendingMessagesAcrossPartitions.

The intention of this change has been to apply backpressure:

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java

Lines 336 to 341 in 2baa1f7

	if (!memoryLimitController.isMemoryLimited()) {
	// set default limits for producers when memory limit controller is disabled
	producerBuilder.maxPendingMessages(NO_MEMORY_LIMIT_DEFAULT_MAX_PENDING_MESSAGES);
	producerBuilder.maxPendingMessagesAcrossPartitions(
	NO_MEMORY_LIMIT_DEFAULT_MAX_PENDING_MESSAGES_ACROSS_PARTITIONS);
	}

The possible root cause could be that this regresses in some way between 3.0.5 and 4.0.9.

[lhotari]

I performed analysis with Claude and based on that, it seems that the NO_MEMORY_LIMIT_DEFAULT_MAX_PENDING_MESSAGES / NO_MEMORY_LIMIT_DEFAULT_MAX_PENDING_MESSAGES_ACROSS_PARTITIONS doesn't get applied for non-partitioned topics.

If we change the default memory limit for pulsar-perf, I believe that the limit should be proportional to the max direct memory provided to pulsar-perf. The reason for this is that it would prevent silent performance regressions caused by the parameter change while capping the memory limit and preventing OOM.
Since pulsar-perf doesn't use direct memory for other purposes than Netty, it could use 50% of available direct memory.

Code example of setting a parameter based on available direct memory:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java

Lines 99 to 103 in 083fd4c

	if (managedLedgerMaxReadsInFlightSizeInMB == null) {
	// When unset, default to 15% of the available JVM direct memory, but never below the maximum
	// size of a single read (dispatcherMaxReadSizeBytes) so that the limiter can never block the
	// completion of one read.
	long fractionOfDirectMemory = (long) (0.15d * DirectMemoryUtils.jvmMaxDirectMemory());

[lhotari]

There's #26342 which fixes the root cause of #26340. As mentioned in the previous commit, I'd suggest taking the path where instead of setting the memory limit to 64M, it would be 50% of available direct memory. This is something that could also be discussed on the dev mailing list.

[AlvaroStream]

Should we make it the default so test can be compared and improve it later?

[lhotari]

Should we make it the default so test can be compared and improve it later?

I'd suggest taking the path where instead of setting the memory limit to 64M, it would be 50% of available direct memory.

This would make pulsar-perf test results more consistent across versions since there hasn't been a limit in the past. Setting a limit changes the behavior significantly for many workloads since there will be less inflight messages. This would mainly impact tests where there are a lot of partitions and the message sizes are relatively large (for example 100 partitions, 32kB message size, very high message rate).
Making the default proportional to the available direct memory is useful since the direct memory would actually get used when there's available memory to use.

The problem with the previous NO_MEMORY_LIMIT_DEFAULT_MAX_PENDING_MESSAGES/NO_MEMORY_LIMIT_DEFAULT_MAX_PENDING_MESSAGES_ACROSS_PARTITIONS solution was that it didn't apply to non-partitioned topics. That's fixed by #26342. Since those limits have been in place for partitioned topics, setting the memory limit to 64M change the behavior and produce different results.

Hopefully this clarifies the reason why setting to 64M isn't something that I support and I'm instead recommending to make it dynamic, based on available direct memory.
It's a very simple change to this PR to make it dynamic. (long) (0.5d * DirectMemoryUtils.jvmMaxDirectMemory()) will return 50% of total direct memory in bytes.

[lhotari]

For Pulsar 4.x, addressing #26340 needs #26371 and #26342 in addition to this one.

On 4.x, pulsar-perf produce always set maxPendingMessages to 0, for partitioned and non-partitioned topics alike: maxOutstanding defaults to ProducerConfigurationData.DEFAULT_MAX_PENDING_MESSAGES, which is 0 since PIP-120 (#13344), and it was passed to the builder unconditionally. maxPendingMessagesAcrossPartitions avoided that only because #15283 made it conditional — but it then simply kept its own default of 0, so neither limit was set.

One correction to what I said earlier: the NO_MEMORY_LIMIT_DEFAULT_MAX_PENDING_MESSAGES* values never reached this tool at all, so they were not what gave partitioned topics backpressure. They are applied in newProducer(Schema), and PerformanceProducer has called the no-argument newProducer() since #1311 (2018) — that overload never got them, which is the first of the two holes #26342 closes. So on 4.x the producer had no message-count bound on either topic shape, and the client memory limit that this PR restores was the only thing that could bound it.

That is why the 4.x fix takes all three: this PR restores the byte-based backpressure; #26371 makes -o / -p Integer so an unset flag falls back to the client's defaults instead of reading as an explicit "no limit", and takes the newProducer(Schema) overload so those defaults exist at all; #26342 turns those defaults into real defaults that a pass-through 0 can no longer overwrite.

#26371 targets branch-4.2; branch-4.0 and branch-4.1 have the identical PerformanceProducer and need the same change.