Search before reporting
Motivation
In topics where messages are roughly uniform in size, a byte-based threshold works as a reasonable approximation: bytes accumulated since last compaction correlates well enough with the number of new key updates.
This assumption breaks down when message sizes are heterogeneous. Consider a topic that mixes small control events (a few hundred bytes) with large snapshot messages (megabytes):
A low threshold is hit after just a handful of large snapshots, triggering compaction runs that are disproportionate to the actual number of stale keys — wasting broker I/O and ledger resources
A high threshold is never reached during stretches of small-message activity, even when thousands of key updates have accumulated and the compacted view is significantly out of date
The core mismatch is that bytes ≠ number of key updates. Compaction's purpose is to retain only the latest value per key; the amount of work it needs to do — and how stale the compacted view is — is determined by how many messages have been written since the last run, not by their total byte size.
A message-count threshold addresses this directly: compaction fires after N new messages regardless of their individual sizes, providing a stable and predictable trigger that reflects actual compaction pressure rather than storage volume.
Solution
No response
Alternatives
Tuning topicCompactionThresholdInBytes per topic — already possible, but operationally expensive on large clusters with heterogeneous topics and does not solve the fundamental mismatch between bytes and message staleness
External compaction triggering via Admin API — works as a workaround but requires additional infrastructure and loses the simplicity of built-in automation
Anything else?
No response
Are you willing to submit a PR?
Search before reporting
Motivation
In topics where messages are roughly uniform in size, a byte-based threshold works as a reasonable approximation: bytes accumulated since last compaction correlates well enough with the number of new key updates.
This assumption breaks down when message sizes are heterogeneous. Consider a topic that mixes small control events (a few hundred bytes) with large snapshot messages (megabytes):
A low threshold is hit after just a handful of large snapshots, triggering compaction runs that are disproportionate to the actual number of stale keys — wasting broker I/O and ledger resources
A high threshold is never reached during stretches of small-message activity, even when thousands of key updates have accumulated and the compacted view is significantly out of date
The core mismatch is that bytes ≠ number of key updates. Compaction's purpose is to retain only the latest value per key; the amount of work it needs to do — and how stale the compacted view is — is determined by how many messages have been written since the last run, not by their total byte size.
A message-count threshold addresses this directly: compaction fires after N new messages regardless of their individual sizes, providing a stable and predictable trigger that reflects actual compaction pressure rather than storage volume.
Solution
No response
Alternatives
Tuning topicCompactionThresholdInBytes per topic — already possible, but operationally expensive on large clusters with heterogeneous topics and does not solve the fundamental mismatch between bytes and message staleness
External compaction triggering via Admin API — works as a workaround but requires additional infrastructure and loses the simplicity of built-in automation
Anything else?
No response
Are you willing to submit a PR?