Ledger defragmentation when messages are not ACKed

[tomaszmichalak]

Case:
We are using Apache Pulsar topic to propagate small files (~30kB) from the producer ("files-generator") to the consumers ("web-server"). We send files via messages with a key, where the key is an unique file path. In case the new consumer subscribes to the topic, only the latest version of the file is important. Old versions of files should be cleaned from the topic to limit the storage size.

Issue:
Apache Pulsar keeps two cursors per subscription (read position and mark delete position). Normally all received messages are ACKed and marked to delete. So mark delete position cursor is close to the read position and old ledgers are cleaned.

In our case, to keep only the latest versions, we acknowledge messages containing the old versions. As a result, we have a lot of acknowledgement holes (we process 1 million files per hour).

Question:
Is there any built-in mechanism in Apache Pulsar (or Bookkeeper) to clean ledgers containing such holes (similar to defragmentation)? For example, not acknowledged messages from the old ledgers are moved to the new ledger, so then the old ledgers can be deleted.