when consumer ack message, the backlog size not reduce, the bookie storage size not release

[FakeKuryr]

1. the backlog-quota use the  limitTime 
pulsar-admin namespaces get-backlog-quotas bsc/spider
message_age    BacklogQuotaImpl(limit=-1, limitSize=-1, limitTime=21600, policy=consumer_backlog_eviction)

2. not set the retention policy
pulsar-admin namespaces get-retention  bsc/spider
{
  "retentionTimeInMinutes" : 0,
  "retentionSizeInMB" : 0
}

3. the pulsar_storage_backlog_size always increase

5. but the pulsar_msg_backlog is always consumer
   

6. only have one subscription and use shared 

  "msgRateIn" : 47.333925891802906,
  "msgThroughputIn" : 76108.15232319059,
  "msgRateOut" : 101.86800154014381,
  "msgThroughputOut" : 168684.02298794116,
  "bytesInCounter" : 376920164455,
  "msgInCounter" : 1882569,
  "bytesOutCounter" : 410604705531,
  "msgOutCounter" : 9426464,
  "averageMsgSize" : 1607.89858202678,
  "msgChunkPublished" : false,
  "storageSize" : 116436386186,
  "backlogSize" : 116391369788,
  "publishRateLimitedTimes" : 0,
  "earliestMsgPublishTimeInBacklogs" : 0,
  "offloadedStorageSize" : 0,
  "lastOffloadLedgerId" : 0,
  "lastOffloadSuccessTimeStamp" : 0,
  "lastOffloadFailureTimeStamp" : 0,
  "ongoingTxnCount" : 0,
  "abortedTxnCount" : 0,
  "committedTxnCount" : 0,

7.  When I use the clear-backlog command, the sotrage size will release

why  consumer ack message, but the bookie not release the backlog storage size?

[dao-jun]

Because pulsar_storage_backlog_size is not an exact value, but an estimated value, it calculated by markDeletedPosition and writePosition. So, although some messages are individualAcked, it also marked as backlog.

It is very difficult to accurately calculate backlog_size, because there are entryFilter and individualACK exists.

clearBacklog will move markDeletedPosition to writePosition, so the pulsar_storage_backlog_size will be set to 0.

You can try consume message with Exclusive/Failover subscription.

[asafm]

@DeyunLuo Don't forget to mark answerred

[asafm]

You can instruct the Backlog Quota check process, which runs periodically, to use a more precise calculation of the backlog size, by using the following config key: https://pulsar.apache.org/reference/#/next/config/reference-configuration-broker?id=precisetimebasedbacklogquotacheck