Using MessageID as a log sequence number downstream? Any better options?

[levand]

One nice property of Pulsar is that all the messages on a topic have a total order that can be used by downstream systems (for example, as the "version" for data).

However, Pulsar's built in MessageIDs are a bit cumbersome. Unlike Kafka or some other systems, they aren't a single integer: they're a struct consisting of a Partition ID (uint32), Ledger ID (uint64), Entry ID (uint64) and Batch ID (uint32).

I have two questions about this:

1. Is it safe to rely on this internal undocumented structure of a MessageID? It seems like it should be safe, since changing it would break many things, but I wanted to double check.

2. Is there a better way to get a log sequence number? I guess we could create our own with a stateful Pulsar Function but that seems like it would add overhead and affect our throughput.

[BewareMyPower]

Is it safe to rely on this internal undocumented structure of a MessageID?

No. Or at least theoretically not.

log sequence number

You can try the BrokerMetadataInterceptor feature, which is already used in KoP.

brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor

Then you can use the Message#getIndex API to get the index, which is similar to Kafka's offset.