RFC: Unified Cursor Format

[alexluong]

Unified Cursor Format

Problem

Two separate cursor implementations with duplicated base62 encoding:

• LogStore: v1:{sortBy}:{sortOrder}:{position} (not yet on main)
• Tenants: tntv01:{position}

Design Decision

Cursor format: {resource}v{version}:{data} → base62 encoded

Resource	Format	Current Data
Events	evtv01:{data}	Position string: 1737123456789_evt_abc
Deliveries	dlvv01:{data}	Position string: 1737123456789_dlv_xyz
Tenants	tntv01:{data}	Timestamp: 1737123456789

Key insight: Cursors only need opaque data. Sort direction comes from the request. The v1:{sortBy}:{sortOrder}:{position} format in the logstore branch is unnecessary.

Evolution

Currently, data is just a position string. If a driver needs richer data, bump the version:

// v01: data = position string
"evtv01:1737123456789_evt_abc"

// v02: data = structured (hypothetical)
"evtv02:1737123456789_evt_abc|extra"

For structured data, avoid JSON (parsing overhead). Use a custom struct with MarshalBinary/UnmarshalBinary for efficient encoding.

Driver handling multiple versions:

data, err := cursor.Decode(req.Next, "evt", 2)
if errors.Is(err, cursor.ErrVersionMismatch) {
    // Fallback to v1
    data, err = cursor.Decode(req.Next, "evt", 1)
    if err != nil {
        return cursor.ErrInvalidCursor
    }
    position = data  // v1: data is just position
} else {
    // v2: custom struct with additional fields
    var c CursorDataV2
    if err := c.UnmarshalBinary([]byte(data)); err != nil {
        return cursor.ErrInvalidCursor
    }
    position = c.Position
    extra = c.Extra
}

Each driver owns its data format. The cursor package is format-agnostic - just handles {resource}v{version}:{data} encoding.

Shared Package

New: internal/cursor/cursor.go

package cursor

var (
    ErrInvalidCursor   = errors.New("invalid cursor")
    ErrVersionMismatch = errors.New("cursor version mismatch")
)

func Encode(resource string, version int, data string) string
func Decode(encoded, resource string, version int) (data string, err error)
func Base62Encode(s string) string
func Base62Decode(s string) (string, error)

Backward Compatibility

Not strictly necessary (beta, few users, cursors are ephemeral), but cheap to include.

Legacy logstore cursors (raw base62'd data, no prefix):

data, err := cursor.Decode(req.Next, "evt", 1)
if errors.Is(err, cursor.ErrVersionMismatch) {
    data, err = cursor.Base62Decode(req.Next)  // legacy: no prefix
}

[alexluong]

Btw, I totally get it if you think this is a bit overengineered in terms of data or versioning. Part of the reason I want to introduce this was because if we ever want to support a 2-store query mechanism, we may need to maintain a composite cursor that has data for 2 or more actual cursors. Just a note from my experience tinkering with pagination in this context.

[alexluong]

implemented by #646