Is your feature request related to a problem? Please describe.
Currently, SerdeUtils in serde_utils.rs uses bincode for serializing and deserializing data stored in RocksDB (including checkpoint metadata, journal batches, inode views, and other persistent data structures). Bincode is a compact binary format but lacks built-in support for schema evolution. This means:
Adding new fields to serialized structs will cause deserialization failures when reading old data
Removing or renaming fields breaks compatibility with previously persisted data
Changing field types requires a full data migration
Rolling upgrades become risky as old nodes cannot read data written by new nodes
This makes it difficult to evolve the data schema over time without requiring a full cluster rebuild or complex migration procedures.
Describe the solution you'd like
Migrate from bincode to a serialization format that supports backwards compatibility, such as:
Protocol Buffers (protobuf): Provides explicit field numbering, optional fields, and well-defined schema evolution rules
MessagePack with schema: Supports field additions/removals gracefully
CBOR: Self-describing format with good schema evolution properties
The new serialization should:
Allow adding new optional fields without breaking existing data
Allow deprecating fields gracefully
Support reading old format data during a migration period (dual-read capability)
Maintain reasonable performance characteristics for high-throughput operations
Is your feature request related to a problem? Please describe.
Currently, SerdeUtils in serde_utils.rs uses bincode for serializing and deserializing data stored in RocksDB (including checkpoint metadata, journal batches, inode views, and other persistent data structures). Bincode is a compact binary format but lacks built-in support for schema evolution. This means:
Adding new fields to serialized structs will cause deserialization failures when reading old data
Removing or renaming fields breaks compatibility with previously persisted data
Changing field types requires a full data migration
Rolling upgrades become risky as old nodes cannot read data written by new nodes
This makes it difficult to evolve the data schema over time without requiring a full cluster rebuild or complex migration procedures.
Describe the solution you'd like
Migrate from bincode to a serialization format that supports backwards compatibility, such as:
Protocol Buffers (protobuf): Provides explicit field numbering, optional fields, and well-defined schema evolution rules
MessagePack with schema: Supports field additions/removals gracefully
CBOR: Self-describing format with good schema evolution properties
The new serialization should:
Allow adding new optional fields without breaking existing data
Allow deprecating fields gracefully
Support reading old format data during a migration period (dual-read capability)
Maintain reasonable performance characteristics for high-throughput operations