HDDS-11898. Design document for leader side execution

[spacemonkd]

What changes were proposed in this pull request?

HDDS-11898. Design document for Leader Side Execution framework

Please describe your PR in detail:

• This PR adds the design document for Leader Side Execution Framework

Based on the proposal by @kerneltime and earlier work by @sumitagrawl on PR #7583

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11898

How was this patch tested?

N/A

[ivandika3]

Please kindly help to research at other production Raft-based / Paxos-based systems (e.g. CockroachDB, TiKV (TiDB), YugabyteDB, ScyllaDB, Cassandra, RheaKV) with respect on how they separate the transaction processing and the state machine replication since these systems should have mature concurrency model while remaining performant. Also it is also good to review some research papers and literatures for this. Based on these systems and literatures, we can inform the way we would evolve the OM architecture.

The leader-side execution requires a more robust (and possibly complex) concurrency model (similar to RDBMS concurrency control) since most write transaction is not going to be serialized in Raft single threaded applier and Ratis is going to be used only for DB mutation replication. From my understanding now is that we need to implement more RDMBS related transaction logic like transaction commit, rollback, etc. However, I'm not sure whether the community has experience in this (e.g. implementing RDBMS MVCC) before.

Also previously @kerneltime and @sumitagrawl worked on this, have you discussed with them regarding the current status and blockers? What is the difference between this and their plan? I'm a bit confused about the status of this feature.

My main point is that we should make sure and convince ourselves that this is a correct decision and make the design a lot more fleshed out (e.g. more detail implementations, how to do concurrency tests to ensure that consistency is not affected). Changing the OM transaction processing framework is a very heavy and time-consuming implementation, so it's better if we can do it right straightaway rather than having future OM transaction processing framework rework again. Personally, I have been wanting to study more on RDBMS concurrency control by implementing https://15445.courses.cs.cmu.edu/fall2025/assignments.html so that I can have the correct fundamentals for evaluate the design.

[spacemonkd]

Thanks @ivandika3 for the detailed insights.
I agree that we should do it once and do it right. I'm going to study the implementation of others using Raft/Paxos and see what they are doing. Probably Claude can help me extract their high level design.

As for this design vs the earlier design from @sumitagrawl and @kerneltime - it isn't different, the older PRs were closed as they didn't have activity, I am planning to pick up the same work and dedicate some time to research and implement these changes. I'll be connecting with them as well to get their inputs and ideas.

Thank you again for the detailed inputs, I really appreciate it.

[kerneltime]

@spacemonkd Thank you for reviving this work. This is probably the largest OM write performance project that is in the pipeline of proposals. I took the time to use Opus and go over all the past spec work, review comments and my original design to generate a detailed a very formal spec here https://github.com/kerneltime/ozone/tree/9c59ae0d66227ce10849b4f914f9efdce9318103/hadoop-hdds/docs/content/design/leader-execution
I am also experimenting using formal methods to validate the spec and the concurrency model.
The docs above include a formal locking proposal that we can base the implementation on. @ivandika3 @spacemonkd has been working with @sumitagrawl and me for this. As for the reason for the performance improvement cause. I believe, that any consensus framework even with batching has a performance envelope in terms of number of concurrent requests in a time frame that it can drive. The size of each slot in the linear write order has some flexibility and adding more requests per slot in the linear history of consensus will bump up the reqs/sec after we reach performance for the consensus framework. This change has a few parts to it

1. Pre compute the DB write and send the DB delta through a consensus round. This simplifies the post Ratis concensus execution and simplifies the caching model. Both these have been a sustained source of bugs and performance challenges.
2. Adds batching when sending a request to Ratis which empirically improves performance more than 3x and reduces overall CPU profile.
3. Eliminates Object ID and other OM correctness from Ratis internals.
   If in the future if Ratis improves the batching such that batching does not lead to a performance improvement we can make the batch size to be one. Thus, this change overall is a good cleanup of the code that simplifies the processing, caching and improves performance.
   The use of AI tooling should help us eliminate a lot of the effort involved while keeping our focus on the correctness.

[ivandika3]

For the RocksDB mutation, we can check Rocksplicator (https://github.com/pinterest/rocksplicator) and Apache KvRocks (https://github.com/apache/kvrocks , inspired by Rocksplicator).

It is also good to take a look at CNCF project CubeFS (https://github.com/cubefs/cubefs) which is also uses Raft based systems (they already supports metadata sharding / partitioning).

[amaliujia]

Per our offline discussion, I suggest breaking this project into at least two sequential milestones. This approach mitigates risk, allows for a gradual rollout, and delivers incremental value.

Milestone 1: Defer Raw Byte Serialization

• Instead of building raw bytes in startTransaction, we should retain the Java objects. This allows us to populate the ObjectID and UpdateID during applyTransaction.
• Benefit: We avoid the immediate complexity of implementing and validating the ManagedIndexService for all edge cases.
• Value Delivered: We can immediately deliver reduced follower execution, eliminate locking and double-buffer complexities, and lower the risk of inconsistency by moving the core business logic to leader planning time.

Milestone 2: Transition to Raw Bytes

• Once Milestone 1 is stable, we can focus on strictly writing raw bytes to the Ratis log and managing the ObjectID/UpdateID entirely during leader planning time.
• Value Delivered: This unblocks Recon to consume the Ratis log directly, decoupling it from leader-specific logic.

[amaliujia]

Following up our conversion:

I am seeing a potential issue of monotonic increasing OM Object ID allocation here. Because we have decoupled the OM Object ID allocation (in StartTransaction) and OM Object ID persistence (In ApplyTransaction), there is a gap in between where an ID is allocated, not yet reach RockDB, and the OM leader crashes.

On concrete example I am thinking is:

1. Old OM leader allocates 1, 2, 3, 4, 5.
2. The DB deltas with ID 1, 2, 3, 4, 5 have been appended to Ratis logs.
3. Only DB deltas with ID 1,2 are applied to DB.

Then assuming old OM leader crashes and new OM leader wins, and assuming last committed Index is 4, then new OM leader does a max(currentIndex, lastCommittedIndex) which is max(2, 4) which becomes 4.

So OM becomes the next Object ID is 5, however 5 is already been used and in the Ratis log, thus cause a conflict.

[spacemonkd]

Thanks for pointing this issue out.
I have updated the design to handle this in a new manner. Could you take another look?

@kerneltime what do you think of the new approach to use an applyFloor method to determine the next ID?