diff --git a/hadoop-hdds/docs/content/design/leader-execution/_index.md b/hadoop-hdds/docs/content/design/leader-execution/_index.md new file mode 100644 index 000000000000..c86a8cd1dfe4 --- /dev/null +++ b/hadoop-hdds/docs/content/design/leader-execution/_index.md @@ -0,0 +1,789 @@ +--- +title: Leader-Planned State Transition Execution +summary: Leader computes DB changes once and sends them to followers, instead of every node repeating the same work +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +author: Abhishek Pal +--- + + +# Leader-Planned State Transition Execution + +## Table of Contents + +1. [Motivation](#1-motivation) +2. [Industry Precedent](#2-industry-precedent) +3. [Proposal](#3-proposal) + * [3.1 High-Level Architecture](#31-high-level-architecture) + * [3.2 Proto Format](#32-proto-format-replicatedstatetransition) + * [3.3 Managed Index Service](#33-managed-index-service) + * [3.4 Leader Planning Framework](#34-leader-planning-framework) + * [3.5 Apply Engine](#35-apply-engine) + * [3.6 State Machine Integration](#36-state-machine-integration) + * [3.7 Granular Locking](#37-granular-locking) + * [3.8 Lock Hold Span](#38-lock-hold-span) +4. [Detailed Concurrency Model](#4-detailed-concurrency-model) +5. [Correctness Arguments](#5-correctness-arguments) +6. [Testing Strategy](#6-testing-strategy) +7. [Comparison: Legacy vs Planned Path](#7-comparison-legacy-vs-planned-path) +8. [Migration Strategy](#8-migration-strategy) +9. [FAQs](#9-faqs) +10. [Future Work and Open Items](#10-future-work-and-open-items) + +--- + +## 1. Motivation + +### Problem with the current OM HA write flow + +In the current architecture, every write request follows this path: + +``` +Client -> OM Leader (preExecute) -> Ratis replication -> All OMs (validateAndUpdateCache) +``` + +Every OM node (leader and followers) runs the full business logic again in +`validateAndUpdateCache()`. This causes several problems: + +| Problem | Impact | +|:--------|:-------| +| **Wasted compute on followers** | Every follower repeats the full request processing — lock acquisition, validation, key building, quota computation | +| **Risk of inconsistency** | `validateAndUpdateCache` reads from a table cache that may have different temporary state on leader vs follower when requests run at the same time | +| **Double buffer complexity** | Results are queued in `OzoneManagerDoubleBuffer`, which adds batching delay and a separate flush thread. Snapshot barriers add more coordination | +| **Tight coupling** | External consumers (such as Recon) must understand all command logic to read state changes | +| **Lock contention** | OBS key operations hold a bucket-level write lock even though most creates/commits do not conflict with each other | + +### What we want + +1. Leader computes a **fixed DB patch** (a list of table puts and deletes) once. +2. The patch is sent to all nodes via Ratis as a self-contained payload. +3. All nodes (leader, followers, listeners) apply the patch directly to RocksDB — no business logic runs again. +4. Locks are made more fine-grained so independent key operations can run in parallel. + +--- + +## 2. Industry Precedent + +Before we detail the proposal, it is useful to see how other production +Raft-based systems solve the same problem. There are two models used in +practice: + +| Model | Who runs the logic? | What goes through Raft? | Used by | +|-------|--------------------|-----------------------|---------| +| **State Machine Replication (SMR)** | All replicas | Commands (the operation) | TiKV, older CockroachDB, existing Ozone replication model | +| **Leader Execution (Primary-Backup)** | Leader only | Computed results (WriteBatch) | YugabyteDB, newer CockroachDB, **this proposal** | + +### 2.1 CockroachDB — Proposer-Evaluated KV + +CockroachDB originally used classic state machine replication (all replicas +evaluate commands independently). They later moved to **Proposer-Evaluated KV +(PEVK)** — the same pattern we propose here. + +**How it works:** +1. The lease-holder (leader) evaluates the request and produces a `WriteBatch` + (the exact bytes to write to storage). +2. The `WriteBatch` + computed response are proposed to Raft as a single entry. +3. Followers apply the pre-computed `WriteBatch` directly — they do not + re-execute command logic. + +**Why they moved to this model** (from their design RFC): +- Removes duplicate execution across replicas +- Makes online migrations simpler (only leader needs new evaluation code) +- Puts all complex logic in one place (the proposer) + +**Their concurrency control:** +- **Latches** (fine-grained per-key read/write locks) are acquired **before** + evaluation and released **after** the WriteBatch is built — but **before** + Raft replication. This is the same pattern as our striped locks. +- An in-memory **lock table** handles transaction-level conflict queueing. +- MVCC timestamps provide version ordering. + +**Key insight:** CockroachDB holds latches only during evaluation, NOT during +replication: +``` +acquire_latch → evaluate(request) → build WriteBatch → release_latch → propose to Raft +``` +This matches our design: `acquireLock → plan() → releaseLock → submit to Ratis`. + +Source: `pkg/kv/kvserver/replica_proposal.go`, `pkg/kv/kvserver/concurrency/` + +### 2.2 YugabyteDB — Leader-Computed Write Batches + +YugabyteDB also uses leader execution: + +1. The tablet leader computes the write batch (exact RocksDB key-value pairs). +2. The batch is replicated via Raft to followers. +3. Followers apply the **pre-computed byte-level changes** to their local + RocksDB — no re-execution. + +Their concurrency uses "provisional records" (intents) written to a separate +RocksDB instance as persistent locks, plus hybrid timestamps for ordering. + +### 2.3 TiKV — Classic State Machine Replication + +TiKV uses the traditional approach: all replicas execute the same Raft log +entries. However, TiKV's "commands" are already low-level Put/Delete operations +on key-value pairs. There is no complex business logic to re-execute. + +This works for TiKV because the transaction coordination layer (Percolator 2PC) +runs **above** Raft. By the time something enters the Raft log, it is already a +simple key-value mutation. + +--- + +## 3. Proposal + +### 3.1 High-Level Architecture + +``` + LEADER FOLLOWER + +------------------+ + Client Request | startTransaction | + -------------> | (leader only) | + | | + | 1. Create | + | PlannedReq | + | 2. preProcess | + | 3. authorize | + | 4. acquireLock | + | 5. plan() | + | -> deltas | + | 6. releaseLock | + | 7. Build proto | + +--------+---------+ + | + | Ratis log entry + | (OMRequest with embedded + | BatchedStateTransitions) + | + +--------v---------+ +------------------+ + | applyTransaction | | applyTransaction | + | (leader) | | (follower) | + | | | | + | 1. Parse proto | | 1. Parse proto | + | 2. For each | | 2. For each | + | Operation: | | Operation: | + | table.put/ | | table.put/ | + | delete/merge | | delete/merge | + | 3. Commit batch | | 3. Commit batch | + | | | | + | (zero business | | (zero business | + | logic) | | logic) | + +------------------+ +------------------+ +``` + +**Important:** The leader does NOT write to RocksDB during `startTransaction`. +It only computes the deltas (the list of puts and deletes). The actual DB write +happens later in `applyTransaction`, which runs on **both leader and followers** +after Ratis commits the log entry. This ensures all nodes have the same DB state. + +The result of planning is a list of raw `{table, key_bytes, value_bytes, +PUT/DELETE}` operations packed into a `ReplicatedStateTransition` proto. The +apply step on all nodes opens a RocksDB batch, loops through the deltas, and +commits. + +### 3.2 Proto Format: ReplicatedStateTransition + +The proto has two layers: + +**Inner layer (domain-agnostic, in `hadoop-hdds/framework`):** +```protobuf +message Operation { + enum Kind { PUT = 1; DELETE = 2; MERGE = 3; CHECKPOINT = 4; } + Kind kind = 1; + bytes column_family = 2; + bytes key = 3; + bytes value = 4; + bytes merge_operator_id = 5; // selects the merge resolver (for MERGE kind) +} + +message Batch { + repeated Operation ops = 1; +} +``` + +**Outer layer (OM envelope, in `hadoop-ozone/interface-client`):** +```protobuf +message ReplicatedStateTransition { + required int64 managedIndex = 1; // OM-managed monotonic ID + required Batch batch = 2; // the inner domain-agnostic patch + required OMResponse response = 3; // pre-computed client response + required Type cmdType = 4; // for barrier detection +} + +message BatchedStateTransitions { + repeated ReplicatedStateTransition transitions = 1; +} +``` + +**Key decisions:** +- **Raw bytes, not domain objects.** Each operation carries the exact bytes + that will be written to the RocksDB column family. Followers do no + deserialization of `OmKeyInfo`, `OmBucketInfo`, etc. +- **Four operation kinds.** `MERGE` is needed for commutative quota updates + (see [OBS quota handling](./obs-design.md#3-quota-handling-decision-required)). + `CHECKPOINT` triggers the snapshot barrier. +- **Inner layer is domain-agnostic.** The `Batch`/`Operation` proto and its + apply engine live in `hadoop-hdds/framework` and import no Ozone OM types. + This makes the apply engine reusable by Recon-as-listener and future + consumers without importing OM command classes. +- **The inner layer never deserializes a domain object.** Apply sees bytes, + writes bytes. This is what makes followers business-logic-free. + +The `BatchedStateTransitions` wrapper is placed on the `OMRequest` at field +number 200. The state machine checks `hasBatchedStateTransitions()` to decide +whether to use the new apply path. + +### 3.3 Managed Index Service + +The OM-managed `AtomicLong` counter that mints objectIDs independently of the Ratis log index. Persisted atomically with each DB batch, and ensures old-path/new-path objectID ranges are disjoint during mixed mode. Every `applyTransaction` (including log replay) calls `advanceFloor` to keep the in-memory counter past all committed values, so the counter is always correct after leader switchover. + +→ [Full component details](./components.md#1-managed-index-service) + +### 3.4 Leader Planning Framework + +Three classes form the planning framework: `PlannedRequest` (abstract base +with a step-iterator model that generalizes single-step and multi-step +operations), `ChangeRecorder` (accumulates a `Batch` proto instead of staging +into the table cache), and `LeaderPlanner` (the orchestrator that drives the +step-iterator to completion, continuation-driven to free worker threads during +Ratis awaits). + +→ [Full component details](./components.md#2-leader-planning-framework) + +### 3.5 Apply Engine + +The `OperationApplier` (in `hadoop-hdds/framework`) writes a `Batch` to +RocksDB. Domain-agnostic — imports no Ozone OM types. Handles barrier splitting +for snapshot operations. A follower that cannot apply a committed patch must +crash and re-sync (fail-stop), same as today. + +→ [Full component details](./components.md#3-apply-engine) + +### 3.6 State Machine Integration + +The `OzoneManagerStateMachine` runs a dual-path model: planned commands +(registered via `registerPlannedCommand`) use the new apply engine; legacy +commands use the existing `runCommand()` + double-buffer path. This allows +step-by-step migration one command at a time. + +→ [Full component details](./components.md#4-state-machine-integration) + +### 3.7 Granular Locking + +Fine-grained, objectID-keyed locking that is rename-stable and ABA-free. +Two lock types: **container locks** (keyed by directory objectID, governs +children) and **slot locks** (keyed by parentObjectID+name, governs one entry). +Implemented as a fixed striped array of non-thread-affine semaphores (~2^20 +stripes). Deadlock-free via uniform total ordering of lock acquisition. + +→ [Full locking details](./locking-and-concurrency.md#1-granular-locking) + +### 3.8 Lock Hold Span + +An open decision: release locks before Ratis submit (Option A — higher +parallelism, matches CockroachDB) or hold through commit+apply (Option B — +removes table cache, trivially correct for multi-step operations). +Recommendation leans toward Option B for FSO and a possible hybrid for OBS. + +→ [Full analysis](./locking-and-concurrency.md#2-lock-hold-span-decision-required) + +--- + +## 4. Detailed Concurrency Model + +The detailed concurrency model covers the life of a request, concurrent +scenarios (parallel creates, same-key commits, CreateKey vs DeleteBucket, +FSO createFile vs rm-rf, crossing renames), quota handling, multi-step +operations, known limitations, and retry strategy. + +→ [OBS concurrency scenarios](./obs-design.md#2-concurrent-scenarios) +→ [FSO concurrency scenarios](./fso-design.md#2-concurrent-scenarios) +→ [Multi-step operations (FSO)](./fso-design.md#3-multi-step-operations) +→ [Quota handling decision](./obs-design.md#3-quota-handling-decision-required) +→ [Known limitations](./locking-and-concurrency.md#5-known-limitations-and-accepted-tradeoffs) +→ [Retry and idempotency](./locking-and-concurrency.md#6-retry-and-idempotency-deferred-decision) + +--- + +## 5. Correctness Arguments + +This section explains why the design is correct — i.e., it produces the same +results as the current serial execution model. + +### 5.1 Argument 1: Ratis Total Order Guarantees Linearizability + +All committed transitions are applied in the same order on all nodes. Ratis +ensures this: each entry has a unique `(term, index)` and entries are applied in +index order. This means: +- If request A is submitted before request B (from Ratis's perspective), A is + applied before B on every node +- The DB state after applying N entries is the same on all nodes + +This is the same guarantee as today. The difference is only in what gets applied +(raw bytes vs re-executed commands). + +### 5.2 Argument 2: Locks Serialize Conflicting Plans + +Two requests that touch the same data will acquire the same striped lock. Since +the lock is exclusive (write lock), only one can plan at a time. This ensures: +- The second request reads committed state that is consistent with what the + first request planned against +- No two plans can produce conflicting deltas for the same key + +**Exception:** CreateKey uses only bucket READ locks and no key lock. Multiple +creates to the same bucket plan in parallel. This is safe because each open key +entry has a unique key (`/vol/bucket/key/clientID`), so parallel creates write +to different rows. + +### 5.3 Argument 3: The Plan-to-Apply Gap is Safe + +Between planning (lock released) and applying (Ratis commits), the leader's DB +does not reflect the planned changes. Could a subsequent request read stale +state and produce wrong results? + +**For non-conflicting requests:** They touch different keys or different tables. +The stale state does not affect them. + +**For conflicting requests:** They need the same lock. Since the first request +still holds the lock during its planning phase, the second request cannot start +planning until the first finishes and releases the lock. By that point, the +first request has already submitted its transition to Ratis (sequentially after +releasing the lock). Ratis log ordering then guarantees the first transition is +applied before the second. + +**Remaining risk — quota drift:** If two requests both read quota=100 and both +plan quota=101, the final value is 101 (not 102). For CreateKey, this is +acceptable (see [OBS Scenario A](./obs-design.md#scenario-a-two-createkey-requests-to-different-keys-in-the-same-bucket)). +For CommitKey, the key write lock prevents this because commits to the same +bucket-and-key are serialized. + +If strict per-create quota accounting is needed, we can fall back to bucket +WRITE lock for CreateKey (same as today). This decision is a tunable tradeoff: +- Bucket READ lock = higher parallelism, slightly relaxed quota at create time +- Bucket WRITE lock = lower parallelism, strict quota at create time + +In both cases, CommitKey enforces final quota correctly because it holds the key +write lock. + +### 5.4 Argument 4: Atomic Apply Prevents Partial State + +Each `ReplicatedStateTransition` is applied as a single RocksDB `WriteBatch`. +RocksDB guarantees that a WriteBatch is atomic — either all writes succeed or +none do. If the process crashes mid-apply, the batch is not committed, and on +restart the entry is replayed from the Ratis log. + +### 5.5 Argument 5: Leader Failure is Handled + +If the leader fails after planning but before Ratis commits: +- The Ratis entry was not committed (no quorum ack). It is lost. +- The client times out and retries. The new leader plans the request fresh with a fresh managed index. + +If the leader fails after Ratis commits but before apply: +- On restart, the Ratis log replays all committed-but-not-applied entries. +- Each replayed `applyTransaction` calls `advanceFloor`, so the in-memory counter advances past all committed values before new requests are accepted. +- No re-planning is needed. + +Managed indices are allocated before commit and persisted after commit. This gap is safe because: (1) uncommitted entries are dropped on leader change and never replayed, so the new leader can reuse those values; (2) committed entries are replayed via `applyTransaction` which calls `advanceFloor`, advancing the counter past all committed values before the new leader accepts new proposals. See [ManagedIndexService details](./components.md#leader-switchover-and-the-allocation-to-persistence-gap) for the full analysis. + +### 5.6 What Could Go Wrong (and How We Prevent It) + +| Risk | Mitigation | +|------|-----------| +| Two threads plan for same key with different base state | Striped key write lock serializes them | +| Quota drift on parallel CreateKey | Acceptable for reservations; enforced at CommitKey. Can use bucket WRITE lock if stricter | +| Bucket deleted while key create is in Ratis log | Bucket WRITE lock + Ratis ordering prevents this (see [Scenario C](./obs-design.md#scenario-c-createkey-and-deletebucket-for-the-same-bucket)) | +| ManagedIndex duplicates after leader change | `advanceFloor` in apply path ensures counter is past all committed IDs before new allocations; uncommitted IDs are lost and safe to reuse | +| Crash during RocksDB WriteBatch | Atomic batch — either fully committed or not. Ratis replays on restart | +| Follower apply fails | Fatal error — follower must catch up via Ratis snapshot (same as today) | +| Snapshot barrier missed | Apply engine checks `cmdType` — CreateSnapshot/SnapshotPurge always get own batch | + +--- + +## 6. Testing Strategy + +Tests at multiple levels ensure correctness: unit tests for each component, +multi-threaded concurrency tests for lock interaction, 3-node HA integration +tests, a determinism verification test (planned path produces byte-identical +DB state to legacy path), a Wing-Gong/Lincheck linearizability harness, and +stress/chaos tests with random node kills. + +→ [Full testing strategy](./testing-strategy.md) + +--- + +## 7. Comparison: Legacy vs Planned Path + +| Aspect | Legacy Path | Planned Path | +|:-------|:------------|:-------------| +| **Who executes logic** | All nodes | Leader only | +| **What is replicated** | Raw `OMRequest` bytes | DB patch (`ReplicatedStateTransition`) | +| **Follower work** | Full `validateAndUpdateCache` | Write raw bytes to column families | +| **Locking** | Bucket write lock (coarse) | Bucket read + striped key/dir (fine) | +| **ID generation** | From Ratis `transactionLogIndex` | From `ManagedIndexService` | +| **Response available** | After `applyTransaction` | Right after planning | +| **Double buffer** | Required (batches + flushes) | Not used | +| **Table cache** | Maintained per-table | Not needed (reads from DB directly) | + +--- + +## 8. Migration Strategy + +Commands are migrated hardest-first across 8 phases (P-0 through P-7). Phase-0 +establishes 3 interlocked prerequisites (shared objectID counter, dual-path +applied-index durability, OMLayoutFeature finalization gate). Each command +migration is self-contained and independently revertible via a dual activation +gate (finalization + per-command runtime flag). + +→ [Full migration playbook](./migration-playbook.md) + +--- + +## 9. FAQs + +### Q: Why not batch multiple planned transitions into one Ratis entry (like the double buffer does)? + +The double buffer batches to spread out the cost of Ratis round-trips. But +Ratis already does its own batching at the replication layer (AppendEntries +groups multiple log entries in one RPC). Batching before Ratis brings back the +same problems as the double buffer: +- Queue management complexity +- Actual batch gain is small (~1.2x in practice) +- Adds delay for the first item in the batch +- One slow item blocks everything behind it + +Each planned request goes to Ratis individually. Ratis's built-in batching at +the log replication layer gives enough throughput. + +### Q: What happens between planning and apply on the leader? Can another request read stale state? + +Between planning (in `startTransaction`) and apply (in `applyTransaction`), +the leader's DB has **not yet been updated** with the planned writes. Another +request that touches the same key would: +- Try to acquire the same striped lock in its own `startTransaction` call +- Wait until the first request releases the lock after planning + +So the lock orders the *planning phase*, not the apply phase. This is safe +because: +- The DB is only changed during `applyTransaction` (after Ratis consensus) +- The second request cannot plan until the first has finished planning and + released the lock +- Between planning and apply, no reader can see a half-written state + +### Q: Why use `ManagedIndexService` instead of just using Ratis index? + +In `startTransaction()`, the Ratis log index for the current entry has not been +assigned yet (it is assigned during log append, which happens after +`startTransaction` returns). We need a unique ID at planning time for +`objectID` generation. The managed index provides this. + +Also, if we ever want to batch in the future, multiple transitions in one log +entry would share a Ratis index but need different object IDs. + +### Q: Why raw bytes in the proto instead of structured domain objects? + +Three reasons: +1. **Performance:** Followers skip all deserialization and codec work. + They write raw bytes directly to column families. +2. **Simplicity:** The apply engine is ~30 lines of code with no command-specific + logic. +3. **Reusability:** Any system that can open the same RocksDB (Recon, backup + tools, external audit systems) can apply the same payload without needing + OM command processing classes. + +### Q: How are snapshots handled? + +`CreateSnapshot` and `SnapshotPurge` are "barrier" operations. The apply engine +detects these by `cmdType` and makes sure they are applied in their own separate +RocksDB batch. This keeps the same guarantee as the current double buffer's +`splitReadyBufferAtCreateSnapshot()`. + +### Q: What about the table cache? + +The legacy path keeps per-table caches (in `TypedTable`) so that +`validateAndUpdateCache` on the leader can see not-yet-committed writes from +earlier requests in the same double-buffer batch. In the planned path: +- There is no double buffer. +- Each request plans on its own, reading from the committed DB state. +- The striped lock ensures requests to the same key run one after another, so + there is no "uncommitted earlier write" to worry about. + +Therefore, **table caches are not needed** for planned commands. They remain +in use for legacy (unmigrated) commands. + +### Q: Can the planned path and legacy path coexist? + +Yes. The state machine routes by command type. Planned commands bypass the +double buffer entirely. Legacy commands use the existing `runCommand` + +double-buffer path unchanged. Both update `lastAppliedTermIndex` correctly. + +### Q: What if planning fails (e.g., key not found, quota exceeded)? + +`LeaderPlanner` catches all exceptions, builds an error `OMResponse`, and +returns a `ReplicatedStateTransition` with **empty deltas** and the error +response. This is still sent through Ratis so all nodes see the same +outcome (no DB changes, index still moves forward). The client receives +the error response. + +### Q: Why bucket read lock instead of no lock for CreateKey? + +The bucket read lock stops the bucket from being deleted or having its +properties changed (e.g., quota, replication policy) while a key operation +is running. It does not block other key operations within the same bucket +(read locks are shared — multiple readers are allowed). + +### Q: How does this relate to removing the double buffer? + +Once all commands are migrated to the planned path, the double buffer is no +longer needed and can be removed entirely. During the migration period, both +paths coexist. The double buffer only serves unmigrated commands. + +### Q: What about audit logging and metrics? + +Not yet connected in the planned path. This is a TODO. Audit events and +OM metrics (`incNumKeyAllocates`, `incNumKeyCommits`, etc.) need to be +called during the planning phase (on the leader) or read from the response +(on followers). + +### Q: How does this compare to other production systems? + +See Section 2 (Industry Precedent). Our design follows the same model as +CockroachDB's Proposer-Evaluated KV and YugabyteDB's leader-computed write +batches. Both are proven at scale in production. The key difference: those +systems also need full MVCC and 2PC because they have multiple Raft groups +(sharded data). We do not — all OM metadata is in one Raft group, so our +concurrency needs are simpler (just latching, no multi-shard transactions). + +### Q: Does this work with Zero-Downtime Upgrade (ZDU)? + +Yes. See the [migration playbook](./migration-playbook.md#5-zero-downtime-upgrade-zdu-compatibility) +for full details. In short: +- The planned execution path is gated behind a component version check. +- Pre-finalization: OMs use the legacy path (compatible with old software). +- Post-finalization: all OMs switch to the planned path together. +- The legacy path stays in the code for one version after introduction, then + can be removed. Users upgrading across multiple versions must pass through + the intermediate version if using ZDU. + +### Q: Do we need to keep the old `validateAndUpdateCache` code forever? + +No. The ZDU versioning policy allows removing old internal APIs after one +version. For example: +- Version N introduces the planned path (legacy path still used pre-finalization) +- Version N+1 keeps the legacy path for pre-finalization only +- Version N+2 removes the legacy path entirely + +Users doing ZDU from version N to N+2 must pass through N+1. Users doing +offline (downtime) upgrades can jump directly. + +### Q: Why not reuse OzoneManagerLock? + +Two reasons: (1) It is built on `ReentrantReadWriteLock` which is thread-affine +— cannot be released on a different thread than acquired. Under Option B of the +lock hold span, the lock is released on the continuation thread after the Ratis +await. (2) It carries eight leveled resource types, per-type striped maps, +trackers, and reentrancy — machinery this design does not need. The replacement +is a single fixed striped array with no resource hierarchy. + +### Q: Why objectID-keyed locks instead of path-keyed? + +A directory rename in FSO is O(1): the directory keeps its objectID, only its +entry changes parent. Children are never touched because they key on the +parent's unchanged objectID. A path-keyed lock would be invalidated by rename; +an objectID-keyed lock survives it. Additionally, objectIDs are never reused +(monotonic from ManagedIndex), so there is no ABA problem where a deleted +node's lock could be confused with a new node's lock. + +### Q: What happens to quota if the Merge operator is chosen? + +The quota counter (`usedBytes`/`usedNamespace`) is always exact — it never +loses an update. Deltas are summed in Ratis order at apply time. However, the +quota **limit check** is still best-effort: two parallel commits can both pass +the check (reading the same pre-increment value) and then both apply. This +means the bucket can transiently exceed its quota limit by up to the number of +in-flight commits. The existing `QuotaRepair` background service reconciles any +drift. This soft-limit behavior is an accepted tradeoff for parallelism. + +### Q: How does FSO createFile with missing parents work as multi-step? + +The leader decomposes `createFile /a/b/c/file` (with `/b` and `/c` missing) +into a chain: create-dir-b → await commit → create-dir-c → await commit → +create-open-file. Each step runs under its own lock set, revalidates the locked +nodes, plans one transition, and submits to Ratis. The worker thread is freed +during each Ratis await (continuation-driven). If the leader crashes mid-chain, +the client retries the whole request on the new leader — already-created +directories are skipped idempotently. + +--- + +## 10. Future Work and Open Items + +These items are scoped out of the current design but are known to need +resolution as the implementation progresses. They are listed here so that +future work does not re-derive them from scratch. + +### 10.1 Exact Quota Enforcement (Open Decision) + +The current design enforces quota limits best-effort (see +[known limitations](./locking-and-concurrency.md#5-known-limitations-and-accepted-tradeoffs)). +The open question: should admission be **exact** (leader-local atomic +reservation) or **approximate** (Merge-only, rely on `QuotaRepair`)? + +Leading candidate for exact enforcement: +- Atomic check-and-reserve in memory on the leader before Ratis submit. +- DB Merge remains the durable truth (no change to the replicated patch). +- Decrement-on-abort if the Ratis submit fails. +- Rebuild-from-DB on failover (the in-memory reservation is leader-local). + +This is layered on top of the commutative Merge without re-migrating any +command. The Merge lands unconditionally in P-1; the enforcement mode is +added later if exact admission is needed. + +### 10.2 Retry/Idempotency Mechanism (Deferred Decision) + +See [retry and idempotency](./locking-and-concurrency.md#6-retry-and-idempotency-deferred-decision) +for the full analysis. The mechanism is deferred pending the per-operation +idempotency audit. The ~10 non-idempotent operations need a durable retry +entry written atomically with the data batch; the ~37 idempotent operations +likely need nothing beyond their natural idempotency. + +### 10.3 Multipart Upload Lock Placement and Large-Value Concern + +MPU (FSO and OBS) lock placement is not yet specified at the same detail as +key operations. Specific open questions: +- Does `CompleteMultiPartUpload` (which assembles N parts) need a multi-step + chain, or is it a single large transition? +- The large-value concern (HDDS-8238): a completed multi-GB MPU object's + `OmKeyInfo` with all block locations is replicated as raw bytes in the + Ratis log. For objects with thousands of blocks, this can be tens of KB per + log entry. Is this acceptable, or does the design need a block-list delta + optimization for MPU complete? + +This is addressed in Phase P-4 of the migration plan. + +### 10.4 hsync / Lease-Recovery Interaction + +`hsync` and lease-recovery (`RecoverLease`) interact with the slot lock +`X(parent, file)` that commit takes. Specific questions: +- Does `RecoverLease` need the same slot lock as commit? +- How does an in-progress hsync stream interact with the planned-execution + model where each `AllocateBlock` is a separate planned transition? +- Does the lease-recovery path need to read the open-key state and plan a + forced-commit transition? + +### 10.5 Snapshot Checkpoint Ordering vs In-Flight Operations + +`CreateSnapshot` emits a `CHECKPOINT` operation that triggers a snapshot +barrier. The open question: how does this interact with fine-grained locks +introduced in P-2? + +If multiple operations are in-flight (planned but not yet applied) when a +`CreateSnapshot` is submitted, the snapshot must capture a consistent point. +The barrier semantics (Checkpoint gets its own batch, applied in isolation) +handle this at the apply layer. But the planning-layer question is: must +`CreateSnapshot` acquire exclusive bucket locks to drain in-flight operations, +or is the Ratis log ordering sufficient to define the snapshot boundary? + +The current design relies on the Ratis log index as the snapshot boundary +(same as today's `splitReadyBufferAtCreateSnapshot`). This is likely +sufficient but needs validation against the fine-grained locking model. + +### 10.6 FSO Directory mtime on Cross-Parent Rename + +When renaming a file from directory A to directory B, both directories' +`mtime` should be updated. Under the current design, this is straightforward +(both parent directories are locked via `S(P1) + S(P2)` and the Batch can +include mtime updates for both). However, if mtime updates ever use a Merge +operator (to allow parallel mtime updates without serialization), the +interaction with the quota Merge operator needs specification. + +Current expectation: mtime is a last-writer-wins property, not a commutative +counter, so a standard PUT of the updated directory entry suffices. No Merge +needed for mtime. + +### 10.7 SetAcl / SetTimes / AllocateBlock Lock Placement + +These operations are expected to follow the pattern: +`S(bucket) + S(parent) + X(parent, name)` — shared bucket, shared container +on the parent, exclusive slot on the target entry. This matches the +deleteFile/commitFile pattern. + +`AllocateBlock` is a special case: it mutates the open-key entry (appending +a block). Since open keys include `clientID` in their key, concurrent +`AllocateBlock` calls from different clients write different rows and do not +conflict. Same-client `AllocateBlock` calls are serialized by the client +(one at a time). So `AllocateBlock` likely needs only `S(bucket)` — same as +`createKey/createFile`. + +### 10.8 Pre-Ratis Batching + +The current design sends each planned request to Ratis individually. Ratis's +built-in log-replication batching (AppendEntries groups multiple entries in +one RPC) provides baseline throughput. + +A future optimization: accumulate multiple independently-planned transitions +in a queue and submit them as a single `BatchedStateTransitions` Ratis entry. +This amortizes the per-entry Ratis overhead (fsync, quorum ack) across +multiple requests. The proto already supports this via +`BatchedStateTransitions { repeated ReplicatedStateTransition }`. + +Trade-offs: +- Adds queue management and flush-interval complexity. +- The first item in the batch incurs the wait time for the batch to fill. +- One slow item does NOT block others (each plans independently; only the + Ratis submit is batched). +- Estimated throughput gain: 2-4x for small operations under high load. + +This is a post-P-1 optimization. The design supports it without structural +changes. + +--- + +## Appendix: File Layout + +``` +hadoop-hdds/framework/src/main/proto/ + ReplicatedDbProtocol.proto - Inner Batch/Operation proto (domain-agnostic) + +hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/replicated/ + OperationApplier.java - Domain-agnostic apply engine + MergeOperatorRegistry.java - Registry for merge resolvers + MergeOperator.java - Interface for merge resolvers + +hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ + execution/ + ManagedIndexService.java - Monotonic index management + LeaderPlanner.java - Orchestrator (drives step-iterator) + ChangeRecorder.java - Accumulates Batch per step + lock/ + StripedSemaphoreLockManager.java - Non-thread-affine striped RW locks + LockHandle.java - AutoCloseable, any-thread release + LockRequest.java - (stripeIndex, mode) pair + execution/request/ + PlannedRequest.java - Abstract base (step-iterator) + key/ + CreateKeyPlannedRequest.java - OBS CreateKey + CommitKeyPlannedRequest.java - OBS CommitKey + CreateFilePlannedRequest.java - FSO CreateFile (multi-step) + CreateDirectoryPlannedRequest.java - FSO CreateDirectory (multi-step) + execution/merge/ + QuotaMergeOperator.java - Commutative quota delta resolver + ratis/ + OzoneManagerStateMachine.java - Dual-path routing (modified) + +hadoop-ozone/interface-client/src/main/proto/ + OmClientProtocol.proto - Outer OM envelope (ReplicatedStateTransition) +``` diff --git a/hadoop-hdds/docs/content/design/leader-execution/components.md b/hadoop-hdds/docs/content/design/leader-execution/components.md new file mode 100644 index 000000000000..80871c1f1e16 --- /dev/null +++ b/hadoop-hdds/docs/content/design/leader-execution/components.md @@ -0,0 +1,284 @@ +--- +title: Leader-Planned Execution — Components +summary: ManagedIndex, LeaderPlanner, ChangeRecorder, ApplyEngine, and State Machine integration +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +author: Abhishek Pal +--- + + +# Components + +This document describes the core components that implement leader-planned execution. For the high-level architecture and motivation, see the [overview](./overview.md). + +## Table of Contents + +1. [Managed Index Service](#1-managed-index-service) +2. [Leader Planning Framework](#2-leader-planning-framework) +3. [Apply Engine](#3-apply-engine) +4. [State Machine Integration](#4-state-machine-integration) + +--- + +## 1. Managed Index Service + +**Problem:** With the current architecture, `objectID` for new keys/volumes comes from the Ratis `transactionLogIndex`. In the new model, planning happens in `startTransaction()` (before the log entry is written), so the Ratis index is not yet available. + +**Why must objectId be decided during planning?** + +The `objectID` is part of the value that gets serialized into the DB delta. For example, when we create a new key: +1. We build an `OmKeyInfo` object — this object contains the `objectID` field +2. We serialize this `OmKeyInfo` to raw bytes via the codec +3. These raw bytes go into an `Operation` message in the `Batch` proto + +The raw bytes are frozen at planning time. They get written as-is to RocksDB during apply. We cannot "fill in" the objectId later during apply because the bytes are already serialized and embedded in the Ratis log entry. + +In the current architecture, this is not a problem: `validateAndUpdateCache` runs during `applyTransaction` where the Ratis log index is already known. So it uses `transactionLogIndex` directly as the objectId. + +In the new architecture, planning runs in `startTransaction` which happens **before** Ratis assigns a log index. So we need our own counter. + +**Solution:** An OM-managed `AtomicLong` counter (`ManagedIndexService`) that: +- Returns a unique, always-increasing value via `getAndIncrement()`. +- Is saved together with each DB batch (stored in `TransactionInfoTable` under key `#MANAGED_INDEX`). +- Advances its in-memory floor on every apply (leader, follower, and replay) to survive leader switchover. +- Is recovered on restart from the saved value. + +```java +public final class ManagedIndexService { + private final AtomicLong currentIndex; + + public long next() { return currentIndex.getAndIncrement(); } + + // Called by applyTransaction on EVERY node (leader, follower, and during + // log replay). Advances the in-memory counter past the applied value. + public void advanceFloor(long appliedManagedIndex) { + currentIndex.updateAndGet(c -> Math.max(c, appliedManagedIndex + 1)); + } + + public void seedFromFinalization(long ratisIndexAtFinalize) { + currentIndex.updateAndGet(c -> Math.max(c, ratisIndexAtFinalize + 1)); + } + + public void persist(BatchOperation batch) { + // write #MANAGED_INDEX into the same atomic batch as the data patch + } +} +``` + +#### Leader switchover and the allocation-to-persistence gap + +Managed indices are allocated in `startTransaction` (before the Ratis entry is committed) and persisted in `applyTransaction` (after commit). Between allocation and persistence, the index exists only in the Ratis log — it is not yet in RocksDB or in the in-memory counter on other nodes. + +**The problem this creates on leader change:** +``` +Old leader: allocates managed IDs 1, 2, 3, 4, 5 +Ratis committed by quorum: entries carrying IDs 1..4 +New leader (was follower): has only applied IDs 1..2 so far + +New leader elected, replay begins: + - Committed entries 3, 4 are replayed via applyTransaction + - Each replay calls advanceFloor(3), advanceFloor(4) + - In-memory counter is now 5 + - Entry for ID 5 was never committed → lost on leader change + - New leader's first allocation → next() returns 5 → no conflict +``` + +The key invariant: `advanceFloor` is called in `applyTransaction` on every node, including during log replay. This guarantees the in-memory counter is always past the highest committed managed index before the new leader accepts its first `startTransaction`. Ratis itself guarantees that all committed entries are replayed before a new leader processes new proposals. + +Uncommitted entries (allocated on the old leader but never quorum-ack'd) are dropped from the log on leader change. They are never replayed anywhere, so the new leader can safely allocate those same values. + +**Mixed-mode objectID safety:** During the rolling-upgrade window, both the +legacy path and the new path must draw objectIDs from this single counter. +The legacy `OzoneManager.getObjectIdFromTxId(trxnId)` is retrofitted to call +`ManagedIndexService.next()` instead of using the raw Ratis transaction index. +This ensures old-path and new-path objectID ranges are disjoint by +construction. The objectID encoding format (`(epoch<<62)|(index<<8)`) is +unchanged — only the source of `index` changes. The low 8 bits go dead-zero +(the 256-wide recursive-directory window is retired for migrated commands). + +**Persistence:** The counter's durable value is written into the same +`BatchOperation` as the data patch and the `#TRANSACTIONINFO` advance. The +triple (patch + transaction info + managed index) lands atomically or not at +all. + +--- + +## 2. Leader Planning Framework + +Three classes form the planning framework: + +### 2.1 PlannedRequest (abstract base with step-iterator) + +```java +public abstract class PlannedRequest { + OMRequest preProcess(OzoneManager om); // validation, normalization, auth-prep + List stepLocks(StepContext ctx); // which stripes this step needs + boolean hasNextStep(); // dynamic iterator (single-step = one step) + StepPlan nextStep(); // returns the step to execute + void advance(CommitResult committed); // consume commit, possibly re-plan next step +} +``` + +The step-iterator model generalizes from "one request → one transition" to +"one request → an ordered chain of transitions." A single-step operation is +the degenerate N=1 case. Multi-step operations (FSO createFile with missing +parents, recursive delete) produce multiple steps dynamically — each step is +planned only after the previous step commits, because the committed state may +affect the next step. + +**Why dynamic, not pre-computed:** A concurrent `rm -rf` can invalidate a +statically-planned chain. The planner must re-resolve per step, under the +lock, and revalidate that the target node still exists. + +### 2.2 ChangeRecorder (request-scoped, replaces TransitionBuilder) + +```java +public final class ChangeRecorder { + void put(String columnFamily, byte[] key, byte[] value); + void delete(String columnFamily, byte[] key); + void merge(String columnFamily, byte[] key, String operatorId, byte[] operand); + void checkpoint(long index); + Batch toBatch(); +} +``` + +The `ChangeRecorder` accumulates a `Batch` (from the inner proto layer) +instead of staging changes into the OM table cache. This is the structural +break from the legacy path: `plan()` records operations into a `Batch`, and +reads go straight to RocksDB. No table cache interaction. + +**How raw bytes are computed:** Ozone already has `Codec` classes for every +value type stored in RocksDB. The recorder uses these codecs to serialize +domain objects to bytes at planning time — the same serialization that +`TypedTable.put()` uses today. + +**How raw bytes flow through the system:** + +1. During planning on the leader: the `ChangeRecorder` uses existing `Codec` + classes (e.g., `OmKeyInfo.getCodec()`) to serialize domain objects to the + exact `byte[]` that would be stored in RocksDB — the same serialization + `TypedTable.put()` uses today. +2. These raw bytes go into `Operation` messages in the `Batch` proto. +3. The `Batch` is embedded in the Ratis log entry. +4. At apply time on all nodes: the `OperationApplier` writes the raw bytes + directly to RocksDB column families — no deserialization needed. + +### 2.3 LeaderPlanner (orchestrator) + +The `LeaderPlanner` drives the request's step-iterator to completion. For +single-step operations, this is one iteration. For multi-step operations (FSO +createFile with missing parents), it loops until the iterator is exhausted. + +```java +public final class LeaderPlanner { + CompletableFuture execute(PlannedRequest request) { + request.preProcess(om); + return driveSteps(request); + } + + private CompletableFuture driveSteps(PlannedRequest req) { + if (!req.hasNextStep()) return completedFuture(req.buildResponse()); + + StepPlan step = req.nextStep(); + List locks = req.stepLocks(step.context()); + LockHandle handle = lockManager.acquire(sorted(locks)); + // Revalidate locked nodes (close resolve-to-lock window) + step.revalidate(om); + long index = indexService.next(); + Batch batch = step.plan(new ChangeRecorder(), om); + // Submit to Ratis (lock still held under Option B) + return submitToRatis(batch, index).thenCompose(commit -> { + handle.release(); // released on continuation thread (non-thread-affine) + req.advance(commit); + return driveSteps(req); // next step, if any + }); + } +} +``` + +**Continuation-driven:** The worker thread is freed during each Ratis await. +The lock is held across the await (under Option B) and released on the +continuation thread — this is why the lock primitive must be non-thread-affine. +Under Option A, the lock is released before `submitToRatis`. + +--- + +## 3. Apply Engine + +The `OperationApplier` (in `hadoop-hdds/framework`) writes a `Batch` to +RocksDB. It is domain-agnostic — it imports no Ozone OM types. + +```java +public final class OperationApplier { + void apply(Batch batch, DBStore store, BatchOperation rocksBatch, + MergeOperatorRegistry registry) { + for (Operation op : batch.getOpsList()) { + switch (op.getKind()) { + case PUT: + store.getTable(op.getColumnFamily()) + .putWithBatch(rocksBatch, op.getKey(), op.getValue()); + break; + case DELETE: + store.getTable(op.getColumnFamily()) + .deleteWithBatch(rocksBatch, op.getKey()); + break; + case MERGE: + MergeOperator resolver = registry.get(op.getMergeOperatorId()); + byte[] current = store.getTable(op.getColumnFamily()).get(op.getKey()); + byte[] merged = resolver.apply(current, op.getValue()); + store.getTable(op.getColumnFamily()) + .putWithBatch(rocksBatch, op.getKey(), merged); + break; + case CHECKPOINT: + // flush current batch, take RocksDB checkpoint at this index + break; + } + } + } +} +``` + +The OM-level apply wrapper handles: +- Splitting at snapshot barriers (`CHECKPOINT` operations get their own batch) +- Writing `#TRANSACTIONINFO` and `#MANAGED_INDEX` in the same atomic batch +- Committing the batch + +**Barrier detection:** `CreateSnapshot` emits a `CHECKPOINT` operation. The +apply wrapper detects this and applies it in its own separate RocksDB batch, +preserving the same semantics as `splitReadyBufferAtCreateSnapshot()`. + +**Failure handling:** A follower that cannot apply a committed patch must crash +and re-sync (fail-stop). The state machine already has this discipline for +`INTERNAL_ERROR` / `METADATA_ERROR`. The planned-apply path inherits it. + +--- + +## 4. State Machine Integration + +The `OzoneManagerStateMachine` runs a **dual-path** model: + +- **Planned commands** (registered via `registerPlannedCommand(Type, creator)`): + - In `startTransaction()`: call `LeaderPlanner.plan()`, embed the result as + `BatchedStateTransitions` in the `OMRequest`, set as log data. + - In `applyTransaction()`: detect `hasBatchedStateTransitions()`, route to + `StateTransitionApplyEngine.applyBatch()`, return pre-computed `OMResponse`. + +- **Legacy commands** (everything not registered): existing `runCommand()` path + via `validateAndUpdateCache()` + `OzoneManagerDoubleBuffer`. + +This dual-path allows step-by-step migration: commands are moved one at a time. +See the [migration playbook](./migration-playbook.md) for the phased migration +plan. diff --git a/hadoop-hdds/docs/content/design/leader-execution/fso-design.md b/hadoop-hdds/docs/content/design/leader-execution/fso-design.md new file mode 100644 index 000000000000..710d7be9e7fd --- /dev/null +++ b/hadoop-hdds/docs/content/design/leader-execution/fso-design.md @@ -0,0 +1,180 @@ +--- +title: Leader-Planned Execution — FSO Design +summary: FSO operations under leader-planned execution — multi-step chains, recursive delete, revalidation, and recovery +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +author: Abhishek Pal +--- + + +# FSO Design + +This document describes how File System Optimized (FSO) bucket-layout +operations work under leader-planned execution. FSO operations are the hardest +to migrate because they involve multi-step chains, recursive deletes, and +directory-tree consistency. For the general locking model, see +[locking and concurrency](./locking-and-concurrency.md). For the high-level +architecture, see the [overview](./overview.md). + +## Table of Contents + +1. [FSO Lock Matrix](#1-fso-lock-matrix) +2. [Concurrent Scenarios](#2-concurrent-scenarios) +3. [Multi-Step Operations](#3-multi-step-operations) +4. [Revalidation After Lock Acquisition](#4-revalidation-after-lock-acquisition) +5. [Recovery](#5-recovery) +6. [FSO Operation Inventory](#6-fso-operation-inventory) + +--- + +## 1. FSO Lock Matrix + +FSO operations use a hierarchical directory namespace. Container locks protect +the parent-child relationship (what children exist under a directory). Slot +locks protect the existence of individual entries. + +| Operation | Bucket | Container | Slot | Notes | +|:----------|:-------|:----------|:-----|:------| +| createFile | S | S(P) | — | open file carries `clientID`; same as OBS | +| createDir | S | S(P) | X(P, dirName) | dir name has no `clientID` so uniqueness needed | +| commitFile | S | S(P) | X(P, fileName) | | +| deleteFile | S | S(P) | X(P, fileName) | | +| deleteDir (empty) | S | S(P) | X(P, dirName) + **X(D)** | X(D) blocks new children during empty-check | +| rename | S | S(P1) + S(P2) | X(P1, srcName) + X(P2, dstName) | no X on the moved node — children untouched | +| recursive delete | S | S(P) | X(P, name) + X(root) | All descendant operations blocked | + +`P` = immediate parent directory; `D` = the directory being deleted; +`P1/P2` = source/dest parent of a rename. + +**Key design choices:** + +- **createFile takes no slot lock** (only `S(parent)`): the open file table + key includes `clientID`, so concurrent creates of the same filename by + different clients write distinct rows. Uniqueness is resolved at commit. +- **rename takes no container lock on the moved node.** FSO rename is O(1) — + children key on the moved node's unchanged objectID and are untouched. + A concurrent create under the moved directory does not conflict with rename. +- **deleteDir takes X(D) (exclusive container lock on the directory being + deleted).** This blocks any new child operations from starting while the + empty-check runs. + +--- + +## 2. Concurrent Scenarios + +### Scenario D: FSO createFile with missing parents vs concurrent rm-rf + +``` +Thread A: createFile /vol/bucket/a/b/c/file Thread B: rm -rf /vol/bucket/a + resolve path: a exists (objectID=10), acquireLock: S(bucket), S(parent-of-a), + b exists (objectID=20), c missing X(parent-of-a, "a"), X(a-container) + Step 1: create dir c under b plan(): tombstone dir a → deletedDirTable + acquireLock: S(bucket), S(b-container=20) releaseLock, submit to Ratis + revalidate: b still exists? YES ... Ratis commits, a is tombstoned ... + plan(): PUT directoryTable c + releaseLock, submit to Ratis + ... Ratis commits dir c ... + Step 2: create open-file under c + acquireLock: S(bucket), S(c-container=30) + revalidate: c still exists? YES (just created) + plan(): PUT openKeyTable + releaseLock, submit to Ratis +``` + +**What happens:** Thread A's step 1 succeeds because it acquired locks before Thread B's tombstone was visible. After Thread B's tombstone commits, the background purge will eventually reach dir `b` and dir `c`. When it tries to purge `c`, it finds the open file and cannot remove `c` until that file is committed/aborted and cleaned up (per the no-orphan guarantee). + +**If Thread A starts step 1 AFTER the tombstone commits:** Path resolution of `/vol/bucket/a/b/c/file` traverses `a`. Since `a` is tombstoned, resolution fails with `DIRECTORY_NOT_FOUND`. Thread A's request fails immediately. + +This is linearizable: either Thread A's operation logically preceded the delete (and completed), or it came after (and failed). + +### Scenario E: FSO two crossing renames + +``` +Thread A: rename /vol/bucket/a/x → /vol/bucket/b/y +Thread B: rename /vol/bucket/b/p → /vol/bucket/a/q +``` + +Both operations need slot locks in directories `a` and `b`. The deadlock +avoidance total order (sort by objectID) ensures both threads acquire locks in +the same order. If `objectID(a) < objectID(b)`, both acquire `a`'s locks first, +then `b`'s. No deadlock is possible. + +--- + +## 3. Multi-Step Operations + +Two operation types are not single transitions. The leader orchestrates them as +an ordered chain of dependent Ratis transitions, each with its own per-step +lock acquisition and release. + +### 3.1 Implicit directory creation (createFile with missing parents) + +`createFile /a/b/c/file` with `/a/b/c` missing is decomposed, OM-internally, +into: `create b` → await commit → `create c` → await commit → `create +open-file`. The client sends one request with the full path; the OM orchestrates +the sub-creates internally. Each sub-create needs the previous one's committed +objectID to resolve the next parent. + +**Dynamic step-iterator, not pre-computed chain.** The planner pulls one step at a time from the request's iterator, acquires that step's locks, revalidates the locked nodes by `(parentObjectID, name)` (confirming they still exist with the expected objectID), plans that step, submits to Ratis, and on commit releases the locks and advances the iterator. The next step may differ from what a static pre-computation would have produced, because a concurrent operation could have changed the state. + +**Non-atomic (accepted).** A failure after creating `b` and `c` leaves empty directories. This matches `mkdir -p` semantics, is idempotent on retry (create dir on an existing dir is a no-op), and is low-harm. + +**ObjectID window retirement.** Because each created object is now its own Ratis transition, each gets exactly one managed index. The 256-wide recursive-dir objectID window (`(epoch<<62)|(txId<<8)|offset`) is retired for migrated commands. + +**Asynchronous orchestration.** The leader registers a continuation on each sub-operation's commit future and frees the worker thread during the Ratis await. It holds the client RPC open, not a worker thread. + +### 3.2 Recursive directory delete + +`rm -rf /a/b` is handled in two phases: + +1. **Synchronous root tombstone:** The root directory `/a/b` is tombstoned immediately (moved to `deletedDirTable`). This makes any new path resolution that traverses `/a/b` fail with `DIRECTORY_NOT_FOUND` immediately — providing fast UX and preventing new descendant operations from starting. + +2. **Asynchronous per-node purge:** A redesigned `DirectoryDeletingService` walks the subtree in the background. Each node removal takes that node's `X(container)` lock + its slot lock, enumerates current children under the lock, and removes the node only when childless. This guarantees no orphan: a late child (from an in-flight create that resolved before the tombstone) is processed before the parent node is removed. + +**Quota release is lazy:** Subtree bytes and namespace are freed as the purge drains each node, not synchronously with the root tombstone. This is eventually consistent. + +--- + +## 4. Revalidation After Lock Acquisition + +After acquiring its locks, every operation re-reads each locked node by `(parentObjectID, name)` using the parent objectIDs captured during path resolution. It confirms the node still exists with the expected objectID before mutating. This closes the window between path resolution and lock acquisition where a concurrent delete could have removed the node. ABA-safe because objectIDs are never reused. + +--- + +## 5. Recovery + +Recovery is client-retry-driven, with no persisted orchestration or saga state. A mid-orchestration leader crash leaves committed sub-steps durable on the quorum and the client's RPC unanswered. The failover retry re-runs the whole request, idempotently skipping already-created directories and completing. The retry-cache entry (`clientId#callId → response`) is written by the terminal step only; intermediate sub-steps are idempotent by structure. + +--- + +## 6. FSO Operation Inventory + +FSO operations in the migration plan (from the +[full operation inventory](./migration-playbook.md#3-full-operation-inventory)): + +| Operation | Tables | Quota? | SCM? | Multi-step? | Phase | +|-----------|--------|--------|------|-------------|-------| +| CreateKey (FSO) | openKeyTable, directoryTable | usage | yes | implicit parents | P-2 | +| CommitKey (FSO) | keyTable, openKeyTable, deletedTable | usage | no | no | P-2 | +| AllocateBlock (FSO) | openKeyTable | no | yes | no | P-2 | +| DeleteKey (FSO) | keyTable, deletedTable | usage | no | recursive | P-2 | +| CreateFile | openKeyTable, directoryTable | usage | yes | yes (mkdir-p) | P-2 | +| CreateDirectory | directoryTable | usage | no | yes (mkdir-p) | P-2 | +| PurgeDirectories | directoryTable, fileTable, deletedDirTable | usage (lazy) | no | yes (background) | P-2 | + +All are migrated in Phase P-2 — the hardest multi-step operations. This phase +retires the showstopper risks (multi-step chains, recursive delete, revalidation +under concurrent modifications). diff --git a/hadoop-hdds/docs/content/design/leader-execution/locking-and-concurrency.md b/hadoop-hdds/docs/content/design/leader-execution/locking-and-concurrency.md new file mode 100644 index 000000000000..99cdad0a1d3f --- /dev/null +++ b/hadoop-hdds/docs/content/design/leader-execution/locking-and-concurrency.md @@ -0,0 +1,400 @@ +--- +title: Leader-Planned Execution — Locking and Concurrency +summary: Granular locking model, lock manager, hold span decision, life of a request, limitations, and retry strategy +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +author: Abhishek Pal +--- + + +# Locking and Concurrency + +This document describes the fine-grained locking model that underpins +leader-planned execution. For operation-specific lock usage, see the +[OBS design](./obs-design.md) and [FSO design](./fso-design.md) documents. +For the high-level architecture, see the [overview](./overview.md). + +## Table of Contents + +1. [Granular Locking](#1-granular-locking) +2. [Lock Hold Span (Decision Required)](#2-lock-hold-span-decision-required) +3. [Life of a Request](#3-life-of-a-request) +4. [Lock Granularity Summary](#4-lock-granularity-summary) +5. [Known Limitations and Accepted Tradeoffs](#5-known-limitations-and-accepted-tradeoffs) +6. [Retry and Idempotency (Deferred Decision)](#6-retry-and-idempotency-deferred-decision) + +--- + +## 1. Granular Locking + +A major benefit of leader-only execution is that locks only need to protect +the planning phase on the leader. This allows fine-grained, objectID-keyed +locking that is rename-stable and ABA-free. + +### 1.1 The lock model: containers and slots + +Every namespace node plays two roles, each with its own lock namespace: + +- **Container lock** — keyed by a directory's **objectID**. Governs "what + children exist under me." Modes: **S** (shared) = "I am adding/touching one + child"; **X** (exclusive) = "I am emptying/removing this directory." +- **Slot lock** — keyed by **(parentObjectID, name)** (the DB key minus the + vol/bucket prefix). Governs "the existence/identity of this one entry." + Taken **X** by whoever creates/commits/deletes/renames that specific entry. + +The **bucket** is the root container. It additionally carries bucket-level +properties (quota limits, default ACLs, replication, encryption) that every +key operation reads, so the bucket lock is taken **S** by every key operation +and **X** by `SetBucketProperty`/`DeleteBucket`. + +**Why objectID-keyed, not path-keyed:** A directory rename is O(1) in FSO — +the directory keeps its objectID; only its own entry changes parent. Children +are never touched because they key on the parent's (unchanged) objectID. A +path-keyed lock would be invalidated by rename; an objectID-keyed lock +survives it. ObjectIDs are never reused (monotonic from ManagedIndex), so +there is no ABA problem. + +### 1.2 Per-operation lock matrix + +`P` = immediate parent directory; `D` = the directory being deleted; +`P1/P2` = source/dest parent of a rename. All locks are acquired in the +total order defined in [Section 1.4](#14-deadlock-avoidance). + +| Operation | Bucket | Container | Slot | Notes | +|:----------|:-------|:----------|:-----|:------| +| createKey (OBS) | S | — | — | open key carries `clientID`; creates never collide | +| createFile (FSO) | S | S(P) | — | open file carries `clientID`; same as OBS | +| createDir (FSO) | S | S(P) | X(P, dirName) | dir name has no `clientID` so uniqueness needed | +| commit (OBS) | S | — | X(bucket, key) | two commits of same key serialize | +| commit (FSO) | S | S(P) | X(P, fileName) | | +| deleteFile | S | S(P) | X(P, fileName) | | +| deleteDir (empty) | S | S(P) | X(P, dirName) + **X(D)** | X(D) blocks new children during the empty-check | +| rename | S | S(P1) + S(P2) | X(P1, srcName) + X(P2, dstName) | no X on the moved node — children untouched | +| SetBucketProperty / DeleteBucket | X | — | — | exclusive bucket lock | + +**Key design choices:** +- **createKey/createFile take no slot lock** (only `S(parent)`): the open + key/file table key includes `clientID`, so concurrent creates of the same + name by different clients write distinct rows and cannot collide. Same-name + resolution is deferred to commit where `X(parent, name)` is taken. +- **rename takes no container lock on the moved node.** Children key on the + moved node's unchanged objectID and are untouched, so a concurrent create + under the moved directory does not conflict with the rename. + +### 1.3 Lock manager implementation + +The existing `OzoneManagerLock` is not reused for this design. It is built on +`ReentrantReadWriteLock` which is thread-affine (cannot be released on a +different thread than acquired). Under Option B of the lock hold span (see +[Section 2](#2-lock-hold-span-decision-required)), the lock must be held +across the Ratis await and released on the continuation thread. + +The new lock manager is a **fixed striped array** of non-thread-affine RW +primitives: + +- **Striped, not per-key.** A key hashes to a stripe index. With a + generously-sized array (~2^20 stripes, ~64 MB), false-contention collisions + are in the low tens at peak and cost only latency, never correctness. +- **Non-thread-affine primitive:** a fair counting `Semaphore` per stripe used + as an RW lock — **S** = acquire 1 permit, **X** = acquire N permits (N = the + per-stripe permit ceiling). Releasable on any thread; fairness prevents + writer starvation. +- **Handle-owned release.** `acquire(sortedReqs)` returns a `LockHandle` + (`AutoCloseable`); the operation releases it in its completion path, possibly + on the continuation thread. +- **No lock timeout.** The timeout lives on the Ratis request (existing). When + the Ratis operation times out or errors, the holder fails and releases the + lock in its `finally`. A genuinely hung OM is handled by failover (locks are + leader-local and discarded on leader change), not by a lock timer. + +```java +public final class StripedSemaphoreLockManager { + private final Semaphore[] stripes; // ~2^20 entries, fair + private final int permitsPerStripe; // N; X drains all, S takes 1 + + public LockHandle acquire(List sortedDedupedReqs) { + // reqs pre-sorted by stripe index; deduped to strongest mode + for (LockRequest req : sortedDedupedReqs) { + stripes[req.stripeIndex()].acquire(req.isExclusive() ? permitsPerStripe : 1); + } + return new LockHandle(sortedDedupedReqs); + } +} + +public interface LockHandle extends AutoCloseable { + void release(); // any thread; reverse order; idempotent +} +``` + +### 1.4 Deadlock avoidance + +All locks for an operation are sorted by a single total order and acquired in +that order; released in reverse. The comparator over lock identities: + +1. By the lock's primary ID — `objectID` for a container, `parentObjectID` for + a slot. The bucket (lowest objectID) sorts first. +2. Tiebreak: container-before-slot when IDs coincide. +3. Tiebreak: slots by `name`. + +This is deadlock-free purely by uniform total ordering — not by an +ancestor-first rule, which rename can invert. + +--- + +## 2. Lock Hold Span (Decision Required) + +There are two valid approaches to when locks are released. **This is a +performance-vs-correctness tradeoff that needs a decision.** + +**Option A — Release before Ratis submit:** +``` +acquireLock → plan() → releaseLock → submit to Ratis → ... → applyTransaction +``` +The lock protects only the planning phase. After planning, the lock is released +and Ratis replication happens without holding any lock. + +**Option B — Hold through Ratis commit and apply:** +``` +acquireLock → plan() → submit to Ratis → ... → applyTransaction → releaseLock +``` +The lock is held until the data is durable in RocksDB. A second request on the +same key blocks until the first request's data is visible in the DB. + +**Trade-offs:** + +| Aspect | Option A (release before) | Option B (hold through apply) | +|--------|--------------------------|-------------------------------| +| Parallelism | Higher — lock held only during CPU work (~microseconds) | Lower — lock held during Ratis round-trip (~2-5ms) | +| Read-your-writes | Requires a cache or conditional ordering for the plan-to-apply gap | Guaranteed — successor reads committed state directly from RocksDB | +| Table cache removal | Cannot fully remove the table cache; reads during the gap may see stale data | Table cache can be fully removed; reads always go to RocksDB | +| Lock primitive | Standard `ReadWriteLock` works (thread-affine OK) | Needs non-thread-affine semaphore (release on continuation thread) | +| Multi-step FSO ops | Each step must handle the gap; createFile chain needs predecessor's objectID from DB but it may not be applied yet | Each step's commit is visible before the next step locks; chain "just works" | +| Complexity | Simpler lock primitive; more complex gap-handling logic | More complex lock primitive; simpler correctness reasoning | +| Quota correctness | Parallel creates can drift (read same base, both increment by 1, final = base+1 not base+2) | No drift — each successor reads committed predecessor state | + +**Arguments for Option A:** +- CockroachDB uses this model (latches released before Raft propose). +- Higher throughput for independent operations. +- Standard Java `ReentrantReadWriteLock` suffices. + +**Arguments for Option B:** +- Removes the table cache entirely (a known source of OM bugs). Read-your-writes is structural, not cache-dependent. +- Multi-step FSO operations (createFile with missing parents) are trivially correct — each step sees the previous step's committed state. +- Eliminates an entire class of plan-to-apply gap bugs. +- The non-thread-affine semaphore is a one-time implementation cost. +- Locks are held only during Ratis round-trip (~2-5ms), not during network I/O to external services. The throughput impact is bounded. + +**Note on multi-step operations:** Under Option A, a `createFile /a/b/c/file` where `/a/b/c` is missing must create `/b` → submit → wait for apply → create `/c` → submit → wait for apply → create open-file. Each intermediate step must wait for its predecessor to be applied before the next step can read the committed objectID. Under Option B, the lock hold span itself provides this guarantee — the next step blocks on the lock until the predecessor's bytes are in RocksDB. + +**Recommendation:** Option B provides stronger guarantees for the same operations that are hardest to get right (multi-step FSO, recursive delete, quota). The non-thread-affine semaphore is a bounded implementation cost. The table cache removal enabled by Option B eliminates a known bug class. However, if throughput testing shows Option B is unacceptable for OBS workloads (which are single-step and do not need read-your-writes across requests), a hybrid approach is possible: Option B for FSO/multi-step commands, Option A for simple OBS commands. + +--- + +## 3. Life of a Request + +This section traces a single request through the planned-execution pipeline. + +``` +Thread A (CreateKey /vol/bucket/key1): + 1. preProcess() — validate key name format + 2. authorize() — check ACLs + 3. acquireLock() — acquire bucket READ lock for /vol/bucket + 4. plan() — read bucket info from DB, build OmKeyInfo, + write deltas to ChangeRecorder: + PUT openKeyTable /vol/bucket/key1/clientA → omKeyInfo bytes + PUT bucketTable /vol/bucket → updated bucket bytes (quota+1) + 5. releaseLock() — release bucket READ lock + 6. Submit ReplicatedStateTransition to Ratis + 7. Ratis replicates to quorum + 8. applyTransaction() on all nodes: + — open RocksDB WriteBatch + — write PUT openKeyTable /vol/bucket/key1/clientA → bytes + — write PUT bucketTable /vol/bucket → bytes + — commit batch atomically + 9. Return OMResponse to client +``` + +--- + +## 4. Lock Granularity Summary + +Restated in container/slot terminology (see [Section 1](#1-granular-locking)): + +| Operation | Bucket | Container | Slot | Conflicts with | +|-----------|--------|-----------|------|----------------| +| OBS CreateKey | S | — | — | DeleteBucket, SetBucketProperty | +| OBS CommitKey | S | — | X(bucket, key) | Same-key Commit/Delete, DeleteBucket | +| OBS DeleteKey | S | — | X(bucket, key) | Same-key Commit/Delete, DeleteBucket | +| FSO createFile | S | S(parent) | — | DeleteBucket (creates never conflict) | +| FSO createDir | S | S(parent) | X(parent, name) | Same-name createDir, DeleteBucket | +| FSO commitFile | S | S(parent) | X(parent, name) | Same-name operations, DeleteBucket | +| FSO deleteFile | S | S(parent) | X(parent, name) | Same-name operations, DeleteBucket | +| FSO deleteDir (empty) | S | S(parent) | X(parent, name) + X(dir) | Children creates, same-name ops | +| FSO rename | S | S(srcParent) + S(dstParent) | X(srcParent, src) + X(dstParent, dst) | Same-name ops at either end | +| FSO recursive delete | S | S(parent) | X(parent, name) + X(root) | All descendant operations | +| DeleteBucket | X | — | — | All key operations in this bucket | +| SetBucketProperty | X | — | — | All key operations in this bucket | +| CreateSnapshot | — | — | — | Barrier in apply (own batch) | + +--- + +## 5. Known Limitations and Accepted Tradeoffs + +These are deliberate design choices, not gaps. They are stated explicitly so +reviewers read them as accepted tradeoffs. + +**Soft quota limit enforcement (accepted):** + +Because key commits take only a shared bucket lock (so commits to different +keys run in parallel — the core throughput goal), and `usedBytes` is updated +by a commutative Merge operator rather than read-modify-written under an +exclusive lock, the quota *limit* is enforced best-effort, not exactly. Two +or more commits in flight can each pass the quota check against the same +pre-increment `usedBytes` and then both apply, transiently over-committing +the limit by up to the in-flight commit count. + +What IS guaranteed: +- The `usedBytes`/`usedNamespace` counter never loses an update — it always + equals the true committed size. No double-count, no lost decrement. +- Only the *limit gate* is soft. The counter itself is exact. + +Mitigation: +- The existing `QuotaRepair` background service reconciles any drift. +- A future leader-local atomic reservation (check-and-reserve in memory, + DB Merge remains the durable truth, decrement-on-abort, rebuild-from-DB + on failover) can enforce exact admission if needed. This is a separate + enhancement tracked under the open quota-enforcement decision. + +**Non-atomic multi-step createFile (accepted):** + +A failed multi-step `createFile` may leave empty intermediate directories. +This matches `mkdir -p` semantics, is idempotent on retry (create-dir on +an existing dir is a no-op), and is low-harm. Not cleaned eagerly. + +**Lazy quota release on recursive delete (accepted):** + +`rm -rf` releases quota as the background purge drains each node, not +synchronously with the root tombstone. The namespace root disappears +synchronously (fast UX) but `usedBytes`/`usedNamespace` converge over time +as nodes are purged. Synchronous release would require maintaining a +subtree-size aggregate on every create/commit/delete up the chain — a +separate enhancement if ever needed. + +**Eventual subtree purge (accepted):** + +Recursive-delete subtree reclamation is asynchronous. Descendants are +reclaimed over time by the `DirectoryDeletingService`. The root tombstone +provides immediate user-visible deletion semantics. + +**No lock timeout (by design):** + +Locks have no timer. The timeout that bounds a wait lives on the Ratis +request (existing). When the Ratis operation times out or errors, the holder +fails and releases the lock in its `finally`. A genuinely hung OM is handled +by failover (locks are leader-local and discarded on leader change), not by +a lock timer. A holder-lease that expired a held lock during an in-flight +Ratis operation would violate mutual exclusion. + +--- + +## 6. Retry and Idempotency (Deferred Decision) + +This section documents the retry/idempotency mechanism as a **deferred +decision** — not omitted by oversight. The framework reserves the seam for +it, and a reader should not mistake its absence for a gap. + +### 6.1 Why it is deferred + +The choice of retry mechanism depends on a per-operation classification that +does not yet exist: which operations are idempotent on re-execution (safe to +re-plan from scratch on failover retry) and which are not (would +double-apply). That classification gates the design. + +### 6.2 The two idempotency questions + +There are two different idempotency questions with opposite answers: + +1. **Is the DB patch idempotent?** Almost always YES. The patch is + whole-object Put/Delete over deterministic keys. Applying the same bytes + twice is a no-op. This is why a follower can crash-and-resync and re-apply + committed entries safely. + +2. **Is re-execution (re-planning from scratch) idempotent?** NO for any + operation that, during planning, consumes a non-idempotent external + resource or emits a commutative Merge. Specifically: + +| Category | Operations | Why non-idempotent on re-exec | +|----------|-----------|-------------------------------| +| SCM block allocation | CreateKey, AllocateBlock, CreateFile | Fresh `allocateBlock` returns new block IDs each call | +| Quota Merge | CommitKey, DeleteKey, CommitMPUPart, CompleteMPU, DeleteKeys | A re-planned `+delta` Merge double-counts | +| Table move | SnapshotMoveDeletedKeys, SnapshotMoveTableKeys | Re-moving already-moved keys double-counts reclaim | + +That is ~10 operations out of ~47 total. The naturally-idempotent pure +Put/Delete operations (all of Group A, plus renames) tolerate weaker +handling. + +### 6.3 Two candidate mechanisms (neither is chosen) + +**(a) Leader-local in-flight registry + durable replicated table:** +- `(clientId, callId) → CompletableFuture` in memory on the leader + deduplicates concurrent retries cheaply. +- `(clientId, callId) → response` table in RocksDB, written **atomically with + the data batch**, survives failover. This is the likely invariant for + non-idempotent operations. + +**(b) In-memory-only retry state:** +- Weaker — loses deduplication on failover. +- Tolerable only for naturally-idempotent operations. + +The atomic-with-data-batch property is the leading candidate for the ~10 +non-idempotent operations. The purely-idempotent operations (Group A) may +need nothing beyond their natural idempotency. + +### 6.4 What IS fixed (regardless of mechanism) + +- The retry-cache entry is written by the **terminal step only** of a + multi-step request. Intermediate sub-steps are idempotent by structure + (create-dir on an existing dir is a no-op) and need no entry. +- The name is `retryCache` (aligned with Ratis terminology). + +### 6.5 Phasing consequence + +Each phase that introduces a non-idempotent operation (P-1: SCM + quota, +P-3: snapshot moves, P-4: MPU, P-5: batch DeleteKeys) leaves the +terminal-step retry-cache write point as a seam. When the retry decision +lands, the durable atomic-with-batch entry is wired in at that seam for the +~10 non-idempotent operations without touching the idempotent ones. + +### 6.6 Items needing clarity before this decision can be taken + +- **Per-operation idempotency audit:** Each of the ~47 write operations must + be classified as idempotent or non-idempotent under re-execution. The + inventory in the [migration playbook](./migration-playbook.md) provides the + structural classification; the audit must verify edge cases (e.g., is a + re-planned DeleteKey that soft-deletes an already-deleted key truly safe, + or does it double-decrement quota?). +- **Retry entry lifetime and eviction:** How long does a durable retry entry + live? Is it scoped to the client session or time-bounded? The existing + Ratis retry cache uses configurable expiry — should the durable table + follow the same model? +- **Interaction with batching:** If multiple transitions are ever batched + into one Ratis entry, each carries its own `(clientId, callId)`. The retry + table must handle per-transition dedup within a batch, not just per-entry. +- **SCM block allocation on retry:** A re-planned CreateKey gets fresh block + IDs from SCM. The old block IDs (from the first plan attempt that was lost) + are now orphaned SCM allocations. How are they reclaimed? Today Ozone has + block-GC for open keys that expire — does the same path cover this, or does + it need explicit cleanup? diff --git a/hadoop-hdds/docs/content/design/leader-execution/migration-playbook.md b/hadoop-hdds/docs/content/design/leader-execution/migration-playbook.md new file mode 100644 index 000000000000..d25b3691d484 --- /dev/null +++ b/hadoop-hdds/docs/content/design/leader-execution/migration-playbook.md @@ -0,0 +1,226 @@ +--- +title: Leader-Planned Execution — Migration Playbook +summary: Phase-0 prerequisites, hard-first phasing, full operation inventory, per-command process, and ZDU compatibility +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +author: Abhishek Pal +--- + + +# Migration Playbook + +This document describes the phased migration strategy from the legacy +`validateAndUpdateCache` path to leader-planned execution. For the high-level +architecture, see the [overview](./overview.md). + +## Table of Contents + +1. [Phase-0 Prerequisites](#1-phase-0-prerequisites) +2. [Hard-First Phasing](#2-hard-first-phasing) +3. [Full Operation Inventory](#3-full-operation-inventory) +4. [Per-Command Migration Process](#4-per-command-migration-process) +5. [Zero-Downtime Upgrade (ZDU) Compatibility](#5-zero-downtime-upgrade-zdu-compatibility) + +--- + +## 1. Phase-0 Prerequisites + +Three prerequisites must land before any command migrates. They are inert +(zero behavior change) but gate every later phase. + +**Prerequisite 1 — Legacy to ManagedIndex objectID retrofit:** + +During the mixed-mode window, some commands execute on the legacy path +(objectID derived from Ratis log index) and others on the new path (objectID +from ManagedIndex). If the two counters are independent, a legacy operation and +a new operation can mint the same objectID — silent namespace corruption. + +Fix: **both** paths draw from the shared `ManagedIndexService` counter during +mixed mode. The legacy `getObjectIdFromTxId` call sites are redirected to +`ManagedIndexService.next()` while keeping the encoding identical (same +`(epoch<<62)|(index<<8)` format, low 8 bits go dead-zero once the 256-window +is retired). + +At first finalization the counter is seeded to `max(committed Ratis index) + 1`, ensuring the new-path range starts strictly above every Ratis-derived objectID ever issued — disjoint by construction. + +**Prerequisite 2 — Dual-path applied-index durability:** + +During mixed mode, both the legacy double-buffer path and the new patch-apply path advance the applied-index concurrently for different commands. A single, monotonic, crash-consistent notion of "last durably applied index" must be maintained that both writers advance correctly. Without this, a crash mid mixed-mode can replay a committed-but-unstamped patch (double-apply) or skip a stamped-but-unflushed legacy batch (lost write). + +**Prerequisite 3 — `OMLayoutFeature` finalization gate:** + +A new layout feature `LEADER_SIDE_EXECUTION` (next ordinal after `SNAPSHOT_DEFRAG`) gates the binary-level safety of the new path. Until finalization, the new proto envelope and `#MANAGED_INDEX` entry are additive and inert. A new leader must not replicate a patch an old follower cannot apply — finalization is the one-way, cluster-wide gate that ensures uniform binary support. + +**These three are interlocked:** Prerequisite 1 makes mixed-mode objectIDs safe; Prerequisite 2 makes mixed-mode crash recovery safe; Prerequisite 3 makes mixed-mode binaries safe. + +--- + +## 2. Hard-First Phasing + +Commands are migrated hardest-first. The rationale is risk: if the easy, +high-visibility commands ship first, the hard bits (multi-step FSO, recursive +delete, commutative quota, snapshot Checkpoint ordering) can stall, leaving the +cluster permanently in mixed mode with showstoppers unsolved. Hard-first +front-loads the work whose feasibility is in doubt so that if the design fails, +it fails early while the legacy path is still fully intact. + +| Phase | Scope | Depends on | +|-------|-------|-----------| +| **P-0** | Framework substrate (12 components) + 3 prerequisites. Inert. | — | +| **P-1** | Hardest single-step OBS: CreateKey, CommitKey, AllocateBlock, DeleteKey. Commutative quota and SCM allocation first bite here. | P-0 | +| **P-2** | Hardest multi-step FSO: CreateFile/CreateDirectory with implicit parents, FSO delete, recursive `rm -rf` + DirectoryDeletingService redesign. Showstoppers retired. | P-1 | +| **P-3** | Snapshot: CreateSnapshot (Checkpoint op), SnapshotPurge, snapshot moves. | P-2 | +| **P-4** | MPU: Initiate, CommitPart, Complete, Abort, AbortExpired. Large-value concern revisited. | P-3 | +| **P-5** | Batch/background: DeleteKeys, RenameKey/Keys, DeleteOpenKeys, PurgeKeys/Directories. | P-2 | +| **P-6** | Easy sweep: ACLs, tagging, SetTimes, secrets, tokens, tenant, snapshot properties, volume/bucket properties, prepare. | P-1 | +| **P-7** | Cleanup: remove double buffer and table cache, delete legacy path, finalize. | All | + +P-5 and P-6 depend on the early hard phases (not the other way around) so +the easy work cannot be used to declare premature victory. + +--- + +## 3. Full Operation Inventory + +All ~47 write operations grouped by migration difficulty: + +### Group C — migration-hard (17 operations, phases P-1 through P-4) + +| Operation | Tables | Quota? | SCM? | Multi-step? | Phase | +|-----------|--------|--------|------|-------------|-------| +| CreateKey (OBS/FSO) | openKeyTable, directoryTable | usage | yes | FSO: implicit parents | P-1/P-2 | +| CommitKey (OBS/FSO) | keyTable, openKeyTable, deletedTable | usage | no | no | P-1/P-2 | +| AllocateBlock (OBS/FSO) | openKeyTable | no | yes | no | P-1/P-2 | +| DeleteKey (OBS/FSO) | keyTable, deletedTable | usage | no | FSO recursive | P-1/P-2 | +| CreateFile (FSO) | openKeyTable, directoryTable | usage | yes | yes (mkdir-p) | P-2 | +| CreateDirectory (FSO) | directoryTable | usage | no | yes (mkdir-p) | P-2 | +| PurgeDirectories | directoryTable, fileTable, deletedDirTable | usage (lazy) | no | yes (background) | P-2 | +| CreateSnapshot | snapshotInfoTable + RocksDB Checkpoint | no | no | no (barrier) | P-3 | +| SnapshotPurge | snapshotInfoTable | no | no | no | P-3 | +| SnapshotMoveDeletedKeys | deletedTable across snapshots | usage | no | no | P-3 | +| SnapshotMoveTableKeys | deletedTable across snapshots | usage | no | no | P-3 | +| InitiateMultiPartUpload | openKeyTable, multipartInfoTable | no | no | no | P-4 | +| CommitMultiPartUpload | multipartInfoTable, openKeyTable, deletedTable | usage | no | no | P-4 | +| CompleteMultiPartUpload | keyTable, openKeyTable, multipartInfoTable | usage | no | yes (assembly) | P-4 | +| AbortMultiPartUpload | openKeyTable, multipartInfoTable, deletedTable | usage | no | no | P-4 | +| AbortExpiredMultiPartUploads | openKeyTable, multipartInfoTable, deletedTable | usage | no | yes (batch) | P-4 | + +### Group B — migration-core (8 operations, phases P-1/P-5) + +| Operation | Tables | Quota? | Multi-step? | Phase | +|-----------|--------|--------|-------------|-------| +| DeleteKeys (batch) | keyTable, deletedTable | usage | yes (N keys) | P-5 | +| RenameKey | keyTable / directoryTable | no | no | P-5 | +| RenameKeys (batch) | keyTable | no | yes (N renames) | P-5 | +| DeleteOpenKeys | openKeyTable, deletedTable | usage | yes (batch) | P-5 | +| PurgeKeys | deletedTable | no | yes (batch) | P-5 | +| CreateBucket | bucketTable, volumeTable | usage (volume) | no | P-1 | +| DeleteBucket | bucketTable, volumeTable | usage (volume) | no | P-1 | +| SetBucketProperty | bucketTable | limit only | no | P-6 | + +### Group A — migration-simple (~22 operations, phase P-6) + +All share the shape: single column family, no quota usage, no SCM, single +transition, whole-row Put/Delete, idempotent on re-execution. + +Includes: CreateVolume, SetVolumeProperty, DeleteVolume, all ACL operations +(Add/Remove/Set for volume/bucket/key/prefix), delegation token operations, +S3 secret operations, tenant operations, SetTimes, PutObjectTagging, +DeleteObjectTagging, RenameSnapshot, SetSnapshotProperty, Prepare, +CancelPrepare, FinalizeUpgrade, RecoverLease, QuotaRepair. + +--- + +## 4. Per-Command Migration Process + +Each command migration is a self-contained, independently revertible unit: + +1. **One `PlannedRequest` subclass** — re-expresses the legacy + `validateAndUpdateCache` body as plan-on-leader. The legacy subclass stays + in the tree untouched as the fallback. +2. **A per-command runtime flag** — one config key (default: legacy/off), + wired into the router that chooses legacy vs planned per command type. + Default-off means landing the subclass changes nothing observable until an + operator opts in. +3. **Tests** — linearizability harness scenarios, per-command unit tests, + determinism test (leader-emitted patch applied on follower yields + byte-identical DB state), and flag-routing test proving both paths produce + identical on-disk results. +4. **Acceptance gate** — the phase does not advance until its tests are green, + flag-routing equivalence holds, and (for performance-sensitive phases) the + benchmark gate clears. + +**Dual activation gate:** A command runs on the planned path only when BOTH +the `OMLayoutFeature.LEADER_SIDE_EXECUTION` is finalized AND the per-command +runtime flag is enabled. Finalization is the binary-safety gate; the flag is +the operational revert knob. Both must be true. + +--- + +## 5. Zero-Downtime Upgrade (ZDU) Compatibility + +The Ozone cluster supports Zero-Downtime Upgrade (ZDU) where nodes are +upgraded one at a time while the cluster stays online (see HDDS-14498). During a +rolling upgrade, different OMs may run different software versions at the same +time. + +**How leader-planned execution works with ZDU:** + +The planned execution path requires two gates: + +1. **OMLayoutFeature.LEADER_SIDE_EXECUTION is finalized** — the binary-safety + gate. Guarantees no un-upgraded follower receives a planned patch it cannot + apply. +2. **Per-command runtime flag is enabled** — the operational revert knob. + Defaults to legacy (off). Operators opt commands in after finalization. + +The routing decision in `OzoneManagerStateMachine`: + +```java +boolean isPlannedPath(Type cmdType) { + if (!versionManager.isAllowed(OMLayoutFeature.LEADER_SIDE_EXECUTION)) { + return false; // pre-finalized: use legacy path + } + if (!perCommandFlagEnabled(cmdType)) { + return false; // flag not set: use legacy path + } + return plannedRequestCreators.containsKey(cmdType); +} +``` + +**Version lifecycle for removing the legacy path:** + +| Version | Behavior | +|---------|----------| +| **Vn** (introduces planned execution) | Both paths exist. Pre-finalized: legacy. Post-finalized + flag on: planned. | +| **Vn+1** | Legacy path still present for pre-finalization and as the flag-off fallback. | +| **Vn+2** | Legacy path removed. ZDU from <= Vn to >= Vn+2 must pass through Vn+1. | + +**Why this works safely:** + +- The ZDU design guarantees all OMs run the same software version at + finalization time. +- The new proto envelope and `#MANAGED_INDEX` entry are additive and inert + until finalization — old software ignores them. +- The per-command flag defaults to legacy, so a freshly finalized cluster still + runs legacy until an operator opts commands in. +- Finalization alone is not sufficient to activate the new path — this is what + gives operators a no-downtime, no-downgrade revert. + +**Non-rolling upgrade path:** + +For clusters that accept downtime: stop all OMs, upgrade, start, finalize, +enable flags. The legacy path is never used post-finalization. diff --git a/hadoop-hdds/docs/content/design/leader-execution/obs-design.md b/hadoop-hdds/docs/content/design/leader-execution/obs-design.md new file mode 100644 index 000000000000..fc21974281c5 --- /dev/null +++ b/hadoop-hdds/docs/content/design/leader-execution/obs-design.md @@ -0,0 +1,210 @@ +--- +title: Leader-Planned Execution — OBS Design +summary: OBS key operations under leader-planned execution — lock usage, concurrency scenarios, and quota handling +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +author: Abhishek Pal +--- + + +# OBS Design + +This document describes how Object Store (OBS) bucket-layout key operations +work under leader-planned execution. For the general locking model, see +[locking and concurrency](./locking-and-concurrency.md). For the high-level +architecture, see the [overview](./overview.md). + +## Table of Contents + +1. [OBS Lock Matrix](#1-obs-lock-matrix) +2. [Concurrent Scenarios](#2-concurrent-scenarios) +3. [Quota Handling (Decision Required)](#3-quota-handling-decision-required) +4. [OBS Operation Inventory](#4-obs-operation-inventory) + +--- + +## 1. OBS Lock Matrix + +OBS operations use a flat key namespace within a bucket. There is no directory +hierarchy, so container locks (which protect parent-child relationships) are +not used. The bucket itself serves as the root container. + +| Operation | Bucket | Container | Slot | Notes | +|:----------|:-------|:----------|:-----|:------| +| CreateKey | S | — | — | open key carries `clientID`; creates never collide | +| CommitKey | S | — | X(bucket, key) | two commits of same key serialize | +| DeleteKey | S | — | X(bucket, key) | same-key Commit/Delete serialize | +| DeleteBucket | X | — | — | exclusive bucket lock blocks all operations | +| SetBucketProperty | X | — | — | exclusive bucket lock blocks all operations | + +**Key design choices:** + +- **CreateKey takes no slot lock** (only bucket `S`): the open key table key + includes `clientID`, so concurrent creates of the same key name by different + clients write distinct rows and cannot collide. Same-name resolution is + deferred to commit where `X(bucket, key)` is taken. +- **CommitKey/DeleteKey take a slot lock** `X(bucket, key)`: two commits or a + commit and a delete of the same key must serialize to prevent conflicting + overwrites. + +--- + +## 2. Concurrent Scenarios + +### Scenario A: Two CreateKey requests to different keys in the same bucket + +``` +Thread A: CreateKey /vol/bucket/key1 Thread B: CreateKey /vol/bucket/key2 + acquireLock(bucket READ) acquireLock(bucket READ) + — Both succeed (READ locks are shared) + plan() — reads bucket, builds key1 plan() — reads bucket, builds key2 + releaseLock() releaseLock() + submit to Ratis submit to Ratis +``` + +**Why this is safe:** Both threads read the same bucket quota from DB. Both +produce a delta that increments quota by 1. When applied in sequence (Ratis +guarantees total ordering of log entries), each apply writes its own open key +entry (different rows: key1/clientA vs key2/clientB). The bucket quota updates +are also applied in order. + +**Quota correctness:** Each planning thread reads the current committed quota from DB (say quota=100). Thread A plans quota=101. Thread B also plans quota=101. After apply, the DB has quota=101, not 102. This is acceptable for CreateKey because: +- CreateKey allocates an open key — it is a reservation, not a final commit +- The actual quota enforcement happens at CommitKey time +- If needed, we can make CreateKey also acquire a striped key lock to serialize quota updates — but this reduces parallelism for a small benefit + +**Alternative (stricter quota):** If exact quota tracking at CreateKey is needed, we can use a bucket WRITE lock for CreateKey (same as today). This serializes all creates within a bucket but keeps the other benefits (no follower execution, no double buffer). The granular locking is an optimization that can be tuned. + +### Scenario B: Two CommitKey requests to the same key + +``` +Thread A: CommitKey /vol/bucket/key1 Thread B: CommitKey /vol/bucket/key1 + acquireLock(bucket READ) acquireLock(bucket READ) + acquireLock(key WRITE for key1) acquireLock(key WRITE for key1) + — BLOCKS (write lock is exclusive) + plan(): + read openKey from DB + read existing key from keyTable (for overwrite) + build deltas: + DELETE openKeyTable/key1/clientA + PUT keyTable/key1 → committed key bytes + PUT deletedTable/old-key1 (if overwrite) + PUT bucketTable → updated quota + releaseLock(key WRITE) — Thread B now acquires key WRITE + releaseLock(bucket READ) + submit to Ratis plan(): + read openKey from DB + ... + releaseLock(key WRITE) + releaseLock(bucket READ) + submit to Ratis +``` + +**Why this is safe:** The key write lock ensures only one thread plans for the +same key at a time. Thread B cannot read stale state because it waits until +Thread A finishes planning and releases the lock. + +**Important detail:** Between Thread A releasing the lock and its transition being applied, Thread B reads from DB. At this point Thread A's changes are NOT yet in the DB (they are in the Ratis log, waiting to be applied). Is this a problem? + +**Answer: No.** Thread B's open key entry is different from Thread A's (different clientID). Thread B's commit produces a different set of deltas based on what it reads from the DB. When both are applied in Ratis order, the last one wins — which is the correct overwrite behavior. + +However, there is a subtlety: if Thread A's commit would move the existing key to the deleted table, and Thread B's commit also tries to move the same existing key to the deleted table (because it reads the same "current key" from DB), we would get a duplicate entry in the deleted table. + +**Solution:** The apply engine processes entries in Ratis log order. Thread A's transition applies first (moves old key to deleted table, writes new key). Thread B's transition applies second (again reads the old key — but now from Thread A's write — and moves it to deleted table). Since deletedTable keys include a unique suffix (objectID or transaction index), both entries are distinct. The background key deletion service handles cleanup. + +### Scenario C: CreateKey and DeleteBucket for the same bucket + +``` +Thread A: CreateKey /vol/bucket/key1 Thread B: DeleteBucket /vol/bucket + acquireLock(bucket READ) acquireLock(bucket WRITE) + — Thread A succeeds — BLOCKS (write lock waits for all + plan() readers to finish) + releaseLock(bucket READ) + — Thread B now acquires bucket WRITE + submit to Ratis plan(): + check bucket is empty + — Reads from DB; Thread A's new key + is NOT yet applied. Bucket appears + empty in DB. + build delta: DELETE bucketTable + releaseLock() + submit to Ratis +``` + +**Problem:** Thread B's delete succeeds because the DB does not yet show Thread A's key. When both apply, we have a key in a deleted bucket. + +**Solution:** This is the same problem that exists today — bucket delete checks the committed DB, and a concurrent create may have a pending entry in the double buffer. The current solution is that bucket delete checks both the DB and the open key table. In the planned path, we handle this the same way: +- DeleteBucket acquires a bucket WRITE lock (exclusive), which blocks all concurrent key creates (they need bucket READ lock) +- By the time DeleteBucket acquires the write lock, all prior key creates have finished planning and submitted to Ratis +- DeleteBucket then reads from DB. If a prior key was created (and already applied), it sees the key and fails the delete +- If the prior key has not yet been applied, the Ratis log ordering ensures: the CreateKey entry was submitted first (before the delete could acquire the write lock), so it appears before DeleteBucket in the log, and is applied first + +**Key guarantee:** The bucket WRITE lock ensures no new keys can start planning while the delete is planning. Combined with Ratis log ordering, this prevents the race. + +--- + +## 3. Quota Handling (Decision Required) + +Bucket quota (`usedBytes`/`usedNamespace`) is a counter that parallel commits +race on. If updating quota requires an exclusive bucket lock, commits serialize +and the throughput benefit of fine-grained locking is lost. + +### Option A — Direct PUT under bucket shared lock (simpler, soft quota limit) + +Each planned commit reads the current `OmBucketInfo` from RocksDB, increments `usedBytes`/`usedNamespace` locally, and emits a whole-row `PUT` of the updated `OmBucketInfo`. Because commits hold only a shared bucket lock, two parallel commits can both read the same `usedBytes=100`, both write `usedBytes=101`, and the second apply overwrites the first — losing one increment. + +- **Quota counter accuracy:** The `usedBytes` counter can drift under parallel commits. The existing `QuotaRepair` background service reconciles it. +- **Quota limit enforcement:** The limit check is best-effort (soft). Two commits near the limit can both pass the check and both apply, transiently exceeding the quota by up to the in-flight commit count. +- **Simplicity:** No new primitive needed. Standard whole-row PUT. + +### Option B — Commutative RocksDB Merge operator (more complex, exact counter) + +Each planned commit records the quota change as a `Merge` operation in the replicated batch: `merge(BUCKET_TABLE, bucketKey, "quota", encode(+delta))`. At apply time on every node, a registered `QuotaMergeOperator` runs in Ratis order: it reads the current `OmBucketInfo` row, applies the delta, and writes the whole row back. + +- **Quota counter accuracy:** The `usedBytes` counter is always exact (never loses an update). Deltas are summed in Ratis order at apply time. +- **Quota limit enforcement:** Still best-effort (soft) because the limit check runs at planning time, before the Merge is applied. Two parallel commits can both pass the check. The counter is exact; only the limit gate is soft. +- **No on-disk format change:** The `bucketTable` value remains the same `OmBucketInfo` bytes — the Merge operator writes the whole row, not an operand list. No RocksDB-native merge operator is needed. +- **Registration invariant:** The Merge operator must be registered on every node before any node can receive a Merge operation. Otherwise a follower hits an unknown operator ID and must crash. + +### Trade-offs + +| Aspect | Option A (direct PUT) | Option B (Merge operator) | +|--------|----------------------|--------------------------| +| Counter accuracy | Can drift under parallel commits | Always exact | +| Limit enforcement | Soft (same as Option B) | Soft (same as Option A) | +| Complexity | None — standard PUT | New Merge framework; operator per counter | +| Recovery | QuotaRepair reconciles drift | No reconciliation needed | +| Bucket serialization | Last-writer-wins on parallel quota PUTs | Commutative — parallel commits safe | + +**Recommendation:** Option B ensures the quota counter is always exact, which eliminates a class of silent-drift bugs and reduces reliance on the `QuotaRepair` background reconciliation. The implementation cost is bounded (one Merge operator for quota, registered at startup on all nodes). The quota **limit** remains soft under both options, which is an accepted tradeoff for parallelism. + +--- + +## 4. OBS Operation Inventory + +OBS operations in the migration plan (from the +[full operation inventory](./migration-playbook.md#3-full-operation-inventory)): + +| Operation | Tables | Quota? | SCM? | Phase | +|-----------|--------|--------|------|-------| +| CreateKey (OBS) | openKeyTable | usage | yes | P-1 | +| CommitKey (OBS) | keyTable, openKeyTable, deletedTable | usage | no | P-1 | +| AllocateBlock (OBS) | openKeyTable | no | yes | P-1 | +| DeleteKey (OBS) | keyTable, deletedTable | usage | no | P-1 | + +All four are single-step operations. They are migrated in Phase P-1 as the +hardest single-step operations (they involve SCM block allocation and/or +commutative quota updates). diff --git a/hadoop-hdds/docs/content/design/leader-execution/testing-strategy.md b/hadoop-hdds/docs/content/design/leader-execution/testing-strategy.md new file mode 100644 index 000000000000..1936ac0e0230 --- /dev/null +++ b/hadoop-hdds/docs/content/design/leader-execution/testing-strategy.md @@ -0,0 +1,186 @@ +--- +title: Leader-Planned Execution — Testing Strategy +summary: Unit, concurrency, integration, determinism, linearizability, and stress/chaos tests +date: 2026-06-12 +jira: HDDS-11898 +status: proposed +--- + + +# Testing Strategy + +To ensure correctness and catch concurrency bugs, we need tests at multiple +levels. For the high-level architecture, see the [overview](./overview.md). +For the locking model being tested, see +[locking and concurrency](./locking-and-concurrency.md). + +## Table of Contents + +1. [Unit Tests](#1-unit-tests) +2. [Concurrency Tests](#2-concurrency-tests) +3. [Integration Tests](#3-integration-tests) +4. [Determinism Verification Test](#4-determinism-verification-test) +5. [Linearizability Harness](#5-linearizability-harness) +6. [Stress/Chaos Tests](#6-stresschaos-tests) + +--- + +## 1. Unit Tests + +| Test class | What it validates | +|-----------|-------------------| +| `TestManagedIndexService` | Monotonic increments, leader switchover, crash recovery | +| `TestChangeRecorder` | Correct proto generation from put/delete/merge calls | +| `TestLeaderPlanner` | Template ordering (preProcess → authorize → lock → plan → unlock), error handling | +| `TestStateTransitionApplyEngine` | Atomic apply, barrier splitting, TransactionInfo update | +| `TestOBSKeyLockManager` | Lock ordering, stripe distribution, timeout behavior | +| `TestFSOKeyLockManager` | Parent-dir lock ordering, stripe distribution | + +--- + +## 2. Concurrency Tests + +These tests use multiple threads to stress the locking and planning path: + +**Test: Parallel CreateKey same bucket** +```java +// 50 threads create keys in the same bucket simultaneously +// Assert: all keys exist in openKeyTable after apply +// Assert: no exceptions, no lost writes +ExecutorService pool = Executors.newFixedThreadPool(50); +for (int i = 0; i < 50; i++) { + pool.submit(() -> createKey("/vol/bucket/key" + threadId)); +} +// Verify all 50 keys in openKeyTable +``` + +**Test: Parallel CommitKey same key (overwrite race)** +```java +// 10 threads commit the same key name simultaneously +// Assert: exactly one wins (the rest get errors or overwrite correctly) +// Assert: deleted table has correct number of entries +// Assert: final key in keyTable is from the last applied commit +``` + +**Test: CreateKey + DeleteBucket race** +```java +// Thread A creates keys in rapid succession +// Thread B tries to delete the bucket +// Assert: either the bucket is deleted and no keys exist, +// OR the bucket still exists with the created keys +// Assert: never a key in a deleted bucket +``` + +**Test: Leader failover during planning** +```java +// Start planning a request, then trigger leader change +// Assert: ManagedIndex moves forward on new leader +// Assert: retried request gets a new managedIndex +// Assert: no duplicate objectIDs +``` + +**Test: Quota consistency under concurrency** +```java +// 100 threads create + commit keys in parallel +// Assert: final bucket quota equals number of committed keys +// Assert: no negative quota, no over-count +``` + +--- + +## 3. Integration Tests + +These run on a `MiniOzoneHAClusterImpl` with 3 OMs: + +| Test | What it validates | +|------|-------------------| +| Write key, read from follower | All nodes have the same state after apply | +| Write key, kill leader, read from new leader | Failover preserves committed state | +| Write key, kill follower, bring back, verify sync | Ratis log replay works with new format | +| Mixed planned + legacy commands | Both paths coexist correctly | +| CreateSnapshot between planned commands | Barrier semantics preserved | +| Bulk write (10k keys), verify all 3 nodes | No lost writes under load | + +--- + +## 4. Determinism Verification Test + +A special test that validates the planned path produces the same DB state as the +legacy path: + +```java +// For each migrated command: +// 1. Run it through the legacy path (validateAndUpdateCache) → capture DB state +// 2. Run the same request through the planned path (plan + apply) → capture DB state +// 3. Assert: both DB states are byte-for-byte identical +``` + +This test gives high confidence that the migration does not change behavior. + +--- + +## 5. Linearizability Harness + +The design allows many interleavings and treats "an error under one ordering" +as a legal outcome. Example-based "operation X always succeeds" tests are not +sufficient. The bar is **linearizability**: every observed concurrent history is +equivalent to some legal sequential order consistent with real-time precedence. + +**Architecture:** + +1. **Sequential reference model** — a single-threaded in-memory filesystem + (path → node with objectIDs) implementing identical operation semantics and + error conditions. The oracle of "what is legal." +2. **Concurrent harness** — randomized concurrent operation mixes against the + real lock + execution path, recording each operation's invocation/response + interval and the final DB state. Generation is weighted toward adversarial + pairs (see scenarios below). +3. **Linearizability checker** — a Wing-Gong / Lincheck-style search verifying + the recorded history is linearizable against the reference model. Accepts + any legal order, including orders in which an operation legitimately errors. +4. **Invariant assertions** layered on top: no orphan, no objectID collision, + quota counter exact (even if limit is soft), deadlock-free completion. + +**Adversarial test scenarios:** + +| Scenario | What it validates | +|----------|-------------------| +| Create under ancestor being rm-rf'd | No orphan; outcome linearizable | +| Already-purged parent | Revalidation fails the operation cleanly | +| Two crossing renames | Deadlock-free (bounded completion) and linearizable | +| Delete-empty vs create-child | X(dir)/S(dir) rendezvous yields linearizable order | +| Deep mkdirs vs ancestor rename | Children follow renamed objectID; chain completes or fails via revalidation | +| Rename vs delete of same node | X(parent, name) slot rendezvous yields exactly one winner | +| Hot-parent throughput | S(parent) allows concurrent creates; no false serialization | +| Leader failover mid-orchestration | Client retry completes idempotently; no double-apply | + +**Scope:** Namespace operations are strictly linearizable. Quota usage and +subtree reclamation are eventually consistent — the checker treats them as +"converges after background work drains," not "correct at every linearization +point." + +--- + +## 6. Stress/Chaos Tests + +- **Thread starvation test:** Run with more concurrent requests than available + lock stripes. Verify no deadlocks and eventual progress. +- **Slow Ratis test:** Add artificial delay in Ratis replication. Verify + correctness under both lock hold span options. +- **Random kill test:** Randomly kill OM nodes during key operations. Verify + all committed operations survive, no partial state. +- **Mixed-mode test:** Run some commands on the planned path and others on the + legacy path simultaneously. Verify objectID disjointness and applied-index + correctness.