HDDS-11900. Introduce fine-grained locking for OBS

[spacemonkd]

What changes were proposed in this pull request?

HDDS-11900. Introduce fine-grained locking for OBS key operations to enable leader-execution model.

Description

This pull request introduces fine-grained key-path-level locking for Ozone Object Store (OBS) key operations as a prerequisite for the leader-execution model (HDDS-11898). Currently, all OBS key operations within a bucket are serialized by a single bucket-level write lock (BUCKET_LOCK), preventing concurrent operations on different keys. This change enables parallel key operations by introducing a lock hierarchy that allows bucket-level read locks (preventing bucket deletion) paired with per-key write locks.

Problem Statement

The leader-execution design requires deterministic, repeated execution of requests: leaders execute request logic and produce a side-effect log, while followers replay the log to apply only the side effects (DB mutations). This model breaks if concurrent mutations on different keys within a bucket are possible—replaying in an arbitrary order risks different outcomes than the leader's execution.

Today's coarse bucket-level write locks serialize all key operations, making determinism straightforward but limiting throughput. The solution is to introduce fine-grained locking: keys within a bucket can be locked independently using striped locks, enabling concurrent operations on different keys while maintaining determinism through ordered multi-key lock acquisition (bulkGet()).

Changes Proposed

Lock Infrastructure:

• Extended OzoneLockStrategy interface with default methods for multi-key locks and bucket-read-only locks, enabling backward-compatible migration of handlers.
• Implemented OBSKeyPathLockStrategy: acquires BUCKET_LOCK read + per-key KEY_PATH_LOCK writes using ordered bulkGet() for multi-key operations; bucket-read-only operations acquire only BUCKET_LOCK read.
• Implemented RegularBucketLockStrategy fallback: preserves BUCKET_LOCK write behavior when key-path locking is disabled (config default: ozone.om.key.path.lock.enabled=false).

Request Handler Migrations:

• Single-key write operations: OMKeyCommitRequest, OMKeyDeleteRequest, OMAllocateBlockRequest, OMKeySetTimesRequest, S3MultipartUploadCompleteRequest, S3MultipartUploadAbortRequest, S3MultipartUploadCommitPartRequest, S3DeleteObjectTaggingRequest, S3PutObjectTaggingRequest, OMKeyAclRequest.
• Bucket-read-only operations (no key lock): OMKeyCreateRequest, S3InitiateMultipartUploadRequest (per design doc, these write only to OpenKey table with unique clientIDs; no key conflict).
• Multi-key ordered locks: OMKeysDeleteRequest (locks all deleted keys), OMKeyRenameRequest (locks source + destination), OMKeysRenameRequest (locks all from/to keys).

Design & Approach

The implementation follows the OBS locking strategy defined in :

• Lock Hierarchy: S3_BUCKET_LOCK(0) → VOLUME_LOCK(1) → BUCKET_LOCK(2) → KEY_PATH_LOCK(5)
• Bucket-read + key-write model: All key operations acquire BUCKET_LOCK read (prevents bucket deletion) + KEY_PATH_LOCK write (per-key serialization).
• Ordered multi-key locking: Batch operations like DeleteKeys and Rename use bulkGet()-based ordered lock acquisition to guarantee deadlock-freedom.
• Backward compatibility: Default config keeps key-path locking disabled; RegularBucketLockStrategy falls back to BUCKET_LOCK write, preserving existing behavior until the feature is enabled.

The migration uses the Strategy pattern (OzoneLockStrategy) to avoid coupling handlers to specific lock schemes, allowing per-bucket-layout lock selection (OBS vs FSO) and per-config feature gating.

References:

@kerneltime's reference specification: https://github.com/kerneltime/ozone/tree/9c59ae0d66227ce10849b4f914f9efdce9318103/hadoop-hdds/docs/content/design/leader-execution
Pre-Ratis Execution design document: #10503

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11900

How was this patch tested?

Unit Tests:

• New TestOBSKeyPathLockStrategy: single-key, multi-key, bucket-read-only lock operations; concurrent lock acquisition on different keys; deadlock-free ordered locking.
• Updated existing tests: fixed mock setup in TestOMKeyRequest and TestS3MultipartRequest base classes to mock getOzoneLockProvider().

Integration Tests (all passing)
Compilation: mvn clean install -U -DskipTests -DskipShade

Feature Flag:

• Default config: ozone.om.key.path.lock.enabled=false (no behavior change; uses RegularBucketLockStrategy).
• When enabled: OBSKeyPathLockStrategy activates per-key locking for OBS buckets, enabling concurrent key operations.