HDDS-14225. Upgrade RocksDB from 7.7.3 to 10.10.1 by smengcl · Pull Request #9813 · apache/ozone

smengcl · 2026-02-24T09:41:04Z

Generated-by: GPT-5.3-Codex, GPT-5.4, Claude Opus 4.8

What changes were proposed in this pull request?

Bump RocksDB version from 7.7.3 to ~~10.4.2~~ 10.10.1. The goal is to upgrade rocksdbjni while keeping
on-disk format and API semantics unchanged.

Since RocksDB 9, BlockBasedTableOptions.format_version=6 is the default. Files written in format_version 6 cannot be read by RocksDB < 8.6.0. This PR explicitly sets the default to version 5 to be compatible with 7.7.3 in the case where Ozone gets downgraded before it gets finalized after an upgrade. ref: https://github.com/facebook/rocksdb/releases/tag/v9.0.0
~~DO NOT MERGE until v10.9.0 or higher RocksDB JNI is available.~~ Note: Potential forward compatibility bug < v10.9.0: https://github.com/facebook/rocksdb/releases/tag/v10.9.1

Fix a bug where compaction with range deletion can persist kTypeMaxValid in MANIFEST as file metadata. kTypeMaxValid is not supposed to be persisted and can change as new value types are introduced. This can cause a forward compatibility issue where older versions of RocksDB don't recognize kTypeMaxValid from newer versions. A new placeholder value type kTypeTruncatedRangeDeletionSentinel is also introduced to replace kTypeMaxValid when reading existing SST files' metadata from MANIFEST. This allows us to strengthen some checks to avoid using kTypeMaxValid in the future.

3. Currently in this PR I am using a rocksdbjni 10.10.1 fatjar I built myself and pushed to maven central snapshot. We need to properly publish it (for example, under Apache Ozone account) before we can merge this.
4. RocksDB#deleteFile is removed upstream (facebook/rocksdb#13322). Callers are migrated to deleteFilesInRanges via ManagedRocksDB.deleteSstFileRange.
5. Adapted RDBTable.isExist/getIfExist to the 9.10.0 KeyMayExist contract by treating a value-less "may exist" hit as inconclusive and confirming with a point-get, and by passing key.duplicate() on the ByteBuffer paths (since KeyMayExist can advance the buffer position), keeping these Ozone existence checks definitive with no behavior change.

DB::KeyMayExist() now follows its function comment, which means value parameter can be null, and it will be set only if value_found is passed in.
ref: https://github.com/facebook/rocksdb/releases/tag/v9.10.0

For hdds-rocks-native. The raw_sst_file_reader patch is updated for RocksDB 10.x, and the upstream source version is derived from rocksdb.version (stripping the packaging suffix 10.10.1.1 -> 10.10.1) so the source download matches the correct upstream tag.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14225

How was this patch tested?

RocksDB related tests are checked and fixed.
Ran full CI and passed.

Generated-by: GPT-5.3-Codex

adoroszlai

Thanks @smengcl for working on this.

We should detect OS at runtime, not build time. Single build (on any OS) should create binaries for all supported operating systems.

dependencyManagement in root POM should include an entry for rocksdbjni artifact with each supported classifier, as well as the non-classified artifact:

  <dependency>
    <groupId>org.rocksdb</groupId>
    <artifactId>rocksdbjni</artifactId>
    <version>${rocksdb.version}</version>
  </dependency>
  <dependency>
    <groupId>org.rocksdb</groupId>
    <artifactId>rocksdbjni</artifactId>
    <version>${rocksdb.version}</version>
    <classifier>linux64</classifier>
  </dependency>
  <dependency>
    <groupId>org.rocksdb</groupId>
    <artifactId>rocksdbjni</artifactId>
    <version>${rocksdb.version}</version>
    <classifier>osx</classifier>
  </dependency>
  ...

hdds-rocks-native should unpack all of these
other modules should continue depending on platform-independent rocksdbjni (no classifier)

Thus changes in most pom.xml files are not needed, nor is rocksdbjni.classifier.

This is a workaround for RocksDB 10.4.2 thin-jar packaging. Ensure classifier JNI artifacts are present at runtime/tests and keep dependency analysis stable. Not needed if mvnrepo rocksdbjni provides a fat jar containing native libs for all supported platforms.

…or change RocksDB 9.10.0 changed DB::KeyMayExist behavior semantics to follow its function comment. Ozone snapshot code paths were treating keyMayExist/getIfExist misses as definitive, which could misclassify existing keys in snapshot-related flows under RocksDB >= 9.10.0 and cause test failures. Treat keyMayExist/getIfExist as hints and fall back to point reads before deciding not-found: - RDBTable: verify with get()/get(ByteBuffer) on inconclusive keyMayExist - TypedTable: in codec-buffer isExist path, fallback to full get before returning false - KeyManagerImpl and ReclaimableRenameEntryFilter: use getSkipCache fallback for snapshot rename lookups RocksDB changelog for reference: https://github.com/facebook/rocksdb/releases/tag/v9.10.0 Behavior Changes DB::KeyMayExist() now follows its function comment, which means value parameter can be null, and it will be set only if value_found is passed in.

RocksDB 9.10+ may consume ByteBuffer state during keyMayExist; treat it as a hint and always use duplicated key buffers for keyMayExist/get fallback paths in RDBTable/RocksDatabase. Add a regression unit test for ByteBuffer fallback behavior. Also make replicas-test.sh restore the whole container.db directory (not just one file) to avoid stale RocksDB WAL/MANIFEST artifacts when recovering from backup.

Replace restored schema-v3 container.db directory

smengcl · 2026-03-12T20:55:28Z

Good news. rocksdb team just published 10.5.1 jars: https://repo1.maven.org/maven2/org/rocksdb/rocksdbjni/10.5.1/

But we need at least 10.9.0 to avoid the forward compatibility bug mentioned in the description. Need to wait a bit more

This reverts commit 4c971a3.

github-actions · 2026-06-22T00:29:48Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions · 2026-06-30T00:26:50Z

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

The RocksDB 10 upgrade replaced the deterministic SST diff assertion with a column-family membership check that passes even when getSSTDiffList returns an empty list for every input, so a regression in the diff engine would go undetected. Restore strong coverage without the version-brittle hard-coded SST file names: - Derive a per-snapshot baseline diff at runtime by tracking all tables. - Assert each table subset's diff equals that baseline filtered to the subset's column families (null-CF files always kept), in order, via containsExactlyElementsOf. - Add a guard that at least one snapshot pair yields a non-empty diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Upgrades Apache Ozone’s embedded RocksDB JNI from 7.7.3 to 10.10.1.1, while aiming to preserve on-disk compatibility and behavioral semantics across upgrades/downgrades, and adapts Ozone’s RocksDB integration to API changes in RocksDB 9/10.

Changes:

Bump RocksDB JNI version and update the native hdds-rocks-native tooling build (including patch updates and source-tag version derivation).
Pin SST block-based table format version to 5 in Ozone-managed configs/writers to keep SSTs readable after downgrade prior to finalization.
Migrate away from removed RocksDB#deleteFile, and adjust keyMayExist/existence-check semantics and tests to match RocksDB 9.10+ contract changes.

Reviewed changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pom.xml	Bumps global `rocksdb.version` to 10.10.1.1.
hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-keywords.robot	Refactors container-state verification keywords for debug smoketests.
hadoop-ozone/dist/src/main/compose/common/replicas-test.sh	Improves debug-compose replica test flow, including container DB backup/restore and pipeline state cleanup.
hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/rocksdiff/TestRocksDBCheckpointDiffer.java	Makes SST-diff/link assertions less dependent on fixed SST filenames across RocksDB versions.
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/hadoop/hdds/utils/db/RDBSstFileWriter.java	Pins SST format version for SST files written via `SstFileWriter`.
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch	Updates native tools patch (eg. `raw_sst_file_reader/iterator`) for RocksDB 10.x API changes.
hadoop-hdds/rocks-native/pom.xml	Derives upstream RocksDB source tag version from `rocksdb.version` and aligns tarball/build paths.
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedRocksDB.java	Replaces file deletion with range-based deletion (`deleteFilesInRanges`) and waits for on-disk removal.
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedDBOptions.java	Adjusts logger setter to match RocksDB 9+ `LoggerInterface` signature change while keeping leak-tracking.
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedBloomFilter.java	Adds `equals`/`hashCode` overrides to satisfy SpotBugs for subclass with additional field.
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedBlockBasedTableConfig.java	Introduces `FORMAT_VERSION=5` constant for downgrade-safe SST compatibility.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTable.java	Adds unit test covering ByteBuffer existence-check fallback behavior.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreCodecBufferIterator.java	Updates Mockito matchers to be explicit with `ByteBuffer` types.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/Table.java	Clarifies that existence/fast-path APIs must remain definitive vs point lookups.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java	Duplicates ByteBuffer for `keyMayExist` safety and migrates SST deletion to range deletion with CF-handle lookup.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java	Updates `isExist/getIfExist` to treat value-less “may exist” as inconclusive and confirm via point-get; fixes ByteBuffer duplication.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/DBProfile.java	Applies pinned SST format version to block-based table configuration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

TestRocksDBCheckpointDiffer: getSSTDiffList returns HashMap values, whose order is not guaranteed and differs between the all-tables baseline call and each per-subset call (they build independent maps from different input sets). Compare with containsExactlyInAnyOrderElementsOf instead of the order-sensitive containsExactlyElementsOf to avoid spurious failures. TestRDBTable: add coverage for the RocksDB "exists with value" fast path of getIfExist(ByteBuffer, ByteBuffer), asserting the returned length, that the value written through outValue.duplicate() is visible in the caller's buffer, and that the fast path does not fall back to a point-get. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ace override Since RocksDB 9.x, DBOptions#setLogger takes a LoggerInterface, so the existing setLogger(Logger) method was only an overload, not an override. A caller invoking setLogger through a DBOptions-typed reference would hit the parent method directly and bypass the leak tracking, leaking the previously set logger. Override setLogger(LoggerInterface) instead so every call path is tracked, and close the previous/current logger when it owns native resources. Add TestManagedDBOptions verifying the logger is closed on replace and on close, including when setLogger is called via a DBOptions reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The subset assertions compare getSSTDiffList's output against a baseline that is itself produced by getSSTDiffList, so a systematic bug returning the wrong files for every input would corrupt both sides and still pass. The only independent check was that at least one diff is non-empty. Add a structural invariant not derived from getSSTDiffList's own output: a snapshot diffed against itself must be empty. Together with the existing non-empty guard, this bounds a broken diff in both directions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The comment justified the last-level guard by the RocksDB#deleteFile API restriction, but the method now uses deleteSstFileRange, which has no such restriction. Update the comment to note the guard is retained to preserve existing pruning behavior rather than because of an API limitation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Note that leakTracker is not part of the filter's identity, so delegating to BloomFilter's equality is correct and the override exists only to declare it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

smengcl · 2026-07-07T19:04:34Z

I have rebased the PR, address all review comments, did self-review with Opus 4.8 and addressed issues found.

CI has passed in my fork. Marking this ready for review.

jojochuang

BTW, we have a few test files using format_version=2 e.g.

ozone/hadoop-hdds/container-service/src/test/resources/123-dn-container.db/OPTIONS-000036

Line 141 in 275653e

format_version=2

that shouldn't be a problem though.

jojochuang · 2026-07-07T22:07:57Z

  public RDBSstFileWriter(File externalFile) throws RocksDatabaseException {
+    // Pin the SST format version so files written here (e.g. snapshot defrag
+    // ingest) stay readable if Ozone is downgraded before finalization.
+    tableConfig.setFormatVersion(ManagedBlockBasedTableConfig.FORMAT_VERSION);


Need an upgrade test later. We'll need it for 2.3.0 release anyway.

Actually, we're missing upgrade tests for 2.1 and 2.2 too https://github.com/apache/ozone/tree/master/hadoop-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/callbacks

Let's file JIRAs for those.

Filed https://issues.apache.org/jira/browse/HDDS-15769 for 2.1 and 2.2

smengcl · 2026-07-08T01:25:30Z

Merged. Thanks @adoroszlai @ptlrs @jojochuang for the reviews.

HDDS-14225. Upgrade RocksDB from 7.7.3 to 10.4.2

f822b30

Generated-by: GPT-5.3-Codex

smengcl changed the title ~~HDDS-14225. Upgrade RocksDB from 7.7.3 to 10.4.2~~ HDDS-14225. [DO NOT MERGE] Upgrade RocksDB from 7.7.3 to 10.4.2 Feb 24, 2026

findbugs

d7e5170

adoroszlai reviewed Feb 24, 2026

View reviewed changes

smengcl added 6 commits February 24, 2026 02:00

Patch rocks-native.patch

e42ca37

Address comment apache#9813 (review)

6c960eb

Merge branch 'master' into rocksdb-v10-upgrade

471e6dc

smengcl force-pushed the rocksdb-v10-upgrade branch from 431f2b2 to c9b83ef Compare March 10, 2026 05:46

smengcl added 2 commits March 10, 2026 11:21

Switch to self-built rocksdbjni 10.10.1 snapshot fat jar

21bc915

Adjust RDBTableStore metrics tests for keyMayExist fallback

7270ac0

smengcl force-pushed the rocksdb-v10-upgrade branch from c9b83ef to 7270ac0 Compare March 10, 2026 18:22

smengcl changed the title ~~HDDS-14225. [DO NOT MERGE] Upgrade RocksDB from 7.7.3 to 10.4.2~~ HDDS-14225. [DO NOT MERGE] Upgrade RocksDB from 7.7.3 to 10.10.1 Mar 10, 2026

smengcl added 8 commits March 10, 2026 16:11

Implement getSkipCache in InMemoryTestTable

84da109

Fix replicas-test.sh

f67327d

Fix replicas-test.sh attempt 2

8ab8f58

Fix replicas-test.sh attempt 3

b3715f0

Replace restored schema-v3 container.db directory

Fix replicas-test.sh attempt 4

602e0db

Fix replicas-test.sh attempt 5

9ef409e

Fix replicas-test.sh attempt 6

d416db1

Handle already-stopped datanodes in replicas-test.sh

1044f33

smengcl added 4 commits March 12, 2026 14:12

Use upstream rocksdbjni 10.5.1 for testing

4c971a3

Revert "Use upstream rocksdbjni 10.5.1 for testing"

ab967ec

This reverts commit 4c971a3.

Restore replicas-test.sh state offline

763440d

Use official rocksdbjni 10.10.1 release

ed1d278

smengcl force-pushed the rocksdb-v10-upgrade branch from fcef039 to ed1d278 Compare March 30, 2026 22:26

Sort pom

7f3abff

github-actions Bot added the stale label Jun 22, 2026

github-actions Bot closed this Jun 30, 2026

smengcl changed the title ~~HDDS-14225. [WIP] Upgrade RocksDB from 7.7.3 to 10.10.1~~ HDDS-14225. [WIP] Upgrade RocksDB from 7.7.3 to 11.1.2 Jul 6, 2026

smengcl reopened this Jul 6, 2026

smengcl changed the title ~~HDDS-14225. [WIP] Upgrade RocksDB from 7.7.3 to 11.1.2~~ HDDS-14225. [WIP] Upgrade RocksDB from 7.7.3 to 10.10.1 Jul 6, 2026

github-actions Bot removed the stale label Jul 7, 2026

Merge branch 'master' into rocksdb-v10-upgrade

946f4ac

smengcl changed the title ~~HDDS-14225. [WIP] Upgrade RocksDB from 7.7.3 to 10.10.1~~ HDDS-14225. Upgrade RocksDB from 7.7.3 to 10.10.1 Jul 7, 2026

smengcl and others added 4 commits July 6, 2026 21:24

Add inline comments

b22b4f3

RDBSstFileWriter should also use pinned format version

8486ae7

Remove redundant key.duplicate()

56a211c

smengcl requested a review from Copilot July 7, 2026 05:54

Copilot started reviewing on behalf of smengcl July 7, 2026 05:55 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java

smengcl and others added 5 commits July 6, 2026 23:10

smengcl marked this pull request as ready for review July 7, 2026 19:03

jojochuang mentioned this pull request Jul 7, 2026

HDDS-15559. Enable cross platform hadoop native library download #10513

Open

jojochuang approved these changes Jul 7, 2026

View reviewed changes

smengcl merged commit b09bcff into apache:master Jul 8, 2026
235 of 240 checks passed

smengcl deleted the rocksdb-v10-upgrade branch July 8, 2026 01:24

smengcl mentioned this pull request Jul 10, 2026

HDDS-15824. Wait for pipeline before writing keys in debug replicas test #10722

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-14225. Upgrade RocksDB from 7.7.3 to 10.10.1#9813

HDDS-14225. Upgrade RocksDB from 7.7.3 to 10.10.1#9813
smengcl merged 41 commits into
apache:masterfrom
smengcl:rocksdb-v10-upgrade

smengcl commented Feb 24, 2026 •

edited

Loading

Uh oh!

adoroszlai left a comment

Uh oh!

smengcl commented Mar 12, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

smengcl commented Jul 7, 2026

Uh oh!

jojochuang left a comment

Uh oh!

Uh oh!

Uh oh!

jojochuang Jul 7, 2026

Uh oh!

smengcl Jul 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

smengcl commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

smengcl commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

smengcl commented Mar 12, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

smengcl commented Jul 7, 2026

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jojochuang Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

smengcl Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smengcl commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

smengcl commented Feb 24, 2026 •

edited

Loading

smengcl Jul 7, 2026 •

edited

Loading