Apache Ozone 2.0 release plan

[jojochuang]

Hi community,

I started a thread to initiate the 2.0 release plan in the mailing list, and I'd just like to copy the discussion over to GitHub disucssion so I can get a wider audience & feedbacks.

I've captured the comments and also reviewed recent feature developments in the JIRA, and took a stab at a release plan:
https://docs.google.com/document/d/1-OLE3IUKZH4zVsPVlUxwNho7MVuoBUwbAgbLMPzouMc/edit?usp=sharing

Let's say
Timeline:

• End of August: Draft release plan
• Mid-September: Scope freeze
• September 30: Code freeze and branching
• October 7-10: Release Candidate 0 (coinciding with Community Over Code NA)
• October 31: Release Candidate 1
• November 22: General Availability (before Thanksgiving week)

The payload of the release is documented at the scope doc: https://docs.google.com/document/d/19QG0fJb5YqvkezUnzV0cJH9VuI8lw4GYfTOyrGpk-Fg/edit?usp=sharing

In the newer release, we will be adding the following features or improvements

• HBase support
• Recon UI revamp
• Website v2
• JDK17 support
• Snapshot improvement

And deprecate or dropping the following features

• Drop/deprecate Hadoop2 support
• Deprecate legacy bucket type
• Remove STAND_ALONE replication type HDDS-6218
• Proto2 to Proto3 migration
• Deprecate o3fs
• Reject creating new LEGACY buckets

• Helm chart HDDS-11027 is a new feature coming out soon but is not part of the main Ozone binary.

Thoughts? Please feel free to comment/edit on the release plan and scope doc, or post the comments below.

[kerneltime]

If I understand correctly, we are not doing proto2 to proto3 migration ? or is it a deprecation of proto2?

[jojochuang]

I feel like we won't have enough bandwidth for it right now. We should deprecate it (not sure how)

[errose28]

Thanks for driving this Wei-Chiu.

• I don't think website v2 will be ready in time and should probably be removed from the scope.
• I think the concept of o3fs only exists on the client side. Can we simply replace that code with an error message that it is no longer supported in 2.x clients?
• I think we should aim to support downgrades from 2.x to 1.x, but not necessarily client/server cross compatibility. This depends on the effects of proto 3 migration though.

[ivandika3]

@jojochuang Thanks for drafting the release plan. I'm curious what changes makes this release a "Major" release (1.4.0 -> 2.0.0) instead of a "Minor" release (1.4.0 -> 1.5.0)? Is it because of the deprecation of features?

[errose28]

Yes this is the biggest reason. In the cases listed so far we should be able to do this with minimal disruption, even though they are technically breaking changes. I also don't think we are aiming to support 1.x <-> 2.x client server cross compatibility. Server side upgrade should be doable without manual migration steps like rewriting data though.

[ivandika3]

@errose28 Thanks for the info.

[errose28]

As discussed in this week's US community sync, we would like to add two more items to the 2.0 release plan:

• HDDS-11754: Drop support for non-Ratis OM and SCM
• HDDS-11753: Remove file per chunk layout from datanode code

Each Jira is open with more details so feel free to comment questions or concerns there.

[errose28]

Can we also remove the LegacyReplicationManager and associated code/configs as part of Ozone 2.0 @sodonnel @siddhantsangwan?

[sodonnel]

It would make sense to remove Legacy RM, as we don't plan to use it or update it any further. There is no point in carrying both versions going forward.

[adoroszlai]

Created HDDS-11759

[smileyboy2019]

According to this plan, how is the current progress and is there any delay? I haven't seen the latest progress of 2.0 yet

[jojochuang]

Hi thanks for checking in. If you look at the dashboard https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12337720 i wanted to send an update last week before end of the year but got distracted.

there are five blockers so far. Among which I think only
Remove file per chunk layout from datanode code
Drop support for non-Ratis OM and SCM

are real blockers. The rest can be moved out. The community's actively working on HDDS-11754 so I think we're getting closer.

[jojochuang]

Hi Ozone community,

Please try out and cast your vote for the Ozone 2.0.0 release candidate 0.

This is a huge release, containing 1691 resolved jiras, numerous features and stability improvements.

Release process: https://ozone-site-v2.staged.apache.org/docs/developer-guide/project/release-guide/
Git tag: https://github.com/apache/ozone/releases/tag/ozone-2.0.0-RC0
All resolved jiras: https://issues.apache.org/jira/issues/?jql=project%20%3D%20HDDS%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.0.0
Source code and binary tarball: https://dist.apache.org/repos/dist/release/ozone/2.0.0-rc0/
The Maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapacheozone-1029/
PGP key: https://dist.apache.org/repos/dist/dev/ozone/KEYS
Fingerprint: 3ED23305D7631918

Per Apache policy, this release candidate vote will open for 7 days until the end of March 28th 2025. PMC members can cast binding votes while committers and community contributors are welcomed to cast non-binding votes.

Best regards,
Weichiu

[jojochuang]

Please take a look at the following proposed release note:

Changelog

All notable changes to Apache Ozone are documented in this file. This project adheres to the Keep a Changelog format and Semantic Versioning.

Apache Ozone 2.0.0 adds 1700 new features, improvements and bug fixes on top of Ozone 1.4.

[2.0.0] - 2025-03-27

Added

• HDDS-7593 Supporting HSync and Lease Recovery: Added support for HDFS-compatible hsync/hflush on Ozone client output streams, enabling durable writes for use cases like HBase and Solr. Also introduced HDFS-style lease recovery (recoverLease()) and the ability to toggle safe mode via the FileSystem API.
• HDDS-7852 SCM Decommissioning Support: The Storage Container Manager now supports decommissioning in an HA cluster. Administrators can gracefully remove an SCM from the HA ring (or force remove if needed).
• HDDS-8829 Symmetric Keys for Delegation Tokens: Added support for symmetric keys to secure delegation tokens, enhancing authentication and token management.
• HDDS-10656 Atomic Key Overwrite and Key Replacement: Introduced atomic operations for key overwrite and key replacement, ensuring consistency during key updates.
• HDDS-6263 ARM64 Support: Added native ARM64 support, enabling Apache Ozone to run efficiently on ARM-based servers.
• HDDS-8246 Java 11/17/21 Support and Testing: Expanded runtime support and testing to include Java 11, 17, and 21, ensuring compatibility with modern Java environments.
• HDDS-11577 AWS SDK v2 Client Support: Added AWS SDK v2 support for improved integration with Amazon Web Services and enhanced S3 gateway functionality.
• HDDS-9307 Ozone Performance and Operational Dashboards: Introduced new dashboards to monitor cluster performance and operational metrics, providing real-time insights into Ozone’s health.
• HDDS-11825 Interactive Mode for Ozone CLI: Added an interactive mode for the Ozone CLI, enhancing the command-line experience with guided prompts and real-time feedback.

Changed

• Default Crypto Compliance Mode: The default value for Ozone’s security compliance mode (ozone.security.crypto.compliance.mode) was changed to a stricter setting. Ozone 2.0 enables compliance mode by default, enforcing stronger cryptographic standards out-of-the-box.
• Kerberos Principal Pattern: Ozone Manager now defines a default principal pattern for Kerberos principals. This change allows cross-realm authentication without additional configuration, preventing errors when Ozone clients interact across different Kerberos realms.
• HSync File Overwrite: It is now possible to overwrite a file that has been hsync-ed. Previously, attempting to overwrite an hsync-flushed file could fail; Ozone 2.0 removes this restriction to better match HDFS semantics.
• DataNode Failure Handling: If a DataNode’s internal Ratis server encounters a critical failure or stops, the DataNode will now shut itself down automatically. This fail-fast behavior ensures consistency and improves cluster stability.
• Performance Improvements: Ozone 2.0 includes multiple optimizations for better throughput and lower latency. Notably, writes through the Hadoop FS interface have improved streaming performance, Ozone clients use less heap memory (avoiding unnecessary byte array copies), and replication between datanodes now leverages zero-copy transfers for efficiency. Data checksum verification was also optimized by using direct buffers and reducing overhead.
• UI Improvements for Recon: Enhanced the Recon UI for improved data visualization and navigation, making it easier for users to analyze cluster data.

Deprecated

• HDDS-11753 Deprecate File-Per-Chunk Container Layout: The legacy FILE_PER_CHUNK container layout (configurable via ozone.scm.container.layout) is now deprecated. Beginning with 2.0.0, new containers cannot be created using this layout. Support for FILE_PER_CHUNK will be removed in a future release, so users should migrate any data off this layout if still in use.

Removed

• Non-HA OM/SCM Modes: Standalone (non-Ratis) deployment modes for Ozone Manager and SCM have been removed. Ozone 2.x now always runs OM and SCM in HA (Ratis) mode. Clusters upgrading from a non-HA setup will be automatically converted to a single-node Ratis quorum. Legacy configuration flags to disable Ratis are no longer applicable.
• HDDS-7307 Move S3 Gateway Web Admin: Moved the S3 Gateway web administration to a dedicated port to isolate admin traffic.
• HDDS-11759 Remove LegacyReplicationManager: Removed the LegacyReplicationManager component, consolidating replication logic into the modern framework.
• (Various internal and unused configurations were also cleaned up—for example, some unused Netty-related SCM config keys were removed.)

Fixed

• Encryption Functionality: Fixed a critical regression where Ozone encryption was unintentionally broken by a prior change. Encryption for buckets/keys now works as expected, restoring data-at-rest encryption capabilities.
• S3 Secret Generation: Resolved an issue that caused an Internal Server Error when generating an S3 secret via the HTTP API. The S3 secret creation endpoint now returns proper status on success or failure.
• OM Crash on S3 Secret: Fixed a bug that could cause the Ozone Manager to terminate with an NPE when handling S3 secret value protos. Generating or fetching S3 credentials will no longer crash the OM.
• S3 MPU ETag Compatibility: Corrected multi-part upload ETag calculation to match AWS S3 behavior, ensuring compatibility with S3 clients.
• Orphaned Blocks on Overwrite: Fixed a scenario that left orphaned data blocks when overwriting an existing key. Ozone now properly cleans up replaced blocks during key overwrite, preventing storage leaks.
• FSO ListKeys Results: Corrected the ListKeys API for FileSystem Optimized (O3FS) buckets. Previously, a key prefix matching multiple keys could yield incomplete results; this issue has been resolved.
• Dynamic Config in Secure Clusters: The ozone admin reconfig command now works in secure (Kerberized) clusters. Earlier, reconfiguration would fail when security was enabled; dynamic updates to configuration now succeed.
• Block Allocation in Safe Mode: Fixed an issue where clients could not allocate blocks if SCM was in safe mode. The client/Ozone Manager now retries or waits when SCM is in safe mode, ensuring smoother startup and write operations.
• SCM Recommission State: Fixed an InvalidStateTransitionException that could occur after recommissioning an SCM in an HA cluster. The SCM state machine now correctly handles recommissioning.
• DataNode Out-of-Memory: Fixed a memory issue on DataNodes that could lead to out-of-memory errors due to excessive mmap allocations. DataNodes now better manage mmap buffers and avoid resource exhaustion.
• OM Memory Leak: Patched a memory leak in Ozone Manager request handling. Previously, per-client thread-local data in OM requests was not properly released, potentially causing memory growth over time.
• DataNode Block Deletion Leak: Fixed a resource leak on DataNodes when the delete blocks queue was full, preventing stale state accumulation and memory buildup.
• Heartbeat Reporting: Resolved an issue where DataNodes could stop reporting heartbeats or container reports for extended periods. Transient failures no longer prevent timely SCM reporting.
• Replication Count Accuracy: Fixed mis-counting of containers on decommissioning or decommissioned DataNodes, preventing false alarms in replication metrics.
• Log Spam on Pipeline Loss: Fixed an infinite-loop bug in SCM that caused log flooding when a pipeline’s nodes could not be found, reducing noise and CPU usage.
• SetTimes on Linked Bucket: Corrected the ozone fs -setTimes command for linked buckets so that setting access or modification times works as intended.

Security

• CVE-2024-45106 (S3 Secret Vulnerability): Fixed an authentication flaw in the S3 Gateway where an HTTP endpoint could allow any authenticated Kerberos user to regenerate another user’s S3 secret. In Ozone 2.0.0, this endpoint is disabled by default and secured.
• Sticky-bit in Shared /tmp: Fixed enforcement of sticky-bit permissions on the shared /tmp bucket when Ranger authorization is used. Ozone now properly upholds the expected Unix semantics for shared temporary storage, closing a potential access control gap.

Developer

• HDDS-6729 JUnit 4 to 5 Upgrade: Upgraded the testing framework from JUnit 4 to JUnit 5 for improved test capabilities and modern annotations.
• HDDS-8113 Dropped Hadoop 2.7–2.9 Support: Removed support for older Hadoop versions (2.7 through 2.9) to streamline compatibility and leverage newer Hadoop features.
• HDDS-10986 Publish SBOM Artifacts: Enabled the publication of SBOM (Software Bill of Materials) artifacts to enhance supply chain transparency.
• HDDS-11617 Hadoop Dependency Updated to 3.4.1: Updated the Hadoop dependency to version 3.4.1, improving compatibility and performance.

[jojochuang]

Update: Ozone 2.0.0 RC2 vote passed. I'm still waiting for our internal QE team who is running tests against the release bits.

[jojochuang]

The version is released: https://github.com/apache/ozone/releases/tag/ozone-2.0.0