HDDS-15845. Design for common authentication/authorization service by spacemonkd · Pull Request #10742 · apache/ozone

spacemonkd · 2026-07-13T17:04:03Z

What changes were proposed in this pull request?

HDDS-15845. Design for common authentication/authorization service

Please describe your PR in detail:
Ozone currently fragments authentication and authorization across multiple client/protocol paths with no unified control plane.

This design proposes a standalone Ozone Auth Service — a separate process that becomes the single owner of client-facing authentication and authorization for all Ozone metadata operations.

Note: Generated using Claude Opus 4.6

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15845

How was this patch tested?

N/A

amaliujia · 2026-07-14T07:03:10Z

+
+---
+
+### 2.10 Edge Cases and Security Considerations


How many auth services do we have in OM HA mode?

Currently the issue with HA mode is that OMClientRequest.getUserInfo() runs inside the Ratis state machine apply path for credential validation.
This means that if we are dependent on an external service like Ranger for validation on the apply path then it is possible that the validation fails for like say Ranger is unavailable when applying to the follower or the latency is high.
We are solving this issue right now by keeping everything in-process in OM.
This means:

We cannot depend on external service for complex policies.

S3 signature verification happens twice (parsed in gateway, re-verified in OM) because OM can't delegate
and many such issues.

The auth service is just going to be sharing the same signing key from SCM's SecretKeyClient.
Each Auth Service independently verifies credentials and issues tokens and no coordination or state synchronization between Auth Service instances is needed as it is independent from the data path.

We can use a single instance only, and since auth service is going to be independent from OM so if token issuance becomes a bottleneck we can scale up as much as we want. It doesn't need to be proportional or equal to the OM instance count.

Sorry for the long paragraph but I hope this bring some more context on how it will be used.

Thanks for the context!

amaliujia

I learned a lot from this doc. Overall LGTM.

At least two big benefits I can tell is:

eliminate and then unify the auth.
Move some logic out of OM Ratis apply path (BTW #10503 will help this as well)

fmorg-git · 2026-07-15T03:45:57Z

hi all, I saw some salient disadvantages with this approach. I posted them in internal communications. Thanks

fapifta

Thank you @spacemonkd for posting the design. I left a few comments after the initial read. It is good to start a discussion on this, let's find the initial steps and start experimenting with this idea as it seems to be really promising and it is really well matches what we have discussed in person earlierfor future ideas that we also touched.

fapifta · 2026-07-15T19:30:20Z

+
+### 2.10 Edge Cases and Security Considerations
+
+#### Token expiry during Ratis replication


I believe the request, if it does not already, should carry a timestamp in the ratis path attached by the leader OM, and expiry should be verified against that later on. That would help to reduce the accepted clock skew and would also work well with transaction replays in case of a huge backlog or during startup replays from the ratis log.

fapifta · 2026-07-15T19:42:38Z

+- OM applies a configurable clock skew tolerance (`ozone.auth.token.clock.skew.ms`, default 30000)
+- The token TTL should be longer than the maximum expected Ratis replication latency
+
+#### Key rotation overlap


Here, we need to calculate with the max token lifetime, where we should apply an absolute maximum (similar to AWS STS tokens), but we also need to prep for arbitrary delays due to whatever failures/delays in checkpointing.
I think the following may work:

SecretKeyManager holds keys for 30 or more days. (This is a security exposure if the keys are stolen from it storage so...)

add support for secret key revocation

assign datetime of issuance and secret key id to the token (part of the signed data)

verify expiry against issuance and secret key, if key is revoked or token is expired reject.

This would resolve overlap problems and there would be no need to guess the key used (this might be there already I don't remember from the top of my head).

fapifta · 2026-07-15T19:49:58Z

+                ↑ rotation point
+```
+
+#### Replay protection sizing


In the current RPC and client behaviour we may want to allow tokens to be used multiple times. If a FileSystem client does an operation, we have 3-16 or such number of rpc calls for one operation, thos may use the same token...
We may need to look at these tokens as multi use tokens like the STS token in AWS and let a client have one token for a longer period of time maybe a full run's duration or for a couple of hours. This will lessen the load on the authn authz service while deliberately allow token reuse so we don't need to worry about our current RPC design problems and the token reuse.

fapifta · 2026-07-15T19:53:57Z

+
+#### Token not logged
+
+The full token (especially signature bytes) must never appear in log output. Only `tokenId` and `subject` are safe to log for debugging and audit purposes.


We may need other things for audit, like authorized operation and maybe a few other thing like client id, source ip etc... Just don't close down the potential to add more. We may define what are the things should definitelly not be logged and allow optionally the rest.

fapifta · 2026-07-15T19:59:23Z

+- Returned user info ≈ `VerifiedIdentity`
+- API server local authz ≈ OM token binding verification
+
+**Key difference:** Kubernetes tokens are session-scoped (valid for any API call). Ozone's `OzoneAuthToken` is operation-bound (valid for exactly one operation type on one resource). This prevents lateral movement — a stolen token for `ReadKey /vol1/b1/key1` cannot be reused for `DeleteVolume /vol1`.


This may be problematic for the same reason I talked about for replays, our current RPC behaviour does prevent us using per operation tokens as one operation spans multiple RPC calls almost always.

fapifta · 2026-07-15T20:03:28Z

+- Session policy intersection with principal permissions → same semantics
+- Configurable duration → `expiryMs` in token
+
+**Key difference:** AWS STS credentials are session-scoped (multiple API calls). Ozone tokens are per-operation. For use cases needing session semantics (e.g., a Spark job running thousands of reads), the client can cache and reuse tokens whose `resourceScope` covers a prefix.


This is prevented by the replay protector isn't it? Do I misunderstand the role of the replay protector?

fapifta · 2026-07-15T20:10:21Z

+
+### Asymmetric signing for cross-domain verification
+
+The current HMAC-SHA256 approach requires OM and Auth Service to share a symmetric key. For future scenarios where external systems need to verify Ozone tokens without possessing the signing key (e.g., cross-cluster federation), an asymmetric algorithm (RSA-PSS or Ed25519) could be added as an alternative `signatureAlgorithm`. The Auth Service signs with its private key; verifiers use the public key.


In case we start to think about federation I think we should come up with some ways where we can keep the symmetric key usage as asymetric will slow things down and would not help with federated perf. But this bridge is far enough to leave this as is.

rakeshadr

Thanks @spacemonkd for the detailed design spec. Added a few comments, please go through it.

rakeshadr · 2026-07-16T11:50:29Z

+### Backward compatibility
+
+- **Auth Service not deployed:** All existing auth paths work unchanged. `OMRequest.authToken` field is never set. Zero code changes required in clients or OM.
+- **Auth Service deployed, legacy clients:** OM accepts both `OzoneAuthToken` (new) and legacy `S3Authentication`/Kerberos UGI (existing). Clients opt in via configuration (`ozone.client.auth.service.enabled`).


Can you describe OzoneFileSystem bootstrap path. How do client obtain the OzoneAuthToken?

Without a client-side implementation, Auth Service is unreachable from any existing Ozone SDK user, right?

For example, we can think from Spark/Hive/Impala application perspective, these are some of the popular workloads that uses OzoneFS client.

rakeshadr · 2026-07-16T13:15:38Z

+| Phase 0 | Deploy Auth Service alongside OM. No clients use it. | Zero — idle process | All existing paths unchanged |
+| Phase 1 | S3 Gateway delegates S3 signature verification to Auth Service | Low | Fallback to legacy path available |
+| Phase 2 | New gRPC clients authenticate via Auth Service | Zero | New capability, not migration |
+| Phase 3 | OzoneFS client optionally exchanges Kerberos ticket for token | Low | Kerberos still works as fallback |


IIUC, Ozone Client → Auth Service (per operation) → OzoneAuthToken → OM looks fine from an S3 workload perspective.

However, OzoneFS + distributed applications (Spark, Hive etc) uses DT approach:

Token (Delegation Token) — session-scoped, distributed by YARN/Spark to all executors, carried in every OM RPC header, verified locally at OM via OzoneDelegationTokenSecretManager.

Token (Block Token) — per-block, issued by OM alongside block locations (OMKeyRequest.java), carried by executors to DataNodes

With Kerberos model, Driver is the only JVM that contacts KDC (once at startup). All executors authenticate to OM using the DT — they have no Kerberos ticket and no credential to call Auth Service.

For Spark/OzoneFS distributed workloads, we need to explore Token as the session credential and change only how the Driver initially obtains it — from Kerberos today to OIDC/token-based tomorrow. The existing two-token architecture (DT → OM, BlockToken → DN) should remain intact. This is critical for seamless integration with OzoneFS applications.

Adding a few code references that we can use for more detailed study. This is critical for the seamless integration of OzoneFS with applications.

Any OIDC JWT connection is rejected immediately with INVALID_AUTH_METHOD before a DT is ever issued.

OzoneManager.java#L2540

/** * @return true if delegation token operation is allowed */ private boolean isAllowedDelegationTokenOp() throws IOException { AuthenticationMethod authMethod = getConnectionAuthenticationMethod(); return !UserGroupInformation.isSecurityEnabled() || (authMethod == AuthenticationMethod.KERBEROS) || (authMethod == AuthenticationMethod.KERBEROS_SSL) || (authMethod == AuthenticationMethod.CERTIFICATE); }

The chain in Ozone source is:

OMGetDelegationTokenRequest.preExecute() → OzoneManager.getDelegationToken() → isAllowedDelegationTokenOp() ← throws INVALID_AUTH_METHOD for OIDC → OMClientRequest.getUserInfo() ← reads Kerberos SASL context only → OzoneDelegationTokenSecretManager.createToken(owner, renewer, realUser)

rakeshadr · 2026-07-16T13:25:19Z

+The `ReplayProtector` maintains a bounded set of recently seen `tokenId` values:
+- Only write operations are tracked (reads are idempotent)
+- Set size is bounded by: `maxTokenTTL / averageWriteInterval`
+- For 30s TTL with 1000 writes/sec = 30,000 entries (~1MB memory)


How does long-running job token renewal happens?

For example, Spark ETL jobs run 12–72 hours. Hive Server2 and Impala daemons run for several days. The 30s OzoneAuthToken TTL, please add token renewal design in the roadmap.

rakeshadr · 2026-07-16T13:29:33Z

+
+In `OzoneManager.checkAcls()`:
+```java
+if (AuthTokenContext.isPresent()) {


Ratis apply path data race:-

Ratis state machine dispatches apply() on its own thread pool. If there is any async handoff between the RPC interceptor setting the thread-local and OzoneManager.checkAcls() reading it, the identity is null, producing either an NPE or silent fallback to unauthenticated access, right?

rakeshadr · 2026-07-16T13:55:52Z

+#### Batch operations
+
+For batch operations (e.g., `DeleteKeys` with multiple keys), the token's `resourceScope` covers the common parent at the appropriate level:
+- `DeleteKeys` on `/vol1/bucket1/key1`, `/vol1/bucket1/key2` → scope is `/vol1/bucket1/*`


Are we widening the scope to /vol1/bucket1/* for same-bucket batch operations? If yes, it exposes a gap between Auth Service's authorization granularity and OM's current per-key ACL enforcement.

Presently, OMKeysDeleteRequest checks ACLs per key inside a loop:

// OMKeysDeleteRequest.java
checkKeyAcls(ozoneManager, volumeName, bucketName, keyName,
IAccessAuthorizer.ACLType.DELETE, OzoneObj.ResourceType.KEY, volumeOwner);
// Per-key failure → PARTIAL_DELETE, key preserved
When AuthTokenContext.isPresent() returns true, checkAcls() exits early and this per-key loop is never reached. A token for /vol1/bucket1/* DELETE effectively grants DELETE on every key in the bucket, including keys the caller has no per-key ACL for.

Reference: OMKeysDeleteRequest.java#L172

HDDS-15845. Design for common authentication/authorization service

07e10e3

spacemonkd self-assigned this Jul 13, 2026

spacemonkd added design AI-gen labels Jul 13, 2026

spacemonkd marked this pull request as ready for review July 13, 2026 17:04

spacemonkd requested review from adoroszlai, amaliujia, fapifta and rakeshadr July 13, 2026 17:04

amaliujia reviewed Jul 14, 2026

View reviewed changes

Comment thread hadoop-hdds/docs/content/design/common-auth-service.md

amaliujia reviewed Jul 14, 2026

View reviewed changes

amaliujia reviewed Jul 15, 2026

View reviewed changes

fapifta reviewed Jul 15, 2026

View reviewed changes

rakeshadr reviewed Jul 16, 2026

View reviewed changes


		### 2.10 Edge Cases and Security Considerations

		#### Token expiry during Ratis replication


		#### Token not logged

		The full token (especially signature bytes) must never appear in log output. Only `tokenId` and `subject` are safe to log for debugging and audit purposes.


		### Asymmetric signing for cross-domain verification

		The current HMAC-SHA256 approach requires OM and Auth Service to share a symmetric key. For future scenarios where external systems need to verify Ozone tokens without possessing the signing key (e.g., cross-cluster federation), an asymmetric algorithm (RSA-PSS or Ed25519) could be added as an alternative `signatureAlgorithm`. The Auth Service signs with its private key; verifiers use the public key.

Uh oh!

Conversation

spacemonkd commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amaliujia left a comment

Choose a reason for hiding this comment

Uh oh!

fmorg-git commented Jul 15, 2026

Uh oh!

fapifta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rakeshadr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

spacemonkd commented Jul 13, 2026 •

edited

Loading