HDDS-14578. Ozone admin command gives inconsistent error messages on expired keytab. by sravani-revuri · Pull Request #10160 · apache/ozone

sravani-revuri · 2026-04-30T04:09:52Z

What changes were proposed in this pull request?

Ozone admin command gives inconsistent error messages on expired keytab for the following commands.

ozone admin safemode status
ozone admin container create
ozone admin container info

bash-5.1$ ozone admin safemode status
Service ID: scmservice
Could not determine leader node

bash-5.1$ ozone admin container create
java.security.cert.CertificateException: org.apache.hadoop.security.AccessControlException: Permission denied.

~# ozone admin container info 69017
Unable to retrieve the container details for 69017
~# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: scm@example.COMValid starting       Expires              Service principal
01/30/2026 10:33:22  01/30/2026 20:33:22  krbtgt/example.COM@example.COM
        renew until 01/31/2026 10:33:10
~# date
Fri Feb  6 12:05:53 PM UTC 2026
root@ve1328:~# ozone admin scm roles
DestHost:destPort ve.example.com:9860 , LocalHost:localPort ve.example.com/10.xx:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
root@ve1328:~# ozone admin container info 69017
Unable to retrieve the container details for 69017
~# kinit -kt scm.keytab scm
~# ozone admin container info 69017
Container id: 69017
Pipeline id: 3c8208ce-46ca-4435-8554-f07a58ec6bf7
Write PipelineId: 7a571825-4751-4876-a6ac-acc4c097f282
Write Pipeline State: CLOSED
Container State: CLOSED

Works again on renewing the Kerberos credentials

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14578

How was this patch tested?

Manual test of ozone admin commands.

bash-5.1$ ozone admin container close 1
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin container report
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin om finalizeupgrade --service-id=omservice
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
bash-5.1$ ozone admin om lof
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
bash-5.1$ ozone admin om lok
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
bash-5.1$ ozone admin om roles
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
bash-5.1$ ozone admin datanode usageinfo --node-id=ba1d4f8a-1b59-4a12-80f8-d78e61a8f85
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin datanode status decommission
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin datanode decommission ba1d4f8a-1b59-4a12-80f8-d78e61a8f85
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin datanode maintenance ba1d4f8a-1b59-4a12-80f8-d78e61a8f85
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin datanode recommission ba1d4f8a-1b59-4a12-80f8-d78e61a8f85
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin containerbalancer start
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin containerbalancer status
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin safemode status
Service ID: scmservice
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin container create
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]

Reconcile command:

manula and unit test.

no keytab:

bash-5.1$ ozone admin container reconcile 1
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ ozone admin container reconcile 1 2 3
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]

non ratis contianer: (failure not form authentication)

bash-5.1$ ozone admin container reconcile 1 2 3
Container #1 not found for reconciliation.
Container #2 not found for reconciliation.
Container #3 not found for reconciliation.
Failed to trigger reconciliation for 3 containers
bash-5.1$ ozone admin container create
Container 1 is created with replication config STANDALONE/ONE.
bash-5.1$ ozone admin container create
Container 2 is created with replication config STANDALONE/ONE.
bash-5.1$ ozone admin container create
Container 3 is created with replication config STANDALONE/ONE.
bash-5.1$ ozone admin container reconcile 1 2 3
Cannot reconcile container 1 in state OPEN.
Cannot reconcile container 2 in state OPEN.
Cannot reconcile container 3 in state OPEN.
Failed to trigger reconciliation for 3 containers
bash-5.1$ ozone admin container close 1
bash-5.1$ ozone admin container close 2
bash-5.1$ ozone admin container close 3
ozone admin container reconcile 1 2 3
Cannot reconcile container #1 with replication type STAND_ALONE. Reconciliation is currently only supported for Ratis containers.
Cannot reconcile container #2 with replication type STAND_ALONE. Reconciliation is currently only supported for Ratis containers.
Cannot reconcile container #3 with replication type STAND_ALONE. Reconciliation is currently only supported for Ratis containers.
Failed to trigger reconciliation for 3 containers

normal:

bash-5.1$ ozone admin container create -t RATIS -r THREE
ozone admin container create -t RATIS -r THREE
ozone admin container create -t RATIS -r THREE
Container 4 is created with replication config RATIS/THREE.
Container 5 is created with replication config RATIS/THREE.
Container 6 is created with replication config RATIS/THREE.
bash-5.1$ ozone admin container close 4 5 6
bash-5.1$ ozone admin container reconcile 4 5 6
Reconciliation has been triggered for container 4
Reconciliation has been triggered for container 5
Reconciliation has been triggered for container 6

…r details on expired keytab

adoroszlai · 2026-04-30T05:10:20Z

Thanks @sravani-revuri for the patch. Are you planning to make admin commands more consistent, or just clarifying admin container info with the exception message?

sravani-revuri · 2026-04-30T05:15:56Z

Thanks @sravani-revuri for the patch. Are you planning to make admin commands more consistent, or just clarifying admin container info with the exception message?

I will be doing it for the other admin commands error messages mentioned in the jira. the jira was initially just for the container info command, but i had found some more inconsistencies. Updated the jira to handle it in this PR itself.

Will update the testing proof once i'm done.

Gargi-jais11 · 2026-04-30T06:11:11Z

@sravani-revuri For the other ozone admin commands which need to have clear error message, please update about it in the PR description.

sravani-revuri · 2026-04-30T15:14:19Z

@Gargi-jais11 could you please take a look at these changes?

sreejasahithi

Thanks @sravani-revuri for working on this
LGTM

Gargi-jais11

Thanks @sravani-revuri fixing this. LGTM!

Gargi-jais11 · 2026-05-05T05:10:03Z

@adoroszlai Do you want to have look into this?

adoroszlai · 2026-05-05T06:09:08Z

Thanks @sravani-revuri for working on this. I think these messages are still inconsistent:

bash-5.1$ ozone admin container info 1
Unable to retrieve the container details for 1. DestHost:destPort scm1.org:9860 , LocalHost:localPort scm1.org/172.25.0.116:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]

bash-5.1$ ozone admin container create
java.security.cert.CertificateException: java.io.IOException: Client cannot authenticate via:[KERBEROS]

bash-5.1$ ozone admin safemode status
Service ID: scmservice
Could not determine leader node. DestHost:destPort scm1.org:9860 , LocalHost:localPort scm1.org/172.25.0.116:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]

different combinations of exceptions (IOException: AccessControlException, CertificateException: IOException)
confusing extra messages (Could not determine leader node, Failed on local exception)
unnecessary details present/absent (DestHost:destPort scm1.org:9860 , LocalHost:localPort scm1.org/172.25.0.116:0.)

Would it be possible to simply print AccessControlException: Client cannot authenticate via:[KERBEROS] or even rely on AccessControlException's default message and print Permission denied?

sarvekshayr · 2026-05-05T06:33:03Z

Would it be possible to simply print AccessControlException: Client cannot authenticate via:[KERBEROS] or even rely on AccessControlException's default message and print Permission denied?

Agreed, just the AccessControlException with a message across all admin commands would be clear and concise.
Similar to below shell commands -

bash-5.1$ ozone sh volume create vol1
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
bash-5.1$ ozone sh bucket create vol1/buck1
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

sravani-revuri · 2026-05-06T12:53:56Z

@sarvekshayr , @adoroszlai could you please take a look at these changes.

sarvekshayr · 2026-05-06T13:08:50Z

I'm seeing the below exception for ozone admin safemode status command. It doesn't match with the output shared in the description. Could you please verify this?

bash-5.1$ ozone admin safemode status
scm:9860 [scmNodeId]: ERROR: Failed to get safe mode status for SCM node: DestHost:destPort scm:9860 , LocalHost:localPort om/172.18.0.9:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]

adoroszlai · 2026-05-06T13:35:22Z

I'm seeing the below exception for ozone admin safemode status command. It doesn't match with the output shared in the description. Could you please verify this?
bash-5.1$ ozone admin safemode status
scm:9860 [scmNodeId]: ERROR: Failed to get safe mode status for SCM node: DestHost:destPort scm:9860 , LocalHost:localPort om/172.18.0.9:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]

FYI this happens in non-HA cluster.

adoroszlai

Thanks @sravani-revuri for updating the patch.

sravani-revuri · 2026-05-08T05:26:02Z

I'm seeing the below exception for ozone admin safemode status command. It doesn't match with the output shared in the description. Could you please verify this?
bash-5.1$ ozone admin safemode status
scm:9860 [scmNodeId]: ERROR: Failed to get safe mode status for SCM node: DestHost:destPort scm:9

@sarvekshayr , the admin commands in non-ha cluster now give the same error message. could you please take a look at these chnages.

ozonesecure % docker compose exec scm bash
bash-5.1$ ozone admin safemode status
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS]
bash-5.1$ exit

adoroszlai

Thanks @sravani-revuri for updating the patch. Please revert changes that are unnecessary now that the command exits on access denied.

sarvekshayr

Thanks for addressing all the comments @sravani-revuri.

LGTM.

sarvekshayr · 2026-05-08T14:05:55Z

Thanks @sravani-revuri for the patch, @adoroszlai, @Gargi-jais11 and @sreejasahithi for the review.

sravani-revuri added 2 commits April 29, 2026 14:48

HDDS-14578. Ozone admin command gives Unable to retrieve the containe…

62f50c2

…r details on expired keytab

change

f5d9640

Gargi-jais11 self-requested a review April 30, 2026 04:21

sravani-revuri changed the title ~~HDDS-14578. Ozone admin command gives Unable to retrieve the container details on expired keytab~~ HDDS-14578. Ozone admin command gives inconsistent error messages on expired keytab. Apr 30, 2026

added container create and safemode status part

cc89366

fixed test

a683d83

sravani-revuri marked this pull request as ready for review May 1, 2026 07:21

sreejasahithi approved these changes May 4, 2026

View reviewed changes

Gargi-jais11 approved these changes May 5, 2026

View reviewed changes

adoroszlai requested a review from errose28 May 5, 2026 06:09

same message for all commands

2b753ab

adoroszlai reviewed May 6, 2026

View reviewed changes

ha cluster comments.

991002f

sravani-revuri requested a review from adoroszlai May 7, 2026 07:10

adoroszlai reviewed May 7, 2026

View reviewed changes

Comment thread hadoop-hdds/cli-common/src/main/java/org/apache/hadoop/hdds/cli/GenericCli.java Outdated

non ha + exit chnage

6b75ec9

sravani-revuri requested a review from adoroszlai May 8, 2026 05:22

adoroszlai reviewed May 8, 2026

View reviewed changes

changes

a695204

changes

f48a9b2

sravani-revuri requested a review from adoroszlai May 8, 2026 09:51

sarvekshayr approved these changes May 8, 2026

View reviewed changes

adoroszlai approved these changes May 8, 2026

View reviewed changes

sarvekshayr merged commit 95867e8 into apache:master May 8, 2026
90 of 91 checks passed

Uh oh!

Conversation

sravani-revuri commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Reconcile command:

no keytab:

non ratis contianer: (failure not form authentication)

normal:

Uh oh!

adoroszlai commented Apr 30, 2026

Uh oh!

sravani-revuri commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gargi-jais11 commented Apr 30, 2026

Uh oh!

sravani-revuri commented Apr 30, 2026

Uh oh!

sreejasahithi left a comment

Choose a reason for hiding this comment

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

Gargi-jais11 commented May 5, 2026

Uh oh!

adoroszlai commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarvekshayr commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sravani-revuri commented May 6, 2026

Uh oh!

sarvekshayr commented May 6, 2026

Uh oh!

adoroszlai commented May 6, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sravani-revuri commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sarvekshayr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sarvekshayr commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sravani-revuri commented Apr 30, 2026 •

edited

Loading

sravani-revuri commented Apr 30, 2026 •

edited

Loading

adoroszlai commented May 5, 2026 •

edited

Loading

sarvekshayr commented May 5, 2026 •

edited

Loading

sravani-revuri commented May 8, 2026 •

edited

Loading