HADOOP-19869 Modernize secret manager default algorithm and key length

[steveloughran]

Description of PR

HADOOP-19869 Modernize secret manager default algorithm and key length

HmacSHA256 and 256 bits.

Also import directly the algorithm in different modules, so it is consistent.

Contains content generated by Coplot + Claude Sonnet 4.6

How was this patch tested?

updated test

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

• The PR includes the phrase "Contains content generated by "
  where is the name of the AI tool used.
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

[cnauroth]

+1 pending pre-commits. Would this need to target 3.6.0 for compatibility concerns?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	2m 5s		Maven dependency ordering for branch
+1 💚	mvninstall	43m 6s		trunk passed
+1 💚	compile	16m 25s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	16m 19s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	5m 40s		trunk passed
+1 💚	mvnsite	5m 17s		trunk passed
+1 💚	javadoc	4m 10s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	4m 13s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	8m 1s		trunk passed
+1 💚	shadedclient	29m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 51s		the patch passed
+1 💚	compile	15m 13s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	15m 13s		the patch passed
+1 💚	compile	16m 16s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	16m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	5m 26s	/results-checkstyle-root.txt	root: The patch generated 2 new + 1449 unchanged - 1 fixed = 1451 total (was 1450)
+1 💚	mvnsite	5m 13s		the patch passed
+1 💚	javadoc	4m 11s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	4m 12s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	8m 43s		the patch passed
+1 💚	shadedclient	29m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	22m 38s		hadoop-common in the patch passed.
-1 ❌	unit	9m 58s	/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt	hadoop-mapreduce-client-core in the patch passed.
+1 💚	unit	1m 34s		hadoop-mapreduce-client-common in the patch passed.
+1 💚	unit	8m 50s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	1m 10s		The patch does not generate ASF License warnings.
		278m 53s

Reason	Tests
Failed junit tests	hadoop.mapreduce.task.reduce.TestFetcher

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8453/1/artifact/out/Dockerfile
GITHUB PR	#8453
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux c201e33b7c49 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 51a998a
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8453/1/testReport/
Max. process+thread count	1594 (vs. ulimit of 10000)
modules	C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8453/1/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

[pan3793]

@steveloughran as we have set up build-only workflow in GitHub Actions (GHA), and it runs stably (I tested it dozens of times and haven't seen failure), do you want to disable the corresponding Jenkins tests, e.g., stop running compile on Debian 13 and Rocky Linux 8 when native code changes, also stop running compile with JDK 21. this way (migrate some jobs to GHA, when it runs stably, shut down these Jenkins jobs), we can speed up the Jenkins pipeline incrementally.

an alternative is to keep Jenkins as-is and run GHA in parallel until we migrate all jobs to GHA, I'm afraid this will take a long time due to lots of flaky tests (~200 classes)

[steveloughran]

@pan3793 +1 for retiring jenkins tests GHA can do

@cnauroth this can be backported. It's just the defaults, We'd log as incompatible and say you can change back.

It only affects the shared secrets between running services (mr AM and workers, for example), and then it is static...the AM doesn't verify it can recreate it, just that it got the same back.

only one I am worried about is HDFS and rolling upgrades. Does a change to keylength/algorithm for new shared secrets cause problems?

[cnauroth]

only one I am worried about is HDFS and rolling upgrades. Does a change to keylength/algorithm for new shared secrets cause problems?

This is the situation I had in mind. SecretManager wraps a javax.crypto.Mac, and it only knows how to create it with one specific algorithm.

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java#L255

If we have some HmacSHA1 tokens in flight, and then we restart NameNode with the new configuration, will it start trying to verify the old tokens as HmacSHA1 and hit password mismatches here?

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L660-L663

[steveloughran]

Lets talk to the hdfs people. Its time to update this