fix(hadoop): Backport HADOOP-18583 & fix OpenSSL native library

[lfrancke]

Description

This was reported as SUP-215 internally.

This fixes

• https://issues.apache.org/jira/browse/HADOOP-12845
• https://issues.apache.org/jira/browse/HADOOP-18583

Together they mean that the native library (OpenSSL) was not used for crypto work:

stackable@hdfs-namenode-default-0 /stackable/hadoop-3.4.0 $ hadoop checknative -a
2025-07-17 21:30:51,368 WARN  bzip2.Bzip2Factory (Bzip2Factory.java:isNativeBzip2Loaded(64)) - Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
2025-07-17 21:30:51,371 INFO  zlib.ZlibFactory (ZlibFactory.java:loadNativeZLib(59)) - Successfully loaded & initialized native-zlib library
2025-07-17 21:30:51,371 WARN  erasurecode.ErasureCodeNative (ErasureCodeNative.java:<clinit>(55)) - ISA-L support is not available in your platform... using builtin-java codec where applicable
2025-07-17 21:30:51,385 INFO  nativeio.NativeIO (NativeIO.java:isPmdkAvailable(192)) - The native code was built without PMDK support.
2025-07-17 21:30:51,448 WARN  crypto.OpensslCipher (OpensslCipher.java:<clinit>(94)) - Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
	at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
	at org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
	at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop:  true /stackable/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false 
bzip2:   false 
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

With only the symlink we get this:

stackable@5df9fee56041 /stackable/hadoop-3.4.1-stackable0.0.0-dev $ hadoop checknative
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Native library checking:
hadoop:  true /stackable/hadoop-3.4.1-stackable0.0.0-dev/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false
bzip2:   false
openssl: false EVP_CIPHER_CTX_block_size
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

Which is why we also need HADOOP-18583 so now the native library works for openssl:

[stackable@0a733a8678ce hadoop-3.4.1-stackable0.0.0-dev]$ bin/hadoop checknative
2025-07-17 22:10:26,164 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
2025-07-17 22:10:26,166 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2025-07-17 22:10:26,166 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2025-07-17 22:10:26,184 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /build/src/hadoop/hadoop/patchable-work/worktree/3.4.1/hadoop-dist/target/hadoop-3.4.1-stackable0.0.0-dev/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false
bzip2:   false
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

Definition of Done Checklist

• Add an entry to the CHANGELOG.md file
• Integration tests ran successfully

[NickLarsenNZ]

LGTM provided @dervoeti is happy with the patch that came in

[lfrancke]

Release Notes

fix: Previously Hadoop wouldn't find and use the native OpenSSL library to speed up crypto operations. This was fixed by adding a missing symlink and backporting HADOOP-18583.