Load the HDF5 JNI library before enum constants read HDF5Constants

[mkitti]

Stacked on #13. The base branch load-library is a copy of #13's commit (04a2568) pushed to this repository, since a pull request here can't use a branch from a fork as its base. When #13 is merged, this can be retargeted to #13's base.

Problem

#13 calls HDF5NativeLibrary.load() from the static initializers of HDF5Factory, HDF5FactoryProvider and HDFHelper. That covers programs whose first contact with JHDF5 is one of those classes. Five public enums initialize their constants from HDF5Constants, though:

• IHDF5WriterConfigurator.FileFormatVersion
• HDF5DataClass
• HDF5StorageLayout
• HDF5EnumerationType.EnumStorageForm
• CharacterEncoding

What goes wrong. Touching any of these first initializes HDF5Constants before load() runs. The same applies to anything that uses them: FileFormatVersionBounds, the *StorageFeatures classes, and switch statements over them. HDF5Constants then runs H5.loadH5Lib(), which can't find hdf5_java, and H5's class initialization fails. A failed class initialization is permanent in Java, so every later HDF5 call in the process fails too, even through HDF5Factory:

java.lang.UnsatisfiedLinkError: 'int hdf.hdf5lib.H5.H5dont_atexit()'
java.lang.NoClassDefFoundError: Could not initialize class ch.systemsx.cisd.hdf5.IHDF5WriterConfigurator$FileFormatVersionBounds

Evidence.

• This is why load H5's JNI library from the main code #13's CI fails 306/348 tests on macOS. HDF5ArchiverTest touches FileFormatVersionBounds before any factory. Linux passes only because its test classes happen to run in a different order. The test-only NativeLibraryBootstrap that load H5's JNI library from the main code #13 removes used to hide this.
• It also hits downstream code: a program that reads CharacterEncoding.UTF8 and then calls HDF5Factory.open(...) fails the same way.

Fix

• Enum constants. Enum constants are created before any static initializer of their enum runs, and their arguments are evaluated before their constructor, so a load() call inside the enum would come too late. Instead, the constants read HDF5 values through two new helpers, HDF5NativeLibrary.intConstant(...) and longConstant(...), which call load() first:

  COMPACT(intConstant(() -> HDF5Constants.H5D_COMPACT))

• HDF5NativeLibrary.load():
  • It marks the library as loaded only after loading succeeded. load H5's JNI library from the main code #13 set the flag first, so if loading failed (e.g. no natives for the platform), later callers skipped it silently. The real error then only surfaced as the obscure UnsatisfiedLinkError above.
  • It no longer calls System.setProperty(..., null), which throws, when Loader.load() returns null (JavaCPP's library loading disabled).

Verification (macOS arm64, local)

• Every class as first contact. Each of the 1,104 classes in the jar was initialized first in a fresh JVM, followed by writing a file through HDF5Factory. 15 failed on load H5's JNI library from the main code #13, 0 with this PR. The 15 were the five enums above plus classes that use them.
• janelia-jhdf5 suite: mvn test passes 2 runs of 2 with JaCoCo, without NativeLibraryBootstrap.
• Downstream: n5-hdf5 toward HDF5 2.2.0 saalfeldlab/n5-hdf5#50 built against this passes 47 tests with 5 skipped, the same as its master baseline. The CharacterEncoding-first program now works.

🤖 Generated with Claude Code

[mkitti]

Windows CI: what's left, and how to check it on a Windows machine

All 300 Windows test failures here (Java 8 and 21) come from one cause: H5Pset_type_conv_cb failed: special zero no error, directly or wrapped. That's bytedeco/javacpp-presets#1813. On Windows, jnihdf5.dll has its own static copy of HDF5, separate from the hdf5.dll that hdf.hdf5lib.H5 uses, so IDs created through H5 are unknown to org.bytedeco.hdf5. bytedeco/javacpp-presets#1814 fixes that, and its own CI passes on windows-x86_64, including a test that passes IDs between the two bindings. What hasn't been tested is janelia-jhdf5's suite on Windows against #1814, since #1814 isn't published yet. These are the steps for an agent or a person on a Windows x64 machine to do that before #1814 is merged.

1. Build #1814 and install it into the local Maven repository

Prerequisites (the same as .github/actions/deploy-windows/action.yml in javacpp-presets):

• Visual Studio 2022 Build Tools (MSVC x64), CMake, Ninja, Maven.
• A bash shell (Git Bash or MSYS2), since cppbuild.sh is a bash script.
• JAVA_HOME pointing at a JDK. Without it, HDF5's CMake fails with JAVA_INCLUDE_PATH-NOTFOUND.
• Run from an environment where cl.exe is on the PATH, e.g. a shell started from the "x64 Native Tools Command Prompt".

git clone -b hdf5-windows-dll-hid-t https://github.com/mkitti/javacpp-presets.git
cd javacpp-presets
mvn -B install -f hdf5/pom.xml -Djavacpp.platform=windows-x86_64

Expected: HDF5BindingsTest (6 tests) and HDF5ConstantsTest pass. The build log should show jnihdf5.dll linking hdf5.lib, hdf5_cpp.lib, hdf5_hl.lib and hdf5_hl_cpp.lib (not libhdf5.lib), and copying hdf5_cpp.dll, hdf5_hl.dll and hdf5_hl_cpp.dll.

2. Make sure the local build is the one that gets used

This installs org.bytedeco:hdf5:2.2.0-1.5.15-SNAPSHOT (plus its windows-x86_64 natives) into ~/.m2. Maven uses it as long as it's newer than the published snapshot. If upstream publishes another snapshot before #1814 is merged, the published one wins and the test is meaningless. Check:

javap -cp ~/.m2/repository/org/bytedeco/hdf5/2.2.0-1.5.15-SNAPSHOT/hdf5-2.2.0-1.5.15-SNAPSHOT.jar org.bytedeco.hdf5.global.hdf5 | grep " H5P_DATASET_XFER;"

It must say long; int means the unfixed bindings. After the build, running the janelia-jhdf5 tests with -o (offline) also keeps Maven from fetching a newer remote snapshot.

Optionally confirm that only one HDF5 is involved: dumpbin /dependents on the extracted jnihdf5.dll, under %USERPROFILE%\.javacpp\cache\...\org\bytedeco\hdf5\windows-x86_64\, should list hdf5.dll, hdf5_cpp.dll, hdf5_hl.dll and hdf5_hl_cpp.dll.

3. Run janelia-jhdf5's suite against it

git clone -b load-library-init-order https://github.com/JaneliaSciComp/janelia-jhdf5.git
cd janelia-jhdf5
mvn -B test -Djavacpp.platform=windows-x86_64

-Djavacpp.platform limits hdf5-platform to the Windows natives. Expected: no H5Pset_type_conv_cb failures.

If other failures remain, they're Windows-specific behaviour that the shared-library error has been hiding until now. Triage them like this:

• The preset itself (e.g. a missing DLL, UnsatisfiedLinkError, an unresolved __imp_ symbol, wrong constant values): fix on the #1814 branch as a new commit.
• janelia-jhdf5 or its tests (paths, permissions, file locking, uid/gid names): fix in a PR stacked on this one.
• Already fixed in Make the tests pass on Windows #12: the time zone (-Duser.timezone, since Java ignores TZ on Windows) and the archiver's \ path separators.

4. Downstream check (optional)

saalfeldlab/n5-hdf5#50 against the result:

1. mvn install -DskipTests -Djavacpp.platform=windows-x86_64 in janelia-jhdf5.
2. mvn test -Denforcer.skip=true in n5-hdf5. The enforcer rejects the SNAPSHOT dependencies otherwise.

On macOS this gives 47 tests, 5 skipped, the same as n5-hdf5 master.

5. After #1814 is merged

bytedeco publishes a new snapshot from master automatically. Rerun this PR's CI, which resolves snapshots from central.sonatype.com/repository/maven-snapshots/. If Windows passes, drop continue-on-error from .github/workflows/build.yml.

🤖 Generated with Claude Code

[mkitti]

Windows verification against bytedeco/javacpp-presets#1814: passes

I ran the steps from the earlier comment on a Windows 11 x64 machine. janelia-jhdf5's suite passes with no H5Pset_type_conv_cb failures.

1. Preset build

• Branch hdf5-windows-dll-hid-t at f0ecc65b41 ("Always run HDF5ConstantsTest").
• Environment: VS 2022 Build Tools (x64 vcvars64), CMake, Ninja, Maven 3.9.16, Git Bash, JDK 17 (JAVA_HOME set).
• mvn -B install -f hdf5/pom.xml -Djavacpp.platform=windows-x86_64 finished with BUILD SUCCESS. HDF5BindingsTest passed 6/6 and HDF5ConstantsTest 1/1.
• It installed org.bytedeco:hdf5:2.2.0-1.5.15-SNAPSHOT and its windows-x86_64 natives into ~/.m2.

2. Local build was the one used

• javap on the installed jar shows public static final long H5P_DATASET_XFER;, so these are the fixed bindings.
• I could not use -o, because this machine had never downloaded the scijava parent POM. I used -nsu instead. The logs show no hdf5 jar downloaded from a remote repository, only the hdf5-platform POM metadata, so Maven used the local build.
• Not done: the optional dumpbin /dependents check on jnihdf5.dll.

3. janelia-jhdf5 tests

mvn -B -nsu test -Djavacpp.platform=windows-x86_64:

Branch	Tests run	Failures	Errors	Skipped
windows-test-fixes (#12, aebf5e4)	348	0	0	5
load-library-init-order (045c663)	348	0	0	5

• H5Pset_type_conv_cb appears 0 times in either log.
• The 5 skips are the expected Windows ones: testDescribeLink (needs Unix user/group names), testGetInfo and three archiver roundtrip tests (need symlinks).
• The same 348 pass on both branches, so the load-order change is not hiding anything on Windows.

Conclusion

With #1814, the single-HDF5-copy problem (bytedeco/javacpp-presets#1813) is resolved for this suite on Windows. Once a snapshot containing #1814 is published, rerun this PR's CI and drop continue-on-error from .github/workflows/build.yml.

Not tested: the n5-hdf5 downstream check (step 4).

🤖 Generated with Claude Code

[mkitti]

Windows follow-up: remaining checks also pass

Follow-up to my earlier comment, using the same locally built #1814 preset (f0ecc65b41) on Windows 11 x64. This covers the checks I had left out.

Check	Result
dumpbin /dependents jnihdf5.dll	Lists hdf5.dll, hdf5_cpp.dll, hdf5_hl.dll, hdf5_hl_cpp.dll (plus the MSVC/UCRT runtime), and no libhdf5.lib. Only one HDF5 copy is involved.
n5-hdf5 PR #50 (8fab88a) against janelia-jhdf5 load-library-init-order (045c663), installed locally	47 run, 0 failures, 5 skipped. Same as macOS and master. Run with -Denforcer.skip=true -Djavacpp.platform=windows-x86_64.
Java 8 (1.8.0_491), via -Djvm=, confirmed from the surefire report	348 run, 0 failures, 5 skipped
JDK 26.0.1, via -Djvm=	348 run, 0 failures, 5 skipped
Fresh JavaCPP cache in a path with a space and a non-ASCII character (cache dir é\jp, set through -Dorg.bytedeco.javacpp.cachedir)	348 run, 0 failures, 5 skipped. The DLLs were extracted there and loaded.

The earlier JDK 17 runs on windows-test-fixes and load-library-init-order gave the same 348 / 0 / 5.

The logs show no UnsatisfiedLinkError traces, so the quieter hdf5_java loading from #1814 works.

Not testable here

mvn test without -Djavacpp.platform (the full hdf5-platform) fails at dependency resolution on this machine. A local install contains only the windows-x86_64 natives, so the android, linux and macosx jars at 2.2.0-1.5.15-SNAPSHOT can't be found, because the local hdf5 pom shadows the remote snapshot. This comes from the local install, not from #1814. Checking it needs a published snapshot containing #1814.

Still to do after #1814 merges

Rerun this PR's CI against the new snapshot, and if Windows passes, drop continue-on-error from .github/workflows/build.yml.

🤖 Generated with Claude Code