latest docker image crashes on MacM4 (prob due to JDK bump)

[tombrus]

Steps to reproduce

docker run neo4j:latest; echo "exit code = $?"

Expected behaviour

a running container

Actual behaviour

exit code = 134

Neo4j image tag being used

neo4j:latest

The output of the docker version command

Client:
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:37:19 2025
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.38.0 (181591)
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:25 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Operating system

chip : Apple M4 Max
macOS: 15.3 (24D60) Sequoia

error messages

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000ffff9fd3fc5c, pid=95, tid=99
#
# JRE version:  (21.0.6+7) (build )
# Java VM: OpenJDK 64-Bit Server VM (21.0.6+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# j  java.lang.System.registerNatives()V+0 [email protected]
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/lib/neo4j/hs_err_pid95.log
[0.007s][warning][os] Loading hsdis library failed
#
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/startup/docker-entrypoint.sh: line 300:    95 Aborted                 ${neo4j_admin_cmd} dbms set-initial-password "${password}" "${extra_args[@]}"

additional info

The following runs fine:

docker run  neo4j:5.26

I noticed that in this version the JDK was bumped from 17 to 21. There might be a relation.

[jennyowen]

@tombrus I think you might be hitting this bug in eclipse temurin 21: adoptium/adoptium-support#1223

To apply their workaround, can you try adding --env NEO4J_server_jvm_additional="-XX:UseSVE=0" to your docker run command?
Like:

docker run -it --rm \
  --env=NEO4J_DEBUG=yes \
  --env NEO4J_server_jvm_additional="-XX:UseSVE=0" \
 neo4j:latest

Does neo4j start?

[tombrus]

Hi @jennyowen, thanks for looking into this.

Unfortunately neo4j does not start (the container keeps restarting):

DEBUGGING ENABLED
Running as root user inside neo4j image
chowning /var/lib/neo4j recursively to neo4j:neo4j
Setting all files in /var/lib/neo4j/conf to permissions 600
Extra plugins were requested. Ensuring the mounted /plugins folder has the required write permissions.
checking /plugins is writable
checking /plugins is readable
checking /logs is writable
checking /data is writable
checking /data/databases is writable
checking /data/dbms is writable
checking /data/transactions is writable
One or more NEO4J_PLUGINS have been requested.
Checking requested plugins are known and can be installed.
Plugin apoc has been requested
apoc is already in the container at /var/lib/neo4j/labs/apoc-*-core.jar
Installing Plugin 'apoc' from /var/lib/neo4j/labs/apoc-*-core.jar to /plugins/apoc.jar
Applying plugin specific configurations.
Applying default values for plugin apoc to neo4j.conf
apoc requires setting dbms.security.procedures.unrestricted=apoc.*
Skipping dbms.security.procedures.unrestricted for plugin apoc because it is already set.
You may need to add apoc.* to the dbms.security.procedures.unrestricted setting in your configuration file.
Setting docker specific configuration overrides
Applying configuration settings that have been set using environment variables.
server.jvm.additional will be appended to /var/lib/neo4j/conf/neo4j.conf without replacing existing settings.
Appended server.jvm.additional=-XX:UseSVE=0 to /var/lib/neo4j/conf/neo4j.conf
Removing existing setting for server.directories.plugins in /var/lib/neo4j/conf/neo4j.conf
Appended server.directories.plugins=/plugins to /var/lib/neo4j/conf/neo4j.conf
Removing existing setting for server.directories.logs in /var/lib/neo4j/conf/neo4j.conf
Appended server.directories.logs=/logs to /var/lib/neo4j/conf/neo4j.conf
Removing existing setting for apoc.import.file.use_neo4j_config in /var/lib/neo4j/conf/apoc.conf
Appended apoc.import.file.use_neo4j_config=true to /var/lib/neo4j/conf/apoc.conf
Removing existing setting for apoc.import.file.enabled in /var/lib/neo4j/conf/apoc.conf
Appended apoc.import.file.enabled=true to /var/lib/neo4j/conf/apoc.conf
Removing existing setting for apoc.export.file.enabled in /var/lib/neo4j/conf/apoc.conf
Appended apoc.export.file.enabled=true to /var/lib/neo4j/conf/apoc.conf
Setting initial password from environment
Making sure /data/dbms is owned by neo4j:neo4j
Setting initial password
su-exec neo4j:neo4j neo4j-admin dbms set-initial-password ***** --verbose
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000ffff9bd3fc5c, pid=185, tid=189
#
# JRE version:  (21.0.6+7) (build )
# Java VM: OpenJDK 64-Bit Server VM (21.0.6+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# j  java.lang.System.registerNatives()V+0 [email protected]
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/lib/neo4j/hs_err_pid185.log
[0.007s][warning][os] Loading hsdis library failed
#
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/startup/docker-entrypoint.sh: line 300:   185 Aborted                 ${neo4j_admin_cmd} dbms set-initial-password "${password}" "${extra_args[@]}"

I don't think it is relevant, but I ran it with docker compose:

services:
  neo4j:
    image: neo4j:latest
    #image: neo4j:5.26.2
    environment:
      NEO4J_DEBUG: yes
      NEO4J_server_jvm_additional: "-XX:UseSVE=0"
      NEO4J_AUTH: xxx/xxx
      NEO4J_apoc_export_file_enabled: true
      NEO4J_apoc_import_file_enabled: true
      NEO4J_apoc_import_file_use__neo4j__config: true
      NEO4J_PLUGINS: '["apoc"]'
    ports:
      - "7474:7474"  # HTTP
      - "7687:7687"  # Bolt
    networks:
      - app_network
    volumes:
      - neo4j-data:/data
      - neo4j-logs:/logs
      - ./tmp/neo4j-import:/var/lib/neo4j/import
      - neo4j-plugins:/plugins
    restart: unless-stopped
...

[jennyowen]

OK interesting. So what happens if you don't set NEO4J_AUTH?

[tombrus]

then it is still restarting but without the SIGILL:

DEBUGGING ENABLED
Running as root user inside neo4j image
chowning /var/lib/neo4j recursively to neo4j:neo4j
Setting all files in /var/lib/neo4j/conf to permissions 600
Extra plugins were requested. Ensuring the mounted /plugins folder has the required write permissions.
checking /plugins is writable
Warning: Folder mounted to "/plugins" is not writable from inside container. Changing folder owner to neo4j.
checking /plugins is readable
checking /logs is writable
checking /data is writable
checking /data/databases is writable
checking /data/transactions is writable
One or more NEO4J_PLUGINS have been requested.
Checking requested plugins are known and can be installed.
Plugin apoc has been requested
apoc is already in the container at /var/lib/neo4j/labs/apoc-*-core.jar
Installing Plugin 'apoc' from /var/lib/neo4j/labs/apoc-*-core.jar to /plugins/apoc.jar
Applying plugin specific configurations.
Applying default values for plugin apoc to neo4j.conf
apoc requires setting dbms.security.procedures.unrestricted=apoc.*
dbms.security.procedures.unrestricted=apoc.* has been added to the configuration file.
Setting docker specific configuration overrides
Appended server.memory.pagecache.size=512M to /var/lib/neo4j/conf/neo4j.conf
Appended server.default_listen_address=0.0.0.0 to /var/lib/neo4j/conf/neo4j.conf
Applying configuration settings that have been set using environment variables.
server.jvm.additional will be appended to /var/lib/neo4j/conf/neo4j.conf without replacing existing settings.
Appended server.jvm.additional=-XX:UseSVE=0 to /var/lib/neo4j/conf/neo4j.conf
Removing existing setting for server.directories.plugins in /var/lib/neo4j/conf/neo4j.conf
Appended server.directories.plugins=/plugins to /var/lib/neo4j/conf/neo4j.conf
Removing existing setting for server.directories.logs in /var/lib/neo4j/conf/neo4j.conf
Appended server.directories.logs=/logs to /var/lib/neo4j/conf/neo4j.conf
Appended apoc.import.file.use_neo4j_config=true to /var/lib/neo4j/conf/apoc.conf
Appended apoc.import.file.enabled=true to /var/lib/neo4j/conf/apoc.conf
Appended apoc.export.file.enabled=true to /var/lib/neo4j/conf/apoc.conf
Setting initial password from environment
getting full neo4j run command
DEBUGGING ENABLED
Running as root user inside neo4j image
chowning /var/lib/neo4j recursively to neo4j:neo4j
Setting all files in /var/lib/neo4j/conf to permissions 600
Extra plugins were requested. Ensuring the mounted /plugins folder has the required write permissions.
checking /plugins is writable
checking /plugins is readable
...etc

[jennyowen]

Restarting? So Neo4j still isn't starting? What's the error message if there's no SIGILL?

[tombrus]

sorry, this is all the output there is....

...
Setting initial password from environment
getting full neo4j run command
>>> restart of container!!!
DEBUGGING ENABLED
Running as root user inside neo4j image
chowning /var/lib/neo4j recursively to neo4j:neo4j
Setting all files in /var/lib/neo4j/conf to permissions 600
...

[tombrus]

when I set restart to no, the container runs once, the status is:

exited (134)

with output:

DEBUGGING ENABLED
Running as root user inside neo4j image
chowning /var/lib/neo4j recursively to neo4j:neo4j
Setting all files in /var/lib/neo4j/conf to permissions 600
Extra plugins were requested. Ensuring the mounted /plugins folder has the required write permissions.
checking /plugins is writable
Warning: Folder mounted to "/plugins" is not writable from inside container. Changing folder owner to neo4j.
checking /plugins is readable
checking /logs is writable
checking /data is writable
checking /data/databases is writable
checking /data/transactions is writable
One or more NEO4J_PLUGINS have been requested.
Checking requested plugins are known and can be installed.
Plugin apoc has been requested
apoc is already in the container at /var/lib/neo4j/labs/apoc-*-core.jar
Installing Plugin 'apoc' from /var/lib/neo4j/labs/apoc-*-core.jar to /plugins/apoc.jar
Applying plugin specific configurations.
Applying default values for plugin apoc to neo4j.conf
apoc requires setting dbms.security.procedures.unrestricted=apoc.*
dbms.security.procedures.unrestricted=apoc.* has been added to the configuration file.
Setting docker specific configuration overrides
Appended server.memory.pagecache.size=512M to /var/lib/neo4j/conf/neo4j.conf
Appended server.default_listen_address=0.0.0.0 to /var/lib/neo4j/conf/neo4j.conf
Applying configuration settings that have been set using environment variables.
server.jvm.additional will be appended to /var/lib/neo4j/conf/neo4j.conf without replacing existing settings.
Appended server.jvm.additional=-XX:UseSVE=0 to /var/lib/neo4j/conf/neo4j.conf
Removing existing setting for server.directories.plugins in /var/lib/neo4j/conf/neo4j.conf
Appended server.directories.plugins=/plugins to /var/lib/neo4j/conf/neo4j.conf
Removing existing setting for server.directories.logs in /var/lib/neo4j/conf/neo4j.conf
Appended server.directories.logs=/logs to /var/lib/neo4j/conf/neo4j.conf
Appended apoc.import.file.use_neo4j_config=true to /var/lib/neo4j/conf/apoc.conf
Appended apoc.import.file.enabled=true to /var/lib/neo4j/conf/apoc.conf
Appended apoc.export.file.enabled=true to /var/lib/neo4j/conf/apoc.conf
Setting initial password from environment
getting full neo4j run command

[jennyowen]

TLDR: this is a java bug and right now there is no workaround. You'll have to stick to neo4j:5.26 until we can fix it.

OK so given the information here, I think this is that bug in temurin java 21. The problem is that the neo4j docker image interacts with the jdk a few times before actually starting the neo4j database:

1. when setting initial password. (NB if you have an existing database already, setting an initial password is a no-op so it can be skipped).
2. When querying neo4j for the full java command to start neo4j eg here.

There's currently no way for users to apply the workaround flag during either of those two steps. I think the second step doesn't even have any way of adding the extra jvm flag, so that may need a change to neo4j itself. I will need to investigate further.

[tombrus]

@jennyowen thanks for the analysis, I understand the situation, I'll stick to 5.26 for now and hope this will be fixed in java 21 some time in the future.

I see that JDK issue 8345296 you indirectly refer to above is in state FIXED on java 24, I guess/hope it will be back ported to java 21 eventually...

Thanks for looking into this @jennyowen and I hope other AppleSilicon users will find this helpful.