EESSI container script hangs

[stevenvdb]

I am following the instructions at https://www.eessi.io/docs/getting_access/eessi_container/#eessi-container-script to launch an EESSI container, but the eessi_container.sh script hangs for hours:

vsc33716@tier2-p-login-3:software-layer$ ./eessi_container.sh
Using /tmp/eessi.n8TF2l4Hve as tmp directory (to resume session add '--resume /tmp/eessi.n8TF2l4Hve').
Pulling container image from docker://ghcr.io/eessi/build-node:debian11 to /tmp/eessi.n8TF2l4Hve/ghcr.io_eessi_build_node_debian11.sif
Mounting 'software.eessi.io' 'read-only' without fuse-overlayfs.
Launching container with command (next line):
singularity -q shell  --fusemount container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch --fusemount container:cvmfs2 software.eessi.io /cvmfs/software.eessi.io /tmp/eessi.n8TF2l4Hve/ghcr.io_eessi_build_node_debian11.sif

Note that CernVM-FS is not installed on this system, but if I understand correctly that should not be necessary for the container approach? Below is some more information about the system that might be relevant:

$ apptainer --version
apptainer version 1.3.2-1
$ uname -a
Linux tier2-p-login-3 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Apr 4 18:13:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
$ fusermount -V
fusermount3 version: 3.3.0

Any ideas what could be the reason for this? Any suggestions to debug this problem, because I don't know how to dig into this.

[boegel]

@stevenvdb Can you try running with the --verbose option, which should produce a bit more information?

[stevenvdb]

vsc33716@tier2-p-login-3:software-layer$ ./eessi_container.sh --verbose
checking for duplicates: 'software.eessi.io,access=ro' and 'software.eessi.io'
Using /tmp/eessi.oXWyJFQEsD as tmp directory (to resume session add '--resume /tmp/eessi.oXWyJFQEsD').
EESSI_TMPDIR=/tmp/eessi.oXWyJFQEsD
HOST_INJECTIONS=/tmp/eessi.oXWyJFQEsD/opt-eessi
SINGULARITY_CACHEDIR=/tmp/eessi.oXWyJFQEsD/singularity_cache
Pulling container image from docker://ghcr.io/eessi/build-node:debian11 to /tmp/eessi.oXWyJFQEsD/ghcr.io_eessi_build_node_debian11.sif
INFO:    Environment variable SINGULARITY_CACHEDIR is set, but APPTAINER_CACHEDIR is preferred
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Copying blob 2c15b1e3e597 done   |
Copying blob f2f58072e9ed done   |
Copying blob 56b3ce5371b3 done   |
Copying blob 6d8db37105cd done   |
Copying blob bb9ca937b24a done   |
Copying blob 96bf55fca17b done   |
Copying blob fc15bab23de9 done   |
Copying blob 2befde2165ea done   |
Copying blob 3c80c851a745 done   |
Copying blob 20857f6099d0 done   |
Copying blob 3f8c525b84f6 done   |
Copying config 4f67a1dc7a done   |
Writing manifest to image destination
2025/06/06 11:13:45  info unpack layer: sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698
2025/06/06 11:13:47  info unpack layer: sha256:96bf55fca17b14e99dfdaa3991a86d85d63509355f2e59cc36664ac49c96a4b6
2025/06/06 11:13:47  info unpack layer: sha256:bb9ca937b24a8cbf8f11e2332ec723bb11147768be4d6993f8edd27a7ed58539
2025/06/06 11:13:47  info unpack layer: sha256:56b3ce5371b38e2a8a2b5e2d303e12e11859f4e14ed4d9cbd6e22897ff0c4b14
2025/06/06 11:13:50  info unpack layer: sha256:6d8db37105cd700bc435faa909e26da46f1e527c7ea4cea700ab5e00ab7a5947
2025/06/06 11:13:53  info unpack layer: sha256:2c15b1e3e597e96da7c24bf102a66a43a5f38d339da8d4ad2592c570fed833c7
2025/06/06 11:13:55  info unpack layer: sha256:fc15bab23de97732242aeb79c61bc1602055a21238ddbe752923004da8c38fd3
2025/06/06 11:13:55  info unpack layer: sha256:2befde2165ea1ae08b26814a60fffb468acb2af9be5811e71922b8298399f731
2025/06/06 11:13:55  info unpack layer: sha256:3c80c851a745b18cf2c30378f5fd4edae8f5a8f18eb40107190a4b0525729c23
2025/06/06 11:13:55  info unpack layer: sha256:20857f6099d0ae25ff0ea1b0365aff7cb762d4d2dfb3d16b103036cae67d8759
2025/06/06 11:13:55  info unpack layer: sha256:3f8c525b84f65ae8fa56c6bb2f12e94bf8d0fc6e525da351c2681a01d666e0d6
INFO:    Creating SIF file...
CONTAINER=/tmp/eessi.oXWyJFQEsD/ghcr.io_eessi_build_node_debian11.sif
EESSI_CVMFS_VAR_LIB=/tmp/eessi.oXWyJFQEsD/var-lib-cvmfs
EESSI_CVMFS_VAR_RUN=/tmp/eessi.oXWyJFQEsD/var-run-cvmfs
SINGULARITY_HOME=/tmp/eessi.oXWyJFQEsD/home:/home/vsc33716
BIND_PATHS=/tmp/eessi.oXWyJFQEsD/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessi.oXWyJFQEsD/var-run-cvmfs:/var/run/cvmfs,/tmp/eessi.oXWyJFQEsD/opt-eessi:/opt/eessi,/tmp/eessi.oXWyJFQEsD:/tmp

BIND_PATHS before processing REPOSITORIES
  BIND_PATHS=/tmp/eessi.oXWyJFQEsD/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessi.oXWyJFQEsD/var-run-cvmfs:/var/run/cvmfs,/tmp/eessi.oXWyJFQEsD/opt-eessi:/opt/eessi,/tmp/eessi.oXWyJFQEsD:/tmp

process CVMFS repo spec 'software.eessi.io,access=ro'
BIND_PATHS after processing 'software.eessi.io,access=ro'
  BIND_PATHS=/tmp/eessi.oXWyJFQEsD/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessi.oXWyJFQEsD/var-run-cvmfs:/var/run/cvmfs,/tmp/eessi.oXWyJFQEsD/opt-eessi:/opt/eessi,/tmp/eessi.oXWyJFQEsD:/tmp

add fusemount options for CVMFS repo 'software.eessi.io,access=ro'
Mounting 'software.eessi.io' 'read-only' without fuse-overlayfs.
SINGULARITY_BIND=/tmp/eessi.oXWyJFQEsD/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessi.oXWyJFQEsD/var-run-cvmfs:/var/run/cvmfs,/tmp/eessi.oXWyJFQEsD/opt-eessi:/opt/eessi,/tmp/eessi.oXWyJFQEsD:/tmp
Launching container with command (next line):
singularity  shell  --fusemount container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch --fusemount container:cvmfs2 software.eessi.io /cvmfs/software.eessi.io /tmp/eessi.oXWyJFQEsD/ghcr.io_eessi_build_node_debian11.sif
INFO:    Environment variable SINGULARITY_BIND is set, but APPTAINER_BIND is preferred
INFO:    Environment variable SINGULARITY_HOME is set, but APPTAINER_HOME is preferred

[boegel]

Maybe you're not correctly seeing a shell prompt appear?
Can you try running a command, like ls /cvmfs/software.eessi.io, even though it still seems to "hang"?

Can you share if there's any activity (output of ps xfu)?

[stevenvdb]

Maybe you're not correctly seeing a shell prompt appear? Can you try running a command, like ls /cvmfs/software.eessi.io, even though it still seems to "hang"?

Can you share if there's any activity (output of ps xfu)?

There is no shell prompt, it is really hanging. There is nothing happening in the apptainer process.

If I run the singularity command that eessi_container.sh executes in debug mode, I see:

$ singularity -d  shell  --fusemount 'container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch' \
    --fusemount 'container:cvmfs2 software.eessi.io /cvmfs/software.eessi.io' \
    /tmp/eessi.5S2XQvl7Kb/ghcr.io_eessi_build_node_debian11.sif
DEBUG   [U=2533716,P=1440994]setValue()                    Updated flag 'tmpdir' value to: /scratch/leuven/337/vsc33716/tmp
DEBUG   [U=2533716,P=1440994]persistentPreRun()            Apptainer version: 1.3.2-1
...
VERBOSE [U=2533716,P=1440994]addCwdMount()                 /tmp/eessi.5S2XQvl7Kb found within container
DEBUG   [U=2533716,P=1440994]mountGeneric()                Mounting /var/lib/apptainer/mnt/session/fuse/0 to /var/lib/apptainer/mnt/session/final/cvmfs/cvmfs-config.cern.ch

and there it hangs. This indicates it is the same issue as apptainer/apptainer#2545 and indeed we have auditd rules enabled on the system where I am trying to run this.

I am not sure if anything needs to change on the EESSI side; if I am correct, this is an apptainer bug for which there is no solution at the moment.

[trz42]

Might not help, but could you try setting EESSI_DO_NOT_MOUNT_CVMFS_CONFIG_CERN_CH=1 and then run eessi_container.sh