Issue starting the EESSI container

[casparvl]

When trying torun the EESSI container with

mkdir -p $TMPDIR/{var-lib-cvmfs,var-run-cvmfs,home}

export SINGULARITY_BIND="$TMPDIR/var-run-cvmfs:/var/run/cvmfs,$TMPDIR/var-lib-cvmfs:/var/lib/cvmfs"
export SINGULARITY_HOME="$TMPDIR/home:/home/$USER"

export EESSI_CONFIG="container:cvmfs2 cvmfs-config.eessi-hpc.org /cvmfs/cvmfs-config.eessi-hpc.org"
export EESSI_PILOT="container:cvmfs2 pilot.eessi-hpc.org /cvmfs/pilot.eessi-hpc.org"

singularity shell --fusemount "$EESSI_CONFIG" --fusemount "$EESSI_PILOT" docker://eessi/client-pilot:centos7-$(uname -m)-2020.10

I ran into this:

CernVM-FS: loading Fuse module... Failed to initialize shared lru cache (12 - quota init failure)

It seems to suggest a quota issue, but my quota seem fine... After clearing out my $TMPDIR, the container starts fine. I think some previous versions of var-lib-cvmfs,var-run-cvmfs,home were left in the $TMPDIR from a previous mount.

I figured I'd just share the issue - and solution - here in case others run into it. We could consider adding a line in the instructions that clears those dirs from $TMPDIR before creating new ones...

[bedroge]

This usually means that there is some weird issue with your local cache (/var/lib/cvmfs), e.g. a permission issue or some kind of corruption. Could it be that your $TMPDIR had run full in some previous Singularity session, or you had two containers using the same location? I also found some issues in the CVMFS bug tracker where the problem was related to selinux, but if that's the case here, I expect you would have seen the issue before.

Anyway, I guess it would be nice to have some common issues page somewhere in the docs where we describe issues like these (and solutions).