Squashfuse by simonpintarelli · Pull Request #164 · eth-cscs/uenv

simonpintarelli · 2026-06-11T12:27:47Z

add mutable root option (following strace -f bwrap)

make --sqfs optional (required for stackinator sandbox)
add --bind-mount, --tmpfs
if --mutable-root is given, non-existent mount points will be created. It's not checked if they reside in the mutable root, thus allows also to created directories in home.

unshare mnt ns, become fake-root
optional create mutable root (disable for setuid)
mount tmpfs
bind mounts
mount sqfs images
exit fake-root

`--join` flag

If --join is passed to uenv run, only the first task per node will setup the namespace and mount the squashfs image (fuse). The remaining tasks will join the namespace (user+mnt) of the first task. It uses the same construction as ch-run --join from charliecloud.

SPANK plugin with fuse

There are two options.

call squashfs-mount after fork via exec and then join the namespace
directly call the squashfuse_ll routines from the spank plugin

Option 2. requires to call prctl(PR_SET_DUMPABLE,1), which has been previously set to zero by the kernel (when slurm switched to the unprivildged user). This might have security implications.
When PR_SET_DUMPABLE is 0, the fakeroot sandbox cannot be created. Furthermore the fuse mount failed too.

Installation

# enable rootless (sqashsufse for squashfs-mount, slurm plugin: root)
meson setup --buildtype=debug -Dsquashfs_mount=true -Dfuse=true -Dfuse_version=3 -Dtests=enabled --wipe builddir

make sure to use -Dfuse_version2 if squashfuse has been built against fuse2. Afaik it will always build against fuse3 if it's present on the system (no matter what flags are given to configure). Mixing fuse versions will cause runtime errors.

bcumming · 2026-06-30T10:42:07Z

@@ -0,0 +1,9 @@
+root = true


How does this interact with clang format? Because we have clang format, most editors should automatically apply the formatting rules on save.

It's redundant. It was just to avoid the nuisance of 2 space indent (my default) when entering a new line.

bcumming · 2026-06-30T10:48:57Z

@@ -1,11 +1,60 @@
 #!/bin/bash


I was thinking about removing this completely, because it is not up to date.

There is a meson wrap db for zstd: https://mensinda.github.io/meson/Wrapdb-projects.html

bcumming · 2026-06-30T10:52:28Z

+
+    if [ ! -d "squashfuse-${squashfuse_version}/install/lib/pkgconfig" ]; then
+        (
+            curl -L https://github.com/vasi/squashfuse/releases/download/${squashfuse_version}/squashfuse-${squashfuse_version}.tar.gz | tar xzf -


Should we be trying to use squashfuse available on Alps systems? The container team install a version from RPM.

squashfuse on alps is 0.5.1. The latest release is 0.6.2.

It is linked to fuse2:

[daint][simonpi@daint-ln001 ~]$ ldd /usr/lib64/libsquashfuse_ll.so.0 linux-vdso.so.1 (0x000040001c560000) libz.so.1 => /usr/lib64/libz.so.1 (0x000040001c5b0000) liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x000040001c5f0000) liblzo2.so.2 => /usr/lib64/liblzo2.so.2 (0x000040001c650000) liblz4.so.1 => /usr/lib64/liblz4.so.1 (0x000040001c690000) libzstd.so.1 => /usr/lib64/libzstd.so.1 (0x000040001c6e0000) libfuse.so.2 => /lib64/libfuse.so.2 (0x000040001c7b0000) libc.so.6 => /lib64/libc.so.6 (0x000040001c810000) /lib/ld-linux-aarch64.so.1 (0x000040001c4f0000) libdl.so.2 => /lib64/libdl.so.2 (0x000040001c9d0000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000040001ca00000) `` Afaik the multithreaded version requires fuse3.

bcumming · 2026-06-30T14:39:47Z

+
+dependencies = [uenv_dep, slurm_dep]
+if slurm_fuse
+    module_src += ['../uenv/rootless.cpp', '../uenv/mutable_root.cpp', 'plugin_fuse.cpp']


The files in src/uenv/ should be part of the uenv dependency, defined in the meson.build file in the root of the project.

bcumming · 2026-06-30T14:41:17Z

+    if get_option('fuse')
+        squashfs_mount_build = executable('squashfs-mount',
+                                          sources: ['src/squashfs-mount/squashfs-mount-rootless.cpp',
+                                                    'src/uenv/rootless.cpp',


The rootless.cpp and mutable_root.cpp belong in the uenv_dep, I think.

bcumming · 2026-06-30T15:18:06Z

+        winner_pid = fork();
+        if (winner_pid.value() == 0) {
+            prctl(PR_SET_PDEATHSIG, SIGHUP);
+            execlp("squashfs-mount", "squashfs-mount", "--sqfs",


Good to see that this works!

How do you feel about implementing the logic locally in the plugin, instead of calling out to squashfs-mount.

It is a little bit confusing (for me at least) to keep track of things.

What happens if squashfs-mount is setuid, and the slurm plugin is using squashfuse... does it still work?

Exiting the fakeroot environment (map_effective_user) fails with

error: failed map effective user unshare(CLONE_NEWUSER|CLONE_NEWNS) failed, Operation not permitted 1000 1000

when called inside slurm_spank_task_init. Also the call to sqfs_ll mount fails with an error. It's not clear why.

~~Instead of calling sqfs_mount in the forked process, one can still try to call squashfuse_ll directly in the fork.~~ Actually, the squashfuse_ll already are in a forked process.

Attaching strace to slurmd revealed that PR_SET_DUMPABLE was set to zero by slurm. When adding prctl(PR_SET_DUMPABLE,1) right after the first unshare call, the fakeroot works and the sqfs_ll calls work as well. I'll change the code back to the squashfuse_ll version.

bcumming · 2026-06-30T15:19:22Z

+                        uenv::mount_list mounts) {
+
+    const uid_t uid = getuid();
+    const uid_t gid = getgid();


These should always work as expected if squashfs-mount is called from within the slurm plugin?

I am guessing yes, because they are called in task_init, which runs as user.

bcumming · 2026-06-30T15:24:23Z

+    }
+
+    // tmps
+    auto tmpfs = uenv::parse_tmpfs(tmpfs_str);


The value of tmpfs_str is parsed/validated after operations like unshare_mount_map_root have been performed.

How about validating all inputs as close to the parsing of the CLI args as possible, and passing pre-validated and parsed values into this function?

bcumming · 2026-06-30T15:33:22Z

+    bool tasks_join = false;
+    std::string raw_mounts;
+    std::vector<std::string> commands;
+    std::vector<std::string> tmpfs_str;


For consistency with how we parse lists of mount points here and in other CLI tools, tmpfs_str and bind_mounts_str could be:

std::optional<std::string> tmpfs_arg; std::optional<std::string> bind_mounts_arg;

Where the argument is a comma-separated list.

The parse calls would take a string instead of vector, and return a std::vector, like parse_mount_list.

You can also then test on the optional value to determine whether the user passed a specific flag.

bcumming · 2026-06-30T15:35:34Z

    meson.add_install_script('./meson-scripts/install-bash-completion.sh')
 endif

+if get_option('fuse') or get_option('slurm_fuse')


The decision of whether to use fuse is in two variables fuse and slurm_fuse.

A single variable fuse would be clearer.

simonpintarelli added 8 commits June 10, 2026 22:39

add rootless version of squashfs-mount

6e267c1

add --priviledged flag for the container

fe8947f

move rootless code to src/uenv

ae84494

fuse mount in the spank plugin

67e6746

fix meson.build

1aae9eb

update

18983ed

fix cli

a984c11

error msg and format

19629f9

simonpintarelli mentioned this pull request Jun 11, 2026

WIP squashfs-mount rootless, mutable root, bind mount #98

Closed

simonpintarelli added 20 commits June 11, 2026 14:56

format

46ffb7f

format

7178b1a

add missing version flag

0dd727b

install shared squashfuse

1fac641

error handling

733e279

WIP: add --join

d4c0653

update

d18cddf

cleanup comment

098fedc

forward join to squashfs-mount via uenv cli

4d6ae4d

option -> flag

e48910c

forward uenv -vvv to squashfs-mount call

214a7b1

update install script

a80770c

add missing pkg

3be8b76

CI: add sudo

e022571

update meson

6c9aca1

add jq to container

8ce0b18

use fork + ns join in spank plugin

56aa7cf

exclude forked squashfs-mount process from join_begin/end

c98aac2

rename variable

8e39781

comments / rename variable

29bd93c

simonpintarelli force-pushed the squashfuse branch from 084dd6b to 29bd93c Compare June 30, 2026 13:02

simonpintarelli added 2 commits June 30, 2026 15:26

remove #ifdef

d4851ef

cleanup includes

4ad79fe

simonpintarelli force-pushed the squashfuse branch from 0447c1f to 4ad79fe Compare June 30, 2026 13:28

simonpintarelli added 3 commits June 30, 2026 15:31

remove ifdef

d106dd0

remove unused header

5e8168a

std::optional for winner_pid

3454617

bcumming requested changes Jun 30, 2026

View reviewed changes

simonpintarelli added 9 commits July 1, 2026 12:05

typo

557d959

add direct sqfs_ll version in plugin, cleanup

73277b6

tmpfs_str -> tmpfs_arg, bind_mounts_str -> bind_mounts_arg

9c03ad1

update docstrings

a7fc8da

cleanup

1d3dce1

prtcl PR_SET_DUMPABLE

d78fd57

sanity check for write/close/openat

b15d33a

refactoring

701d1c4

fix ns_join for setuid

7d9a7bf

simonpintarelli force-pushed the squashfuse branch from ce676bc to 7d9a7bf Compare July 8, 2026 07:47

simonpintarelli added 9 commits July 9, 2026 07:57

cleanup

6afed54

cleanup make_mutable_root

dec2487

format

f1b2363

fuse plugin: missing args

c068ee2

skip remount of proc/sys/tmp

4ccb29e

don't skip squashfs-mount -r

2a25b17

disable mutable_root sandbox for setuid

4aff578

add comments

ed3f159

remove some ifdefs

af63c8a

Uh oh!

Conversation

simonpintarelli commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

--join flag

SPANK plugin with fuse

Installation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonpintarelli Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simonpintarelli commented Jun 11, 2026 •

edited

Loading

`--join` flag

simonpintarelli Jul 1, 2026 •

edited

Loading