storage/overlay: allow native overlay on Lustre in user namespaces

[tim-day-387]

From the commit message:

Native overlay is rejected on network file system backing stores when running in a user namespace. However, Lustre can potentially support idmapped mounts, which is enough for native overlay to work rootless.

Add an allow list of network file systems with idmapped mounts support, currently just Lustre, and only reject network file systems that are not on it.

Not all version of Lustre will support either native overlayfs or ID mapping, so we still need to probe for support at runtime.

I've tested this on my own Lustre system - and podman will correctly use native overlayfs and ID mapping.

I think the code could be structured better. I'm open to suggestions. isNetworkFileSystem() seems to be a proxy for "this filesystem doesn't support either native overlayfs or ID mapping". In an upcoming release, Lustre will support both native overlayfs and ID mapping. A given version of Lustre may have one, the other, or both. I'd want to fall back to the existing behavior (i.e. emitting a helpful warning) when the support is incomplete. I know podman usually does runtime feature checks for this kind of stuff - but network filesystem seem like an odd edge case where we want to give the user extra guidance. I'm wary of removing Lustre from the network filesystem list outright - most Lustre filesystems in production won't support native overlayfs or ID mapping. So I have this compromise solution. Thoughts?

[giuseppe]

how does it work from a user namespace?

Have you tested it with rootless?

EDIT:

what happens if you run podman run --rm --cap-add DAC_OVERRIDE --user 10:10 fedora touch /usr/sbin/foo?

[tim-day-387]

I tested rootless podman primarily. I've done container builds and run existing container-based development workflows, and everything seems to work.

See:

$ podman run --rm --cap-add DAC_OVERRIDE --user 10:10 fedora touch /usr/sbin/foo
$ echo $?
0

and:

$ podman run --rm --user 10:10 fedora touch /usr/sbin/foo
touch: cannot touch '/usr/sbin/foo': Permission denied
$ echo $?
1

I'm using the latest Lustre development branch with ID map support and overlayfs support patches applied.

[giuseppe]

it seems to work, can we just drop LUSTRE from the list in isNetworkFileSystem and add a comment about that?

[mtrmac]

it seems to work, can we just drop LUSTRE from the list in isNetworkFileSystem and add a comment about that?

Should Lustre still trigger the Network file system detected as backing store. Enforcing overlay option force_mask="%o"` code path? If it should, we do need to separate “is network FS” vs. “supports user namespaces”.

---

Either way I think isIdmapNetworkFileSystem can be inlined into the only caller.

---

A given version of Lustre may have one, the other, or both. I'd want to fall back to the existing behavior (i.e. emitting a helpful warning) when the support is incomplete. I know podman usually does runtime feature checks for this kind of stuff - but network filesystem seem like an odd edge case where we want to give the user extra guidance.

I don’t see how this text relates to the PR. The text advocates for “extra guidance” but in fact that’s not what the PR is doing; if we do need “extra guidance” then we probably should have that (cached) runtime check.

As for whether we need that “extra guidance”, that probably depends on what is the failure mode when running against the older versions which don’t support ID mapping.

[tim-day-387]

We could drop LUSTRE from isNetworkFileSystem() and that'd work fine for a version of Lustre that supports ID map and overlayfs. But it'd result in a behavior change for all previous versions of Lustre. Looking at the code block:

        if opts.mountProgram == "" {
                if supported, err := SupportsNativeOverlay(home, runhome); err != nil {
			return nil, err
                } else if !supported {
                        if path, err := exec.LookPath("fuse-overlayfs"); err == nil {
	                        opts.mountProgram = path
                        }
                }
        }

        if opts.mountProgram != "" {
                if unshare.IsRootless() && isNetworkFileSystem(fsMagic) && opts.forceMask == nil {
                        m := os.FileMode(0o700)
                        opts.forceMask = &m
                        logrus.Warnf("Network file system detected as backing store.  Enforcing overlay option `force_mask=\"%o\"`.  Add it to storage.conf to silence this warning", m)
                }

                if err := os.WriteFile(getMountProgramFlagFile(home), []byte("true"), 0o600); err != nil {
                        return nil, err
                }
        } else {
                // check if they are running over btrfs, aufs, overlay, or ecryptfs
                switch fsMagic {
                case graphdriver.FsMagicAufs, graphdriver.FsMagicOverlay, graphdriver.FsMagicEcryptfs:
                        return nil, fmt.Errorf("'overlay' is not supported over %s, a mount_program is required: %w", backingFs, graphdriver.ErrIncompatibleFS)
                }
                if unshare.IsRootless() && isNetworkFileSystem(fsMagic) {
                        return nil, fmt.Errorf("a network file system with user namespaces is not supported.  Please use a mount_program: %w", graphdriver.ErrIncompatibleFS)
                }
        }

Once a filesystem is removed from isNetworkFileSystem(), you'll never see the "Network file system detected as backing store" or "a network file system with user namespaces is not supported" messages on older filesystem versions. That was the guidance I was thinking about. I don't think we'd want to silence these on old versions of Lustre just because a new version has more features - if the given version of Lustre supports neither overlayfs or ID map, the behavior should be unchanged.

In the first code branch (mountProgram != ""), we're using fuse-overlayfs on a network filesystem. In this case, we want to enforce the fuse-overlayfs option and warn.

In the second code branch, we're using native overlayfs. But we assume that all network filesystems don't support user namespace. And that this remains true across all versions of those filesystems. We could have a check like:

unshare.IsRootless() && isNetworkFileSystem(fsMagic) && !filesystemSupportsIdmap()

but I decided to skip Lustre with isNonIdmapNetworkFileSystem and have podman fall back to it's other checks. In this case, podman should still emit an error if Lustre doesn't support ID map (although I'm not sure if it'd suggest using fuse-overlayfs, I have to double check).