overlay: a metacopy=on probe failure makes the driver unusable, even though metacopy is never supported for unprivileged mounts

[michelemodolo]

Description

On a host where unprivileged overlay mounts are permitted but metacopy=on is not, the
overlay graphdriver refuses to initialise at all, even though it would work perfectly with
metacopy disabled.

This is the normal state of affairs inside an unprivileged user namespace: metacopy records
copy-up metadata in trusted.overlay.* extended attributes, and an unprivileged mount cannot
set trusted.* xattrs, so the kernel rejects metacopy=on. A rootless container engine
therefore can never use metacopy — which the driver already knows how to cope with, since
metacopy is optional and doesMetacopy() returning false is a perfectly good answer.

The bug is that a probe failure is treated as fatal rather than as "unsupported".

Steps to reproduce

Any environment where an unprivileged overlay mount succeeds but metacopy=on is denied. Ours
is a Kubernetes/OpenShift pod under a restricted SCC (SELinux type container_t, all
capabilities dropped at the container level, no subuid/subgid ranges so buildah's own user
namespace maps a single ID):

# storage root must be on a non-overlayfs filesystem (the kernel refuses an overlay
# upperdir on overlay); in our pods only the /builds emptyDir (xfs) qualifies.
buildah --root /builds/store/root --runroot /builds/store/run \
        --storage-driver overlay --storage-opt overlay.ignore_chown_errors=true \
        info

Actual result

Error: failed to mount overlay for metacopy check with "" options: permission denied

The driver is unusable. Every subsequent build fails.

Crucially, the runroot shows that the basic overlay mount had already succeeded:

$ ls /builds/store/run/overlay/
overlay-true

checkAndRecordOverlaySupport() runs before the metacopy check, performs a real overlay
mount, and records overlay-true. Reaching the metacopy error therefore proves that
unprivileged overlay works in this environment — only the metacopy=on variant is refused.

Expected result

doesMetacopy() returning EPERM should be recorded as metacopy unsupported and the
driver should initialise with metacopy off, exactly as it does on hosts where the probe
returns a clean negative.

Analysis

In drivers/overlay/overlay.go, Init():

supportsDType, err = checkAndRecordOverlaySupport(home, runhome)   // real overlay mount
if err != nil {
    return nil, err                                               // different error text
}
feature := fmt.Sprintf("metacopy(%s)", opts.mountOptions)
metacopyCacheResult, _, err := cachedFeatureCheck(runhome, feature)

The first check is fatal-on-failure and produces a distinct message
(kernel does not support overlay fs / 'overlay' is not supported over %s). Because we see
the metacopy message instead, the basic mount succeeded. The metacopy probe's error is then
propagated out of Init() rather than being interpreted.

Evidence that this is a probe-classification problem and not an environment problem: on an
ordinary developer workstation (no SELinux, rootless podman), the same probe yields a clean
negative and the cache file metacopy()-false is written — overlay then works normally, with
metacopy off. The pod differs only in that the refusal surfaces as EPERM rather than as an
unsupported result.

Suggested fix

Treat a permission error from the metacopy probe as false rather than as a hard error —
i.e. record the negative result and continue, in the same way a clean negative is handled.
EPERM from metacopy=on is expected and unremarkable for any unprivileged mount, so it
carries no information beyond "metacopy is not available here".

Workaround (in use)

The probe result is cached in the runroot as a zero-length file named
<feature>-<result>, and cachedFeatureCheck() short-circuits when it is present. Seeding it
before the first buildah call skips the failing probe entirely:

mkdir -p "$RUNROOT/overlay"
: > "$RUNROOT/overlay/metacopy()-false"

The parentheses are literal and the argument is empty because mountOptions is empty; setting
--storage-opt overlay.mountopt=<opts> changes the key to metacopy(<opts>) and the probe
runs again, so that is not an alternative workaround.

This is an undocumented internal and we would rather not depend on it — hence this report.

Version

buildah v1.43.1 (vendoring go.podman.io/storage v1.62.0), compiled from the upstream
release tarball. Node kernel 5.14.0-570.117.1.el9_6 (RHCOS), UBI9 runner image.

Environment

• GitLab Runner Kubernetes executor on OpenShift, restricted SCC.
• Pod: container_t SELinux type (unchanged inside buildah's own user namespace), container
  CapEff 0000000000000000, buildah's namespace single-ID (uid_map: 0 1000 1) with
  /proc/self/setgroups = deny.
• --isolation chroot, --format docker, overlay.ignore_chown_errors=true.
• Storage root on an xfs emptyDir; the container rootfs, /tmp and /var/tmp are all
  overlayfs, so the store must be relocated for overlay to be legal at all.
• /dev/fuse absent, so fuse-overlayfs is not an alternative.

[giuseppe]

doesMetacopy() returning EPERM should be recorded as metacopy unsupported and the driver should initialise with metacopy off, exactly as it does on hosts where the probe returns a clean negative.

have you straced the process to be sure about that error code? If this is the case, we need to just ignore it and return metacopy is not supported.

But from your error message, it looks like EACCES, so that might be related to the mount options difference between the overlay mount and the metacopy check

[michelemodolo]

Thanks @giuseppe — I straced it. Two things: the measured errno, and a correction to my own report.

The errno is EACCES

strace -f -e trace=mount of buildah … info in the failing environment, with the storage root on xfs and no seeded feature-cache marker. Both mounts, verbatim (paths shortened, flags are 0 in both):

mount("overlay", ".../compat<rand>/merged", "overlay", 0,
      "lowerdir=<c>/lower1:<c>/lower2,upperdir=<c>/upper,workdir=<c>/work,userxattr") = 0
mount("overlay", ".../metacopy-check<rand>/merged", "overlay", 0,
      "lowerdir=<m>/l1,upperdir=<m>/l2,workdir=<m>/work,userxattr") = -1 EACCES (Permission denied)

Reproduced identically in two separate runs. You were right that the message implied EACCES; my report said EPERM, which I had inferred from the words "permission denied" rather than measured. Apologies — that is corrected below.

To rule out an instrument artefact, the same harness in the same job returns EPERM(1) for the same mount outside the user namespace, ENOENT(2) for a non-existent lowerdir, and EINVAL(22) for metacopy=on. So it distinguishes the codes.

Correction to my original report

My filing said the probe mount carries metacopy=on, and that the kernel refuses it because metacopy needs trusted.overlay.* xattrs an unprivileged mount cannot set. That was wrong, and I should have read the code before writing it. doesMetacopy() mounts a plain rootless overlay and tests metacopy afterwards with chmod + Lgetxattr; metacopy=on never reaches mount(2) when mountOptions is empty. Please disregard that paragraph of the issue — the failure has nothing to do with the metacopy mount option.

On the mount-options hypothesis — I could not confirm it

You suggested the difference might be between the options of the overlay mount and the metacopy check. From the source the two differ in four ways: two lowerdirs vs one, a conditional ,context="…", a whiteout char device in upper, and dir modes 0700 vs 0755. I tested each.

A small C program issuing the identical mount(2) — same data string, flags 0, same single-UID user namespace, same xfs filesystem, same parent-directory modes, and even inside the same graph-root overlay directory — succeeds. A 14-arm ladder varying one property at a time never reproduced the refusal, except when ,context="…" was added, which returns EACCES. But context= is not in play here. Measured in the same job: the process runs as container_t, /sys/fs/selinux is not mounted inside the container, and the write that SecurityCheckContext() performs fails — so c/storage adds context= to neither mount. The strace above confirms it is absent from both.

So: the two mounts do not differ in any option that matters, and an identical syscall from another process in the same namespace is permitted. Whatever decides it seems to be state in the calling process rather than the mount options, and that is past what I can see from outside. Happy to run anything specific you want measured — the harness is cheap to re-run.

Environment, unchanged from the original report: rootless buildah v1.43.1 (vendoring go.podman.io/storage v1.62.0), Kubernetes pod under a restricted SCC, SELinux type container_t, container CapEff all-zero, no subuid/subgid ranges so the user namespace maps a single ID (uid_map: 0 1000 1, /proc/self/setgroups = deny), kernel 5.14.0-570.117.1.el9_6, --isolation chroot, /dev/fuse absent. checkAndRecordOverlaySupport() succeeds and writes overlay-true before the failure, so plain unprivileged overlay works here.

The fix

Unchanged, and now a one-liner: doesMetacopy() already treats EINVAL as "metacopy not supported". Widening that to the permission errnos is enough:

if errors.Is(err, unix.EINVAL) || errors.Is(err, unix.EACCES) || errors.Is(err, unix.EPERM) {
    logrus.Infof("overlay: metacopy option not supported on this kernel, checked using options %q", mountOpts)
    return false, nil
}

A probe that cannot complete carries no information beyond "metacopy is not available here", and the driver is fully usable without it — which is exactly the state we run in today, by pre-seeding the metacopy()-false feature-cache file to skip the probe. I would rather not depend on an undocumented internal, hence this issue.

Glad to test a patch if you put one up.

[giuseppe]

@nalind are you fine to relax this check?

[nalind]

@giuseppe yes.

[giuseppe]

PR here: #1068

[mtrmac]

You suggested the difference might be between the options of the overlay mount and the metacopy check. From the source the two differ in four ways: two lowerdirs vs one, a conditional ,context="…", a whiteout char device in upper, and dir modes 0700 vs 0755. I tested each.
…
So: the two mounts do not differ in any option that matters

So what is the reason the test mount is being refused? If we are doing nothing special for the test mount, shouldn’t it be expect that the same kinds of rejections could happen during normal operation as well?

[giuseppe]

So what is the reason the test mount is being refused?

we are passing metacopy=on to the mount, the previous mount without metacopy=on worked, so we know we can use overlay. It can fail with EPERM if it is running without CAP_SYS_ADMIN in the initial user namespace and no userxattr specified.

$ sudo podman run --tmpfs /tmp -w /tmp --rm -ti fedora bash -c 'mkdir upper lower workdir merged; unshare -rm mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=workdir,metacopy=off merged && echo ok'
ok

$ sudo podman run --tmpfs /tmp -w /tmp --rm -ti fedora bash -c 'mkdir upper lower workdir merged; unshare -rm mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=workdir,metacopy=on merged && echo ok'
mount: /tmp/merged: permission denied.
       dmesg(1) may have more information after failed mount system call.

$ sudo podman run --cap-add CAP_SYS_ADMIN --tmpfs /tmp -w /tmp --rm -ti fedora bash -c 'mkdir upper lower workdir merged; mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=workdir,metacopy=on merged && echo ok'
ok

[mtrmac]

Except that

• the strace fragment does not show metacopy=on
• the workaround : > "$RUNROOT/overlay/metacopy()-false" suggests mountOptions == ""

I’m fine with the code change, and I believe you it can be clearly beneficial in some environments (… under the current behavior where get silently drops metacopy=on if we detect that it is not supported; it would be plausible to instead reject such configurations).

I’m just worried it might not actually help the reporter. (OTOH the reporter[’s AI] did ask for this exact change…)

[giuseppe]

well my assumption was that the reporter had tested the suggested change, that is why I (blindly) assumed it works.

I believe the strace is not complete though, that is the check for userxattr

[michelemodolo]

Thanks @giuseppe and @mtrmac for your care, review and doubts (doubts are crucially important, always). Yes, I reported this through AI (which I harnessed to thoroughly test claims in our environment before reporting anything) and I purposely left its typical "style" without any humanization so that it's trivially clear that it's an output from AI. From my side the crucial point is that claims are tested before being reported. The proposed fix was inferred from the thorough troubleshooting so it has not been tested at the time of proposal (I'll strengthen the rules so that any proposed fix, whenever possible, must be successfully tested before being proposed, otherwise it should clearly be flagged as untested), yet it was tested a hour ago and it worked.

That said, the AI output is here below (I constrained it a little bit so that the typical AI verbosity is a bit curbed, yet I kept its "direct" style):

---

Below: what was tested, the full trace, and a direct answer to @mtrmac's two doubts.

What was tested

No released buildah carries this yet, and the one that eventually will is newer than 1.45, which I can't run in this environment for an unrelated reason (podman-container-tools/buildah#6910). So I applied the merged #1068 hunk, unchanged, to the vendored storage v1.62.0 copy of drivers/overlay/check.go in buildah v1.43.1 — the version from the original report — and built two binaries from the same tree with the same toolchain, differing only in that hunk. Same pod, same store layout, same invocation. No metacopy()-false marker is seeded in any run below.

binary	result
without #1068	exit 125, failed to mount overlay for metacopy check with "" options: permission denied
with #1068	exit 0, GraphDriverName=overlay, and c/storage records metacopy()-false itself

A second run reuses that cached negative and does not re-probe. A real marker-free build — FROM scratch, two COPY steps, --layers, pushed to docker-archive: — builds and pushes cleanly with the patched binary.

The full trace

Every mount(2) of that build under the patched binary (strace -f -e trace=mount, paths shortened, strace PIDs kept):

494  mount("overlay", "<root>/compat<rand>/merged", "overlay", 0,
       "lowerdir=<c>/lower1:<c>/lower2,upperdir=<c>/upper,workdir=<c>/work,userxattr") = 0
494  mount("overlay", "<root>/metacopy-check<rand>/merged", "overlay", 0,
       "lowerdir=<m>/l1,upperdir=<m>/l2,workdir=<m>/work,userxattr") = -1 EACCES (Permission denied)
494  mount("overlay", "<root>/opaque-bug-check<rand>/merged", "overlay", 0,
       "lowerdir=<o>/l2:<o>/l1,upperdir=<o>/l3,workdir=<o>/work,userxattr") = -1 EACCES (Permission denied)
503  mount("overlay", "<root>/volatile-check<rand>/merged", "overlay", 0,
       "volatile,lowerdir=<v>/lower,upperdir=<v>/upper,workdir=<v>/work,userxattr") = -1 EACCES (Permission denied)
503  mount("overlay", "<root>/<layer1>/merged", "overlay", 0,
       "lowerdir=<layer1>/empty,upperdir=<layer1>/diff,workdir=<layer1>/work,userxattr") = 0
509  mount("overlay", "<root>/<layer2>/merged", "overlay", 0,
       "lowerdir=l/<link>,upperdir=<layer2>/diff,workdir=<layer2>/work,userxattr") = 0
509  mount("overlay", "<root>/<layer3>/merged", "overlay", MS_RDONLY,
       "lowerdir=<layer3>/diff:<layer3>/empty,userxattr") = 0

@giuseppe — on your two points: there is no metacopy=on anywhere, because mountOptions is empty here, so doesMetacopy() appends only userxattr; and userxattr is not a separate check, it is the option doesSupportNativeDiff, doesMetacopy and doesVolatile each append when rootless. The earlier two-mount trace was complete for that run — the later probes appear only once Init() gets past the metacopy error, which before #1068 it never did.

@mtrmac — your two doubts, directly

"I'm just worried it might not actually help the reporter."
It does, and that is the A/B above: same pod, same store, same command, two binaries differing only in this hunk. Without it the driver cannot initialise at all (exit 125). With it the driver opens on overlay, c/storage records the negative itself, and a full build completes and pushes. Nothing was seeded to make that happen.

"If we are doing nothing special for the test mount, shouldn't it be expect that the same kinds of rejections could happen during normal operation as well?"
The measurement says: the same rejection does happen elsewhere — and it has never affected normal operation, because two of the three places it happens are already non-fatal upstream. Concretely, in the seven-mount build above:

• 3 refusals, all of them feature probes: doesMetacopy, doesSupportNativeDiff, doesVolatile — the same EACCES, same shape.
• 0 refusals among the operational mounts: all three layer mounts succeed, and the build and push complete.
• Of those three probes, doesSupportNativeDiff and doesVolatile already treat the failure as "unsupported" (useNaiveDiffOnly = true; usingVolatile = false). doesMetacopy was the only one that returned the error to Init().

So what you describe is already the mechanism here, not a future risk. The same refusal hits the two sibling probes in the run above, and because both are already non-fatal it could not have surfaced anywhere before — a trace like this is the first place it could be seen. #1068 therefore isn't suppressing a signal that predicts a runtime failure: in this very build the same signal is already suppressed twice, in the same code path, and the build still completes. It makes the third case consistent with the other two.

If it had been the other way round — if the refusal reached the operational mounts — the build in the table above could not have completed, with or without the patch.

What I still can't explain

Why those three probes are refused while compat — also a probe, also a fresh temp dir under the same parent — and every operational mount are granted. It is not a clean "probe vs. real work" split, and I don't want to imply it is. Two constraints are measured:

• Not the mount options. compat and opaque-bug-check have the same shape — two lowerdirs, upperdir, workdir, userxattr, flags 0 — and opposite outcomes.
• Not a latched process state. PID 503 is refused volatile-check and then immediately granted a layer mount.

So it is decided per mount, on something not visible in the mount arguments. I would rather not guess again — the last time I did, the guess was wrong.

Same cluster and pod configuration as the original report: rootless buildah in a Kubernetes pod under a restricted SCC, SELinux container_t, container CapEff all-zero, no subuid/subgid ranges so the user namespace maps a single ID, --isolation chroot, no /dev/fuse, store on xfs.