Expose standalone read-only erofs image mounts or mount specs for container snapshotters

[mike-sul]

I am working on a containerd snapshotter using composefs and found a gap in the cfsctl mount API, both CLI and varlink.

IIUC, the current API creates a complete composefs mount: an overlayfs mount combining the erofs image with the repo objects directory. For writable mounts, the caller provides upperdir and workdir, while composefs-rs assembles the lower side.

A containerd snapshotter needs one level lower: a standalone, read-only erofs mount of the composefs image, without mounting the objects directory or creating the final overlayfs mount.

The snapshotter needs this because:

1. It must assemble the complete overlay lower stack itself, including committed snapshots, Docker’s init layer, or other runtime-specific lowers.

2. It normally returns mount specifications, while the mounts are performed later by the containerd engine in the target mount namespace.

3. The image-level erofs mount should be reusable across multiple per-container overlays.

container-libs follows a similar model: it mounts the composefs erofs blob separately and later includes it in the final overlay.

The containerd erofs snapshotter also follows similar pattern: it returns erofs mount specs followed by the final overlay mount spec. Also, this flow is described in the snapshotter's doc.

I see two possible API shapes:

1. Open and verify the composefs image and return a detached, read-only erofs mount of the image only.
2. Return the resolved and verified image information needed to construct a containerd erofs mount spec, including the image source and verity requirements.

Since the varlink API is being stabilized in #322, it may be worth considering this request there as well.

[mike-sul]

@hsiangkao, I would appreciate your view on the containerd side of this.

My assumption is that a composefs snapshotter returns a final overlay mount spec whose lowers include:

1. The composefs erofs metadata image.
2. The objects directory as a :: data-only lower.
3. Any additional snapshot lowers, such as committed uppers or Docker’s init layer.

What I am less sure about is how the erofs metadata image should be handled:

• Should the snapshotter return an [erofs mount spec](https://github.com/containerd/containerd/blob/b6aca07dd50288a598baadab38dcc8656c5e5327/plugins/snapshots/erofs/erofs.go#L331-L345), as the erofs snapshotter does, and let containerd perform the mount?
• Or should the snapshotter mount it once per image and reuse that mount across per-container overlays, since the metadata lower is identical for all containers using the same image?

This also affects the composefs API: the first approach only requires verified image and objects paths, while the second requires an image-level mount primitive.

[hsiangkao]

@mike-sul , I really cannot represent containerd but according to containerd in-tree practice, snapshotters shouldn't take care of mount management (and that is what mount manager was introduced and used.)
However, since you're working on a remote snapshotter, I don't think it does any matter to you since basically those snapshotters can do whatever they do as long as the functionality can work, and the existing remote snapshotters already work as is.

[mike-sul]

@hsiangkao I understand a remote snapshotter can do whatever it wants, the constraint isn't who mounts, it's the shape of the mount spec the snapshotter returns: the engine mounts the container rootfs itself, as an overlay whose lowers the snapshotter assembles (committed snapshots, Docker's init layer, etc).

A composefs mount (what cfsctl mount produces) is itself an overlayfs (erofs metadata + data-only objects lower). Passing such a mountpoint as a lowerdir makes the rootfs mount overlay-over-overlay, which hits FILESYSTEM_MAX_STACK_DEPTH = 2 (file-backed erofs on ≥ 6.12 costs a level too).

In my testing it fails with overlayfs: maximum fs stacking depth exceeded on 6.17 regardless of mount options, and happens to work on 6.19. Even where it mounts, it uses the last stacking level, and when the snapshotter's directories themselves live on overlayfs (e.g. containerd running in a container), the extra level makes it fail even on kernels where the bare-host case works.

What works is the same shape the erofs snapshotter and containers/storage use: mount only the erofs metadata read-only (once per image) and return the mount spec for one flat overlay: lowerdir=<committed uppers>:<erofs meta>::<objects>, verity=require.

Hence the ask in this issue: cfsctl (and its varlink API) should either:

1. provide that metadata-only mount, or
2. expose and lock the resolved, verified image so the snapshotter can do the erofs mount itself on the locked image (this GC can delete images that are currently mounted #346 is kind of related to it).

[cgwalters]

What works is the same shape the erofs snapshotter and containers/storage use: mount only the erofs metadata read-only (once per image) and return the mount spec for one flat overlay: lowerdir=:::, verity=require.

This seems fine to me. That said, we added support for specifying the overlayfs upper, which I think is probably enough for many use cases (and is the cause of the overlay-on-overlay failure).

[mike-sul]

What works is the same shape the erofs snapshotter and containers/storage use: mount only the erofs metadata read-only (once per image) and return the mount spec for one flat overlay: lowerdir=:::, verity=require.

This seems fine to me. That said, we added support for specifying the overlayfs upper, which I think is probably enough for many use cases (and is the cause of the overlay-on-overlay failure).

Yep, with the upper handed to cfsctl the result is a single overlay, so the stacking problem disappears. That covers clients that need "image + one writable layer".

However, the containerd snapshotter API implies the snapshotter does not perform the container rootfs mount itself: the Prepare() call returns []mount.Mount, i.e. the mount specs that the containerd engine later uses to do the actual mount (the same true for View() and Mounts() API methods).

In addition, when the ctrd's snapshotter is used through docker (the containerd storage backend is default is docker since v29), there is an auxiliary "docker-init" layer between the image rootfs and the container's writable upper, so the final overlay is container upper over the docker-init upper over cfs image: the docker-init layer enters as an extra lowerdir, which "image + one upper" can't express.

FWIW, it looks like the functionality behind option 1 already exists internally, see [mount::erofs_mount()] (

composefs-rs/crates/composefs/src/mount.rs

Line 116 in 35a59a4

pub fn erofs_mount(image: OwnedFd) -> Result<OwnedFd> {

), it returns exactly the read-only erofs mount afaiu.