macos: podman fails to mount folders from another partition: statfs ... no such file or directory

[ssbarnea]

Issue Description

$ podman version
Client:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.23.0
Git Commit:   fcee48106a12dd531702d729d17f40f6e152027f
Built:        Wed Aug 21 18:43:11 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.6
Built:        Wed Aug 21 01:00:00 2024
OS/Arch:      linux/arm64

Steps to reproduce the issue

Steps to reproduce the issue

1. install podman
2. create a partition (case-sensitive), change current directory to it
3. try to mount it with podman run

Describe the results you received

$ podman run -it -v $PWD:$PWD bash  bash
Resolving "bash" using unqualified-search registries (/etc/containers/registries.conf.d/999-podman-machine.conf)
Trying to pull docker.io/library/bash:latest...
Getting image source signatures
Copying blob sha256:c8f7b110368c306b89b39c4c6d5c4d5d5ab978a935627bc6ac47f582ffbd0165
Copying blob sha256:cf04c63912e16506c4413937c7f4579018e4bb25c272d989789cfba77b12f951
Copying blob sha256:f94f09caab421436260f6f97d02ff2d2651bbca0f4dcc7fd3c020cffd1b7cefa
Copying config sha256:3ccbdbc2e7a42d8cf9e1ed614f7b31d9a9bcc16fe80e15d0f3dbf6bfdd6089e8
Writing manifest to image destination
Error: statfs /Users/ssbarnea/code/a/vscode-ansible: no such file or directory

Describe the results you expected

Podman should be able to mount local partitions.

podman info output

$  podman info
host:
  arch: arm64
  buildahVersion: 1.37.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 98.59
    systemPercent: 0.71
    userPercent: 0.7
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2036
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.9.12-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 129171456
  memTotal: 2044416000
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.20240819115418474394.main.6.gc2cd0be.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.0-dev
    package: netavark-1.12.1-1.20240819170533312370.main.26.g4358fd3.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0-dev
  ociRuntime:
    name: crun
    package: crun-1.16-1.20240813143753154884.main.16.g26c7687.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 158b340ec38e187abee05cbf3f27b40be2b564d0
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240726.g57a21d2-1.fc40.aarch64
    version: |
      pasta 0^20240726.g57a21d2-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 11m 37.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 6093688832
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724198400
  BuiltTime: Wed Aug 21 01:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/arm64
  Version: 5.2.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

$ uname -a
Darwin m1 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6000 arm64

Additional information

It should be noted that the partition that I am an APFS Volume created with case-sensitive switch on, because that is recommended for keeping code that works on linux (default APFS is case insensitive).

I am not sure yet if that case-insensitive is the cause for failure or any kind of partition would trigger the same bug.

Initially I believed it was the fact that this partition is symlinked to ~/code location and that this might confuse PWD, but I tried resolving the path and the issue is still present.

statfs reports it as not being able to find it.

I also destroyed and recreated the podman machine as part of debugging, no change in behavior. Out of curiosity I also tested docker behavior, but docker does not seem to have any problem mounting it.

[ssbarnea]

That was the first thing I tried once i discovered the issue. Yep, it does not mount:

$ podman --log-level=debug run -it -v $PWD:$PWD bash bash INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run -it -v /Volumes/code/p:/Volumes/code/p bash bash) DEBU[0000] SSH Ident Key "/Users/ssbarnea/.config/containers/podman/machine/machine" SHA256:mPYRAtUq5IqO9+5xiW1+SX4xKCY+gKT+zM9Lx67C9h0 ssh-ed25519 DEBU[0000] DoRequest Method: GET URI: http://d/v5.2.2/libpod/_ping DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Found credentials for brew.registry.redhat.io in credential helper containers-auth.json in file /Users/ssbarnea/.config/containers/auth.json DEBU[0000] Found credentials for registry.redhat.io in credential helper containers-auth.json in file /Users/ssbarnea/.config/containers/auth.json DEBU[0000] DoRequest Method: POST URI: http://d/v5.2.2/libpod/images/pull DEBU[0000] User or group ID mappings not available: open /proc/self/uid_map: no such file or directory DEBU[0000] User or group ID mappings not available: open /proc/self/uid_map: no such file or directory DEBU[0000] User mount /Volumes/code/p:/Volumes/code/p options [] DEBU[0000] DoRequest Method: GET URI: http://d/v5.2.2/libpod/images/bash/json DEBU[0000] DoRequest Method: POST URI: http://d/v5.2.2/libpod/containers/create Error: statfs /Volumes/code/p: no such file or directory DEBU[0000] Shutting down engines FAIL: 125

[ssbarnea]

Based on my research it seems that by default podman machine is not created with volume mounts needed for properly mount code from macos.

$ podman machine list
NAME                     VM TYPE     CREATED      LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  applehv     11 days ago  Currently running  5           2GiB        100GiB

ssbarnea@m1: ~/other/ansible-dev-tools feat/devspaces
$ podman machine inspect
[
     {
          "ConfigDir": {
               "Path": "/Users/ssbarnea/.config/containers/podman/machine/applehv"
          },
          "ConnectionInfo": {
               "PodmanSocket": {
                    "Path": "/var/folders/32/1xrphgzd4xv777syxjtkpdw80000gn/T/podman/podman-machine-default-api.sock"
               },
               "PodmanPipe": null
          },
          "Created": "2024-09-16T14:44:13.466193+01:00",
          "LastUp": "2024-09-26T17:11:06.325768+01:00",
          "Name": "podman-machine-default",
          "Resources": {
               "CPUs": 5,
               "DiskSize": 100,
               "Memory": 2048,
               "USBs": []
          },
          "SSHConfig": {
               "IdentityPath": "/Users/ssbarnea/.config/containers/podman/machine/machine",
               "Port": 62790,
               "RemoteUsername": "core"
          },
          "State": "running",
          "UserModeNetworking": true,
          "Rootful": false,
          "Rosetta": true
     }
]

I am going to follow their recommendation and test the podman machine init --volume /Users --volume /Volumes which may seem as generic approach. If it works we should really consider making this implicit on macos. As I explained if user keeps code on another partition, podman machine will not be usable.

I know that they could in theory use other location for mounting, but that is highly unlikely, due to how macos works.

Results: it works.

@baude Should we rename this issue to mark changing default volume mount on macos? -- That is quite important especially from UX point of view, as we do want a setup that "just works".

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[LethargicDeveloper]

This post has saved me from ripping all my hair out.

I had to add the /tmp directory to it as well, but this worked for me.

podman machine init --volume /Users --volume /Volumes --volume /tmp