machine: Volume ops test: statfs /private/tmp/ci/ginkgoNNN: no such file or directory

[edsantiago]

Not quite as frequent or annoying as #22551, but still causing wasted runs:

run basic podman commands
  Volume ops
....
  Trying to pull quay.io/libpod/alpine_nginx:latest...
  ...
  Writing manifest to image destination
  WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
  Error: statfs /private/tmp/ci/ginkgo630638030: no such file or directory

• darwin : machine-mac podman darwin rootless host sqlite
  • PR ExitWithError() - pod_xxx tests #22552
    • 05-01 12:16 in run basic podman commands Volume ops
  • PR ExitWithError() - yet more low-hanging fruit #22489
    • 04-24 17:00 in run basic podman commands Volume ops
    • 04-24 12:06 in run basic podman commands Volume ops
  • PR ExitWithError() - more low-hanging fruit #22486
    • 04-24 10:09 in run basic podman commands Volume ops
  • PR fix(deps): update module github.com/docker/docker to v26.1.0+incompatible #22461
    • 04-22 18:43 in run basic podman commands Volume ops

x	x	x	x	x	x
machine-mac(5)	podman(5)	darwin(5)	rootless(5)	host(5)	sqlite(5)

[edsantiago]

@cevich is there any chance whatsoever that https://github.com/containers/podman/blob/c9644ebccf14309a77769cba00833cd139509e4a/contrib/cirrus/mac_cleanup.sh is getting invoked in the middle of a running CI job? I just can't understand this bug and am grasping at straws.

[Luap99]

Unlikely, there is a extra level of indirectness here given the dir is mounted in the machine VM. As such maybe the machine mount failed silently?

[cevich]

is there any chance whatsoever that

What Paul said. And the Mac's are single-task/single-user. Is it possible the running VM really is x86_64 via some emulation and/or is the pull command specifying --arch or --platform (just double-checking)?

Sorry no hack/get_ci_vm.sh support here, that's just way to complex with this environment to do safely. But in case it helps and is supported (I never checked), the re-run in terminal may be an option (with cleanup temporarily disabled).

Otherwise, there is a way to isolate (for a few hours) one of the Macs and dedicate it to servicing a single PR. In that PR, the end-of-task cleanup could be disabled, so that a human may ssh in and check out the state of things. This is all manual, and a bit of a chore to pull off, but it's technically possible.

[Luap99]

@edsantiago Not sure if you are testing machine in your non flake retry testing PR but if you do could you give this a go:

diff --git a/pkg/machine/apple/apple.go b/pkg/machine/apple/apple.go
index 93201407e..04db7638b 100644
--- a/pkg/machine/apple/apple.go
+++ b/pkg/machine/apple/apple.go
@@ -124,7 +124,7 @@ func GenerateSystemDFilesForVirtiofsMounts(mounts []machine.VirtIoFs) ([]ignitio
        mountPrep.Add("Service", "Type", "oneshot")
        mountPrep.Add("Service", "ExecStartPre", "chattr -i /")
        mountPrep.Add("Service", "ExecStart", "mkdir -p '%f'")
-       mountPrep.Add("Service", "ExecStopPost", "chattr +i /")
+       // mountPrep.Add("Service", "ExecStopPost", "chattr +i /")
 
        mountPrep.Add("Install", "WantedBy", "remote-fs.target")
        mountPrepFile, err := mountPrep.ToString()

[edsantiago]

Oops, no, I long ago disabled machine tests in #17831. I will look into reenabling this one.

FWIW here's the current flake list. I don't think there's any useful info in this list, i.e., I haven't seen any logs that look different or provide interesting new data, but am posting anyway.

• darwin : machine-mac podman darwin rootless host sqlite
  • PR libpod: wait for healthy on main thread #22658
    • 05-13 16:18 in run basic podman commands Volume ops
  • PR ExitWithError() - s files #22582
    • 05-02 20:13 in run basic podman commands Volume ops
  • PR ExitWithError() - pod_xxx tests #22552
    • 05-01 12:16 in run basic podman commands Volume ops
  • PR ExitWithError() - yet more low-hanging fruit #22489
    • 04-24 17:00 in run basic podman commands Volume ops
    • 04-24 12:06 in run basic podman commands Volume ops
  • PR ExitWithError() - more low-hanging fruit #22486
    • 04-24 10:09 in run basic podman commands Volume ops
  • PR fix(deps): update module github.com/docker/docker to v26.1.0+incompatible #22461
    • 04-22 18:43 in run basic podman commands Volume ops

x	x	x	x	x	x
machine-mac(7)	podman(7)	darwin(7)	rootless(7)	host(7)	sqlite(7)

[Luap99]

The alternative is I instrument the tests to do some checks. Basically I it would have to ssh into the machine VM and run systemctl status on all the mount units. I think the race here is the most likely cause.

One interesting point would be the new machine init with volume test, if this never fails then I am sure this is a race due the parallel running chattr -i and chattr +i in different units. Reason this tests mounts only one path so there cannot be a race, however the default volumes are several paths thus the chance for the race.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Luap99]

@edsantiago Any conclusions?

[edsantiago]

No. This isn't something I can look into, and our PR merge rate is too low these days; not many flakes to report.

[edsantiago]

However, I just ran my afternoon flake catchup, and here's a new one

[Luap99]

I was mostly interested if you ever saw it in the no flake retry PR (edsantiago@28882ca)

However as I have a mac now I can try to reproduce locally and see where I go from there.

[edsantiago]

Still active

• darwin : machine-mac podman darwin rootless host sqlite
  • PR pkg/machine/e2e: Remove unnecessary copy of machine image. #23068
    • 06-21 15:03 in run basic podman commands Volume ops
  • PR cirrus.yml: implement skips based on source changes #23030
    • 06-19 09:12 in run basic podman commands Volume ops
  • PR Update module github.com/checkpoint-restore/checkpointctl to v1.2.1 #23021
    • 06-17 17:01 in run basic podman commands Volume ops
  • PR New Windows makefile (winmake.ps1) targets and Windows build documentation update #22913
    • 06-06 08:52 in run basic podman commands Volume ops

[Luap99]

So I have been running this script all day without luck. So either my script is wrong or I was not able to reproduce.

#!/bin/bash

set -e

while :; do
dirs=()
for i in {1..20}; do dir="$TMPDIR$i"; dirs+=($dir); mkdir -p $dir; done
args=() 
for dir in "${dirs[@]}"; do args+=("--volume" "$dir:$dir"); done
podman machine init --now "${args[@]}"
podman machine ssh ls "${dirs[@]}"
podman machine ssh mount | grep $TMPDIR
podman machine ssh systemctl list-units --failed | grep fail && break

podman machine rm -f
done

I did manage to hit a quay.io flake though

Error: reading manifest sha256:a7775864b05f6402c7ca071446f8a50ce94e456e85c3dbe8d94b3a8bf2a2c81d in quay.io/podman/machine-os: authentication required

So my best bet is to try to instrument the CI tests to give out more logs when it happens.

[edsantiago]

I still suspect this is related to the weird Macintosh CI setup, and as such will only trigger in CI. But I have no evidence to back that up.

[Luap99]

Well my PR did triggered it first try so I can tell you now that the bug is in the VM
https://api.cirrus-ci.com/v1/artifact/task/4925116956540928/html/machine-mac-podman-darwin-rootless-host-sqlite.log.html#t--run-basic-podman-commands-Volume-ops--1

[Luap99]

#23118 should contain a proper fix now