podman system prune -a and podman info very slow

[nickdesaulniers]

Issue Description

Smells like #6288 , but I'm running:

$ podman version
Client:       Podman Engine
Version:      4.9.4
API Version:  4.9.4
Go Version:   go1.22.7
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64
$ podman info
WARN[0001] Found incomplete layer "3c350344f166c20784bdbf4812549210d5ff7c222c14e63d0937d2580434e940", deleting it 
<sits forever, doesn't finish, ctrl+c>
$ 

### Steps to reproduce the issue

`podman system prune -a`
`podman info`

### Describe the results you received

Both commands seem to hang forever. `prune` ran overnight. I'm on a high end workstation...

### Describe the results you expected

Clean up in a relatively acceptable amount of time.

### podman info output

```yaml
x86_64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

No response

Additional information

In GDB I can see multiple threads getting spawned repeatedly. But in top, it looks like the process is pegging 1 cpu core.

[nickdesaulniers]

Same for podman images...

[nickdesaulniers]

FWIW, I was able to unstick myself somehow via: podman system reset

[giuseppe]

@nickdesaulniers could you try with a newer version? 4.x is not supported upstream and there are some fixes in the last version that could affect what you are seeing

[Luap99]

podman system prune -a can be slow if you have a ton of unused images that all need to be removed. Many of these operations have to hold locks so it make sense that other commands might hang waiting for them.

Now if the podman system prune -a process really did end up in an endless loop while holding a lock that certainly would be quite bad. The base case if you can reproduce this again is to send SIGABRT to the process which should kill it and print the stack traces which might give us a clue what the process was doing.
If you you used gdb you should also be able to dump the stack traces to see what it is doing.