Debug what is happening during 10min "exporting layers"

[hamishfagg]

Hi there,

I am trying to debug an issue where I am using a buildkit to build a reasonably large image (~4GB), and the "exporting to image" step takes ~10 minutes to complete. This is the same whether I build on our remote buildkit agent in kubernetes, or on my local machine using the docker-container driver.

I have turned on trace logging in buildkit, and it doesn't print anything but healthchecks during this 10mins. I've looked at the CPU/mem/swap/IO usage, and none of them are particularly stressed during this step. It doesn't seem like much is happening at all. Ive checked this issue, and my filesystem has non-zero nanos for files.

But how can I find out what buildkit is actually doing for all this time? It seems to be running through a bunch of files in /tmp/containerd-mountxxxxxx, but it takes forever and the disk is not tapped out.

Hoping someone can point me in the right direction, thanks

[hamishfagg]

requirements.txt
test.Dockerfile.txt
Here's a repro. Installing all the pip packages takes a while as expected, but then "exporting image" takes 10+ mins when building with:
docker buildx build -f test.Dockerfile --load .