Random "Is the docker daemon running?" with Docker-in-Docker Feature

[metaskills]

I make heavy use of docker-in-docker and have noticed across a few projects that use (https://github.com/devcontainers/ci) randomly failing with the following message. NOTE: I never see this message in Codespaces, only when using the CLI via CI project.

ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

At first I thought I needed to wait for docker to start up and I tried adding something like this to my postCreateCommand script:

timeout 60s bash -c 'until docker info; do sleep 5; done'

However it turns out that something lower level was going wrong. The logs always showed this error when the issue happened shortly after the devcontainer was created:

2023-01-22T19:03:04.2136493Z [2023-01-22T19:03:04.213Z] Container started
2023-01-22T19:03:04.5278182Z [2023-01-22T19:03:04.527Z] Running the postCreateCommand from devcontainer.json...
2023-01-22T19:03:04.5818191Z [2023-01-22T19:03:04.581Z] sed: couldn't flush stdout: Device or resource busy

This led me to investigate the following issues which may be related:

• Regression: Feature entrypoints run as remoteUser #70
• Correct USER after features stage: Investigate other cases #78

I tried adding "privileged": true, to my JSON in the hopes this would help and it did not. Random failures still occur. If it helps, here are some log outputs to compare. These are pulled from

• GitHub Devcontainer w/Docker-in-Docker Action: Success
• GitHub Devcontainer w/Docker-in-Docker Action: Random Failure

Success	Failure

In the work I am doing for this project (https://github.com/customink/crypteia/tree/MultiPlatform) I have set each job's test to do a simple docker inspect to fail fast and you can see where it fails further up in my postCreate script which is doing a docker run command. But make no mistake, even if that postCreate were removed the random failures would happen because the devcontainer somehow is not in a good state with docker-in-docker. Been seeing this on other project's CI as well.

[metaskills]

OK, this is straight up a bug with the Docker-in-Docker feature with its usage with the CI feature. It could be a CLI bug since I have not seen this issue with Codespaces. I made a very basic repo that demonstrates this happens ~20% (crude math) of the time. See here:

• https://github.com/customink/dnd-demo (repo)
• https://github.com/customink/dnd-demo/actions/runs/3981703221 (failure demo)

[metaskills]

Sorry, not sure why this was closed, reopened it. Since I am un-sure if the issue should be in one of these places or not:

• Random "Is the docker daemon running?" with Docker-in-Docker Feature ci#192
• docker-in-docker fails with the ubuntu:latest image actions/runner-images#6980

[chrmarti]

Continuing in devcontainers/ci#192.