Persist host docker host during upgrade

[yang-wei]

No description provided.

[rheinschiene]

I used the playbook to install my ECE environment in version 3.7.3 using the /run/podman/podman.sock socket. I have also noticed that the parameter --host-docker-host still needs to be added to the upgrade part of the playbook. I've added this parameter in the same way to my local playbook as this commit suggests and tried to upgrade to version 3.8.1. The upgrade starts as expected, but there seems to be a bug regarding the frc-upgraders-upgrader container, which is started during the upgrade process. All containers running with podman are using the expected podman socket, except for this upgrade container. The --host-docker-host parameter is ignored for the upgrade container and the upgrade fails, because the default /run/docker.sock does not exist.

This is what I get during the upgrade:

podman inspect frc-upgraders-upgrader | grep "\.sock"
                    "Source": "/run/docker.sock",
                    "Destination": "/run/docker.sock",
                    "HOST_DOCKER_HOST=/run/podman/podman.sock",
                    "/run/docker.sock:/run/docker.sock:rw,rprivate,nosuid,nodev,rbind"

This is what I was expecting:

podman inspect frc-upgraders-upgrader | grep "\.sock"
                    "Source": "/run/podman/podman.sock",
                    "Destination": "/run/docker.sock",
                    "HOST_DOCKER_HOST=/run/podman/podman.sock",
                    "/run/podman/podman.sock:/run/docker.sock:rw,rprivate,nosuid,nodev,rbind"

I've opened a Elastic Support ticket to address this issue and I told the support that I was using this playbook to install my environment. The ticket is still open and I'am waiting for a response from the developement team. But support clearly stated that you should adhere to the standard, which is specified in the Elastic documentation: https://www.elastic.co/docs/deploy-manage/deploy/cloud-enterprise/migrate-ece-to-podman-hosts

I started replacing /run/podman/podman.sock with /run/docker.sock in my local playbook to stay as close as possible to the official documentation and avoid any problems with support.

[yang-wei]

@rheinschiene are you able to set environment variable ENABLE_DEBUG_LOGGING to true during the upgrade and show me what does the debug log says about

update: what you can do is to add --debug flag to the script

    --docker-registry {{ ece_docker_registry }} \Add a comment on line R7Add diff commentMarkdown input:  edit mode selected.WritePreviewAdd a suggestionHeadingBoldItalicQuoteCodeLinkUnordered listNumbered listTask listMentionReferenceSaved repliesAdd FilesPaste, drop, or click to add filesCancelCommentStart a reviewReturn to code
    --ece-docker-repository {{ ece_docker_repository }} \
    --user admin \
    --pass {{ adminconsole_root_password }} \
    --debug \ 
    {% if container_engine == "Podman" %}
    --podman \
    --host-docker-host /run/podman/podman.sock \
    --force
    {% endif %}

[rheinschiene]

I used the --debug parameter when I was first investigating with the support team, but unfortunetely, I can't retrieve the file again. But I found this, maybe it will help:

The frc-upgraders-upgrader container showed the following error message:
2025/08/19 11:22:57 socat[83] E UNIX-CLIENT: /run/docker.sock: Permission denied
The container was restarted about every 30 seconds and ended with the permission denied message.

Environment variables from the log with debug mode:

CLIENT_FORWARDER_OBSERVERS_TAG=${?CLIENT_FORWARDER_OBSERVERS_TAG}
LATEST_STACK_PRE_RELEASE=
LATEST_VERSIONS_DOCKER_NAMESPACE=cloud-release
PREVIOUS_STACK_PRE_RELEASE=-0
PREVIOUS_VERSIONS_DOCKER_NAMESPACE=cloud-assets
CLOUD_ENTERPRISE_VERSION=3.8.1
ECE_DOCKER_REPOSITORY=cloud-enterprise
MASTER_RUNNER_ID=u999eslog023t
RUNNER_ID=${RUNNER_ID}
RUNNER_REGION=${RUNNER_REGION}
RUNNER_HOST_IP=${RUNNER_HOST_IP}
ROLE=upgrade-additional
ELASTIC_UID=${ELASTIC_UID}
ELASTIC_GID=${ELASTIC_GID}
HOST_STORAGE_PATH=${HOST_STORAGE_PATH}
HOST_STORAGE_ROOT_VOLUME_PATH=${HOST_STORAGE_ROOT_VOLUME_PATH}
SKIP_PENDING_PLAN_CHECK=false
HOST_DOCKER_HOST=/run/podman/podman.sock
ELASTIC_CLOUD_PLATFORM=aws
ENABLE_DEBUG_LOGGING=true
FOUND_UPGRADERS_SHARED_ZK=upgrader-services:c2GJ0kYqnUElvppYoymqPSLLTzww9Ctcp6H0ubhkz6d
FOUND_UPGRADERS_SHARED_ZK_ENCODED=upgrader-services:lTyTuo4ZovNuCTHPFlCpcNi1VpQ=
DOCKER_REGISTRY=artifactory-internal/it-infra_log-service_cir_abeq2_remote
ECE_TIMEOUT_FACTOR=10.0 {}

The support mentioned this finding in the upgrade.log:

[2025-08-19 11:22:59,035][ERROR][no.found.docker.DockerApi] Docker InspectContainer request failed {"ec_container_kind":"docker","ec_container_group":"directors","ec_container_name":"director","tags":["InspectContainer",
[2025-08-19 11:22:57,609][DEBUG][no.found.docker.DockerApi] Docker API request: HttpRequest(HttpMethod(GET),http://localhost:2375/v1.39/containers/frc-directors-director/json,List(),HttpEntity.Strict(none/none,0 bytes total),HttpProtocol(HTTP/1.1)) {"ec_container_kind":"docker","ec_container_group":"directors","ec_container_name":"director"} [2025-08-19 11:22:57,997][ERROR][no.found.docker.DockerApi] Docker InspectContainer request failed {"ec_container_kind":"docker","ec_container_group":"directors","ec_container_name":"director","tags":["InspectContainer","ClosedConnection"]} no.found.docker.DockerApiException: The http server closed the connection unexpectedly before delivering responses for 1 outstanding requests Category: [InspectContainer] Error kind: [ClosedConnection] Docker API request: [HttpRequest(HttpMethod(GET),http://localhost:2375/v1.39/containers/frc-directors-director/json,List(),HttpEntity.Strict(none/none,0 bytes total),HttpProtocol(HTTP/1.1))], response: [None]

When I created a symlink from /run/podman/podman.sock to /run/docker.sock, the upgrade was successful.

[yang-wei]

thank you @rheinschiene

We found out that our codebase that boot up the frc-upgraders-upgrader container has this hardcoded

 "/run/docker.sock:/run/docker.sock"

I am changing them to

            "${HOST_STORAGE_PATH}:/run/docker.sock"

We will need the change on our codebase to allow this PR to be merged