Adding image hash id when checking and pulling the image

[ipajumets]

What's the problem?

When Watchtower is checking and pulling images from our private registry it's adding image hash id to image name and tag. That way there is no way to identify if newer image is available or not.

Linked issues

The closest issue to my problem I've found is #1050.

Environment

Containers are running as services in a stack. There are 4 different stacks in a 3 manager, 2 worker node Docker swarm. Watchtower is deployed on every single node.

Logs

...
2021-12-21 09:50:47 | time="2021-12-21T07:50:47Z" level=debug msg="No new images found for /image-name.1.nm365xc8qhci7ii6lkfpm5wrm"
2021-12-21 09:50:47 | time="2021-12-21T07:50:47Z" level=debug msg="Pulling image" container=/image-name.1.nm365xc8qhci7ii6lkfpm5wrm image="private-registry.com/image-name:latest@sha256:3e3b4b0fb385f3965e18359bdf846e79f32027899fdf9af95b4f337f502a5187"
...
2021-12-21 09:50:47 | time="2021-12-21T07:50:47Z" level=debug msg="Parsing image ref" host=private-registry.com image=image-name normalized="private-registry.com/image-name:latest@sha256:3e3b4b0fb385f3965e18359bdf846e79f32027899fdf9af95b4f337f502a5187" tag="latest@sha256:3e3b4b0fb385f3965e18359bdf846e79f32027899fdf9af95b4f337f502a5187"
2021-12-21 09:50:47 | time="2021-12-21T07:50:47Z" level=debug msg="Credentials found."
2021-12-21 09:50:47 | time="2021-12-21T07:50:47Z" level=debug msg="Setting scope for auth token" image=private-registry.com/image-name scope="repository:image-name:pull"
...
2021-12-21 09:50:47 | time="2021-12-21T07:50:47Z" level=debug msg="Checking if pull is needed" container=/image-name.1.nm365xc8qhci7ii6lkfpm5wrm image="private-registry.com/image-name:latest@sha256:3e3b4b0fb385f3965e18359bdf846e79f32027899fdf9af95b4f337f502a5187"
...
2021-12-21 09:50:47 | time="2021-12-21T07:50:47Z" level=debug msg="Got image name: private-registry.com/image-name:latest@sha256:3e3b4b0fb385f3965e18359bdf846e79f32027899fdf9af95b4f337f502a5187"
...

Expected behaviour

As far as I understand it should only be checking the image tag in order to identify newer images.

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[piksel]

It's not watchtower that adds the hash. Some tool you are using is probably pinning the image version to that hash to prevent unexpected upgrades. Watchtower could just ignore the hash, but that would circumvent the explicitly stated "don't upgrade this container" flag that someone put on it. Also, it might break the connection to that tool that initially put it there in the first place, since it probably doesn't expect it to suddenly run another image without any pinned hash.

[ipajumets]

@piksel Could it be related to the fact that services are running in a swarm? Also I tried deploying with and without "latest" tag. No difference there. Also I tried to deploy only the stack that is related to our private registry leaving out services like Portainer, Grafana, Loki, Consul - again, no difference. I'm kind of clueless.

[piksel]

@ipajumets Yes, docker swarm pins images to hashes. It's probably possible to make watchtower able to update those images and play nicely with swarm, but none of the maintainers currently use swarm and have a good enough idea about how it should behave.

[wmandra]

It should be documented that Watchtower is not compatible with docker swarm if this isn't going to be fixed. I wasted hours attempting to get this to work until I stumbled on this issue.

[piksel]

@wmandra There isn't any fix. Swarm pins the container images to specific hashes which means that the latest version of the image is always the same. Even if watchtower would ignore that it's pinned, it has no way of telling what tag/image/repo to check for a later version, since that field just contains the image hash instead.

If you are running watchtower v1.5.0+ it explicitly logs that it cannot update pinned containers:

❯ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock sha256:d6b5c7ec581c5e365b6860ab65556a7b3363dcc8d0ab5df2f5324cdece09d6e4 --run-once
INFO[0001] Watchtower 1.5.1
INFO[0001] Using no notifications
INFO[0001] Checking all containers (except explicitly disabled with label)
INFO[0001] Running a one time update.
INFO[0001] Unable to update container "/trusting_grothendieck": container uses a pinned image, and cannot be updated by watchtower. Proceeding to next.
INFO[0001] Session done                                  Failed=0 Scanned=1 Updated=0 notify=no
INFO[0001] Waiting for the notification goroutine to finish  notify=no

Update: Actually, just removing the @hash might work for docker swarm images, as they are not regular (hash-only) pins, just the image tag combined with a hash.
That being said, I still have no idea how docker swarm will react to watchtower updating a container. It's untested waters, and I don't use swarm personally. If someone wants to investigate, I'd be happy to look at a PR for the added support.

[wmandra]

@piksel I'd be happy to test it. On my installation Watchtower does see the image tag with the hash appended to the end. Swarm behavior should be fine depending on what command Watchtower uses. On a swarm install all that's needed is to call: docker service update <service_name> --image <image_tag>

I'll tray to take a look at the Watchtower code over the next few days and see if something might be possible.

[piksel]

Watchtower doesn't use commands, it's communicating directly with the docker daemon through the api.

If you want to run swarm-specific stuff, it seems like it would be easier to use watchtower with --porcelain v1 and then run docker service update commands based on it's output instead.

[t-keller]

Hi there,

I've been through the same issue with a Portainer based stack (environment is Swarm)
Here is how it is defined in my docker-compose.yml:

myservice1:
    image: ghcr.io/mycompany/myservice1:latest

(docker ps doesn't show any sha version, just latest tag)

But when Watchtower is checking for new version, it includes the sha version:

time="2023-09-06T16:36:14+02:00" level=debug msg="Pulling image" container=/myservice.1.u2303gjudtet5wdm2pxlwbp5k image="ghcr.io/mycompany/myservice1:latest@sha256:4654ae5a4e76f3af32920aaedc4e9002a70f7d93cc70a70e6f35524495da8541"
time="2023-09-06T16:36:14+02:00" level=debug msg="No new images found for /myservice.1.u2303gjudtet5wdm2pxlwbp5k"

And as it includes sha version, Watchtower won't notice a newest version.
At this point, I'm not sure to understand which component in the stack is adding the version hash.

Any idea?

Thank you!

[piksel]

@t-keller docker swarm.

Even though it's not visible in docker ps, it's still present in docker inspect (which gives the same information that the API/watchtower gets).

[t-keller]

Indeed, you're right :(
I had to switch to Sheperd.

But I remain surprised this requirement can't be addressed natively by Portainer or ArgoCD.
To me, it sounds like a common requirement for a development environment (having always the last "latest")

Any different approach?