Could not do a head request: invalid character

[modem7]

Describe the bug

Issues with the crowdsec image.

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: Crowdsec

Error:

Could not do a head request for "crowdsecurity/crowdsec:latest", falling back to regular pull.
Reason: invalid character 'S' looking for beginning of value

Steps to reproduce

Watchtower config:

  watchtower:
    image: containrrr/watchtower
    container_name: Watchtower
    hostname: Watchtower
    networks:
      - isonet
    labels: 
      - autoheal=true
    environment:
      TZ: $TZ
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_REMOVE_VOLUMES: "false"
      WATCHTOWER_INCLUDE_STOPPED: "true"
      WATCHTOWER_NO_STARTUP_MESSAGE: "true"
      WATCHTOWER_TIMEOUT: 30s
      WATCHTOWER_SCHEDULE: "0 0 6 * * *" # Everyday at 04:30
      WATCHTOWER_NOTIFICATIONS: shoutrrr
      WATCHTOWER_NOTIFICATION_URL: "telegram://$TGRAM_BOT_TOKEN@telegram?channels=$TGRAM_CHAT_ID"
      WATCHTOWER_NOTIFICATIONS_LEVEL: info
      DOCKER_HOST: unix:///var/run/docker.sock
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
    logging:
      driver: "local"
      options:
        max-size: 10m
        max-file: "3"
    restart: always
    mem_limit: 250m
    mem_reservation: 60m

Expected behavior

To update.

Screenshots

Environment

• Ubuntu 22.04
• x86
• Docker version 20.10.22, build 3a2c30b
• Crowdsec

Your logs

time="2022-12-26T06:00:35Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-27T06:00:39Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-28T06:00:04Z" level=warning msg="Could not do a head request for \"crowdsecurity/crowdsec:latest\", falling back to regular pull." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:04Z" level=warning msg="Reason: invalid character 'S' looking for beginning of value" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:15Z" level=info msg="Found new linuxserver/bookstack:latest image (f532d54a179c)"
time="2022-12-28T06:00:23Z" level=info msg="Found new linuxserver/sonarr:latest image (10d61a66d74e)"
time="2022-12-28T06:00:51Z" level=info msg="Found new linuxserver/netbootxyz:latest image (2d07183b1077)"
time="2022-12-28T06:00:53Z" level=info msg="Stopping /Netbootxyz (14c6cda3588b) with SIGTERM"
time="2022-12-28T06:01:34Z" level=info msg="Stopping /Sonarr (d19f614e614d) with SIGTERM"
time="2022-12-28T06:01:45Z" level=info msg="Stopping /Bookstack (7a1565f59d9e) with SIGTERM"
time="2022-12-28T06:01:51Z" level=info msg="Creating /Bookstack"
time="2022-12-28T06:01:52Z" level=info msg="Creating /Sonarr"
time="2022-12-28T06:01:53Z" level=info msg="Creating /Netbootxyz"
time="2022-12-28T06:01:59Z" level=info msg="Removing image 455a86d2b778"
time="2022-12-28T06:01:59Z" level=info msg="Removing image f4e288f594a6"
time="2022-12-28T06:02:00Z" level=info msg="Removing image 4866c5508f26"
time="2022-12-28T06:02:00Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=3 notify=no

Additional context

Unsure if related to #1050

Doing a manual run does not trigger any issues.

docker run --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower \
    --debug --run-once \
    Crowdsec
time="2022-12-29T05:56:32Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2022-12-29T05:56:33Z" level=debug msg="Making sure everything is sane before starting"
time="2022-12-29T05:56:33Z" level=info msg="Watchtower 1.5.1"
time="2022-12-29T05:56:33Z" level=info msg="Using no notifications"
time="2022-12-29T05:56:33Z" level=info msg="Only checking containers which name matches \"Crowdsec\""
time="2022-12-29T05:56:33Z" level=info msg="Running a one time update."
time="2022-12-29T05:56:33Z" level=debug msg="Checking containers for updated images"
time="2022-12-29T05:56:33Z" level=debug msg="Retrieving running containers"
time="2022-12-29T05:56:35Z" level=debug msg="Trying to load authentication credentials." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials for crowdsecurity found" config_file=/config.json
time="2022-12-29T05:56:35Z" level=debug msg="Got image name: crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Checking if pull is needed" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-12-29T05:56:35Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-12-29T05:56:35Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-12-29T05:56:35Z" level=debug msg="Setting scope for auth token" image=crowdsecurity/crowdsec scope="repository:crowdsecurity/crowdsec:pull"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials found."
time="2022-12-29T05:56:36Z" level=debug msg="Parsing image ref" host=index.docker.io image=crowdsecurity/crowdsec normalized="docker.io/crowdsecurity/crowdsec:latest" tag=latest
time="2022-12-29T05:56:36Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/crowdsecurity/crowdsec/manifests/latest"
time="2022-12-29T05:56:36Z" level=debug msg="Found a remote digest to compare with" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg=Comparing local="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg="Found a match"
time="2022-12-29T05:56:36Z" level=debug msg="No pull needed. Skipping image."
time="2022-12-29T05:56:36Z" level=debug msg="No new images found for /Crowdsec"
time="2022-12-29T05:56:36Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no
time="2022-12-29T05:56:36Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

[piksel]

Seems like there was some kind of temporary network error. During your first session, it only warned you that it could not perform a "cheap" check for updates, falling back to a normal pull.

Whatever error message the server responded with is probably now gone and it works normally.

[UniverseXXX]

I'm having the same issue on Raspberry Pi 4 (AARCH64). I couldn't revert back to docker-compose 1.28.2, as no binary for aarch64 existed. It looks like there is a problem with docker-compose v.2.

[hoorna]

I have exactly the same issue. Occasionally I get this error-message for alternating images (not the same each time). I have the impression that this message has been occurring since the last Watchtower update.

[piksel]

@UniverseXXX and @hoorna what part do you mean is the same?
Do you also get the invalid character 'S' looking for beginning of value reason?

I don't see how docker-compose version is related to this issue. Nor should there be anything in the last release that alters this behaviour...

[hoorna]

@piksel, I also get the "invalid character 'S' looking for beginning of value".

Today I got for example an email message with the following text:
"Could not do a head request for "containrrr/watchtower:latest", falling back to regular pull.
Reason: invalid character 'S' looking for beginning of value"

Before the latest Watchtower update I never got such a message.

[piksel]

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error.
Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

[modem7]

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error.
Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

If the message is cryptic and making it difficult to troubleshoot, I wonder if it would be worthwhile investigating a way to make that particular error more verbose (but not run the entire container verbose as that might just cause noise, even more so if it's an intermittent issue (seems to be in my case))?

[piksel]

Since #1528 was opened for the 404 errors, let's use it for those errors, since it seems to be something different from the original issue here.

[lordraiden]

I still get this error

Notifications:
Container errors
Streams: Vector - Containers
--- [Event Definition] ---
Title: Container errors
Type: aggregation-v1
Timestamp: 2023-10-12T23:00:08.000Z

--- [Log] ---
Source: docker_logs
Container: Watchtower
Level: warning
Message:
Message: Could not do a head request for
Message Link: https://10.10.40.190:9000/messages/vector_7/17144332-6953-11ee-819f-0242ac170004

Container errors
Streams: Vector - Containers
--- [Event Definition] ---
Title: Container errors
Type: aggregation-v1
Timestamp: 2023-10-12T23:00:08.000Z

--- [Log] ---
Source: docker_logs
Container: Watchtower
Level: warning
Message:
Message: Reason: invalid character 'S' looking for beginning of value
Message Link: https://10.10.40.190:9000/messages/vector_7/17146a40-6953-11ee-819f-0242ac170004

[piksel]

@lordraiden It's not an error, but rather a warning. If you don't care about the warning you can turn it off. It means that watchtower gets an invalid response from the repository and can't perform a cheap change check and falls back to a regular pull.

This issue go hijacked a bit for the occurring 404 errors (which was due to a change in docker buildx). I'll reopen this and clean it from unrelated comments.

[piksel]

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error.
Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

If the message is cryptic and making it difficult to troubleshoot, I wonder if it would be worthwhile investigating a way to make that particular error more verbose (but not run the entire container verbose as that might just cause noise, even more so if it's an intermittent issue (seems to be in my case))?

The response should be logged in full to aid in further investigation. Reopening this.