watchtower: Could not do a head request: invalid character

Describe the bug

Issues with the crowdsec image.

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: Crowdsec

Error:

Could not do a head request for "crowdsecurity/crowdsec:latest", falling back to regular pull.
Reason: invalid character 'S' looking for beginning of value

Steps to reproduce

Watchtower config:

  watchtower:
    image: containrrr/watchtower
    container_name: Watchtower
    hostname: Watchtower
    networks:
      - isonet
    labels: 
      - autoheal=true
    environment:
      TZ: $TZ
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_REMOVE_VOLUMES: "false"
      WATCHTOWER_INCLUDE_STOPPED: "true"
      WATCHTOWER_NO_STARTUP_MESSAGE: "true"
      WATCHTOWER_TIMEOUT: 30s
      WATCHTOWER_SCHEDULE: "0 0 6 * * *" # Everyday at 04:30
      WATCHTOWER_NOTIFICATIONS: shoutrrr
      WATCHTOWER_NOTIFICATION_URL: "telegram://$TGRAM_BOT_TOKEN@telegram?channels=$TGRAM_CHAT_ID"
      WATCHTOWER_NOTIFICATIONS_LEVEL: info
      DOCKER_HOST: unix:///var/run/docker.sock
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
    logging:
      driver: "local"
      options:
        max-size: 10m
        max-file: "3"
    restart: always
    mem_limit: 250m
    mem_reservation: 60m

Expected behavior

To update.

Screenshots

image

Environment

  • Ubuntu 22.04
  • x86
  • Docker version 20.10.22, build 3a2c30b
  • Crowdsec

Your logs

time="2022-12-26T06:00:35Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-27T06:00:39Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-28T06:00:04Z" level=warning msg="Could not do a head request for \"crowdsecurity/crowdsec:latest\", falling back to regular pull." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:04Z" level=warning msg="Reason: invalid character 'S' looking for beginning of value" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:15Z" level=info msg="Found new linuxserver/bookstack:latest image (f532d54a179c)"
time="2022-12-28T06:00:23Z" level=info msg="Found new linuxserver/sonarr:latest image (10d61a66d74e)"
time="2022-12-28T06:00:51Z" level=info msg="Found new linuxserver/netbootxyz:latest image (2d07183b1077)"
time="2022-12-28T06:00:53Z" level=info msg="Stopping /Netbootxyz (14c6cda3588b) with SIGTERM"
time="2022-12-28T06:01:34Z" level=info msg="Stopping /Sonarr (d19f614e614d) with SIGTERM"
time="2022-12-28T06:01:45Z" level=info msg="Stopping /Bookstack (7a1565f59d9e) with SIGTERM"
time="2022-12-28T06:01:51Z" level=info msg="Creating /Bookstack"
time="2022-12-28T06:01:52Z" level=info msg="Creating /Sonarr"
time="2022-12-28T06:01:53Z" level=info msg="Creating /Netbootxyz"
time="2022-12-28T06:01:59Z" level=info msg="Removing image 455a86d2b778"
time="2022-12-28T06:01:59Z" level=info msg="Removing image f4e288f594a6"
time="2022-12-28T06:02:00Z" level=info msg="Removing image 4866c5508f26"
time="2022-12-28T06:02:00Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=3 notify=no

Additional context

Unsure if related to https://github.com/containrrr/watchtower/issues/1050

Doing a manual run does not trigger any issues.

docker run --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower \
    --debug --run-once \
    Crowdsec
time="2022-12-29T05:56:32Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2022-12-29T05:56:33Z" level=debug msg="Making sure everything is sane before starting"
time="2022-12-29T05:56:33Z" level=info msg="Watchtower 1.5.1"
time="2022-12-29T05:56:33Z" level=info msg="Using no notifications"
time="2022-12-29T05:56:33Z" level=info msg="Only checking containers which name matches \"Crowdsec\""
time="2022-12-29T05:56:33Z" level=info msg="Running a one time update."
time="2022-12-29T05:56:33Z" level=debug msg="Checking containers for updated images"
time="2022-12-29T05:56:33Z" level=debug msg="Retrieving running containers"
time="2022-12-29T05:56:35Z" level=debug msg="Trying to load authentication credentials." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials for crowdsecurity found" config_file=/config.json
time="2022-12-29T05:56:35Z" level=debug msg="Got image name: crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Checking if pull is needed" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-12-29T05:56:35Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-12-29T05:56:35Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-12-29T05:56:35Z" level=debug msg="Setting scope for auth token" image=crowdsecurity/crowdsec scope="repository:crowdsecurity/crowdsec:pull"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials found."
time="2022-12-29T05:56:36Z" level=debug msg="Parsing image ref" host=index.docker.io image=crowdsecurity/crowdsec normalized="docker.io/crowdsecurity/crowdsec:latest" tag=latest
time="2022-12-29T05:56:36Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/crowdsecurity/crowdsec/manifests/latest"
time="2022-12-29T05:56:36Z" level=debug msg="Found a remote digest to compare with" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg=Comparing local="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg="Found a match"
time="2022-12-29T05:56:36Z" level=debug msg="No pull needed. Skipping image."
time="2022-12-29T05:56:36Z" level=debug msg="No new images found for /Crowdsec"
time="2022-12-29T05:56:36Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no
time="2022-12-29T05:56:36Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 27 (12 by maintainers)

Most upvoted comments

I’m not sure if related, but for the past 2 weeks I’ve been received the following notification (never had an issue previously)

Could not do a head request for "nicolargo/glances:latest-full", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

Sometimes there is then a subsequent message, e.g.,

Found new nicolargo/glances:latest-full image (ce3e6aaf669f)

Everything seems to be present OK on Docker Hub - https://hub.docker.com/r/nicolargo/glances/tags I’ve also not noticed any issues with more generally with updates.

Same for me with the image koenkk/zigbee2mqtt:latest.

The log says:

time="2023-01-13T23:53:08Z" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" 
time="2023-01-13T23:53:08Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"

Does anyone know the reason why this is happening? Is it a problem from dockerhub, is there something wrong with the published images, or is it a problem with watchtower?

+8
ochstobi on Oct 13, 2023

@notDavid it’s closed because the issue is fixed in our code base. If you want to test it before it’s released you can use the containrrr/watchtower:latest-dev, but since it’s bleeding edge, it’s not recommended for daily use.

+2
piksel on Oct 13, 2023

Since #1528 was opened for the 404 errors, let’s use it for those errors, since it seems to be something different from the original issue here.

+2
piksel on Jan 20, 2023

Plenty of other repos are also broken …

netdata/netdata:latest amir20/dozzle:latest shlinkio/shlink:latest zabbix/zabbix-*:latest

Could not do a head request for "zabbix/zabbix-agent:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-server-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-web-nginx-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "amir20/dozzle:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "shlinkio/shlink:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
+2
xpliz on Oct 13, 2023

Plenty of other repos are also broken …

netdata/netdata:latest amir20/dozzle:latest shlinkio/shlink:latest zabbix/zabbix-*:latest

Could not do a head request for "zabbix/zabbix-agent:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-server-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-web-nginx-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "amir20/dozzle:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "shlinkio/shlink:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

Pretty sure it’s not the repos at this point.

+1
modem7 on Oct 13, 2023
Read more comments on GitHub
← youki: Unable to start rootless container under Podman when Youki used as runtime
watchtower: Docker containers with hash-only image cannot be updated →