watchtower: Error Could not do a head request for

NOTE: See https://github.com/containrrr/watchtower/discussions/1529 for workarounds until this gets properly fixed

Describe the bug

Hello, since 48h hour, when auto updating I get this for every container that need an update:

Could not do a head request for "xxxx", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

thanks for help

Steps to reproduce

docker run -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once

Expected behavior

update with correct message pull

Note: The image does get updated, you only get a warning that it couldn’t do so without using up the docker API rate limit -@piksel

Environment

photon 4
Docker version 20.10.14, build a224086

Your logs

Logs

2023-01-20T21:19:57Z [I] Watchtower 1.5.1
2023-01-20T21:19:57Z [I] Using no notifications
2023-01-20T21:19:57Z [I] Checking all containers (except explicitly disabled with label)
2023-01-20T21:19:57Z [I] Running a one time update.
2023-01-20T21:19:59Z [W] Could not do a head request for "pihole/pihole:latest", falling back to regular pull.
                         container: /pihole
                         image: pihole/pihole:latest
2023-01-20T21:19:59Z [W] Reason: registry responded to head request with "404 Not Found", auth: "not present"
                         container: /pihole
                         image: pihole/pihole:latest
2023-01-20T21:20:07Z [I] Session done
                         Failed: 0
                         Scanned: 13
                         Updated: 0
                         notify: no
2023-01-20T21:20:07Z [I] Waiting for the notification goroutine to finish
                         notify: no

About this issue

Original URL
State: closed
Created a year ago
Reactions: 3
Comments: 30 (4 by maintainers)

Commits related to this issue

Temporary workaround for watchtower https://github.com/containrrr/watchtower/issues/1528 — committed to strayer/game-server-deployment-discord-bot by strayer a year ago
Fix 404 for images with provenance information See https://github.com/containrrr/watchtower/issues/1528#issuecomment-1399419983 — committed to I-Al-Istannen/Lighthouse by I-Al-Istannen a year ago

Most upvoted comments

I have verified these images of the reports from the linked thread and all have the above issue:

koenkk/zigbee2mqtt:latest netdata/netdata:latest ~~pihole/pihole:latest~~ crowdsecurity/crowdsec:latest

darthShadow on Jan 22, 2023

This actually looks to be quite an easy fix, since only adding the application/vnd.oci.image.index.v1+json mime to type the Accepted: header seems to make it work correctly.

We don’t supporting parsing the response, but that is irrelevant, since the only thing we are interested in is if it’s been changed since the image was pulled.

piksel on Jan 22, 2023

Confirmed, seeing it on crowdsecurity/crowdsec:latest

dreary-ennui on Jan 20, 2023

esphome/esphome:latest is also affected by the issue.

There also is an explicit warning for this issue in the release notes of buildx 0.10.0. Seams like many have updated without checking them or just use the latest version in their CI. I do the later myself. Would not expect that a breaking change like that is released until at least dockers own registry is ready for it.

evilscientress on Jan 24, 2023

I have verified these images of the reports from the linked thread and all have the above issue:

koenkk/zigbee2mqtt:latest netdata/netdata:latest ~pihole/pihole:latest~ crowdsecurity/crowdsec:latest

I would like to add the following to the list

amir20/dozzle:latest jokobsk/pi.alert:latest

tamimology on Jan 23, 2023

The pi-hole issue is no more Thanks to @PromoFaux

stanthewizzard on Jan 21, 2023

With the homarr image is the same 😦 https://github.com/ajnart/homarr

vilmotz on Jan 22, 2023

They (pihole) can pin the buildx version to v0.9.1 to fix their images (Example: https://github.com/cilium/cilium/pull/23206)

darthShadow on Jan 20, 2023

This may be due to the recent buildx change due to the Github Ubuntu Runner being upgraded:

https://github.com/docker/build-push-action/issues/755

This results in images being pushed whose manifests cannot be parsed due to these issues:

https://github.com/moby/moby/issues/43126

https://github.com/docker/buildx/issues/1509

I didn’t get time to verify via the API for a failing image (https://github.com/AnalogJ/scrutiny, specifically the collector) but the docker manifest inspect command does fail for it and docker buildx imagetools inspect shows that it’s an oci image.

darthShadow on Jan 20, 2023

Hi there! 👋🏼 As you’re new to this repo, we’d like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

github-actions[bot] on Jan 20, 2023