watchtower: Log "Spam": "Could not do a head request

Describe the bug I recently updated to the latest watchtower version. But now I seem to get spammed with the error message:

Could not do a head request, falling back to regular pull.

which gets logged as well as creates an email notification every time.

To Reproduce Steps to reproduce the behavior:

Updated watchtower to latest image
Deploy a container from a gitlab hosted registry
Let it run
Monitor logs and emails

Expected behavior Ether a way to disable email notifications in case of this error. Or a fix for why the HEAD request is failing.

Environment

Platform: Ubuntu (Linux)
Architecture: x86_64
Docker version: 19.03.13, build 4484c46d9d

Logs from running watchtower with the --debug option

2020-12-20T22:04:01Z [I] Running a one time update.
2020-12-20T22:04:01Z [D] Checking containers for updated images
2020-12-20T22:04:01Z [D] Retrieving running containers
2020-12-20T22:04:01Z [D] Trying to load authentication credentials.
                         container: /play-for-today-website
                         image: registry.gitlab.com/***/website:latest
2020-12-20T22:04:01Z [D] Loaded auth credentials for user gitlab+deploy-token-32212, on registry registry.gitlab.com/***/website:latest, from file /config.json
2020-12-20T22:04:01Z [D] Got image name: registry.gitlab.com/***/website:latest
2020-12-20T22:04:01Z [D] Credentials loaded
2020-12-20T22:04:01Z [D] Checking if pull is needed
                         container: /play-for-today-website
                         image: registry.gitlab.com/***/website:latest
2020-12-20T22:04:01Z [D] Building challenge URL
                         URL: https://registry.gitlab.com/v2/
2020-12-20T22:04:02Z [D] Got response to challenge request
                         header: Bearer realm="https://gitlab.com/jwt/auth",service="container_registry"
                         status: 401 Unauthorized
2020-12-20T22:04:02Z [D] Checking challenge header content
                         realm: https://gitlab.com/jwt/auth
                         service: container_registry
2020-12-20T22:04:02Z [D] Setting scope for auth token
                         image: registry.gitlab.com/***/website
                         scope: repository:registry.gitlab.com/***/website:pull
2020-12-20T22:04:02Z [D] Credentials found.
                         credentials: ******
2020-12-20T22:04:02Z [D] Parsing image ref
                         host: registry.gitlab.com
                         image: cogindo/***/website
                         normalized: registry.gitlab.com/***/website:latest
                         tag: latest
2020-12-20T22:04:02Z [D] Doing a HEAD request to fetch a digest
                         url: https://registry.gitlab.com/v2/***/website/manifests/latest
2020-12-20T22:04:02Z [I] Could not do a head request, falling back to regular pull.
2020-12-20T22:04:02Z [D] Reason: registry responded to head request with 401
2020-12-20T22:04:02Z [D] Pulling image
                         container: /play-for-today-website
                         image: registry.gitlab.com/***/website:latest
2020-12-20T22:04:04Z [D] No new images found for /play-for-today-website

Additional context

The issue seem to happen only for the gitlab registry. Maybe that helps fixing it.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 10
Comments: 55 (23 by maintainers)

Most upvoted comments

I can confirm it works for me without an issue with version 1.1.4. No more notifications about head request fail for my gitlab repo.

fkrauthan on Dec 23, 2020

v1.1.4 in CI now. Should be with you in an hour or so. Please keep the updates coming regarding whether you get spammed or not, as well as whether HEAD requests work (if you’re on quay.io, dockerhub.com, or ghcr.io)

simskij on Dec 23, 2020

latest and v1.1.4 both point at the same hash so shouldn’t be any difference. However, if you already had v1.1.3 as latest, it would require a self-update before you’d end up at v1.1.4.

Worst case, running docker pull containrrr/watchtower:latest and then restarting watchtower should solve the issue. 👍🏼

Thanks for letting us know! 🙏🏼

simskij on Dec 23, 2020

@simskij sorry, I wanted to write GitLab registry. Nowadays all services have a registry so I got confused. Let me know if I can post any log output that might help you.

jwillmer on Dec 22, 2020

Alright! Lets proceed with your suggestion then. 👍🏻 I’ll have a look tomorrow. Thanks for letting me know!

simskij on Dec 21, 2020

@deividaspetraitis:

I might be wrong but it seems like Watchtower appends this prefix to the URL when sending HEAD request to obtain resource information.

You’re totally correct in that observation! It does this for all images not stored under an org or user, as this is how “official images” are resolved. We could probably hotfix this so it only does this for dockerhub images. In the meantime, if you push the image as someorg/somerepo it should work with head requests as well.

And yes, as someone said previously, being able to opt out of head requests should likely be an option. Initially globally, but down the line even per repo or registry.

simskij on Dec 21, 2020

Yeah, this is because Github Packages (not Github Container Registry) uses non-standard URLs for their images so it’s not possible to normalize the names. This shouldn’t matter as neither the rate limits, nor the HEAD requests apply to Github Packages. There definitely should be a way to opt out of HEAD-based checking for repos that does not conform to the OCI Distribution spec.

Edit: Saw that this was about Git_Lab_ and not Github. But the same thing applies. They also use a three-part URL, but it seems like you could omit the image name?

I have the issue on dockerhub, not github or gitlab. Or am I missing something?

I have no limits since I have a pro account on dockerhub, so I am good either way. If it makes no difference, maybe it’s a good idea to get rid of the spam message and move it to debug?

eilandert on Dec 21, 2020

@simskij works via docker pull [image]. I suspect that GitLab might block to many calls at once since I have 12 different images hosted on GitLab that all get checked by watchtower. I will play around with it an open a new issue if I can confirm my suspicion. In the meantime I can confirm that I don’t get any log entries about Could not do a head request anymore.

jwillmer on Dec 23, 2020

Well done, thank you very much, no longer receiving Could not do a head request, falling back to regular pull. messages as notifications for private registry based on https://docs.docker.com/registry/ ! 🙏🏼

deividaspetraitis on Dec 23, 2020

Hey @fkrauthan!

v1.1.3, which is the current live version should have all the fixes. Although the PR provided by @tkalus will further improve things as it will push all failed HEAD request logs to debug except when the registry is DockerHub.

@yonisolo and @deividaspetraitis, this should improve things for you as well, given that the GitLab Registry hasn’t implemented the HEAD option.

simskij on Dec 22, 2020

Seems alright now! thanks

eilandert on Dec 21, 2020

@piksel Yeah my problem is less rate limiting (as you are correct at least so far they did not introduce anything like docker) as the amount of noice this (by the sounds of it expected) behaviour brings.

fkrauthan on Dec 21, 2020