image: gracefully handle HTTP 429 return codes (Too Many Requests)

Downstream BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1702519

In the above BZ, the RHCOS nodes are attempting to pull a container from quay.io via skopeo and the registry is returning HTTP 429 (Too Many Requests). skopeo reacts with:

msg="Error determining repository tags: Invalid status code returned when fetching tags list 429 (Too Many Requests)"

I think the error message is misleading since 429 is a valid code (https://tools.ietf.org/html/rfc6585#section-4), so at a minimum the message should be changed.

cc: @runcom

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 23 (14 by maintainers)

Most upvoted comments

We can add a Retry-After header too, if that helps

josephschorr on Apr 24, 2019

If you send enough requests in a brief burst to Quay.io, you’ll get a 429 response – other users not bursting requests will not.

Alternatively, perhaps set up a local nginx proxy with Quay.io as the backend. Something like:

events {}

http {
    limit_req_zone $binary_remote_addr zone=mylimit:1m rate=1r/s;
 
    server {
        listen 443;
        location / {
            limit_req zone=mylimit;
            limit_req_status 429;
            limit_conn_status 429;
            proxy_pass https://quay.io;
        }
    }
}

(may need some tweaking)

and podman run --name nginx -v .../nginx.conf:/etc/nginx/nginx.conf:Z -p 8443:443 nginx

twaugh on Sep 13, 2019

We are running into this when using skopeo from the command line from a Python program which wants to manipulate tags in Quay.io.

Is anyone working on fixing this? Is there anything I can do to help?

twaugh on Sep 10, 2019

The spec says:

...MAY include a Retry-After header indicating how long to wait before making a new request.

So it is unfortunately up to the registry/server if they decide to tell the client how long to back-off. Maybe retry according to Retry-After if it is present in the response?

miabbott on Apr 24, 2019