go-containerregistry: `crane delete` requests token with insufficient scope

I’m the lead developer on a cloud-hosted registry implementation called Keppel. I received a complaint from a customer that they cannot crane delete images stored in Keppel. I reproduced the issue; it boils down to Crane guessing the wrong token scope. When invoked as crane delete keppel.example.com/foo/bar:qux, it just assumes the required token scope is repository:foo/bar:push,pull, even though repository:foo/bar:delete would be required.

For reference, here’s a trace of what crane delete keppel.example.com/foo/bar:qux does, as recorded by mitmdump. (The actual hostnames, repository names, and tokens have been redacted.) Note that the final request, the actual DELETE, indicates the correct token scope, but Crane ignores this information.

$ mitmdump -p 8888 --ssl-insecure --flow-detail 3
Proxy server listening at http://*:8888
[::1]:48300: clientconnect
[::1]:48300: GET https://keppel.example.com/v2/ HTTP/2.0
    :authority: keppel.example.com
    accept-encoding: gzip
    user-agent: Go-http-client/2.0

 << 401  104b
    server: nginx/1.15.6
    date: Wed, 27 May 2020 14:04:27 GMT
    content-type: application/json
    content-length: 104
    docker-distribution-api-version: registry/2.0
    vary: Origin
    www-authenticate: Bearer realm="https://keppel.example.com/keppel/v1/auth",service="keppel.example.com"
    strict-transport-security: max-age=15724800; includeSubDomains

    {
        "errors": [
            {
                "code": "UNAUTHORIZED",
                "detail": null,
                "message": "no bearer token found in request headers"
            }
        ]
    }

[::1]:48300: GET https://keppel.example.com/keppel/v1/auth?scope=repository%3Afoo%2Fbar%3Apush%2Cpull&service=keppel.example.com HTTP/2.0
    :authority: keppel.example.com
    authorization: Basic password-redacted
    user-agent: go-containerregistry
    accept-encoding: gzip

    scope:   repository:foo/bar:push,pull
    service: keppel.example.com

 << 200  1.65k
    server: nginx/1.15.6
    date: Wed, 27 May 2020 14:04:27 GMT
    content-type: application/json
    vary: Accept-Encoding
    vary: Origin
    strict-transport-security: max-age=15724800; includeSubDomains
    content-encoding: gzip

    {
        "expires_in": 3600,
        "issued_at": "2020-05-27T14:04:27Z",
        "token": "token-was-redacted"
    }

[::1]:48300: DELETE https://keppel.example.com/v2/foo/bar/manifests/qux HTTP/2.0
    :authority: keppel.example.com
    authorization: Bearer token-was-redacted
    user-agent: go-containerregistry
    accept-encoding: gzip

 << 403  117b
    server: nginx/1.15.6
    date: Wed, 27 May 2020 14:04:27 GMT
    content-type: application/json
    content-length: 117
    docker-distribution-api-version: registry/2.0
    vary: Origin
    www-authenticate: Bearer realm="https://keppel.example.com/keppel/v1/auth",service="keppel.example.com",scope="repository:foo/bar:delete",error="insufficient_scope"
    strict-transport-security: max-age=15724800; includeSubDomains

    {
        "errors": [
            {
                "code": "DENIED",
                "detail": null,
                "message": "token does not cover scope repository:foo/bar:delete"
            }
        ]
    }

[::1]:48300: clientdisconnect

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 3
  • Comments: 17

Commits related to this issue

Most upvoted comments

Use of the delete scope is widespread enough that it might be worth just adopting it… this was changed a while back: https://github.com/docker/distribution/commit/ccb839e0e30c3b6992fb4084dfd6550d0ddd4d1a

Annoyingly, this would break crane delete for GCR. We should adopt the approach of parsing the challenge and requesting the correct scope.

is this something you could change in Keppel?

Done. See commit linked above.