harbor: Harbor Registry does not support AssumeRoleWithWebIdentity

Given that the AWS SDK supports assuming a role through a WebIdentity, pods running in EKS should be able to assume a role with a web identity as documented here

Expected behavior and actual behavior:

  • Expected: Pods using ServiceAccounts annotated to assume a role with a web identity should have access/denial to resources as specified in the policies attached to the role.

  • Actual: Since pods are not assuming the role, one cannot, for instance, use s3 as a storage backend for the registry.

Steps to reproduce the problem: On EKS, annotate the serviceAccount and cycle the pods; you will see the environment variables AWS SDK needs, but the role is not assumed. Instead, the ec2 instance role is used.

Versions: Please specify the versions of following systems.

  • harbor version: [2.0.2] (via helm chart)
  • kubernetes 1.17

Additional context:

time="2020-08-26T14:06:37.339344937Z" level=info msg="redis: connect cache.registry.ops.internal.penneo.com:6379" go.version=go1.14.5 instance.id=a88be6db-6417-4ae5-8aeb-a8fb6339ba41 redis.connect.duration=2.041877ms service=registry version=v2.7.1.m
time="2020-08-26T14:06:37.367875774Z" level=debug msg="s3aws.GetContent("/docker/registry/v2/repositories/library/doggo-api/_layers/sha256/d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7/link")" auth.user.name="harbor_registry_user" go.version=go1.14.5 http.request.host=registry.host http.request.id=fbfa917d-a7f4-41bd-8a82-8438fadc5f25 http.request.method=HEAD http.request.remoteaddr=172.30.11.169 http.request.uri="/v2/library/doggo-api/blobs/sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" trace.duration=26.84362ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=4d98e52c-a803-4b85-b92e-d51b8fa2fd5d trace.line=95 vars.digest="sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7" vars.name="library/doggo-api"
time="2020-08-26T14:06:37.367975651Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="s3aws: AccessDenied: Access Denied
	status code: 403, request id: 4DD7564DBB51808B, host id: QBjvn4SoNmDkk8u2y9MDqTV/e8nnTlonTAlj7oKLxnxcYxiVDn8LliG7hiQYcmhfch9xY5EDlnc=" err.message="unknown error" go.version=go1.14.5 http.request.host=registry.host http.request.id=fbfa917d-a7f4-41bd-8a82-8438fadc5f25 http.request.method=HEAD http.request.remoteaddr=172.30.11.169 http.request.uri="/v2/library/doggo-api/blobs/sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=106.022665ms http.response.status=500 http.response.written=123 vars.digest="sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7" vars.name="library/doggo-api"
127.0.0.1 - - [26/Aug/2020:14:06:37 +0000] "HEAD /v2/library/doggo-api/blobs/sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7 HTTP/1.1" 500 123 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
`

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 16
  • Comments: 26 (8 by maintainers)

Most upvoted comments

My PR https://github.com/goharbor/harbor/pull/18686 solves using Harbor with IRSA.

@Tejuvmware @GowriRegistry Other than using a patched version of the registry like in my PR, I’d like to suggest allocating resources to the upstream distribution/distribution so that the next release (v3) can be cut soon. The current release branch will not have IRSA fixed, but it is already fixed on the main branch of distribution/distribution.

Opened this issue in docker’s distribution/distribution project (which is what Harbor uses as its registry service) having confirmed that building the harbor/registry image using the registry binary that can be compiled using the main branch (or edge tag) of that project resolves the problems we were having using IRSA, which were the same problems as those shared in this issue and elsewhere. To be absolutely clear though, these are the errors we were seeing prior to rebuilding the goharbor/registry image using the very latest registry binary from distribution:

time="2022-10-25T20:55:40.080713951Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="s3aws: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors" err.message="unknown error" go.version=go1.17.7 http.request.host=<REDACTED> http.request.id=721d9cad-4893-4fa1-8704-db965a992ead http.request.method=HEAD http.request.remoteaddr=<REDACTED> http.request.uri="/v2/thanos/thanos/blobs/sha256:9ca801fd774b0039343d2b85a7f628cbdb23dfdb90c4d390cdc3230f5e3af836" http.request.useragent="docker/20.10.17 go/go1.17.11 git-commit/a89b842 kernel/3.10.0-1160.11.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.17 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=137.472283ms http.response.status=500 http.response.written=104 vars.digest="sha256:9ca801fd774b0039343d2b85a7f628cbdb23dfdb90c4d390cdc3230f5e3af836" vars.name="thanos/thanos"

How did you solve this? I have the same problem reported by @rsilva-nk. In the values.yaml, Do I add the block config from Registry and in credentials I use the aws accesskey and secretkey ? Like: registry: serviceAccountName: "" registry: ... ... ... credentials: username: "accesskey" password: "secretkey"