harbor: Not able to use AWS IAM role in harbor registry with pre-release v2.5.0-rc1 for AWS S3 backend

Expected behavior and actual behavior: Should be able to replicate the harbour images to an AWS S3 storage bucket. As per this PR https://github.com/goharbor/harbor/pull/16435 which is included in pre-release v2.5.0-rc1 there should now be support for IAM roles for Service Accounts in AWS EKS. This does now work for harbor chartmuseum but we still have an issue with harbor registry (see error log below).

Steps to reproduce the problem: Create a replication rule to pull from AWS ECR with a destination endpoint of an S3 bucket. Fails to write the images due to an AWS credentials issue. The harbor registry pod has the correct service account mounted with the correct IAM privileges to access and write to the S3 bucket.

Versions:

harbor version: [v2.5.0-rc1]

goharbor/registry-photon:v2.5.0-rc1 goharbor/harbor-registryctl:v2.5.0-rc1

Additional context: harbor helm values:

persistence: imageChartStorage: type: s3 s3: bucket: my-aws-s3-bucket region: eu-west-1

Error message in harbor registry pod when trying to replicate image to an AWS S3 backend using a service account and IAM policy:

level=info msg="authorized request" go.version=go1.17.7 http.request.host="harbor-core:80" http.request.id=ea5f89c9-677e-4dee-ae0a-4f9a60323086 http.request.method=HEAD http.request.remoteaddr=100.64.248.123  │
│ time="2022-03-08T21:38:53.230230565Z" level=error msg="response completed with error" auth.user.name=admin err.code=unknown err.detail="s3aws: NoCredentialProviders: no valid providers in chain. Deprecated.
"HEAD /v2/dce/test-bats-2ef5098/blobs/sha256:0bd848d9f7a107ab36d5c05f0c881b3edfc39f9a239f018f8013cb685c54ed5b HTTP/1.1" 500 104 "" "harbor-registry-client"

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 15
Comments: 23 (9 by maintainers)

Most upvoted comments

It seems that distribution v2.8 supports this feature, the Harbor replication needs to change for this feature

stonezdj on Mar 9, 2022

We just ran into this when setting up harbor. Quite a bummer to be honest. Given the mentioned comment from the discussion, I second the idea of pinning an edge tag for distribution.

Jasper-Ben on Mar 20, 2023

Same Issue has been raised here https://github.com/goharbor/harbor/issues/18699. Closed as duplicate.

Hello Team,

Given that the AWS SDK supports assuming a role , pods running in EKS/GKE with the storage target as AWS S3 should be able to assume a role to connect to the S3 buckets.

Example or Brief can be found here : https://confluence.eng.vmware.com/display/public/AEAV/Service+User+Model

Versions: Please specify the versions of following systems.

harbor version: [2.3.3] (via helm chart) kubernetes 1.20.6 Cluster : GKE Storage : AWS S3

Expected behavior and actual behavior:

Expected: Pods using ServiceAccounts annotated to assume a role with should have access/denial to resources as specified in the policies attached to the role. These assume role credentials generates session token with the validity of 12 hours. Need a mechanism to re-establish connection with the AWS before the session token expiry.
Actual: Since pods are not assuming the role, one cannot, for instance, use s3 as a storage backend for the registry. Currently Harbor doesn’t support this kind of AWS connectivity and it is a blocker for us to get on-boarded with VMware CloudGate.

It would be great if this feature request can be prioritised as

BLOKER : Currently It is a Hard blocker from Harbor for us to get TanzuNet Production AWS accounts to get on-boarded with VMware CloudGate. Having this feature request implemented resolves our blocker.

Let me know if any further details are required.

Thanks,

Tejuvmware on Jul 10, 2023

Hi @wy65701436, any update on if this is possible? This would really help us move through this long standing blocker!

daemon-ian on Mar 23, 2023

as per the distribution discussion:

The TL;DR is: 2.8 is so far behind main that keeping it up to date with main or even close to it 
is a colossal amount of work, not to mention that it's not even using go modules.
We made a decision that 2.8 will only receive security patches from then on 
and most of the work should focus on v3 release.

@wy65701436 is it at all possible to pin to an edge tag with the aws-sdk-go updates included? Distribution will never push this update (until v3.0, but who knows when that will be ready)

chrisminton on Mar 20, 2023

Hi @MinerYang @wy65701436 @YangJiao0817, quite a few people seem to be seeing this issue now. Can you share an update please?

daemon-ian on Jun 16, 2022

I am still unsure if we should keep it open or only close it once it is resolved upstream. Given the velocity of Docker Distribution this won’t happen within the next 90 days.

Vad1mo on Jul 13, 2022