charts: [ISSUE] Could not use AWS3V3 with EKS and service account role

Is this a request for help?: YES

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

it’s bug report

Version of Helm and Kubernetes: kubernetes

Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

helm 3.02 Which chart: artifactory-ha the last one in master Artifactory version

version:  7.4.1  
 Revision: 70401900

What happened: We want to use S3, so we setup AWS3V3 with eks iam roles. So we annotate the service account with the right iam roles with policy s3:* for all resource * but we could not upload or use S3

What you expected to happen: See my blob to be uploaded to S3.

How to reproduce it (as minimally and precisely as possible): Use this charts, configure s3v3 with EKS roles and try to upload an object to artifactory

Anything else we need to know: We try a pod with an AWS cli and the same service account and it works, I can dowload, list upload etc… to my bucket. Pod contains the right environment variable

AWS_ROLE_ARN=arn:aws:iam::MYAWSACCOUNTID:role/dev-artifactory-ha-pod-role

Token files exist

cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Logs

2020-04-21T08:41:07.754Z [jfrt ] [ERROR] [a7f842cf0d07ead4] [.s.b.p.RetryBinaryProvider:126] [-cluste
r-s3-worker-0] - Failed to check if blob '1c368c86f69048176f60b9ce43345f10f102d648' exist in next bin
ary provider
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; E
rror Code: 403 Forbidden; Request ID: 3E7787E73F42B2F1; S3 Extended Request ID: DA5KEGvqOdLPxSdNaF8xC
wwuoPlfJmKUmVr0YfYedwk58m489lH0LV/qwt6gROijlam+X+fcsfM=)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.j
ava:1799)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpC
lient.java:1383)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.jav
a:1359)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:11
39)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java
:764)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.j
ava:680)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1335)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1309)
	at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1390)
	at org.artifactory.addon.filestore.type.s3.S3ClientStorageService.exists(S3ClientStorageServi
ce.java:59)
	at org.artifactory.addon.filestore.type.s3.S3AwsBinaryProvider.exists(S3AwsBinaryProvider.jav
a:334)
	at org.artifactory.addon.filestore.type.s3.S3AwsBinaryProvider.exists(S3AwsBinaryProvider.jav
a:155)
	at org.jfrog.storage.binstore.providers.RetryBinaryProvider.existWithRetries(RetryBinaryProvi
der.java:118)
	at org.jfrog.storage.binstore.providers.RetryBinaryProvider.existWithRetries(RetryBinaryProvi
der.java:124)
	at org.jfrog.storage.binstore.providers.RetryBinaryProvider.existWithRetries(RetryBinaryProvi
der.java:124)
	at org.jfrog.storage.binstore.providers.RetryBinaryProvider.existWithRetries(RetryBinaryProvi
der.java:124)
	at org.jfrog.storage.binstore.providers.RetryBinaryProvider.existWithRetries(RetryBinaryProvi
der.java:124)
	at org.jfrog.storage.binstore.providers.RetryBinaryProvider.existWithRetries(RetryBinaryProvi
der.java:124)
	at org.jfrog.storage.binstore.providers.RetryBinaryProvider.exists(RetryBinaryProvider.java:5
8)
	at org.artifactory.addon.filestore.eventual.cluster.EventualDownstreamWorker.handleAdd(Eventu
alDownstreamWorker.java:85)
	at org.artifactory.addon.filestore.eventual.cluster.EventualDownstreamWorker.call(EventualDow
nstreamWorker.java:64)
	at org.artifactory.addon.filestore.eventual.cluster.EventualDownstreamWorker.call(EventualDow
nstreamWorker.java:25)
	at org.artifactory.opentracing.TraceableCallableDecorator.call(TraceableCallableDecorator.jav
a:24)```


binarystore.xml

<config version="2"> <chain> <provider id="cache-fs-eventual-s3" type="cache-fs"> <provider id="sharding-cluster-eventual-s3" type="sharding-cluster"> <sub-provider id="eventual-cluster-s3" type="eventual-cluster"> <provider id="retry-s3" type="retry"> <provider id="s3-storage-v3" type="s3-storage-v3"/> </provider> </sub-provider> <dynamic-provider id="remote-s3" type="remote"/> </provider> </provider> </chain>

<provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
    <readBehavior>crossNetworkStrategy</readBehavior>
    <writeBehavior>crossNetworkStrategy</writeBehavior>
    <redundancy>3</redundancy>
    <property name="zones" value="local,remote"/>
</provider>

<provider id="remote-s3" type="remote">
    <zone>remote</zone>
</provider>

<provider id="eventual-cluster-s3" type="eventual-cluster">
    <zone>local</zone>
</provider>

<!-- Set max cache-fs size -->
<provider id="cache-fs-eventual-s3" type="cache-fs">
    <maxCacheSize>5e+10</maxCacheSize>
    <cacheProviderDir>cache</cacheProviderDir>
</provider>
<provider id="s3-storage-v3" type="s3-storage-v3">
    <testConnection>false</testConnection>
    <region>ap-northeast-1</region>
    <bucketName>dev-artifactory-ha-storage-bucket</bucketName>
    <path>artifactory/filestore</path>
    <endpoint></endpoint>
    <useInstanceCredentials>true</useInstanceCredentials>
    <usePresigning>false</usePresigning>
    <signatureExpirySeconds>300</signatureExpirySeconds>
</provider>