chartmuseum: Access Denied with EKS Service Account

I’m using the Helm chart with version 2.12.0 which says cahrtmuseum version is 0.12.0.

I followed exactly the instructions for linked service accounts. I have done this many times to other Helm charts and it always has worked before doing the exact process described.

Here is the error I get:

2020-05-01T20:52:29.225970789Z {“L”:“ERROR”,“T”:“2020-05-01T20:52:29.225Z”,“M”:“AccessDenied: Access Denied\n\tstatus code: 403, request id: someid, host id: /someid/someid/someid/someid=”,“repo”:“”} 2020-05-01T20:52:29.226080384Z 2020-05-01 20:52:29.226031 I | AccessDenied: Access Denied 2020-05-01T20:52:29.226089093Z status code: 403, request id: someid, host id: /someid/someid/someid/someid=

Here is the policy I used:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::my-helm-repo",
            "Effect": "Allow",
            "Sid": "AllowListObjects"
        },
        {
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::my-helm-repo/*",
            "Effect": "Allow",
            "Sid": "AllowObjectsCRUD"
        }
    ]
}

The trust relationship looks good too:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::someid:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/someid"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-west-2.amazonaws.com/id/someid": "system:serviceaccount:default:chartmuseum-chartmuseum"
        }
      }
    }
  ]
}

here are all my values too:

env:
  open:
    STORAGE: amazon
    STORAGE_AMAZON_PREFIX: charts
    STORAGE_AMAZON_REGION: us-west-2
    STORAGE_AMAZON_BUCKET: my-helm-repo
    CHART_URL: https://charts.example.com/museum
    DISABLE_API: false
    ALLOW_OVERWRITE: true
    DISABLE_METRICS: false
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
  - name: charts.example.com
    path: "/museum"
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: the-correct-role-arn
serviceMonitor:
  enabled: true

Any idea why I still get access denied?

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 5
Comments: 22 (7 by maintainers)

Most upvoted comments

@jdolitsky @outime Your suggestions worked! Thank you. The following settings now successfully connect with AWS using IRSA

env:
  open:
    AWS_SDK_LOAD_CONFIG: true
serviceAccount: 
  create: true
  annotations: 
    "eks.amazonaws.com/role-arn": myRoleArn
securityContext: 
  enabled: true
  fsGroup: 65534

+21

kferrone on Jul 14, 2020

I was able to workaround this by adding a truthy environment variable to an EKS deployment:

AWS_SDK_LOAD_CONFIG="true"

+13

terry-sm on May 29, 2020

I think by default in the latest chart it uses fsGroup: 1000. Set the pod to local storage but keep the iam annotation, and you can see:
/ $ whoami
chartmuseum
/ $ ls -alsh /var/run/secrets/eks.amazonaws.com/serviceaccount
total 0
     0 drwxrwsrwt    3 root     chartmus     100 Jul 14 16:03 .
     0 drwxr-xr-x    3 root     root          28 Jul 14 16:03 ..
     0 drwxr-sr-x    2 root     chartmus      60 Jul 14 16:03 ..2020_07_14_16_03_16.1
     0 lrwxrwxrwx    1 root     root          31 Jul 14 16:03 ..data -> ..2020_07_14_16_03_16.1
     0 lrwxrwxrwx    1 root     root          12 Jul 14 16:03 token -> ..data/token
and when you cat the token it works. I think the only change needed is to set the env variable AWS_SDK_LOAD_CONFIG=true

You make my day! Thank you!!! I spent about 2 days with my s3 bucket configuration and cross-account access settings and role configuration for the EKS service account without results. And this little environment variable resolves the issue!

My current configuration works perfect without securityContext in it. Just adding the AWS_SDK_LOAD_CONFIG: true resolves the issue!

env:
  open:
    STORAGE: "amazon"
    STORAGE_AMAZON_BUCKET: "${bucket_name}"
    STORAGE_AMAZON_PREFIX: "${bucket_prefix}"
    STORAGE_AMAZON_REGION: "${bucket_region}"
    AWS_SDK_LOAD_CONFIG: true
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "${iam_role_arn}"

P.S. It was not is official chart documentation, I think it must be added there to save a lot or human lives 😃

artem-kosenko on Oct 5, 2021

I think by default in the latest chart it uses fsGroup: 1000. Set the pod to local storage but keep the iam annotation, and you can see:

/ $ whoami
chartmuseum
/ $ ls -alsh /var/run/secrets/eks.amazonaws.com/serviceaccount
total 0
     0 drwxrwsrwt    3 root     chartmus     100 Jul 14 16:03 .
     0 drwxr-xr-x    3 root     root          28 Jul 14 16:03 ..
     0 drwxr-sr-x    2 root     chartmus      60 Jul 14 16:03 ..2020_07_14_16_03_16.1
     0 lrwxrwxrwx    1 root     root          31 Jul 14 16:03 ..data -> ..2020_07_14_16_03_16.1
     0 lrwxrwxrwx    1 root     root          12 Jul 14 16:03 token -> ..data/token

and when you cat the token it works. I think the only change needed is to set the env variable AWS_SDK_LOAD_CONFIG=true

jdfalk on Jul 14, 2020

@ridicule777 It’s all about the security context. Very specifically the 65534.

securityContext: 
  enabled: true
  fsGroup: 65534

kferrone on Nov 28, 2020

Hi,

I have the exact same problem. These are my helm values.
env:
  open:
    AWS_SDK_LOAD_CONFIG: true
    STORAGE: amazon
    STORAGE_AMAZON_BUCKET: bucket-name
    STORAGE_AMAZON_PREFIX:
    STORAGE_AMAZON_REGION: region-name
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::${account-id}:role/${role-name}"
Tried with AWS_SDK_LOAD_CONFIG: true and AWS_SDK_LOAD_CONFIG: 1. No luck.

@ridicule777

You need to quote “true”, otherwise it will not be deployed (https://github.com/chartmuseum/charts/blob/main/src/chartmuseum/templates/deployment.yaml#L64)

kj187 on Aug 29, 2021

@jdolitsky that solves it partially. You also need to add a securityContext to the deployment since by default, chartmuseum is ran by a non-root used. By adding a fsGroup with value 65534 it makes it possible for the non-root user to read the files that will finally allow to obtain proper auth for AWS calls.

outime on Jun 4, 2020