secrets-store-csi-driver-provider-aws: Failure getting secret values from provider type secretsmanager: Failed to fetch secret from all regions

Describe the bug

k logs -n kube-system csi-secrets-store-provider-aws-nnjvn

I0126 07:28:50.650576       1 auth.go:123] Role ARN for default:govplt-deployment-sa is arn:aws-cn:iam::655418457877:role/KONE-EKS-govplt-secrets-readonly
E0126 07:28:50.892712       1 server.go:151] Failure getting secret values from provider type secretsmanager: Failed to fetch secret from all regions: /kone/govplt/elasticache/secret
I0126 07:30:53.184809       1 server.go:124] Servicing mount request for pod nginx-deployment-k8s-secrets-66d5f46844-v4w92 in namespace default using service account govplt-deployment-sa with region(s) cn-north-1
I0126 07:30:53.247115       1 auth.go:123] Role ARN for default:govplt-deployment-sa is arn:aws-cn:iam::655418457877:role/KONE-EKS-govplt-secrets-readonly
E0126 07:30:53.609110       1 server.go:151] Failure getting secret values from provider type secretsmanager: Failed to fetch secret from all regions: /kone/govplt/elasticache/secret
I0126 07:32:55.771473       1 server.go:124] Servicing mount request for pod nginx-deployment-k8s-secrets-66d5f46844-v4w92 in namespace default using service account govplt-deployment-sa with region(s) cn-north-1
I0126 07:32:55.777593       1 auth.go:123] Role ARN for default:govplt-deployment-sa is arn:aws-cn:iam::655418457877:role/KONE-EKS-govplt-secrets-readonly
E0126 07:32:56.183717       1 server.go:151] Failure getting secret values from provider type secretsmanager: Failed to fetch secret from all regions: /kone/govplt/elasticache/secret

k logs -n kube-system secrets-store-csi-driver-wwfxn

I0126 07:32:55.708010       1 server.go:151] "request" method="/csi.v1.Node/NodePublishVolume" req="{\"readonly\":true,\"target_path\":\"/var/lib/kubelet/pods/76bee6b4-1e9d-421b-a401-88c08c8f328f/volumes/kubernetes.io~csi/secrets-store-inline/mount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{}},\"access_mode\":{\"mode\":1}},\"volume_context\":{\"csi.storage.k8s.io/ephemeral\":\"true\",\"csi.storage.k8s.io/pod.name\":\"nginx-deployment-k8s-secrets-66d5f46844-v4w92\",\"csi.storage.k8s.io/pod.namespace\":\"default\",\"csi.storage.k8s.io/pod.uid\":\"76bee6b4-1e9d-421b-a401-88c08c8f328f\",\"csi.storage.k8s.io/serviceAccount.name\":\"govplt-deployment-sa\",\"secretProviderClass\":\"nginx-deployment-spc-k8s-secrets\"},\"volume_id\":\"csi-9a2de871afbdd7628333c241e09fb619c5824541fffce76450288400e8fa5b8e\"}" deadline="1m59.997897253s"
I0126 07:32:55.708114       1 nodeserver.go:144] "node publish volume" target="/var/lib/kubelet/pods/76bee6b4-1e9d-421b-a401-88c08c8f328f/volumes/kubernetes.io~csi/secrets-store-inline/mount" volumeId="csi-9a2de871afbdd7628333c241e09fb619c5824541fffce76450288400e8fa5b8e" attributes=map[csi.storage.k8s.io/ephemeral:true csi.storage.k8s.io/pod.name:nginx-deployment-k8s-secrets-66d5f46844-v4w92 csi.storage.k8s.io/pod.namespace:default csi.storage.k8s.io/pod.uid:76bee6b4-1e9d-421b-a401-88c08c8f328f csi.storage.k8s.io/serviceAccount.name:govplt-deployment-sa secretProviderClass:nginx-deployment-spc-k8s-secrets] mount flags=[]
I0126 07:32:55.708169       1 mount_linux.go:219] Mounting cmd (mount) with arguments (-t tmpfs tmpfs /var/lib/kubelet/pods/76bee6b4-1e9d-421b-a401-88c08c8f328f/volumes/kubernetes.io~csi/secrets-store-inline/mount)
I0126 07:32:55.717713       1 nodeserver.go:359] "Using gRPC client" provider="aws" pod="nginx-deployment-k8s-secrets-66d5f46844-v4w92"
E0126 07:32:56.184444       1 nodeserver.go:242] "failed to mount secrets store object content" err="rpc error: code = Unknown desc = Failed to fetch secret from all regions: /kone/govplt/elasticache/secret" pod="default/nginx-deployment-k8s-secrets-66d5f46844-v4w92"
I0126 07:32:56.185002       1 nodeserver.go:88] "unmounting target path as node publish volume failed" targetPath="/var/lib/kubelet/pods/76bee6b4-1e9d-421b-a401-88c08c8f328f/volumes/kubernetes.io~csi/secrets-store-inline/mount" pod="default/nginx-deployment-k8s-secrets-66d5f46844-v4w92"
I0126 07:32:56.185069       1 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/76bee6b4-1e9d-421b-a401-88c08c8f328f/volumes/kubernetes.io~csi/secrets-store-inline/mount
I0126 07:32:56.193452       1 server.go:155] "response" method="/csi.v1.Node/NodePublishVolume" deadline="1m59.997897253s" duration="485.54268ms" status.code="Unknown" status.message="failed to mount secrets store objects for pod default/nginx-deployment-k8s-secrets-66d5f46844-v4w92, err: rpc error: code = Unknown desc = Failed to fetch secret from all regions: /kone/govplt/elasticache/secret"

To Reproduce

Steps to reproduce the behavior:

nginx-deployment-spc-k8s-secrets.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-spc-k8s-secrets
spec:
  provider: aws
  parameters:
    region: cn-north-1
    objects: |
      - objectName: "/kone/govplt/auroramysql/connectionstr"
        objectAlias: "db_conn_str"
        objectType: "ssmparameter"
      - objectName: "/kone/govplt/elasticache/secret"
        objectType: "secretsmanager"
        jmesPath:
          - path: endpoints
            objectAlias: redis_endpoint
          - path: port
            objectAlias: redis_port

  secretObjects:
    - secretName: kone-govplt-secret # name of the Kubernetes Secret object
      type: Opaque # type of the Kubernetes Secret object e.g. Opaque, kubernetes.io/tls
      labels:
        provider: "csi-driver"
      data:
        - objectName: redis_endpoint # name of the mounted content to sync. this could be the object name or the object alias
          key: redis_endpoint # data field to populate
        - objectName: redis_port
          key: redis_port
        - objectName: db_conn_str
          key: db_conn_str

nginx-deployment-k8s-secrets.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-k8s-secrets
  labels:
    app: nginx-k8s-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-k8s-secrets
  template:
    metadata:
      labels:
        app: nginx-k8s-secrets
    spec:
      serviceAccountName: govplt-deployment-sa
      containers:
      - name: nginx-deployment-k8s-secrets
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 80
        volumeMounts:
          - name: secrets-store-inline
            mountPath: "/mnt/secrets"
            readOnly: true
        # I have turn on syncSecret.enable = true
        env:
          - name: DB_CONN_STR
            valueFrom:
              secretKeyRef:
                name: kone-govplt-secret
                key: db_conn_str
          - name: REDIS_PORT
            valueFrom:
              secretKeyRef:
                name: kone-govplt-secret
                key: redis_port

          - name: REDIS_ENDPOINT
            valueFrom:
              secretKeyRef:
                name: kone-govplt-secret
                key: redis_endpoint

      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: nginx-deployment-spc-k8s-secrets

aws-provider-installer.yml

# https://kubernetes.io/docs/reference/access-authn-authz/rbac
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-secrets-store-provider-aws
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csi-secrets-store-provider-aws-cluster-role
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csi-secrets-store-provider-aws-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csi-secrets-store-provider-aws-cluster-role
subjects:
- kind: ServiceAccount
  name: csi-secrets-store-provider-aws
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: csi-secrets-store-provider-aws
  labels:
    app: csi-secrets-store-provider-aws
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-aws
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-aws
    spec:
      serviceAccountName: csi-secrets-store-provider-aws
      hostNetwork: false
      containers:
        - name: provider-aws-installer
          image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-35-g41dc61e-2022.12.16.20.38
          imagePullPolicy: Always
          args:
              - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
          volumeMounts:
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: HostToContainer
      volumes:
        - name: providervol
          hostPath:
            path: "/etc/kubernetes/secrets-store-csi-providers"
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
      nodeSelector:
        kubernetes.io/os: linux

Environment:

Secrets Store CSI Driver version: (use the image tag): csi-secrets-store/driver v1.3.0 csi-node-driver-registrar v2.6.2 livenessprobe v2.8.0
Kubernetes version: (use kubectl version): 1.23

Additional context the strangest thing that deployment pod can run up success if I comment out the objectType: “secretsmanager” section

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-spc-k8s-secrets
spec:
  provider: aws
  parameters:
    # region: cn-north-1
    objects: |
      - objectName: "/kone/govplt/auroramysql/connectionstr"
        objectAlias: "db_conn_str"
        objectType: "ssmparameter"
#      - objectName: "/kone/govplt/elasticache/secret"
#        objectType: "secretsmanager"
#        jmesPath:
#          - path: endpoints
#            objectAlias: redis_endpoint
#          - path: port
#            objectAlias: redis_port

  secretObjects:
    - secretName: kone-govplt-secret # name of the Kubernetes Secret object
      type: Opaque # type of the Kubernetes Secret object e.g. Opaque, kubernetes.io/tls
      labels:
        provider: "csi-driver"
      data:
        - objectName: db_conn_str
          key: db_conn_str
#        - objectName: redis_endpoint # name of the mounted content to sync. this could be the object name or the object alias
#          key: redis_endpoint # data field to populate
#        - objectName: redis_port
#          key: redis_port

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-k8s-secrets
  labels:
    app: nginx-k8s-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-k8s-secrets
  template:
    metadata:
      labels:
        app: nginx-k8s-secrets
    spec:
      serviceAccountName: govplt-deployment-sa
      containers:
      - name: nginx-deployment-k8s-secrets
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 80
        volumeMounts:
          - name: secrets-store-inline
            mountPath: "/mnt/secrets"
            readOnly: true
        #   syncSecret.enable = true
        env:
          - name: DB_CONN_STR
            valueFrom:
              secretKeyRef:
                name: kone-govplt-secret
                key: db_conn_str
#          - name: REDIS_PORT
#            valueFrom:
#              secretKeyRef:
#                name: kone-govplt-secret
#                key: redis_port
#
#          - name: REDIS_ENDPOINT
#            valueFrom:
#              secretKeyRef:
#                name: kone-govplt-secret
#                key: redis_endpoint

      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: nginx-deployment-spc-k8s-secrets

i am sure serviceAccountName: govplt-deployment-sa have secretsmanager permission

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 25

Most upvoted comments

For us, we used a secret that contained a hyphen which needed to be quoted. The newer image seemed to be hiding the error message and saying “Failed to fetch secret from all regions” but the actual error was a failing jmespath query.

https://github.com/jmespath/jmespath.py/issues/109#issuecomment-346917804

 apiVersion: secrets-store.csi.x-k8s.io/v1
 kind: SecretProviderClass
 metadata:
   name: secrets-provider
 spec:
   provider: aws
   parameters:
     region: us-east-1
     objects: |
      - objectName: "secret/api-key"
        objectType: secretsmanager
        jmesPath:
          - path: "\"api-key\""
            objectAlias: apiKey

our json in the secret

{ "api-key": "apipaipaiapiapia" }

using jp

✗ echo '{ "api-key": "apipaipaiapiapia" }' | jp 'api-key'
SyntaxError: Unexpected token at the end of the expression: tNumber
api-key
   ^

✗ echo '{ "api-key": "apipaipaiapiapia" }' | jp '"api-key"'
"apipaipaiapiapia"

Older image that showed the real error for us is the image 1.0.r2-6-gee95299-2022.04.14.21.07. After we resolved it, we went back to the latest 1.0.r2-35 image.

https://gallery.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws

nitrocode on Feb 4, 2023

We have enhanced the log output with change https://github.com/aws/secrets-store-csi-driver-provider-aws/pull/190 and believe this should help with the majority of issues in this thread. Please open a new issue if you continue to have any issues.

jbct on Apr 18, 2023

well, finally i choose https://github.com/external-secrets/external-secrets solution. it is more easy and less bug

mafeifan on Mar 5, 2023

@daithi-walker In our environment we were unable to access any AWS services without a proper Egress network policy. The secret store driver would not say anything about inability to reach the secret manager IP, etc, so it was pretty misleading. It would be pretty neat if we could set some verbosity level (as far as I can tell the helm chart doesn’t support that) to see the root causes in the logs.

Once I fixed that, I had to add to the permission policy of the assumed role kms:Decrypt and kms:DescribeKey on the encryption key used for the secret. As far as I can tell, the guides out there don’t mention this, but I was getting decrypt errors (properly logged at least 😃 on the pod startup (on the pod with mounted secret that). Once I fixed that mount + env var started to work.

I hope this helps.

istvan-fodor on Feb 11, 2023

i was having the same issue as @mafeifan, and was able to narrow the issue down to the fact that one of my secrets, which stores key/value pairs in JSON format, had a quoted integer value for one of its keys. i also saw that the issue only manifests when attempting to use jmesPath: to access these keys directly, as mentioned above.

in particular, it will fail when trying to access a key which begins with an integer. if you comment only that key out under jmesPath, and retrieve other keys which don’t begin with (or consist entirely of) numeric chars, it will work. also if i simply write the entire secret to a file in the container (i.e. comment out the entire jmesPath: section), it works, and writes the JSON content as expected. this shows that the issue doesn’t have anything to do with IAM privs or even invalid JSON. i suspect this is just a bug in the way the jmesPath mechanism parses JSON keys in secrets.

unfortunately this may not solve the OPs problem, as it appears they are not using any keys which begin with integers, but based on their posts, it seems safe to say that the issue is contained within the jmesPath mechanism, and i would guess that there’s something about one or more of your JSON keys that it doesn’t like.

AdamStaudt on Jan 31, 2023