actions-runner-controller: Runner Pod can't start up with ServiceAccount on EKS 1.15.10

We already have the documentation for this at https://github.com/summerwind/actions-runner-controller#using-eks-iam-role-for-service-accounts so it should be straightforward today. Closing as resolved!

No without the annotation I was not able to get IAM working with IAM. I needed to include:

• The annotation to the base IAM role in my service account, without this auth wouldn’t work
• Add in envs, volumeMounts and volume that the annotation automatically injects. The apps manager seemed to kill the pod if you let the annotation do this, I pressume because it fails some sort of validation?
• Add in the fsGroup so the token is readable, 65534 or 27 seemed to work. 65534 broke sudo though whereas 27 didn’t

My final yaml (with aws account id’s replaced with $AWS_ACCOUNT_ID)

kind: ServiceAccount
metadata:
  name: sre-actions-runner-system-sa-ajh
  namespace: actions-runner-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::$AWS_ACCOUNT_ID:role/eksctl-sre-eks-test-addon-iamserviceaccount-Role1-1244USHBUKLEY 
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-ajh
  namespace: actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      organization: org
      serviceAccountName: sre-actions-runner-system-sa-ajh
      env:
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::$AWS_ACCOUNT_ID:role/eksctl-sre-eks-test-addon-iamserviceaccount-Role1-1244USHBUKLEY
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      volumeMounts:
      - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
        name: aws-iam-token 
        readOnly: true
      volumes:
      - name: aws-iam-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
      securityContext:
        fsGroup: 27
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: hra-1-ajh
  namespace: actions-runner-system
spec:
  scaleDownDelaySecondsAfterScaleUp: 10
  scaleTargetRef:
    name: runner-ajh
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
    repositoryNames:
    - test-repo

After that it was just normal the AWS assume role setup.

I think it’s related to EKS IRSA. It works for me when I remove the annotation eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE>.

But with that annotation present, the controller keeps looping creating and destroying the runner pod.

EKS IRSA introduces envvars and volumes automatically in the pod:

  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token

volumeMounts:
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true

env
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::xxxxxx:role/xxxxx
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Maybe those collide with some validation at the controller.

I’ve checked the pod definitions when using EKS IRSA and they’re fine. It’s the controller who destroys the pod.

I tested this on vanilla Kubernetes setup and service account was mounted to Runner Container without any issues.