velero: kops master is unauthorized to attach ark-restored volumes

Hi, I’m trying to solve the nginx use-case but using S3 for object storage. I’m trying to restore a backup which is created by running this command ark backup create nginx-backup --selector app=nginx --snapshot-volumes. The command used for restoring is ark restore create nginx-backup --restore-volumes.

The backup however is being created successfully and the backup files are uploaded to the object storage and the snapshot is getting created. The issue that I’m facing is that it is pointing to the same PV while restoring in a different k8s cluster. And the pod that is supposed to restore is struck in STATUS ContainerCreating. Is there anyway that I can get it to create a new PV while restoring in a different cluster?

Output of ark backup describe nginx-backup

Name:         nginx-backup
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app=nginx

Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Phase:  Completed

Backup Format Version:  1

Expiration:  2018-04-07 19:10:11 +0000 UTC

Validation errors:  <none>

Persistent Volumes:
  pvc-2db45bb0-22f8-11e8-82f2-0e21f011a24c:
    Snapshot ID:        snap-0c9aad251b280516d
    Type:               gp2
    Availability Zone:  us-east-1a
    IOPS:               <N/A>

Output of ark restore describe nginx-backup-20180308194129

Name:         nginx-backup-20180308194129
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Backup:  nginx-backup

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  true

Phase:  Completed

Validation errors:  <none>

Warnings:  <none>
Errors:    <none>

yaml file used to create nginx-example

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx-example
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: nginx-logs
          persistentVolumeClaim:
           claimName: nginx-logs
      containers:
      - image: nginx:1.7.9
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/var/log/nginx"
            name: nginx-logs
            readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: my-nginx
  namespace: nginx-example
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

NOTE: I’m using the k8s nodes iam instance profile to provide EC2 and S3 access to the ark server instead of a secret as mentioned in the example use-case.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 34 (24 by maintainers)

Most upvoted comments

If you’re using an IAM policy for Ark in AWS, make sure you add ec2:DescribeSnapshots to the policy.

Hey @ncdc,

Yes, the two clusters are in the same region us-east-1 and the same account as well.

Output of kubectl -n nginx-example describe pod

 Warning  FailedMount            29m                 attachdetach                            AttachVolume.Attach failed for volume "pvc-2db45bb0-22f8-11e8-82f2-0e21f011a24c" : Error attaching EBS volume "vol-03d0039cdcca2fc9a" to instance "i-09b03f4a4bf903960": "UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: S-a2ASLzwJ9nBVU6f0aibSyj2D3pZgyJOyINWAq2dPNTNh9zIfR18JmUHovsgW3V5v0VprZLG7Ih-Xr_7TDgntbeITrxxG1jJ6kLAtNYpAOnjvPmBvuc8skXP5NSg5-lx8NHXtDM6URR0-SX3QjY1bAUtrVEs9iCEbB5TyiIpcxJ9KB8d25XFSDjCrxD6pW69zsEZwiDz3roPizswzWKTdu-zg2J3C0N-IoqtgkTMfwVPwQXigJBdDWYbzO2JkbIGOq_3T-i46fDEJaBR_7MpXRjymYyHC-QAmAoriQU4Ompmlg9cJFBj0hhjPimHr1By9xgy0O7KdrBpPpGj40FVK5XgPBTMrpaAhGk6oJJEGiXCPiuwubl4l3APYGtYtVj6SGWiqJfODq3PyNj7R2g6JiyAEMW2r36E-c56Ezo-8cqdmGkKEeFuFSyVebpabnQpO9IkZTqCUX7RViI73yH7bHBtboKT0gwPd7zg76_IpfUYkiaoqbkLWUyP03E1VWL7HGXfUSe6K2Ix_bM8-qHl6VCUB8oxbVDleHO0uiH6Jm-PmSe-I3WDMpF8MlJTsgOxLQp2Yx9pnEA6Js-MGe9UdsjYpyJ_pKC2vpyMJxRZ-fKszyhVon5frBsbtZS48Isam38BgIK-qZBc7_41B7rBKbBjdG6NZJV5R-VYhYW-dE-R5VHcEcRxBiqXtCoSw7X3Ix2zkOyqvdCNAM\n\tstatus code: 403, request id: 4b8cf55d-872c-4103-b0b1-7255c540a1c9"
  Warning  FailedMount            29m                 attachdetach                            AttachVolume.Attach failed for volume "pvc-2db45bb0-22f8-11e8-82f2-0e21f011a24c" : Error attaching EBS volume "vol-03d0039cdcca2fc9a" to instance "i-09b03f4a4bf903960": "UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: w0GQMhnWgO3aIGEe35lnsOpwS63v21ouwE-BPntKb_uhYZuj5NVwrPzs0vSFE-2Fqh2XKqrHAuMZ220gsjP51yxmLmdcRK4_4ZmfY9UyKG5PTamND34W65_6uADj2C_kZVrh28E5ut4N5gDd8WNF0jpjabBu8D27Ym8P_f2C0rHe3Vu-1IpuwHHMrjashgEW-7FK8-yymrhQW3822swy8ycGQQbViBiG8Himybev2wz5-ni5MN0LVZS8ifoTTOq4l7JiTBogQQ9bSPFWA96F_h0Rt-czFhZdyljXKxdPXgmffCG-PDoFCEo0zqWzoMccmdQd8M458Pz8YfUW0EI4WM64SdkVIwhX5yqsADWODU9ZDAS_9ClLca4tA5Th92HhV-2p4k2VbpDNLxL9jw_kLSY9K-SUtBn0woGiR9Mjljfx303XrlyFppSIYd0yptkyxLGWxm2Dk_cbBLvZf26iGt0fv0RzOVE4LA0jHu8jGT1YbMp62Cqz9-qtIwnwl18kA8FbQL0oGwPznhrKqTVlCjkAuUeWChZs2XpOpzvb13v-d2Ttg1SGjt8i2W7qJBrM2sV5X6Ir2CyxRqRKbOr4EwEa4QQRKrTkRcp5qVb3_p7QPQ7zdYMPxLmzMn7zpvUaLYbi_hatBG4c5ptyJQ4cHuj8Yl8Klk8d4EjN3AWU7nTKqPme5xHwh8s8xLG01N8z1bybv-9U9IbfkgA\n\tstatus code: 403, request id: f56d6588-a117-4ded-b02c-8b5b868d82ba"
  Warning  FailedMount            28m                 attachdetach                            AttachVolume.Attach failed for volume "pvc-2db45bb0-22f8-11e8-82f2-0e21f011a24c" : Error attaching EBS volume "vol-03d0039cdcca2fc9a" to instance "i-09b03f4a4bf903960": "UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: Nw_DiAjFtK-wC88Xzi5dC_1BUdncs1YycTIhGUfX9jwLfOowJveeMlR9dCL5zQV_whY30_UHejiXy39cAGnNs0Pr8YkatBuPHBcBngiy_cSxhzp2wNbhNyJxx3jZ3pF_0HMNuzJmn5oSRl2v7NBcHR9WQuADr7fzrAJs21nZBGgZjGT201_AE8iNZyu2yGqPNMmJX0ZaOdaMqy2XDPvR795WUGnaqrqr7D_mQNQohrNLAzQvN0iJa30iCGQYtg4FnDJy0QV65Zlr-U640hIOf08lVfMjc8Tn5X_a3XkqOhBtittW9tYoN0YO_W0_YuGvrnZM5xx0py1Vk57saQaBfmZz_Ii1wAqjmaVhLSJz7vwrgUJQrF6OUfqfvSrWOEIxisC3hXDHOYu3chJMD63UOFyhdoUuu3hGxyjb-LfgO2TaMZl0bnXlj1p4YXAq2wTP1xQbv2bOXsdanOUvrlwhpt_0p2SIDCWZkCs2y92SfQV6rUAhRIRBvyi_sCc2v6dbae9or6hyMLlqDRAoZuh5bivW4JrbkbilfU01A-tQ46PHgcsWYWzYOb6_aNgVaigaZSF4DLbORAl1HG41u5SpjtF8Om1lGOiCtb4EFenEHCdjqp7MpL744jbxd2Se45yuKhKfVh0IMXcF75zPt42cg5TnA0lXYOQ-J9EwTA63gV-ELx2n_hh5SL0_0c98PxevXIP7eIOqklkw0s4\n\tstatus code: 403, request id: c004f9d0-cee8-4979-927c-508f986f29b9"
  Warning  FailedMount            27m                 attachdetach                            AttachVolume.Attach failed for volume "pvc-2db45bb0-22f8-11e8-82f2-0e21f011a24c" : Error attaching EBS volume "vol-03d0039cdcca2fc9a" to instance "i-09b03f4a4bf903960": "UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: pZtXYGQz9FOeEercFJxYzcLJnueLAZWw6ztjrziTUym342AL4B4LL7qqWrTTLQSnq3tYHtQHeCPZlteda4C6VLGKOjvC62dIxJxW88b27jfwlDpYp3nUYQApLqRbxWezafaBjpkCdyS9t2-jJ5HX39fzHiIWQLFsJ67xG1H9n7TVoHGMTeUDkqml3eY8wLZa4C_KUC0XwKODU-A9Eo57sp-EXgddfSwAthGTsdK0oM_ncJu8KlWWpOoTf9qb0D1BDtDRBdSVZ3W82D8Lg5T35zX5ENdcFWcfAeF7mMwOcxMT36aA8Gs5s88isXjzR7aMvrrmRpy0gWtqPLHbq3thOp1070NMCVj6YdOSb8ya5wp4CYNUsj_kG8d_caDD-D4DKOzXgKHs7ads72UK05H5z04wcBIhKZmC7GfijBl_c_ySWv4q-UiKtOSwLb0ANTWqgPoF98XfdmdNAwfoo070hQTlc_wfaab6hNx2epFcglu1mEN-cA2nEBlm7t4LRt8b2Sk9LyqPWv_Fw7OajH_mVgq20G_y0lBo_8eOWW0F9pL_bjkcAWQHCDNBMqyGcbk8ja_VPfEDOvTQW8MtDQqEJo-MlPuItgouRs_thvPHevwuMwrflVFq6ewklfmwmgsnHVPb84OcFQQnCDJIbu4pp7So2po0c35VHjtEWRhrgiMhJknEhNq47jE0JMP6nvG0RtnODMpm5i-PJnQ\n\tstatus code: 403, request id: 6bbb9bfd-8c92-4ce1-bf03-2e16d071969d"
  Warning  FailedMount            1m (x13 over 25m)   attachdetach                            (combined from similar events): AttachVolume.Attach failed for volume "pvc-2db45bb0-22f8-11e8-82f2-0e21f011a24c" : Error attaching EBS volume "vol-03d0039cdcca2fc9a" to instance "i-09b03f4a4bf903960": "UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: VJ9jW-_EPhGBd2Syur9KvvqPClxfPZib3vjnh7r31sDaeaoviZPb0dajrJSSm8WOeIJAFzdOfN-Qd8l7lXMY1fUP1VOWLsr1oEV-ziS-cgePKSoaFEEV2Qa7uLGO0n2EUOc-YWqz9mECbUST756o8P1uIJwnhtCliDU2iTzs7CxXY_Q2W8k4rkhQiYYRoIJ7ATbbS-oB69vU8ZCugflr09TG0JhPTHybCaLVtJHUnSJiSNuuTGw8YpYmtiiz_6TRtxbN6hGkZJtnb8OD1bluvqRPb9w8Pb9f58vs6du-Y9K-bCz7gJo7qK1sW9WW3Xf8fvU0nP_iqgOhJ_RQbRcIylibtZzDOfpLE8jFKySy5Nay5Z3QnBDQy4L5FeJWk0PrXhsRcZJIS8CMteJdh6G1D1VI0k9c2DuMS-dNEOfu1pnG8FH0i0oEL_pR4ZDm_ArsgZVw6n7bT0Elz8a4XhEiKDme_mWlNVA8VQPRSebZL9vL6IG9edD-sJdXHZQCrmKsowbeyhWBXKeIGJjUQFaRBj_NkditbNWvbVoes2BIgiNW37VsSedzX0SoZe3Kt0dB3RMgPQcN_Q7Mft0JJ5R8BCym9DwQE2AKT0eIhSWB7tMxqBaX1xou-CGHIbJ-stt0Xt077XghB9EFAk2pkmMXBpydvZZuXDvJJQyj3Z00CVf-iCS6UKgzAvfN8jB7SeWVvpKyNNanVn4t\n\tstatus code: 403, request id: 8447989d-7052-4b0d-9102-b5c6d1be36c2"
  Warning  FailedMount            44s (x13 over 28m)  kubelet, ip-172-20-49-240.ec2.internal  Unable to mount volumes for pod "nginx-deployment-644cf84b7f-6m6wh_nginx-example(b599b5ee-2308-11e8-82f8-0e8c84c7abe0)": timeout expired waiting for volumes to attach/mount for pod "nginx-example"/"nginx-deployment-644cf84b7f-6m6wh". list of unattached/unmounted volumes=[nginx-logs]
  Warning  FailedSync             44s (x13 over 28m)  kubelet, ip-172-20-49-240.ec2.internal  Error syncing pod