kiam: failed to load system roots and no roots provided - TLS error

{"generation.metadata":0,"level":"error","msg":"error warming credentials: RequestError: send request failed\ncaused by: Post https://sts.amazonaws.com/: x509: failed to load system roots and no roots provided","pod.iam.role":"arn:aws:iam::###########:role/chrisiamtest1","pod.name":"aws-cli3","pod.namespace":"default","pod.status.ip":"100.112.74.130","pod.status.phase":"Running","resource.version":"4849725","time":"2018-02-14T17:59:54Z"}

{"generation.metadata":0,"level":"error","msg":"error warming credentials: RequestError: send request failed\ncaused by: Post https://sts.amazonaws.com/: x509: failed to load system roots and no roots provided","pod.iam.role":"chrisiamtest1","pod.name":"aws-cli","pod.namespace":"default","pod.status.ip":"100.112.74.129","pod.status.phase":"Running","resource.version":"4849455","time":"2018-02-14T17:59:54Z"}

{"level":"error","msg":"error requesting credentials: RequestError: send request failed\ncaused by: Post https://sts.amazonaws.com/: x509: failed to load system roots and no roots provided","pod.iam.role":"arn:aws:iam::############:role/chrisiamtest1","time":"2018-02-14T18:00:54Z"}

I’ve tried with just using the role name and the full ARN in the pod deployment. Can someone help me understand what this error means? Is there documentation on how to specify the base-arn or is autodetect the best solution?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 15 (5 by maintainers)

Most upvoted comments

@pingles Thanks for sending me down the correct path. (the kops hosts have a /usr/share/ca-certificates with a bunch of mozilla certificates but it doesn’t seem to be what is needed here so update is necessary. Maybe include ca-certificates in the docker image to avoid different underlying hosts having ca-certificates in the wrong places or create config-map for easier/documented update depending upon the k8s install you might have?

For anyone who might find this issue and needs help here is what I ended up needing to do in order to get this to work with kops

updates to server.yaml

diff --git a/deploy/server.yaml b/deploy/server.yaml
index 9cd82ad..6d08710 100644
--- a/deploy/server.yaml
+++ b/deploy/server.yaml
@@ -17,10 +17,14 @@ spec:
       serviceAccountName: kiam-server
       nodeSelector:
         kubernetes.io/role: master
+      tolerations:
+      - key: "node-role.kubernetes.io/master"
+        effect: "NoSchedule"
+        operator: "Exists"
       volumes:
         - name: ssl-certs
           hostPath:
-            path: /usr/share/ca-certificates
+            path: /etc/ssl/certs
         - name: tls
           secret:
             secretName: kiam-server-tls

trust-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      },
      {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::{{Account ID}}:role/masters.k8s.clusterdomain.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }

+15

cdenneen on Feb 16, 2018

@cdenneen Your comments and patches saved the day for me. Thank you very much!

luispollo on May 4, 2018

Maybe include ca-certificates in the docker image to avoid different underlying hosts having ca-certificates in the wrong places or create config-map for easier/documented update depending upon the k8s install you might have?

I’d rather continue to mount from the host. I know this is somewhat host/cluster specific but it means that updates can be processed as the nodes are upgraded, rather than relying on each container having to

pingles on Feb 16, 2018