spire: Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = DeadlineExceeded desc = workloadattestor(k8s): no selectors found after max poll attempts"

  • Version: 1.0.1

  • Platform: k8s + istio

  • Subsystem: spire-server:

apiVersion: v1
kind: Namespace
metadata:
  name: spire

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spire-server
  namespace: spire

---

apiVersion: v1
kind: Secret
metadata:
  name: spire-server
  namespace: spire
type: Opaque
data:
  bootstrap.key: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1JR2tBZ0VCQkRBZzJMYnVsWHpRWDFORisyRGkwUkt6TVdmRUdpb0JoaC9mRnB4N3lPRXFrYS8vVHBhZVUzTzUKUUpSWlhkV0hLdWFnQndZRks0RUVBQ0toWkFOaUFBUmFNSDZkSVpMRWhpTE9kdnpqRzdsWVlObVB6U2N2dGJWegpmTi9qeGFITFNacnRqdVlJRXJOOUNTdUFPQzRqaVBSbjdUKzBNZit2eUMwNjBzdXNpbTR6QlllaDdpOXRVRVcxCjdXK1BwZTNwWjRUeVZmQndLOHV6K1p5YTgrcFVyMk09Ci0tLS0tRU5EIEVDIFBSSVZBVEUgS0VZLS0tLS0K

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-server
  namespace: spire
data:
  server.conf: |
    server {
      bind_address = "0.0.0.0"
      bind_port = "8081"
      trust_domain = "example.org"
      data_dir = "/run/spire/data"
      log_level = "DEBUG"
      default_svid_ttl = "1h"
      ca_subject = {
        country = ["US"],
        organization = ["SPIFFE"],
        common_name = "",
      }
    }

    plugins {
      DataStore "sql" {
        plugin_data {
          database_type = "sqlite3"
          connection_string = "/run/spire/data/datastore.sqlite3"
        }
      }

      NodeAttestor "k8s_sat" {
        plugin_data {
          clusters = {
            "demo-cluster" = {
              use_token_review_api_validation = true
              service_account_whitelist = ["spire:spire-agent"]
            }
          }
        }
      }

      NodeResolver "noop" {
        plugin_data {}
      }

      KeyManager "disk" {
        plugin_data {
          keys_path = "/run/spire/data/keys.json"
        }
      }

      UpstreamAuthority "disk" {
        plugin_data {
          key_file_path = "/run/spire/secrets/bootstrap.key"
          cert_file_path = "/run/spire/config/bootstrap.crt"
        }
      }
    }

    health_checks {
      listener_enabled = true
      bind_address = "0.0.0.0"
      bind_port = "8080"
      live_path = "/live"
      ready_path = "/ready"
    }
  bootstrap.crt: |
    -----BEGIN CERTIFICATE-----
    MIIBzDCCAVOgAwIBAgIJAJM4DhRH0vmuMAoGCCqGSM49BAMEMB4xCzAJBgNVBAYT
    AlVTMQ8wDQYDVQQKDAZTUElGRkUwHhcNMTgwNTEzMTkzMzQ3WhcNMjMwNTEyMTkz
    MzQ3WjAeMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGU1BJRkZFMHYwEAYHKoZIzj0C
    AQYFK4EEACIDYgAEWjB+nSGSxIYiznb84xu5WGDZj80nL7W1c3zf48Why0ma7Y7m
    CBKzfQkrgDguI4j0Z+0/tDH/r8gtOtLLrIpuMwWHoe4vbVBFte1vj6Xt6WeE8lXw
    cCvLs/mcmvPqVK9jo10wWzAdBgNVHQ4EFgQUh6XzV6LwNazA+GTEVOdu07o5yOgw
    DwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwGQYDVR0RBBIwEIYOc3Bp
    ZmZlOi8vbG9jYWwwCgYIKoZIzj0EAwQDZwAwZAIwE4Me13qMC9i6Fkx0h26y09QZ
    IbuRqA9puLg9AeeAAyo5tBzRl1YL0KNEp02VKSYJAjBdeJvqjJ9wW55OGj1JQwDF
    D7kWeEB6oMlwPbI/5hEY3azJi16I0uN1JSYTSWGSqWc=
    -----END CERTIFICATE-----

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: spire-server
  namespace: spire
  labels:
    app: spire-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spire-server
  serviceName: spire-server
  template:
    metadata:
      namespace: spire
      labels:
        app: spire-server
    spec:
      serviceAccountName: spire-server
      containers:
        - name: spire-server
          image: xxx/spire-server:1.0.1
          args: ["-config", "/run/spire/config/server.conf"]
          ports:
            - containerPort: 8081
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-secrets
              mountPath: /run/spire/secrets
              readOnly: true
            - name: spire-data
              mountPath: /run/spire/data
              readOnly: false
          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            failureThreshold: 2
            initialDelaySeconds: 15
            periodSeconds: 60
            timeoutSeconds: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: spire-config
          configMap:
            name: spire-server
        - name: spire-secrets
          secret:
            secretName: spire-server
        - name: spire-data
          emptyDir: {}
---

apiVersion: v1
kind: Service
metadata:
  name: spire-server
  namespace: spire
spec:
  type: ClusterIP
  ports:
    - name: grpc
      port: 8081
      targetPort: 8081
      protocol: TCP
  selector:
    app: spire-server

spire-agent:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spire-agent
  namespace: spire

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-agent
  namespace: spire
data:
  agent.conf: |
    agent {
      data_dir = "/run/spire"
      log_level = "DEBUG"
      server_address = "spire-server"
      server_port = "8081"
      socket_path = "/run/spire/sockets/agent.sock"
      trust_bundle_path = "/run/spire/config/bootstrap.crt"
      trust_domain = "example.org"
    }

    plugins {
      NodeAttestor "k8s_sat" {
        plugin_data {
          cluster = "demo-cluster"
        }
      }

      KeyManager "memory" {
        plugin_data {
        }
      }

      WorkloadAttestor "k8s" {
        plugin_data {
          # Defaults to the secure kubelet port by default.
          # Minikube does not have a cert in the cluster CA bundle that
          # can authenticate the kubelet cert, so skip validation.
          skip_kubelet_verification = "true"
          #kubelet_read_only_port = "10255"
          node_name_env = "MY_NODE_NAME"
        }
      }

      WorkloadAttestor "unix" {
          plugin_data {
          }
      }
    }

    health_checks {
      listener_enabled = true
      bind_address = "0.0.0.0"
      bind_port = "8080"
      live_path = "/live"
      ready_path = "/ready"
    }
  bootstrap.crt: |
    -----BEGIN CERTIFICATE-----
    MIIBzDCCAVOgAwIBAgIJAJM4DhRH0vmuMAoGCCqGSM49BAMEMB4xCzAJBgNVBAYT
    AlVTMQ8wDQYDVQQKDAZTUElGRkUwHhcNMTgwNTEzMTkzMzQ3WhcNMjMwNTEyMTkz
    MzQ3WjAeMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGU1BJRkZFMHYwEAYHKoZIzj0C
    AQYFK4EEACIDYgAEWjB+nSGSxIYiznb84xu5WGDZj80nL7W1c3zf48Why0ma7Y7m
    CBKzfQkrgDguI4j0Z+0/tDH/r8gtOtLLrIpuMwWHoe4vbVBFte1vj6Xt6WeE8lXw
    cCvLs/mcmvPqVK9jo10wWzAdBgNVHQ4EFgQUh6XzV6LwNazA+GTEVOdu07o5yOgw
    DwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwGQYDVR0RBBIwEIYOc3Bp
    ZmZlOi8vbG9jYWwwCgYIKoZIzj0EAwQDZwAwZAIwE4Me13qMC9i6Fkx0h26y09QZ
    IbuRqA9puLg9AeeAAyo5tBzRl1YL0KNEp02VKSYJAjBdeJvqjJ9wW55OGj1JQwDF
    D7kWeEB6oMlwPbI/5hEY3azJi16I0uN1JSYTSWGSqWc=
    -----END CERTIFICATE-----

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spire-agent
  namespace: spire
  labels:
    app: spire-agent
spec:
  selector:
    matchLabels:
      app: spire-agent
  template:
    metadata:
      namespace: spire
      labels:
        app: spire-agent
    spec:
      hostPID: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccountName: spire-agent
      initContainers:
        - name: init
          # This is a small image with wait-for-it, choose whatever image
          # you prefer that waits for a service to be up. This image is built
          # from https://github.com/lqhl/wait-for-it
          image: xxx/wait-for-it
          args: ["-t", "30", "spire-server:8081"]
          env:
            - name: MY_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
      containers:
        - name: spire-agent
          image: xxx/spire-agent:1.0.1
          args: ["-config", "/run/spire/config/agent.conf"]
          env:
            - name: MY_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-agent-socket
              mountPath: /run/spire/sockets
              readOnly: false
          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            failureThreshold: 2
            initialDelaySeconds: 15
            periodSeconds: 60
            timeoutSeconds: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: spire-config
          configMap:
            name: spire-agent
        - name: spire-agent-socket
          hostPath:
            path: /run/spire/sockets
            type: DirectoryOrCreate

When I get the certificate through spire-agent, I get an error. the logs of spire-agent:

time="2022-05-18T11:48:58Z" level=warning msg="Container id not found" attempt=7 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:58Z" level=warning msg="Container id not found" attempt=8 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:59Z" level=warning msg="Container id not found" attempt=9 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:59Z" level=warning msg="Container id not found" attempt=10 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:59Z" level=error msg="Received error from stream secrets server" error="<nil>" method=StreamSecrets pid=1193 service=SDS.v3 subsystem_name=endpoints
time="2022-05-18T11:48:59Z" level=error msg="Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = Canceled desc = workloadattestor(k8s): context canceled" pid=1193 subsystem_name=workload_attestor
time="2022-05-18T11:48:59Z" level=debug msg="PID attested to have selectors" pid=1193 selectors="[type:\"unix\" value:\"uid:1337\" type:\"unix\" value:\"gid:1337\"]" subsystem_name=workload_attestor
time="2022-05-18T11:48:59Z" level=error msg="Failed to attest the workload" error="rpc error: code = Unauthenticated desc = could not verify existence of the original caller: caller is no longer being watched" method=StreamSecrets pid=1193 service=SDS.v3 subsystem_name=endpoints

the logs of spire-server:

time="2022-05-18T11:07:39Z" level=warning msg="Current umask 0022 is too permissive; setting umask 0027"
time="2022-05-18T11:07:39Z" level=info msg="Data directory: \"/run/spire/data\""
time="2022-05-18T11:07:39Z" level=info msg="Opening SQL database" db_type=sqlite3 subsystem_name=sql
time="2022-05-18T11:07:39Z" level=info msg="Initializing new database" subsystem_name=sql
time="2022-05-18T11:07:39Z" level=info msg="Connected to SQL database" read_only=false subsystem_name=sql type=sqlite3 version=3.34.0
time="2022-05-18T11:07:39Z" level=warning msg="The \"noop\" NodeResolver is not required, is deprecated, and will be removed from a future release" subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=warning msg="The `service_account_whitelist` configurable is deprecated and will be removed in a future release. Please use `service_account_allow_list` instead." external=false plugin_name=k8s_sat plugin_type=NodeAttestor subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=k8s_sat plugin_type=NodeAttestor subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=disk plugin_type=KeyManager subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=disk plugin_type=UpstreamAuthority subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=debug msg="Loading journal" path=/run/spire/data/journal.pem subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="Journal loaded" jwt_keys=0 subsystem_name=ca_manager x509_cas=0
time="2022-05-18T11:07:39Z" level=debug msg="Preparing X509 CA" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="X509 CA prepared" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" self_signed=false slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="X509 CA activated" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=debug msg="Successfully rotated X.509 CA" subsystem_name=ca_manager trust_domain_id="spiffe://example.org" ttl=86399.559774583
time="2022-05-18T11:07:39Z" level=debug msg="Preparing JWT key" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=warning msg="UpstreamAuthority plugin does not support JWT-SVIDs. Workloads managed by this server may have trouble communicating with workloads outside this cluster when using JWT-SVIDs." plugin_name=disk subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="JWT key prepared" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="JWT key activated" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=debug msg="Rotating server SVID" subsystem_name=svid_rotator
time="2022-05-18T11:07:39Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:39Z" spiffe_id="spiffe://example.org/spire/server" subsystem_name=ca
time="2022-05-18T11:07:39Z" level=info msg="Building in-memory entry cache" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=info msg="Completed building in-memory entry cache" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=debug msg="Initializing API endpoints" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=info msg="Starting TCP server" address="[::]:8081" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=info msg="Starting UDS server" address=/tmp/spire-server/private/api.sock subsystem_name=endpoints
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=catalog.datastore subsystem_name=health
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server.ca subsystem_name=health
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server.ca.manager subsystem_name=health
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server subsystem_name=health
time="2022-05-18T11:07:40Z" level=info msg="Serving health checks" address="0.0.0.0:8080" subsystem_name=health
time="2022-05-18T11:07:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:47Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/1be11f43-e9e7-41b6-80a4-4dc816b1e889" subsystem_name=ca
time="2022-05-18T11:07:47Z" level=info msg="Agent attestation request completed" address="192.168.1.7:5204" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/1be11f43-e9e7-41b6-80a4-4dc816b1e889" caller_addr="192.168.1.7:5204" method=AttestAgent node_attestor_type=k8s_sat request_id=7bd55951-34d2-4042-b3ac-18f165a0dc9b service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:07:48Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:48Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/64823730-d786-4277-a9bc-106b8eb2f4ff" subsystem_name=ca
time="2022-05-18T11:07:48Z" level=info msg="Agent attestation request completed" address="192.168.1.6:5510" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/64823730-d786-4277-a9bc-106b8eb2f4ff" caller_addr="192.168.1.6:5510" method=AttestAgent node_attestor_type=k8s_sat request_id=76052c90-a828-4a94-a16a-5d6be221e603 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:11:32Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:32Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:11:32Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:32Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:13:55Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:13:55Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:13:55Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:13:55Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:28:36Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:36Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/be1300e4-e93d-40f6-b987-6cb42d20e237" subsystem_name=ca
time="2022-05-18T11:28:36Z" level=info msg="Agent attestation request completed" address="192.168.1.7:26058" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/be1300e4-e93d-40f6-b987-6cb42d20e237" caller_addr="192.168.1.7:26058" method=AttestAgent node_attestor_type=k8s_sat request_id=b00ead65-d001-46e3-89f5-5234442697c9 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/bd4ae7d9-4fbf-4ba0-931c-efc4da0b6e08" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=info msg="Agent attestation request completed" address="192.168.1.6:37898" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/bd4ae7d9-4fbf-4ba0-931c-efc4da0b6e08" caller_addr="192.168.1.6:37898" method=AttestAgent node_attestor_type=k8s_sat request_id=7967a14e-ab5c-4bf9-aaef-21e2d43cf413 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:30:53Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:53Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/93fe2161-95cc-4454-84d4-a510616a5dcf" subsystem_name=ca
time="2022-05-18T11:30:53Z" level=info msg="Agent attestation request completed" address="192.168.1.6:42279" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/93fe2161-95cc-4454-84d4-a510616a5dcf" caller_addr="192.168.1.6:42279" method=AttestAgent node_attestor_type=k8s_sat request_id=c3bf5aac-0c6b-4ae5-9fb1-86bcd668f8ec service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:30:57Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:57Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/ca6a5d08-3517-43a1-8ffc-dc0519a2dea7" subsystem_name=ca
time="2022-05-18T11:30:57Z" level=info msg="Agent attestation request completed" address="192.168.1.7:51286" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/ca6a5d08-3517-43a1-8ffc-dc0519a2dea7" caller_addr="192.168.1.7:51286" method=AttestAgent node_attestor_type=k8s_sat request_id=9dd8a63e-5ace-4381-8a5a-80e2a1e5f0d1 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:32:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:02Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/32c3378f-b8b0-4df6-b6a0-18fb3bf824f7" subsystem_name=ca
time="2022-05-18T11:32:02Z" level=info msg="Agent attestation request completed" address="192.168.1.7:20470" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/32c3378f-b8b0-4df6-b6a0-18fb3bf824f7" caller_addr="192.168.1.7:20470" method=AttestAgent node_attestor_type=k8s_sat request_id=37844d8e-27f5-42c6-a7be-019bfe6f9c82 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:32:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:02Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/7e068ae9-00f4-4786-9191-2b61cb7f08e9" subsystem_name=ca
time="2022-05-18T11:32:02Z" level=info msg="Agent attestation request completed" address="192.168.1.6:24604" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/7e068ae9-00f4-4786-9191-2b61cb7f08e9" caller_addr="192.168.1.6:24604" method=AttestAgent node_attestor_type=k8s_sat request_id=60c6024e-eaa6-4f21-8d24-7f4dfff0eb24 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:37:34Z" level=debug msg="Rotating server SVID" subsystem_name=svid_rotator
time="2022-05-18T11:37:34Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:37:34Z" spiffe_id="spiffe://example.org/spire/server" subsystem_name=ca

the agent list in spire-server.

/opt/spire/bin # /opt/spire/bin/spire-server agent list
Found 2 attested agents:

SPIFFE ID         : spiffe://example.org/spire/agent/k8s_sat/demo-cluster/56d7ed58-14ae-4cf5-9880-ffc123b314c1
Attestation type  : k8s_sat
Expiration time   : 2022-05-18 13:02:00 +0000 UTC
Serial number     : 319040273546637254342952962498965679004

SPIFFE ID         : spiffe://example.org/spire/agent/k8s_sat/demo-cluster/63588790-f382-492b-a324-359999d3a394
Attestation type  : k8s_sat
Expiration time   : 2022-05-18 13:02:00 +0000 UTC
Serial number     : 277908708266485423488447531291378329815

the entry in spire-server.

Found 5 entries
Entry ID         : 7ec60e3e-147d-444b-b026-a244279a03c6
SPIFFE ID        : spiffe://example.org/ns/default/sa/default
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:default
Selector         : k8s:sa:default

Entry ID         : cc99cf5e-cfd7-44ca-966d-b6313a540447
SPIFFE ID        : spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:istio-system
Selector         : k8s:sa:istio-ingressgateway-service-account

Entry ID         : a689cef7-6972-46b1-8277-b3212f31c230
SPIFFE ID        : spiffe://example.org/ns/spire/sa/spire-agent
Parent ID        : spiffe://example.org/spire/server
Revision         : 0
TTL              : default
Selector         : k8s_sat:agent_ns:spire
Selector         : k8s_sat:agent_sa:spire-agent
Selector         : k8s_sat:cluster:demo-cluster

Entry ID         : be9bd775-85cb-4ffa-b914-5493c651d264
SPIFFE ID        : spiffe://example.org/ns/test/sa/httpbin
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:test
Selector         : k8s:sa:httpbin

Entry ID         : 3615702c-293e-4d39-9785-16a546c2462b
SPIFFE ID        : spiffe://example.org/ns/test/sa/sleep
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:test
Selector         : k8s:sa:sleep

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 31 (16 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks @tanjunchen - yes I understand what the feature does, I am wondering why you need it

It doesn’t look like this issue will be resolved on the k8s side any time soon. The next best thing could be to convince istio to implement the feature differently … that also seems like a long shot but is worth it if we can find an alternate mechanism (one that does not lean on these lifecycle hooks).

In the meantime, I raised https://github.com/spiffe/spire/pull/3443 which adds this as a known issue to the plugin docs

+2
evan2645 on Sep 21, 2022

IIRC, removing the container ID check is not tenable. We use that check to sharp shoot the correct container from the pod to generate accurate selectors for the workload. If we cannot identify the container in the pod, then attestation is weakened.

+2
azdagron on Aug 9, 2022

@loveyana @radoslav-tomov @azdagron
I found the reason for the failure, because I turned on the holdApplicationUntilProxyStarts=true parameter when I installed istio.

apiVersion: operator.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  profile: default
  meshConfig:
    trustDomain: example.org
    accessLogFile: /dev/stdout
    defaultConfig:
      holdApplicationUntilProxyStarts: true
  values:
    gateways:
      istio-egressgateway:
        autoscaleEnabled: false
      istio-ingressgateway:
        autoscaleEnabled: false
    global:
      tag: 1.11.4
      proxy:
        privileged: true
        enableCoreDump: false
        resources:
          requests:
            cpu: 10m
            memory: 40Mi

This should be a bug in spire.

+2
tanjunchen on May 23, 2022

This error message indicates that the K8s workload attestor was unable to locate the pod containing the workload container after a configurable period of time. The process is basically as follows:

  1. Determine workload pod UID and container ID by inspecting the cgroups for the workload PID
  2. Query the kubelet for the pod listing and attempt to locate the workload pod by matching up the UID and container ID
  3. Repeat step 2 for 60 attempts, once every 500ms (for a total of 30 seconds) until successful. The number of attempts and inverval between attempts is configurable.
  4. If unsuccessful, fail with the error above
  5. If successful, gather selectors for the workload

To debug this, it may be necessary to perform these steps yourself to determine where the breakdown lies. From the agent container, you can query /proc/<workload pid>/cgroups to locate the pod UID and container ID. You can then curl the kubelet to get the pod listing, e.g.

$ curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://127.0.0.1:10250/pods

(caveat: i haven’t tested this command)

You can then see if you can identify the workload pod.

ok, Thanks for your explanation, thank you very much.

+1
tanjunchen on May 22, 2022

This error message indicates that the K8s workload attestor was unable to locate the pod containing the workload container after a configurable period of time. The process is basically as follows:

  1. Determine workload pod UID and container ID by inspecting the cgroups for the workload PID
  2. Query the kubelet for the pod listing and attempt to locate the workload pod by matching up the UID and container ID
  3. Repeat step 2 for 60 attempts, once every 500ms (for a total of 30 seconds) until successful. The number of attempts and inverval between attempts is configurable.
  4. If unsuccessful, fail with the error above
  5. If successful, gather selectors for the workload

To debug this, it may be necessary to perform these steps yourself to determine where the breakdown lies. From the agent container, you can query /proc/<workload pid>/cgroups to locate the pod UID and container ID. You can then curl the kubelet to get the pod listing, e.g.

$ curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://127.0.0.1:10250/pods

(caveat: i haven’t tested this command)

You can then see if you can identify the workload pod.

+1
azdagron on May 19, 2022
Read more comments on GitHub
← spider: Scraped html does not match the url - chrome [with_wait_for_idle_network]
spire: Verify spire-agent high cpu load caused by k8s JSON parsing →