dapr: Dapr sidecard does not created on prod env

/area runtime

Hey! we have weird issue on production environment Dapr sidecard after deployment is not create and node is crashed with: “Dapr sidecar is unhealthy” message. We can’t find something from dapr logs, seems all is deployed and healthy.

Our environment is using EKS on aws v1.22 and we tried to use 1.24, and different version of Dapr 1.9.0, 1.9.5 installed from helm/terraform

for example I can deploy node app, but the same for our pods with different namespaces

kind: Service
apiVersion: v1
metadata:
  name: nodeapp
  labels:
    app: node
spec:
  selector:
    app: node
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000
  type: LoadBalancer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodeapp
  labels:
    app: node
spec:
  replicas: 1
  selector:
    matchLabels:
      app: node
  template:
    metadata:
      labels:
        app: node
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "nodeapp"
        dapr.io/app-port: "3000"
        dapr.io/enable-api-logging: "true"
    spec:
      containers:
      - name: node
        image: ghcr.io/dapr/samples/hello-k8s-node:latest
        env:
        - name: APP_PORT
          value: "3000"
        ports:
        - containerPort: 3000
        imagePullPolicy: Always

logs:

kubectl logs -l app=dapr-sidecar-injector -n dapr-system
time="2023-01-18T16:30:31.428593518Z" level=info msg="log level set to: info" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.428779364Z" level=info msg="metrics server started on :9090/" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.metrics type=log ver=1.9.0
time="2023-01-18T16:30:31.431027033Z" level=info msg="starting Dapr Sidecar Injector -- version 1.9.0 -- commit fdce5f1f1b76012291c888113169aee845f25ef8" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.432520224Z" level=info msg="Healthz server is listening on :8080" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.459415595Z" level=warning msg="Unable to get SA tekton-pipelines-controller:tekton-pipelines UID" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.459548492Z" level=info msg="Sidecar injector is listening on :4000, patching Dapr-enabled pods" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0

kubectl logs dapr-operator-86b7fd6467-zxzff -n dapr-system

time="2023-01-18T17:56:54.932837588Z" level=info msg="DaprWatchdog started checking pods" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:56:54.93321807Z" level=warning msg="Pod default/nodeapp-c98c589fb-mv8zp does not have the Dapr sidecar and will be deleted" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:56:54.938490881Z" level=info msg="Deleted pod default/nodeapp-c98c589fb-mv8zp" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:56:54.938518278Z" level=warning msg="Pod default/nodeapp-c98c589fb-djv8n does not have the Dapr sidecar and will be deleted" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:56:54.947082799Z" level=info msg="Deleted pod default/nodeapp-c98c589fb-djv8n" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:56:57.938915763Z" level=info msg="DaprWatchdog completed checking pods" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:14.932910571Z" level=info msg="DaprWatchdog started checking pods" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:14.933301186Z" level=warning msg="Pod default/nodeapp-c98c589fb-5c6v4 does not have the Dapr sidecar and will be deleted" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:14.946847335Z" level=info msg="Deleted pod default/nodeapp-c98c589fb-5c6v4" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:14.94687685Z" level=warning msg="Pod default/nodeapp-c98c589fb-djv8n does not have the Dapr sidecar and will be deleted" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:14.962092116Z" level=info msg="Deleted pod default/nodeapp-c98c589fb-djv8n" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:17.947051125Z" level=info msg="DaprWatchdog completed checking pods" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:34.932753675Z" level=info msg="DaprWatchdog started checking pods" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:34.933136442Z" level=warning msg="Pod default/nodeapp-c98c589fb-5c6v4 does not have the Dapr sidecar and will be deleted" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:34.937751855Z" level=info msg="Deleted pod default/nodeapp-c98c589fb-5c6v4" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:34.937797477Z" level=warning msg="Pod default/nodeapp-c98c589fb-lcvrg does not have the Dapr sidecar and will be deleted" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:34.951630393Z" level=info msg="Deleted pod default/nodeapp-c98c589fb-lcvrg" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0
time="2023-01-18T17:57:37.937886097Z" level=info msg="DaprWatchdog completed checking pods" instance=dapr-operator-86b7fd6467-zxzff scope=dapr.operator type=log ver=1.9.0

kubectl logs dapr-placement-server-0 -n dapr-system
time="2023-01-18T16:30:31.402857974Z" level=info msg="starting Dapr Placement Service -- version 1.9.0 -- commit fdce5f1f1b76012291c888113169aee845f25ef8" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:31.408061102Z" level=info msg="log level set to: info" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:31.40816798Z" level=info msg="metrics server started on :9090/" instance=dapr-placement-server-0 scope=dapr.metrics type=log ver=1.9.0
time="2023-01-18T16:30:31.408473599Z" level=info msg="Raft server is starting on 127.0.0.1:8201..." instance=dapr-placement-server-0 scope=dapr.placement.raft type=log ver=1.9.0
time="2023-01-18T16:30:31.408499314Z" level=info msg="mTLS enabled, getting tls certificates" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:31.408530828Z" level=info msg="tls certificate not found; waiting for disk changes" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:31.408559198Z" level=info msg="starting watch for certs on filesystem: /var/run/dapr/credentials" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:32.901350335Z" level=info msg="tls certificates loaded successfully" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:32.901407427Z" level=info msg="placement service started on port 50005" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:32.901541213Z" level=info msg="cluster leadership acquired" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:32.901608303Z" level=info msg="leader is established." instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0
time="2023-01-18T16:30:32.901752178Z" level=info msg="Healthz server is listening on :8080" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.9.0

kubectl logs dapr-sentry-77dbf4fb47-d4r2g -n dapr-system
time="2023-01-18T16:30:31.397393346Z" level=info msg="starting sentry certificate authority -- version 1.9.0 -- commit fdce5f1f1b76012291c888113169aee845f25ef8" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:31.397540182Z" level=info msg="log level set to: info" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:31.397711581Z" level=info msg="metrics server started on :9090/" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.metrics type=log ver=1.9.0
time="2023-01-18T16:30:31.431884821Z" level=info msg="configuration: [port]: 50001, [ca store]: default, [allowed clock skew]: 15m0s, [workload cert ttl]: 24h0m0s" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry.config type=log ver=1.9.0
time="2023-01-18T16:30:31.431924863Z" level=info msg="starting watch on filesystem directory: /var/run/dapr/credentials" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:31.431949907Z" level=info msg="certificate authority loaded" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:31.432578624Z" level=info msg="Healthz server is listening on :8080" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:31.436576307Z" level=info msg="root and issuer certs not found: generating self signed CA" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry.ca type=log ver=1.9.0
time="2023-01-18T16:30:31.474833529Z" level=info msg="self signed certs generated and persisted successfully" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry.ca type=log ver=1.9.0
time="2023-01-18T16:30:31.474933659Z" level=info msg="trust root bundle loaded. issuer cert expiry: 2024-01-18 16:30:31 +0000 UTC" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:31.475264357Z" level=info msg="validator created" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:31.475302684Z" level=info msg="sentry certificate authority is running, protecting ya'll" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:34.90132616Z" level=warning msg="issuer credentials changed; reloading" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:34.901417783Z" level=info msg="sentry certificate authority is restarting" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:34.901430297Z" level=info msg="sentry certificate authority is shutting down" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:35.101873083Z" level=info msg="certificate authority loaded" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:36.110213309Z" level=info msg="trust root bundle loaded. issuer cert expiry: 2024-01-18 16:30:31 +0000 UTC" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:36.110585063Z" level=info msg="validator created" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:36.11062681Z" level=info msg="sentry certificate authority is running, protecting ya'll" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:38.211029397Z" level=warning msg="issuer credentials changed; reloading" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:38.211084217Z" level=info msg="sentry certificate authority is restarting" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:38.211096344Z" level=info msg="sentry certificate authority is shutting down" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:38.411506769Z" level=info msg="certificate authority loaded" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:39.419911537Z" level=info msg="trust root bundle loaded. issuer cert expiry: 2024-01-18 16:30:31 +0000 UTC" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:39.420230495Z" level=info msg="validator created" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:39.420273626Z" level=info msg="sentry certificate authority is running, protecting ya'll" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:41.52086733Z" level=warning msg="issuer credentials changed; reloading" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:41.520901489Z" level=info msg="sentry certificate authority is restarting" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:41.520915093Z" level=info msg="sentry certificate authority is shutting down" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:41.722130574Z" level=info msg="certificate authority loaded" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:42.729539713Z" level=info msg="trust root bundle loaded. issuer cert expiry: 2024-01-18 16:30:31 +0000 UTC" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:42.729927977Z" level=info msg="validator created" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0
time="2023-01-18T16:30:42.729965535Z" level=info msg="sentry certificate authority is running, protecting ya'll" instance=dapr-sentry-77dbf4fb47-d4r2g scope=dapr.sentry type=log ver=1.9.0

kubectl logs dapr-sidecar-injector-754cc79c4c-zn9nf -n dapr-system
time="2023-01-18T16:30:31.428593518Z" level=info msg="log level set to: info" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.428779364Z" level=info msg="metrics server started on :9090/" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.metrics type=log ver=1.9.0
time="2023-01-18T16:30:31.431027033Z" level=info msg="starting Dapr Sidecar Injector -- version 1.9.0 -- commit fdce5f1f1b76012291c888113169aee845f25ef8" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.432520224Z" level=info msg="Healthz server is listening on :8080" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.459415595Z" level=warning msg="Unable to get SA tekton-pipelines-controller:tekton-pipelines UID" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0
time="2023-01-18T16:30:31.459548492Z" level=info msg="Sidecar injector is listening on :4000, patching Dapr-enabled pods" instance=ip-10-35-3-193.eu-west-1.compute.internal scope=dapr.injector type=log ver=1.9.0