dapr: Operator stream EOF error every 60 minutes in daprd sidecar - Dapr v1.5.0, K8s 1.20.9

Across multiple kubernetes clusters, in both HA and Single instance configurations we see this error every 60 minutes on the dot. This then causes all components to be reloaded when the connection is re-established and exacerbates an issue with connections.

Expected Behavior

Should not see an EOF error or reload components

Actual Behavior

EOF logged every 60 minutes "error from operator stream: rpc error: code = Unavailable desc = error reading from server: EOF","scope":"dapr.runtime","time":"2021-11-25T12:06:38.347853275Z","type":"log","ver":"1.5.0"}

Causes components to reload which exacerbates this issue - https://github.com/dapr/components-contrib/issues/1318

Steps to Reproduce the Problem

  • Dapr v1.5.0
  • K8s 1.20.9
  • Run a POD with multiple components
  • Wait 60+ minutes, observe that EOF is displayed in daprd logs

Release Note

Fix operator EOF

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (14 by maintainers)

Most upvoted comments

I think I found the issue, will be fixed for 1.6.