longhorn: [BUG] Instance managers and Pods with attached volumes restarted every hour

Describe the bug On a number of different clusters I’ve had over the past few months (k3s of various versions, on various clouds and OKE (Oracle Cloud)) on 1.19 and 1.20, I’ve had an issue where all instance managers and Pods with attached volumes get restarted precisely every hour. Sometimes redploying the whole of Longhorn from scratch and restoring from backup resolves the issue. The nodes are all healthy in this scenario and the rest of the cluster is stable and unchanged. However like clockwork, every hour it restarts all my pods.

To Reproduce I need to do further testing to see if it happens with completely fresh cluster without my backups restored. However it is currently happening for me on a two node ARM-based OKE cluster on Kubernetes 1.20, with 6 small volumes (100-500MB) restored from S3 backups and 3 larger 1-10GB volumes which were created fresh.

Expected behavior To not have everything restart every hour.

Log There is nothing in the logs indicating this is about to happen. The only thing which gets logged is the recovery.

time="2021-07-24T08:53:13Z" level=debug msg="Polling backup store for new volume backups" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185
time="2021-07-24T08:53:13Z" level=debug msg="Refreshed all volumes last backup based on backup store information" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="Stop monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=error msg="error receiving next item in engine watch: rpc error: code = Canceled desc = context canceled" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=error msg="error receiving next item in engine watch: rpc error: code = Unavailable desc = transport is closing" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="Stop monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Try to get requested log for pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c on node 10.0.121.185"
time="2021-07-24T09:00:48Z" level=warning msg="cannot get requested log for instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c on node 10.0.121.185, error invalid Instance Manager instance-manager-r-07214e36, state: error, IP: "
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state error, new state stopped"
time="2021-07-24T09:00:48Z" level=warning msg="Try to get requested log for pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 on node 10.0.121.185"
time="2021-07-24T09:00:48Z" level=warning msg="cannot get requested log for instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 on node 10.0.121.185, error invalid Instance Manager instance-manager-r-07214e36, state: error, IP: "
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 state, old state error, new state stopped"
time="2021-07-24T09:00:48Z" level=warning msg="Try to get requested log for pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 on node 10.0.121.185"
time="2021-07-24T09:00:48Z" level=warning msg="cannot get requested log for instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 on node 10.0.121.185, error invalid Instance Manager instance-manager-r-07214e36, state: error, IP: "
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state error, new state stopped"
time="2021-07-24T09:00:58Z" level=info msg="Created instance manager pod instance-manager-e-04771971 for instance manager instance-manager-e-04771971"
time="2021-07-24T09:00:58Z" level=info msg="Created instance manager pod instance-manager-r-07214e36 for instance manager instance-manager-r-07214e36"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=debug msg="Start monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:59Z" level=debug msg="Start monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031, will mark the instance as state ERROR"
time="2021-07-24T09:01:02Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state error, new state stopped"
time="2021-07-24T09:01:04Z" level=debug msg="Instance handler updated instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 state, old state error, new state stopped"
time="2021-07-24T09:01:06Z" level=debug msg="Instance handler updated instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e state, old state error, new state stopped"
time="2021-07-24T09:01:07Z" level=debug msg="Instance handler updated instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e state, old state error, new state stopped"
time="2021-07-24T09:01:09Z" level=debug msg="Instance handler updated instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c state, old state error, new state stopped"
time="2021-07-24T09:01:13Z" level=debug msg="Instance handler updated instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 state, old state error, new state stopped"
time="2021-07-24T09:01:16Z" level=debug msg="Stop monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:01:16Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:01:16Z" level=error msg="error receiving next item in engine watch: rpc error: code = Canceled desc = context canceled" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:01:17Z" level=debug msg="Stop monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:17Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:17Z" level=error msg="error receiving next item in engine watch: rpc error: code = Canceled desc = context canceled" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:19Z" level=warning msg="Error syncing Longhorn replica longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9" controller=longhorn-replica error="fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP: " node=10.0.121.185
time="2021-07-24T09:01:19Z" level=warning msg="Error syncing Longhorn replica longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9" controller=longhorn-replica error="fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP: " node=10.0.121.185
E0724 09:01:19.450376       1 replica_controller.go:178] fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP:
time="2021-07-24T09:01:19Z" level=warning msg="Dropping Longhorn replica longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 out of the queue" controller=longhorn-replica error="fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP: " node=10.0.121.185
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3453 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4564 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1?action=detach HTTP/1.1" 200 2229 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3525 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338?action=detach HTTP/1.1" 200 3362 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4568 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-7be8efab-347f-463a-b507-3875c8e369fc detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc?action=detach HTTP/1.1" 200 3366 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6?action=detach HTTP/1.1" 200 3376 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3465 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72?action=detach HTTP/1.1" 200 2324 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e?action=detach HTTP/1.1" 200 2244 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3454 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4584 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4568 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3526 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3466 "" "Go-http-client/1.1"
time="2021-07-24T09:01:28Z" level=info msg="Created instance manager pod instance-manager-r-07214e36 for instance manager instance-manager-r-07214e36"
time="2021-07-24T09:01:28Z" level=info msg="Created instance manager pod instance-manager-e-04771971 for instance manager instance-manager-e-04771971"
10.244.2.136 - - [24/Jul/2021:09:01:29 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3454 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:29 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3506 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:29 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3446 "" "Go-http-client/1.1"
time="2021-07-24T09:01:29Z" level=debug msg="Start monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:30Z" level=debug msg="Start monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3434 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3526 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
time="2021-07-24T09:01:31Z" level=info msg="Volume pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "POST /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6?action=attach HTTP/1.1" 200 3376 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3466 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3454 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3506 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3466 "" "Go-http-client/1.1"
time="2021-07-24T09:01:34Z" level=debug msg="Prepare to create instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031"
time="2021-07-24T09:01:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031\", UID:\"908ed51e-b8a2-4e57-8bc0-c519bb40c3c9\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015760\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031"
time="2021-07-24T09:01:35Z" level=debug msg="Instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:35Z" level=debug msg="Instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 starts running, Port 10000"
time="2021-07-24T09:01:35Z" level=debug msg="Instance handler updated instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3506 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3429 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4548 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4544 "" "Go-http-client/1.1"
time="2021-07-24T09:01:37Z" level=info msg="Volume pvc-7be8efab-347f-463a-b507-3875c8e369fc attachment to 10.0.117.182 with disableFrontend false requested"
time="2021-07-24T09:01:37Z" level=info msg="Volume pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "POST /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338?action=attach HTTP/1.1" 200 3362 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "POST /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc?action=attach HTTP/1.1" 200 3366 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
time="2021-07-24T09:01:37Z" level=info msg="Volume pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "POST /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72?action=attach HTTP/1.1" 200 3360 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4544 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4548 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
time="2021-07-24T09:01:40Z" level=debug msg="Prepare to create instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066"
time="2021-07-24T09:01:40Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066\", UID:\"6ad0043e-62ff-4ec5-b0d2-399c22c3b563\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015878\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066"
10.244.2.136 - - [24/Jul/2021:09:01:40 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:40 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3377 "" "Go-http-client/1.1"
time="2021-07-24T09:01:41Z" level=debug msg="Instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:41Z" level=debug msg="Instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 starts running, Port 10015"
time="2021-07-24T09:01:41Z" level=debug msg="Instance handler updated instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 state, old state stopped, new state running"
time="2021-07-24T09:01:41Z" level=debug msg="Prepare to create instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:01:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015897\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4524 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4599 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
time="2021-07-24T09:01:41Z" level=debug msg="Prepare to create instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e"
time="2021-07-24T09:01:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e\", UID:\"47166af1-dc55-4945-aabf-b68034792aa1\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015902\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e starts running, IP 10.244.2.205"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e starts running, Port 10045"
time="2021-07-24T09:01:42Z" level=debug msg="Instance handler updated instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e state, old state stopped, new state running"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, Port 10030"
time="2021-07-24T09:01:42Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4595 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4599 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3505 "" "Go-http-client/1.1"
time="2021-07-24T09:01:44Z" level=debug msg="Prepare to create instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:01:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015945\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:01:44Z" level=debug msg="Instance process pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c had been created, need to wait for instance manager update"
10.244.2.136 - - [24/Jul/2021:09:01:44 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:44 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 4462 "" "Go-http-client/1.1"
time="2021-07-24T09:01:44Z" level=info msg="Volume pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:44 +0000] "POST /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e?action=attach HTTP/1.1" 200 3280 "" "Go-http-client/1.1"
time="2021-07-24T09:01:45Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, IP 10.244.2.205"
time="2021-07-24T09:01:45Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, Port 10060"
time="2021-07-24T09:01:45Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state stopped, new state running"
time="2021-07-24T09:01:45Z" level=debug msg="Prepare to create instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:01:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015973\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3538 "" "Go-http-client/1.1"
time="2021-07-24T09:01:45Z" level=debug msg="Prepare to create instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e"
time="2021-07-24T09:01:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e\", UID:\"0b2f24bc-2883-4164-b608-25a150d1876a\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015978\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3542 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3505 "" "Go-http-client/1.1"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e starts running, IP 10.244.2.205"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e starts running, Port 10090"
time="2021-07-24T09:01:46Z" level=debug msg="Instance handler updated instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e state, old state stopped, new state running"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, Port 10075"
time="2021-07-24T09:01:46Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:46 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 4533 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3613 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3623 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3576 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4450 "" "Go-http-client/1.1"
time="2021-07-24T09:01:48Z" level=info msg="Volume pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "POST /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1?action=attach HTTP/1.1" 200 3265 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 4533 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3623 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3588 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:50 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4450 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:50 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3476 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3621 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3588 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:52 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4450 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:52 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3621 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3588 "" "Go-http-client/1.1"
time="2021-07-24T09:01:53Z" level=debug msg="Prepare to create instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c"
time="2021-07-24T09:01:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c\", UID:\"c28e49bc-2bea-49fb-a581-076b13e8f67e\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016091\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c"
time="2021-07-24T09:01:54Z" level=debug msg="Instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c starts running, IP 10.244.2.205"
time="2021-07-24T09:01:54Z" level=debug msg="Instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c starts running, Port 10105"
time="2021-07-24T09:01:54Z" level=debug msg="Instance handler updated instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:54 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4521 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:54 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3648 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3417 "" "Go-http-client/1.1"
time="2021-07-24T09:01:55Z" level=info msg="Volume pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3651 "" "Go-http-client/1.1"
time="2021-07-24T09:01:55Z" level=info msg="Volume pvc-65580647-e44a-4210-8545-2aff63ff0fe2 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "POST /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2?action=detach HTTP/1.1" 200 2316 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3682 "" "Go-http-client/1.1"
time="2021-07-24T09:01:55Z" level=info msg="Volume pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "POST /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270?action=detach HTTP/1.1" 200 2320 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3659 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "POST /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa?action=detach HTTP/1.1" 200 2053 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:56 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3464 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:56 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3648 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3679 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3727 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 5456 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3462 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:58 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3535 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:58 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3692 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3659 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3662 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3695 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 5456 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3442 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:00 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:00 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3692 "" "Go-http-client/1.1"
time="2021-07-24T09:02:01Z" level=debug msg="Prepare to create instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483"
time="2021-07-24T09:02:01Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483\", UID:\"acbef7c7-8e56-4687-933f-718fbac4b78a\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016207\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 5493 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3659 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3706 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3695 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 5636 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3791 "" "Go-http-client/1.1"
time="2021-07-24T09:02:02Z" level=debug msg="Instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 starts running, IP 10.244.2.205"
time="2021-07-24T09:02:02Z" level=debug msg="Instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 starts running, Port 10120"
time="2021-07-24T09:02:02Z" level=debug msg="Instance handler updated instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:02:02 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:02 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3586 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3692 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 5493 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3703 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3695 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3791 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:04 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:04 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3586 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 5489 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 5842 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3703 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3624 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4182 "" "Go-http-client/1.1"
time="2021-07-24T09:02:06Z" level=debug msg="Prepare to delete instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016263\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:06Z" level=debug msg="Prepare to delete instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016264\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
10.244.2.136 - - [24/Jul/2021:09:02:06 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:06 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3630 "" "Go-http-client/1.1"
time="2021-07-24T09:02:07Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state running, new state stopped"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 5489 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4052 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 5852 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3553 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4166 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:08 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:08 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3630 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 5838 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4052 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3553 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4046 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:10 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3618 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:10 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 5427 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:11 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4443 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:11 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3429 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:11 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4046 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:12 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 5764 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:12 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 5427 "" "Go-http-client/1.1"
time="2021-07-24T09:02:13Z" level=debug msg="Prepare to delete instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016350\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:13Z" level=debug msg="Prepare to delete instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016353\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
10.244.2.136 - - [24/Jul/2021:09:02:13 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4427 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:13 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3358 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:13 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3235 "" "Go-http-client/1.1"
time="2021-07-24T09:02:14Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state running, new state stopped"
10.244.2.136 - - [24/Jul/2021:09:02:14 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 5776 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4307 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 4538 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3164 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 4538 "" "Go-http-client/1.1"
time="2021-07-24T09:02:15Z" level=info msg="Volume pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "POST /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270?action=attach HTTP/1.1" 200 3356 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:17 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3496 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:17 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3093 "" "Go-http-client/1.1"
time="2021-07-24T09:02:17Z" level=debug msg="Prepare to create instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016418\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
10.244.2.136 - - [24/Jul/2021:09:02:17 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 4538 "" "Go-http-client/1.1"
time="2021-07-24T09:02:17Z" level=debug msg="Prepare to delete instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016420\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:17Z" level=debug msg="Prepare to delete instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016421\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:18Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, IP 10.244.2.205"
time="2021-07-24T09:02:18Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, Port 10030"
time="2021-07-24T09:02:18Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state stopped, new state running"
time="2021-07-24T09:02:18Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state running, new state stopped"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4534 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4273 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4534 "" "Go-http-client/1.1"
time="2021-07-24T09:02:19Z" level=info msg="Volume pvc-65580647-e44a-4210-8545-2aff63ff0fe2 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4273 "" "Go-http-client/1.1"
time="2021-07-24T09:02:19Z" level=info msg="Volume pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "POST /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2?action=attach HTTP/1.1" 200 3352 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "POST /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa?action=attach HTTP/1.1" 200 3089 "" "Go-http-client/1.1"
time="2021-07-24T09:02:20Z" level=debug msg="Prepare to create instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016464\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:20Z" level=debug msg="Prepare to create instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016469\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, IP 10.244.2.205"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, Port 10075"
time="2021-07-24T09:02:21Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state stopped, new state running"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, IP 10.244.2.205"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, Port 10060"
time="2021-07-24T09:02:21Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:02:21 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3662 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:21 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3548 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:21 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3358 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:23 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:23 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3658 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:23 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3397 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:25 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:25 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 5499 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:25 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 5238 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:27 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:27 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 5499 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:27 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 5238 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:29 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 6243 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:29 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 5499 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:29 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 5978 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:31 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 6239 "" "Go-http-client/1.1"
time="2021-07-24T09:03:13Z" level=debug msg="Polling backup store for new volume backups" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185
time="2021-07-24T09:03:13Z" level=debug msg="Refreshed all volumes last backup based on backup store information" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185

You can also attach a Support Bundle here. You can generate a Support Bundle using the link at the footer of the Longhorn UI. longhorn-support-bundle_d4963230-e637-4991-9edd-4526a0295afe_2021-07-22T18-24-07Z.zip

Environment:

Longhorn version: v1.1.2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: OKE 1.20
- Number of management node in the cluster: Unknown. Managed.
- Number of worker node in the cluster: 2
Node config
- OS type and version: Oracle Linux 7.9
- CPU per node: 2 (ARM64)
- Memory per node: 12GB
- Disk type(e.g. SSD/NVMe): Network attached SSDs of some kind
- Network bandwidth between the nodes: Unknown but unlikely to be a factor
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Oracle Cloud OKE
Number of Longhorn volumes in the cluster: 9

This has happened to me on single node (Intel) k3s clusters in AWS, a 3 node (Intel) k3s cluster on Civo Cloud as well as my current two node ARM64 Oracle setup. All with replicas set appropriately for the number of nodes of course.

Additional context If the support bundle comes up with nothing useful I’ll spin up a few different clusters and see if I can get any more idea about what triggers it.

FAO @joshimoo

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 37 (19 by maintainers)

Most upvoted comments

We were experiencing the same symptoms and log messages. The restarts were caused by having a global-default priority class defined for the cluster. Any instance managers that inherited this priority class started to restart at regular intervals. As soon as the priority class was changed (and the change inherited by a new instance) the restarting stopped.

Hopefully this saves someone a bit of trouble as the cause was not clear from the logs. My guess it has something to do with the manager definition of the pod no longer matching once kubernetes adds the priority class. However I could not find where this is in the source code.

germondc on Jan 12, 2022

Actually code on https://github.com/longhorn/longhorn-manager/blob/840ae441597ac6af0904946db7d40916d959f9f8/controller/setting_controller.go#L496-L559 and https://github.com/longhorn/longhorn-manager/blob/840ae441597ac6af0904946db7d40916d959f9f8/controller/setting_controller.go#L225-L228 are quite clear. If priority class does not match then those pods will be deleted and re-created so if you define global priority class you need also set exactly same value with kubectl -n longhorn-system edit setting priority-class

and there is similar logic with node selector. As I have node selector set to longhorn-system namespace I need to set exactly same value with kubectl -n longhorn-system edit setting system-managed-components-node-selector

To avoid that issue would need modify settings controller logic on way that it will check those only if they have non-default value defined.

Now I just need to wait hour to verify that theory…

EDIT1: I already see that issue which I mentioned on https://github.com/longhorn/longhorn/issues/3042#issuecomment-1021203337 that deleting pod longhorn-manager-... on any single node did trigger instance-manger-... pods termination on all nodes does not happen anymore. So that most probably can be used to verify your configuration changes and/or code changes without waiting that one hour.

EDIT2: Confirmed 60 minutes restart issue disappeared by adding that setting.

olljanat on Jan 26, 2022

@germondc @tbomberg May I ask which Longhorn version are you using?

Is it possible that you can provide us with a support bundle for investigating and fixing? longhorn-support-bundle@rancher.com

@PhanLe1010 , v1.1.2 but upgrading to v1.2.3 did not fix the problem. I can rebuild a dev cluster to replicate the problem and provide the support bundle if that will help. But the problem can replicated with the following steps:

have Longhorn installed and running
apply a K8s default priority class:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: test-default
value: 1000000 
globalDefault: true

restart an instance manager and check the new pod definition has inherited the priority class:

priority: 1000000
priorityClassName: test-default

at this point the instance manager will be restarted regularly. Restarting the longhorn-manager will also result the instance manager being restarted. Worse still if all instance manager inherit the default priority class, restarting any longhorn manager will restart all instance managers.

I had a search in the code but as it isn’t familiar would take me some till to pin point where this could be happening.

Please let me know regarding generating a support bundle.

germondc on Jan 21, 2022

Yes, that as exactly the problem on our side. Instance Managers are up now for >14h.

tbomberg on Jan 13, 2022

After thinking and discussing with @yasker , we think that we shouldn’t change the controller logic here because:

This problem looks like related to a specific user env configuration. We can create a KB note on the documentation to notify the user that if they have a default priority class in the cluster, they should set the same value for Longhorn.
We were thinking about having Longhorn auto-detect the default priority class and use it when the priority class setting in Longhorn is empty. However, the problem with this approach is what if the user deletes the default priority class. In this case, a seemly harmless operation (delete the default priority class) results in Longhorn restarting all the instance manager pods and therefore crashes all volumes. The damage, in this case, is more likely to be more severe than the above case because at this point users probably already have many running Longhorn volumes.

Action items:

Create a KB note on the documentation to notify the user that if they have a default priority class in the cluster, they should set the same value for Longhorn.
Print out the log before deleting instance manager pods. This way, users can quickly find out why the instance manager pods are restarted in the early state after installing Longhorn.

How do you think @innobead @joshimoo @shuo-wu

PhanLe1010 on Feb 24, 2022

Many thanks @olljanat @germondc for the investigation.

Yeah, using your investigation I was able to see the problem (both from reproducing and from the code logic):

From the code logic, like @olljanat pointed out, the setting controller will delete any pods that don’t have the miss-match priority class as specified in the priority-class setting inside Longhorn UI. Some more data to match the behavior with the buggy code logic:

Why this problem happen every hour? It is because the setting controller has 1 hour resync. It means the setting controller will touch and reconcile every setting CR every hour interval. When setting controller process the priority-class setting CR, it deletes the instance manager pod that with the miss-match priority class. After deleting the instance manager pod, the pod is recreated with the same default priority class (which is different than the setting inside Longhorn UI). As the result, the cycle continues at the next hour.
Why are the instance manager pod with the miss-match priority class get deleted when a longhorn-manager pod is restarted? It is because when Longhorn manager is restarted, the setting controller probably touches all setting CRs. This again led to the deletion behavior.

Workaround:

Yeah, like @olljanat mentioned. Change the priority-class setting have the same name as the default one

Proposal:

This problem is similar to the problem that we have with default strogeclass before. The problem is that the manifest that we apply to Kubernetes cluster is different than the actual manifest in the cluster due to default values. Then, we think that we need to re-apply the manifest. This cycle continues infinitely.
To solve this general problem, we can remember the last apply manifest inside a annotation and only reapply the manifest if the annotation is different than value we want. This is similar to how kubectl apply command does. It is also similar to what we did with default storageclass

PhanLe1010 on Feb 9, 2022

Logging the reason the instance manager pods are being restarted would make a big difference.

germondc on Feb 25, 2022

@jenting I am still experiencing this issue in my prod k8s cluster. At almost 60 minutes on the mark all the instance manager pods will terminate. I have completely torn down longhorn twice and the issue still exists. I have it running In our dev K8s cluster and the issue doesn’t exist there.

Let me me know any information I can provide or gather for you.

sschroed99 on Aug 9, 2021

@joshimoo We have priority class test - https://github.com/longhorn/longhorn-tests/blob/master/manager/integration/tests/test_settings.py#L505 in our integration test.

khushboo-rancher on Jan 26, 2022

Hi,

@germondc @tbomberg May I ask which Longhorn version are you using?

Is it possible that you can provide us with a support bundle for investigating and fixing? longhorn-support-bundle@rancher.com

@PhanLe1010 We started with 1.2.2 and upgraded to 1.2.3 I created a support bundle after the 1.2.3 upgrade back in Dec. I just sent it.

tbomberg on Jan 21, 2022