longhorn: [Question] longhorn-driver-deployer can not start

kubectl get pods \
> --namespace longhorn-system
NAME                                       READY   STATUS             RESTARTS   AGE
engine-image-ei-eee5f438-s7lb4             1/1     Running            0          10m
instance-manager-e-2c134851                1/1     Running            0          10m
instance-manager-r-100de490                1/1     Running            0          10m
longhorn-driver-deployer-cd74cb75b-dlgvt   0/1     Init:0/1           0          10m
longhorn-manager-8g48d                     1/1     Running            0          10m
longhorn-ui-8486987944-r78hc               0/1     CrashLoopBackOff   6          10m

kubectl describe pod longhorn-driver-deployer-cd74cb75b-dlgvt   --namespace longhorn-system

Events:
  Type    Reason     Age        From                              Message
  ----    ------     ----       ----                              -------
  Normal  Scheduled  <unknown>  default-scheduler                 Successfully assigned longhorn-system/longhorn-driver-deployer-cd74cb75b-dlgvt to izj6cco39nfexbhvl3qk7oz
  Normal  Pulled     11m        kubelet, izj6cco39nfexbhvl3qk7oz  Container image "longhornio/longhorn-manager:v1.0.0" already present on machine
  Normal  Created    11m        kubelet, izj6cco39nfexbhvl3qk7oz  Created container wait-longhorn-manager
  Normal  Started    11m        kubelet, izj6cco39nfexbhvl3qk7oz  Started container wait-longhorn-manager

kubectl logs longhorn-driver-deployer-cd74cb75b-dlgvt   --namespace longhorn-system                                                                                                      
Error from server (BadRequest): container "longhorn-driver-deployer" in pod "longhorn-driver-deployer-cd74cb75b-dlgvt" is waiting to start: PodInitializing

kubectl logs longhorn-ui-8486987944-r78hc  --namespace longhorn-system
2020/07/04 09:17:33 [warn] 1#1: duplicate MIME type "text/html" in /etc/nginx/nginx.conf:7
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/nginx.conf:7
2020/07/04 09:17:33 [emerg] 1#1: host not found in upstream "longhorn-backend" in /etc/nginx/nginx.conf:32
nginx: [emerg] host not found in upstream "longhorn-backend" in /etc/nginx/nginx.conf:32

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 13
Comments: 48 (14 by maintainers)

Most upvoted comments

I met the similar issue ,but all looks good ,coredns is also fine . longhorn-manager log like this : time=“2020-12-10T12:51:14Z” level=debug msg=“Requeue longhorn-system/lab1 due to conflict: Operation cannot be fulfilled on nodes.longhorn.io "lab1": the object has been modified; please apply your changes to the latest version and try again”

OS: ubuntu 20.04 , k8s : rke2 1.18

*Update After ipv6 was disabled ,seems it is not pending , not sure why ipv6 cause this issue .

thomasdba on Dec 10, 2020

Ok, here is probably the problem: Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host The package iscsiadm/open-iscsi is probably missing on the worker nodes. Can you install it on the systems and check if the deployment gets a step further?

jaisers on Mar 30, 2021

@Donatas-M

Hi, can you provide a little more detail with this. I mean what were the logs exactly and how did you enabled --selinux flag.

The one item I should have added is this would only affect hosts that have a distro using it. So Fedora/RHEL/Centos/Fedora Coreos…etc. I am using Fedora Coreos. I missed a section referencing the topic on the k3s install documents. https://rancher.com/docs/k3s/latest/en/advanced/#selinux-support

As for the actual log message it is going to be on the host not the longhorn manager. I don’t have it saved, but I found it on the hosts in the actual k3s log, and the k3s-agent log. It was only one line that shows up on a startup or restart. The line actually states something to the regard of you are using SELINUX but do not have the --selinux flag set. As for setting the flag I just ran the normal install/upgrade command for the k3s server / k3s agents again one by one adding --selinux into the command. On the k3s master/control planes it looked like this.

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.20.6+k3s1 sh -s - server --datastore-endpoint="mysql://user:password@tcp(database-ip:3306)/k3s" --node-taint CriticalAddonsOnly=true:NoExecute --selinux

roygabriel on Aug 23, 2021

What would be the best workaround at the moment?

PhilParisot on Mar 30, 2021

I’m having this same problem:

pod/longhorn-driver-deployer-658fdf45cc-vxcpq   0/1     Init:0/1   0          48m

When I try to curl, I get this:

kubectl -n longhorn-system exec engine-image-ei-ee18f965-qtnzr -- curl -v http://longhorn-backend:9500
* Rebuilt URL to: http://longhorn-backend:9500/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.43.21.124...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:--  0:00:32 --:--:--     0* connect to 10.43.21.124 port 9500 failed: Connection timed out
* Failed to connect to longhorn-backend port 9500: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to longhorn-backend port 9500: Connection timed out
command terminated with exit code 7

My service is this:

service/longhorn-backend    ClusterIP   10.43.21.124   <none>        9500/TCP   115m

Since my curl tried to connect to the correct IP, that implies that DNS is working within the cluster, right?

Here’s the logs from the longhorn-manager:

kubectl logs -f longhorn-manager-dt76s -n longhorn-system
time="2020-12-15T04:30:52Z" level=info msg="Start overwriting built-in settings with customized values"
time="2020-12-15T04:30:52Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: Failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access '/var/lib/rancher/longhorn/engine-binaries/*': No such file or directory\n, error exit status 2"
time="2020-12-15T04:30:52Z" level=info msg="Start upgrading"
time="2020-12-15T04:30:52Z" level=info msg="No API version upgrade is needed"
time="2020-12-15T04:30:52Z" level=info msg="Finish upgrading"
time="2020-12-15T04:30:52Z" level=info msg="Upgrade leader lost: master"
E1215 04:30:52.647751       1 kubernetes_node_controller.go:244] Couldn't get nodes master: node "master" not found
E1215 04:30:52.652063       1 kubernetes_node_controller.go:256] Couldn't get nodes master: node "master" not found
time="2020-12-15T04:30:52Z" level=debug msg="Engine image longhornio/longhorn-engine:v1.0.2 is ready"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn Kubernetes node controller"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn Engine Image controller"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn replica controller"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn volume controller"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn Setting controller"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn node controller"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn websocket controller"
time="2020-12-15T04:30:52Z" level=info msg="Starting Longhorn instance manager controller"
time="2020-12-15T04:30:52Z" level=info msg="Start Longhorn engine controller"
time="2020-12-15T04:30:52Z" level=info msg="Start kubernetes controller"
time="2020-12-15T04:30:52Z" level=info msg="Listening on 10.42.0.22:9500"
time="2020-12-15T04:30:52Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"master\", UID:\"9111993c-17ed-42ef-8fde-c3c423387a75\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2802\", FieldPath:\"\"}): type: 'Warning' reason: 'Ready' Node master is down: the manager pod longhorn-manager-dt76s is not running"
time="2020-12-15T04:30:52Z" level=debug msg="Start monitoring instance manager instance-manager-r-3edc5d33"
time="2020-12-15T04:30:52Z" level=debug msg="Start monitoring instance manager instance-manager-e-05789d64"
time="2020-12-15T04:30:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"master\", UID:\"9111993c-17ed-42ef-8fde-c3c423387a75\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2837\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node master is ready"

Here are the events for the manager:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  54m   default-scheduler  Successfully assigned longhorn-system/longhorn-manager-dt76s to master
  Normal  Pulled     54m   kubelet            Container image "longhornio/longhorn-manager:v1.0.2" already present on machine
  Normal  Created    54m   kubelet            Created container longhorn-manager
  Normal  Started    54m   kubelet            Started container longhorn-manager

So, I’m very confident it is up.

I installed this from the helm chart, with very few values overridden (only set replicaSoftAntiAffinity to true, since I have a single node cluster at the moment)

Here’s the service description:

kubectl describe -n longhorn-system service/longhorn-backend
Name:              longhorn-backend
Namespace:         longhorn-system
Labels:            app=longhorn-manager
                   app.kubernetes.io/instance=longhorn
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=longhorn
                   app.kubernetes.io/version=v1.0.2
                   helm.sh/chart=longhorn-1.0.2
Annotations:       meta.helm.sh/release-name: longhorn
                   meta.helm.sh/release-namespace: longhorn-system
Selector:          app=longhorn-manager
Type:              ClusterIP
IP:                10.43.21.124
Port:              manager  9500/TCP
TargetPort:        manager/TCP
Endpoints:         10.42.0.22:9500
Session Affinity:  ClientIP
Events:            <none>

Any and all help is appreciated.

---- EDIT ----

I deleted the Session Affinity from the service (service/longhorn-backend)

This block:

  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

And it got past the infinite hang, along with other things. I sincerely wonder if this timeout was achieved for @lexfrei and then his cluster started. If I read this correctly, the session affinity lasts for 3 hours?

Is this correct? Feels like a long time for a session affinity.

Bwvolleyball on Dec 15, 2020

I had the same problem and actually could track it down to CoreDNS not being able to resolv the DNS Adresses. The init container could access http://longhorn-backend:9500/v1 as long as the Cluster IP was used instead of the name.

So the problem is not longhorn related.

The CoreDNS had a lot of error messages like this:

[ERROR] plugin/errors: 2 www.example.com. A: read udp "CoreDNS Pod IP":41873->"IP of external DNS Server":53: i/o timeout
docker showed the following errors on the host
Nov 12 11:14:48 Hostname dockerd[1488]: time="2020-11-12T11:14:48.141346136+01:00" level=warning msg="IPv4 forwarding is disabled. Networking will not work."

There we have the problem - forwarding wasn’t set: sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 0 Add to /etc/sysctl.conf net.ipv4.ip_forward = 1 and restart networking on the host (systemctl restart network.service)

Afterwards it worked for me.

jaisers on Oct 26, 2021

[root@k3s-master ~]# kubectl logs -f longhorn-manager-2zgrn -nlonghorn-system
time="2020-07-24T10:00:26Z" level=info msg="Start overwriting built-in settings with customized values"
time="2020-07-24T10:00:26Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: Failed to execute: nsenter [--mount=/host/proc/1483/ns/mnt --net=/host/proc/1483/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access /var/lib/rancher/longhorn/engine-binaries/*: No such file or directory\n, error exit status 2"
time="2020-07-24T10:00:27Z" level=info msg="Start upgrading"
time="2020-07-24T10:00:28Z" level=warning msg="Cannot verify current CRD version, assume it's not v1alpha1: unable to verify if version matches v1alpha1: settings.longhorn.rancher.io \"default-engine-image\" is forbidden: User \"system:serviceaccount:longhorn-system:longhorn-service-account\" cannot get resource \"settings\" in API group \"longhorn.rancher.io\" in the namespace \"longhorn-system\""
time="2020-07-24T10:00:30Z" level=info msg="Initialized CRD API Version to longhorn.io/v1beta1"
time="2020-07-24T10:00:31Z" level=info msg="Finish upgrading"
E0724 10:00:31.521131       1 leaderelection.go:282] Failed to release lock: Lease.coordination.k8s.io "longhorn-manager-upgrade-lock" is invalid: spec.leaseDurationSeconds: Invalid value: 0: must be greater than 0
time="2020-07-24T10:00:31Z" level=info msg="Upgrade leader lost: k3s-node1"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn Engine Image controller"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn node controller"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn Setting controller"
time="2020-07-24T10:00:31Z" level=info msg="Start kubernetes controller"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn engine controller"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn volume controller"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn Kubernetes node controller"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn replica controller"
time="2020-07-24T10:00:31Z" level=info msg="Start Longhorn websocket controller"
time="2020-07-24T10:00:31Z" level=info msg="Starting Longhorn instance manager controller"
time="2020-07-24T10:00:35Z" level=debug msg="Updated setting default-engine-image to longhornio/longhorn-engine:v1.0.1"
time="2020-07-24T10:00:36Z" level=debug msg="Updated setting default-instance-manager-image to longhornio/longhorn-instance-manager:v1_20200514"
time="2020-07-24T10:00:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node1\", UID:\"b15128f2-b877-4e1c-bbc9-bc827578d0fa\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907743\", FieldPath:\"\"}): type: 'Warning' reason: 'Ready' Node k3s-node1 is down: the manager pod longhorn-manager-2zgrn is not running"
time="2020-07-24T10:00:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node1\", UID:\"b15128f2-b877-4e1c-bbc9-bc827578d0fa\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907743\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' "
time="2020-07-24T10:00:36Z" level=debug msg="Prepare to create default instance manager instance-manager-e-fc2e789b, node: k3s-node1, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: engine"
time="2020-07-24T10:00:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node1\", UID:\"b15128f2-b877-4e1c-bbc9-bc827578d0fa\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907743\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Disk default-disk-fd0000000000(/var/lib/longhorn/) on node k3s-node1 is ready"
time="2020-07-24T10:00:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node1\", UID:\"b15128f2-b877-4e1c-bbc9-bc827578d0fa\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907743\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' Disk default-disk-fd0000000000(/var/lib/longhorn/) on node k3s-node1 is schedulable"
time="2020-07-24T10:00:36Z" level=debug msg="Prepare to create default instance manager instance-manager-r-b816d5eb, node: k3s-node1, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: replica"
time="2020-07-24T10:00:36Z" level=debug msg="Created engine image ei-3bd16bdf (longhornio/longhorn-engine:v1.0.1)"
time="2020-07-24T10:00:36Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.1 to be ready"
time="2020-07-24T10:00:36Z" level=debug msg="Instance Manager Controller k3s-node1 picked up instance-manager-e-fc2e789b"
time="2020-07-24T10:00:36Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-e-fc2e789b with state stopped"
time="2020-07-24T10:00:37Z" level=debug msg="Engine Image Controller k3s-node1 picked up ei-3bd16bdf (longhornio/longhorn-engine:v1.0.1)"
time="2020-07-24T10:00:37Z" level=info msg="Created daemon set engine-image-ei-3bd16bdf for engine image ei-3bd16bdf (longhornio/longhorn-engine:v1.0.1)"
time="2020-07-24T10:00:37Z" level=info msg="Created instance manager pod instance-manager-e-fc2e789b for instance manager instance-manager-e-fc2e789b"
time="2020-07-24T10:00:37Z" level=debug msg="Instance Manager Controller k3s-node1 picked up instance-manager-r-b816d5eb"
time="2020-07-24T10:00:37Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-r-b816d5eb with state stopped"
time="2020-07-24T10:00:38Z" level=info msg="Created instance manager pod instance-manager-r-b816d5eb for instance manager instance-manager-r-b816d5eb"
time="2020-07-24T10:00:42Z" level=debug msg="Start monitoring instance manager instance-manager-r-b816d5eb"
time="2020-07-24T10:00:42Z" level=debug msg="Start monitoring instance manager instance-manager-e-fc2e789b"
time="2020-07-24T10:00:42Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.1 to be ready"
time="2020-07-24T10:00:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"EngineImage\", Namespace:\"longhorn-system\", Name:\"ei-3bd16bdf\", UID:\"0b1686b0-c89d-4751-b934-c099f4a3eddb\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907811\", FieldPath:\"\"}): type: 'Normal' reason: 'ready' Engine image ei-3bd16bdf (longhornio/longhorn-engine:v1.0.1) become ready"
time="2020-07-24T10:00:48Z" level=debug msg="Engine image longhornio/longhorn-engine:v1.0.1 is ready"
time="2020-07-24T10:00:48Z" level=info msg="Listening on 172.16.190.144:9500"
time="2020-07-24T10:00:54Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node2\", UID:\"b9c6c253-056a-4dc4-aafa-0b3effbe49da\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907959\", FieldPath:\"\"}): type: 'Warning' reason: 'Ready' Node k3s-node2 is down: the manager pod longhorn-manager-jf7zg is not running"
time="2020-07-24T10:00:54Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node2\", UID:\"b9c6c253-056a-4dc4-aafa-0b3effbe49da\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907959\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' "
time="2020-07-24T10:00:55Z" level=debug msg="Instance Manager Controller k3s-node1 picked up instance-manager-r-24090113"
time="2020-07-24T10:00:55Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node1\", UID:\"b15128f2-b877-4e1c-bbc9-bc827578d0fa\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907839\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node k3s-node1 is ready"
time="2020-07-24T10:00:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node2\", UID:\"b9c6c253-056a-4dc4-aafa-0b3effbe49da\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2908000\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node k3s-node2 is ready"
time="2020-07-24T10:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T10:01:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-master\", UID:\"78dcaafc-5471-4060-9036-2f2521859a25\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2908108\", FieldPath:\"\"}): type: 'Warning' reason: 'Ready' Node k3s-master is down: the manager pod longhorn-manager-gsqgj is not running"
time="2020-07-24T10:01:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-master\", UID:\"78dcaafc-5471-4060-9036-2f2521859a25\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2908108\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' "
time="2020-07-24T10:01:16Z" level=debug msg="Requeue longhorn-system/k3s-master due to conflict: Operation cannot be fulfilled on nodes.longhorn.io \"k3s-master\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-07-24T10:01:24Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-master\", UID:\"78dcaafc-5471-4060-9036-2f2521859a25\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2908164\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node k3s-master is ready"
time="2020-07-24T11:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T12:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T13:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T14:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T15:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T16:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T17:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T18:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T19:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T20:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T21:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T22:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T23:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T00:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T01:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T02:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T03:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T04:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T05:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T06:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T07:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T08:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T09:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T10:01:01Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T11:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T12:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T13:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T14:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T15:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T16:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T17:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T18:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T19:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T20:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T21:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T22:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T23:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T00:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T01:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T02:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T03:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T04:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T05:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T06:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T07:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T08:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T09:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T10:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T11:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T12:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T13:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T14:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T15:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T16:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T17:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T18:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T19:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T20:01:02Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"

[root@k3s-master ~]# kubectl logs -f longhorn-manager-gsqgj -nlonghorn-system time=“2020-07-24T10:00:36Z” level=info msg=“Start overwriting built-in settings with customized values” time=“2020-07-24T10:00:36Z” level=info msg=“cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: Failed to execute: nsenter [–mount=/host/proc/1476/ns/mnt --net=/host/proc/1476/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/], output , stderr, ls: cannot access /var/lib/rancher/longhorn/engine-binaries/: No such file or directory\n, error exit status 2” time=“2020-07-24T10:00:36Z” level=info msg=“New upgrade leader elected: k3s-node1” time=“2020-07-24T10:00:56Z” level=info msg=“New upgrade leader elected: k3s-node2” time=“2020-07-24T10:01:16Z” level=info msg=“Start upgrading” time=“2020-07-24T10:01:16Z” level=info msg=“No API version upgrade is needed” time=“2020-07-24T10:01:16Z” level=info msg=“Finish upgrading” E0724 10:01:16.521264 1 leaderelection.go:282] Failed to release lock: Lease.coordination.k8s.io “longhorn-manager-upgrade-lock” is invalid: spec.leaseDurationSeconds: Invalid value: 0: must be greater than 0 time=“2020-07-24T10:01:16Z” level=info msg=“Upgrade leader lost: k3s-master” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn Engine Image controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn websocket controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn replica controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn volume controller” time=“2020-07-24T10:01:16Z” level=info msg=“Starting Longhorn instance manager controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn Kubernetes node controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn Setting controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn node controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start Longhorn engine controller” time=“2020-07-24T10:01:16Z” level=info msg=“Start kubernetes controller” time=“2020-07-24T10:01:16Z” level=debug msg=“Prepare to create default instance manager instance-manager-e-3d67a14b, node: k3s-master, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: engine” time=“2020-07-24T10:01:16Z” level=info msg=“Event(v1.ObjectReference{Kind:"Node", Namespace:"longhorn-system", Name:"k3s-master", UID:"78dcaafc-5471-4060-9036-2f2521859a25", APIVersion:"longhorn.io/v1beta1", ResourceVersion:"2908110", FieldPath:""}): type: ‘Normal’ reason: ‘Ready’ Disk default-disk-fd0000000000(/var/lib/longhorn/) on node k3s-master is ready” time=“2020-07-24T10:01:16Z” level=info msg=“Event(v1.ObjectReference{Kind:"Node", Namespace:"longhorn-system", Name:"k3s-master", UID:"78dcaafc-5471-4060-9036-2f2521859a25", APIVersion:"longhorn.io/v1beta1", ResourceVersion:"2908110", FieldPath:""}): type: ‘Normal’ reason: ‘Schedulable’ Disk default-disk-fd0000000000(/var/lib/longhorn/) on node k3s-master is schedulable” time=“2020-07-24T10:01:16Z” level=debug msg=“Engine image longhornio/longhorn-engine:v1.0.1 is ready” time=“2020-07-24T10:01:16Z” level=info msg=“Listening on 172.16.68.79:9500” time=“2020-07-24T10:01:16Z” level=debug msg=“Prepare to create default instance manager instance-manager-r-542a82f3, node: k3s-master, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: replica” time=“2020-07-24T10:01:16Z” level=debug msg=“Instance Manager Controller k3s-master picked up instance-manager-e-3d67a14b” time=“2020-07-24T10:01:16Z” level=warning msg=“Starts to clean up then recreates pod for instance manager instance-manager-e-3d67a14b with state stopped” time=“2020-07-24T10:01:17Z” level=info msg=“Created instance manager pod instance-manager-e-3d67a14b for instance manager instance-manager-e-3d67a14b” time=“2020-07-24T10:01:18Z” level=debug msg=“Instance Manager Controller k3s-master picked up instance-manager-r-542a82f3” time=“2020-07-24T10:01:18Z” level=warning msg=“Starts to clean up then recreates pod for instance manager instance-manager-r-542a82f3 with state stopped” time=“2020-07-24T10:01:18Z” level=info msg=“Created instance manager pod instance-manager-r-542a82f3 for instance manager instance-manager-r-542a82f3” time=“2020-07-24T10:01:18Z” level=debug msg=“Start monitoring instance manager instance-manager-e-3d67a14b” time=“2020-07-24T10:01:19Z” level=debug msg=“Start monitoring instance manager instance-manager-r-542a82f3” time=“2020-07-24T10:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T10:01:24Z” level=info msg=“Event(v1.ObjectReference{Kind:"Node", Namespace:"longhorn-system", Name:"k3s-master", UID:"78dcaafc-5471-4060-9036-2f2521859a25", APIVersion:"longhorn.io/v1beta1", ResourceVersion:"2908164", FieldPath:""}): type: ‘Normal’ reason: ‘Ready’ Node k3s-master is ready” time=“2020-07-24T10:01:24Z” level=debug msg=“Requeue longhorn-system/k3s-master due to conflict: Operation cannot be fulfilled on nodes.longhorn.io "k3s-master": the object has been modified; please apply your changes to the latest version and try again” time=“2020-07-24T11:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T12:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T13:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T14:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T15:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T16:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T17:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T18:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T19:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T20:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T21:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T22:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-24T23:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T00:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T01:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T02:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T03:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T04:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T05:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T06:01:20Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T07:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T08:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T09:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T10:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T11:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T12:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T13:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T14:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T15:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T16:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T17:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T18:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T19:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T20:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T21:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T22:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-25T23:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T00:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T01:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T02:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T03:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T04:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T05:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T06:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T07:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T08:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T09:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T10:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T11:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T12:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T13:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T14:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T15:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T16:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T17:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T18:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T19:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T20:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T21:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T22:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-26T23:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-27T00:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving" time=“2020-07-27T01:01:21Z” level=debug msg=“Failed to check for the latest upgrade: Post "https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\”: dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: server misbehaving"

[root@k3s-master ~]# kubectl logs -f longhorn-manager-jf7zg  -nlonghorn-system
time="2020-07-24T10:00:32Z" level=info msg="Start overwriting built-in settings with customized values"
time="2020-07-24T10:00:32Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: Failed to execute: nsenter [--mount=/host/proc/1476/ns/mnt --net=/host/proc/1476/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access /var/lib/rancher/longhorn/engine-binaries/*: No such file or directory\n, error exit status 2"
time="2020-07-24T10:00:32Z" level=info msg="New upgrade leader elected: k3s-node1"
time="2020-07-24T10:00:54Z" level=info msg="Start upgrading"
time="2020-07-24T10:00:54Z" level=info msg="No API version upgrade is needed"
time="2020-07-24T10:00:54Z" level=info msg="Finish upgrading"
E0724 10:00:54.839645       1 leaderelection.go:282] Failed to release lock: Lease.coordination.k8s.io "longhorn-manager-upgrade-lock" is invalid: spec.leaseDurationSeconds: Invalid value: 0: must be greater than 0
time="2020-07-24T10:00:54Z" level=info msg="Upgrade leader lost: k3s-node2"
E0724 10:00:54.862469       1 kubernetes_node_controller.go:244] Couldn't get nodes k3s-node2: node "k3s-node2" not found 
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn Setting controller"
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn Kubernetes node controller"
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn Engine Image controller"
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn engine controller"
time="2020-07-24T10:00:54Z" level=info msg="Starting Longhorn instance manager controller"
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn replica controller"
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn websocket controller"
time="2020-07-24T10:00:54Z" level=info msg="Start kubernetes controller"
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn node controller"
time="2020-07-24T10:00:54Z" level=info msg="Start Longhorn volume controller"
time="2020-07-24T10:00:55Z" level=debug msg="Engine image longhornio/longhorn-engine:v1.0.1 is ready"
time="2020-07-24T10:00:55Z" level=info msg="Listening on 172.16.63.28:9500"
time="2020-07-24T10:00:55Z" level=debug msg="Prepare to create default instance manager instance-manager-e-0bcebec3, node: k3s-node2, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: engine"
time="2020-07-24T10:00:55Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node2\", UID:\"b9c6c253-056a-4dc4-aafa-0b3effbe49da\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907961\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Disk default-disk-fd0000000000(/var/lib/longhorn/) on node k3s-node2 is ready"
time="2020-07-24T10:00:55Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node2\", UID:\"b9c6c253-056a-4dc4-aafa-0b3effbe49da\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907961\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' Disk default-disk-fd0000000000(/var/lib/longhorn/) on node k3s-node2 is schedulable"
time="2020-07-24T10:00:55Z" level=debug msg="Instance Manager Controller k3s-node2 picked up instance-manager-e-0bcebec3"
time="2020-07-24T10:00:55Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-e-0bcebec3 with state stopped"
time="2020-07-24T10:00:55Z" level=info msg="Created instance manager pod instance-manager-e-0bcebec3 for instance manager instance-manager-e-0bcebec3"
time="2020-07-24T10:00:55Z" level=debug msg="Prepare to create default instance manager instance-manager-r-24090113, node: k3s-node2, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: replica"
time="2020-07-24T10:00:55Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-node1\", UID:\"b15128f2-b877-4e1c-bbc9-bc827578d0fa\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2907839\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node k3s-node1 is ready"
time="2020-07-24T10:00:56Z" level=debug msg="Instance Manager Controller k3s-node2 picked up instance-manager-r-24090113"
time="2020-07-24T10:00:56Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-r-24090113 with state stopped"
time="2020-07-24T10:00:56Z" level=debug msg="Requeue longhorn-system/k3s-node1 due to conflict: Operation cannot be fulfilled on nodes.longhorn.io \"k3s-node1\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-07-24T10:00:56Z" level=info msg="Created instance manager pod instance-manager-r-24090113 for instance manager instance-manager-r-24090113"
time="2020-07-24T10:00:57Z" level=debug msg="Requeue longhorn-system/k3s-node2 due to conflict: Operation cannot be fulfilled on nodes.longhorn.io \"k3s-node2\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-07-24T10:00:58Z" level=debug msg="Start monitoring instance manager instance-manager-e-0bcebec3"
time="2020-07-24T10:00:59Z" level=debug msg="Start monitoring instance manager instance-manager-r-24090113"
time="2020-07-24T10:01:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-master\", UID:\"78dcaafc-5471-4060-9036-2f2521859a25\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2908108\", FieldPath:\"\"}): type: 'Warning' reason: 'Ready' Node k3s-master is down: the manager pod longhorn-manager-gsqgj is not running"
time="2020-07-24T10:01:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-master\", UID:\"78dcaafc-5471-4060-9036-2f2521859a25\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2908108\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' "
time="2020-07-24T10:01:16Z" level=debug msg="Instance Manager Controller k3s-node2 picked up instance-manager-e-3d67a14b"
time="2020-07-24T10:01:16Z" level=debug msg="Requeue longhorn-system/instance-manager-e-3d67a14b due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-e-3d67a14b\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-07-24T10:01:17Z" level=debug msg="Instance Manager Controller k3s-node2 picked up instance-manager-r-542a82f3"
time="2020-07-24T10:01:24Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"k3s-master\", UID:\"78dcaafc-5471-4060-9036-2f2521859a25\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2908164\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node k3s-master is ready"
time="2020-07-24T10:01:24Z" level=debug msg="Requeue longhorn-system/k3s-master due to conflict: Operation cannot be fulfilled on nodes.longhorn.io \"k3s-master\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-07-24T10:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T11:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T12:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T13:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T14:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T15:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T16:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T17:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T18:01:24Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T19:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T20:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T21:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T22:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-24T23:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T00:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T01:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T02:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T03:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T04:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T05:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T06:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T07:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T08:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T09:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T10:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T11:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T12:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T13:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T14:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T15:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T16:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T17:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T18:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T19:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T20:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T21:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T22:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-25T23:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T00:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T01:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T02:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T03:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T04:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T05:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T06:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T07:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T08:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T09:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T10:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T11:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T12:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T13:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T14:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T15:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T16:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T17:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T18:01:25Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T19:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T20:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T21:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T22:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-26T23:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-27T00:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-27T01:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-27T02:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-27T03:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-27T04:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"
time="2020-07-27T05:01:26Z" level=debug msg="Failed to check for the latest upgrade: Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout"

Crystal-w on Jul 27, 2020

@shuo-wu Sorry it was actually my coredns needed to be restarted. I had switched from calico to weave and there were some lingering files on the control plane node. Thanks for the fast reply!

jonmahoney15 on Jun 28, 2022

For me, it was a misconfigured DNS. My router was configured to supply an AdGuard IP as first DNS Server, which itself was previously running in K3S, before I needed a recovery of my master node. So this DNS was down. I testet the curl from an engine pod to the backend service and could not reach it (even if that should stay cluster intern). My resolution was indeed to fix the provided DNS IPs and the restart of CoreDNS. As soon as I restartet CoreDNS, all csi-attacher, resizer, provisioner etc. popped up instantly.

Bizzl on Aug 5, 2021

Snap I think that worked! Thanks @jaisers!

PhilParisot on Mar 30, 2021