calico: Calico tries to reassign IP addresses already in use

Calico tries to reassign IP addresses already in use

Expected Behavior

Calico should assign a unused IP address to a new created pod

Current Behavior

Calico trise several IP addresses, already in use by other pods, leading to multiple errors like

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9f90580e9dd409a5ab621ef44c1637972ec6fb5d55db732c1943647ce1cf9e00" network for pod "nginx-ingress-controller-6664bc7855-8hxwv": networkPlugin cni failed to set up pod "nginx-ingress-controller-6664bc7855-8hxwv_eai-selfservice-dev" network: error adding host side routes for interface: calicfa03c2c5b1, error: route (Ifindex: 436, Dst: 100.80.1.62/32, Scope: 253) already exists for an interface other than 'calicfa03c2c5b1'

with changing IP addresses. At some point the container starts with a free IP address assigned.

Steps to Reproduce (for bugs)

I’m not sure on how to reproduce this, but I’ll provide as many informations as I can: The events on a pods looks like this:

  Warning  FailedCreatePodSandBox  43s                kubelet, k8s-worker-01.example.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9f90580e9dd409a5ab621ef44c1637972ec6fb5d55db732c1943647ce1cf9e00" network for pod "nginx-ingress-controller-6664bc7855-8hxwv": networkPlugin cni failed to set up pod "nginx-ingress-controller-6664bc7855-8hxwv_eai-selfservice-dev" network: error adding host side routes for interface: calicfa03c2c5b1, error: route (Ifindex: 436, Dst: 100.80.1.62/32, Scope: 253) already exists for an interface other than 'calicfa03c2c5b1'
  Warning  FailedCreatePodSandBox  37s                kubelet, k8s-worker-01.example.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b9ecd16415de567bdad4523f42ee9e0a1b7ebf609eed03a61e88f117b2b5f2d1" network for pod "nginx-ingress-controller-6664bc7855-8hxwv": networkPlugin cni failed to set up pod "nginx-ingress-controller-6664bc7855-8hxwv_eai-selfservice-dev" network: error adding host side routes for interface: calicfa03c2c5b1, error: route (Ifindex: 437, Dst: 100.80.1.63/32, Scope: 253) already exists for an interface other than 'calicfa03c2c5b1'
  Warning  FailedCreatePodSandBox  26s                kubelet, k8s-worker-01.example.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "667a2b553cf6598f7fc0da75dfb9d5bd18163c87348db3a03e42be98d9a0762c" network for pod "nginx-ingress-controller-6664bc7855-8hxwv": networkPlugin cni failed to set up pod "nginx-ingress-controller-6664bc7855-8hxwv_eai-selfservice-dev" network: error adding host side routes for interface: calicfa03c2c5b1, error: route (Ifindex: 439, Dst: 100.80.1.65/32, Scope: 253) already exists for an interface other than 'calicfa03c2c5b1'

You can see here, that it tries to assign the IP addresses 100.80.1.62, .63 and .64, which are all in use. Next it startet the pod successfull with 100.80.1.67.

Next I checked one of the exisiting pods, but everything looks good as far as I can tell:

k get pod -owide --all-namespaces|grep 100.80.1.63
eai-dev               default-http-backend-6fd4978fb-2hkx4                           1/1     Running                      0          37h     100.80.1.63      k8s-worker-01.example.com    <none>           <none>

And to be sure I check the ipamhandle of this pod too:

k get ipamhandle k8s-pod-network.5a1d961664df5e922c225bc06e69d0fc60aedf6eaa96670ecd5febc7cb160262 -oyaml
apiVersion: crd.projectcalico.org/v1
kind: IPAMHandle
metadata:
  annotations:
    projectcalico.org/metadata: '{"creationTimestamp":null}'
  creationTimestamp: "2020-08-25T23:45:08Z"
  generation: 1
  name: k8s-pod-network.5a1d961664df5e922c225bc06e69d0fc60aedf6eaa96670ecd5febc7cb160262
  resourceVersion: "971980875"
  selfLink: /apis/crd.projectcalico.org/v1/ipamhandles/k8s-pod-network.5a1d961664df5e922c225bc06e69d0fc60aedf6eaa96670ecd5febc7cb160262
  uid: 467c0bca-4a32-4179-b214-3e286cadb783
spec:
  block:
    100.80.1.0/26: 1
  handleID: k8s-pod-network.5a1d961664df5e922c225bc06e69d0fc60aedf6eaa96670ecd5febc7cb160262

(even though I’m not sure whether this helps.)

I checked the ipamblock of the node, which is huge and shortened here, but you can see, that the IP addresses are not listet under unallocated:

apiVersion: crd.projectcalico.org/v1
kind: IPAMBlock
metadata:
  annotations:
    projectcalico.org/metadata: '{"creationTimestamp":null}'
  creationTimestamp: "2020-07-12T11:09:10Z"
  generation: 155
  name: 100-80-1-0-26
  resourceVersion: "971980877"
  selfLink: /apis/crd.projectcalico.org/v1/ipamblocks/100-80-1-0-26
  uid: 1c670855-c430-11ea-b730-00505684b2d0
spec:
  affinity: host:k8s-worker-00.example.com
  allocations:
  - null
  - null
  - 15
  - 11
  - 29
  - 40
  - 51
  - 12
  - 13
  - 14
  - null
  - 0
  - null
  - 1
  - 2
  - null
  - null
  - null
  - null
  - 3
  - 16
  - 4
  - 5
  - 6
  - 7
  - 8
  - 9
  - 17
  - 10
  - 18
  - 19
  - 20
  - 21
  - 22
  - 23
  - 24
  - 25
  - 26
  - 27
  - 28
  - 30
  - 31
  - 32
  - 33
  - 34
  - 35
  - 36
  - 37
  - 38
  - 39
  - 41
  - 42
  - 43
  - 44
  - 45
  - 46
  - 47
  - 48
  - 49
  - 50
  - 52
  - 53
  - 54
  - 55
  attributes:
  - handle_id: "k8s-pod-network.dd372304a3d85e8900e0b940ce4d98936543fd403312ee151f6f9a178d08636d\r\neth0"
    secondary:
      namespace: rook-ceph-stage-primary
      node: k8s-worker-00.example.com
      pod: rook-ceph-stage-primary-backup-pdm7z
  - handle_id: "k8s-pod-network.518cee30324d9bee2c1a80a78fb41055ea5609298833e0a042ab9097d51be239\r\neth0"
    secondary:
      namespace: cxa-stage
      node: k8s-worker-00.example.com
      pod: release-name-zookeeper-1
[...]
  - handle_id: k8s-pod-network.44333bb9ad1fa32a354121431cb75c4e4b8996587793434eeb0c493ea6e4893d
    secondary:
      namespace: eai-dev
      node: k8s-worker-01.example.com
      pod: comp00000006-66c689ff6f-x9472
  - handle_id: k8s-pod-network.5a1d961664df5e922c225bc06e69d0fc60aedf6eaa96670ecd5febc7cb160262
    secondary:
      namespace: eai-dev
      node: k8s-worker-01.example.com
      pod: default-http-backend-6fd4978fb-2hkx4
  cidr: 100.80.1.0/26
  deleted: false
  strictAffinity: false
  unallocated:
  - 0
  - 1
  - 10
  - 12
  - 15
  - 16
  - 17
  - 18

One strange thing I noticed here, is that under attributes it lists another node (k8s-worker-00.example.com)which has been deleted and is no longer part of the cluster. I’m not sure if this is a problem. Maybe this happend because calico reused the CIDR range for this node. I checked the node ipamhandle too, but this looks good to me too:

apiVersion: crd.projectcalico.org/v1
kind: IPAMHandle
metadata:
  annotations:
    projectcalico.org/metadata: '{"creationTimestamp":null}'
  creationTimestamp: "2020-08-24T23:28:51Z"
  generation: 1
  name: ipip-tunnel-addr-k8s-worker-01.example.com
  resourceVersion: "970070259"
  selfLink: /apis/crd.projectcalico.org/v1/ipamhandles/ipip-tunnel-addr-k8s-worker-01.example.com
  uid: d80ea18c-4490-446a-b079-ef286a17cfe0
spec:
  block:
    100.80.1.64/26: 1
  handleID: ipip-tunnel-addr-k8s-worker-01.example.com

And last here are the calico-node logs for this particular pod, but it seems to only log the successful assignment of the final IP address:

k logs -n kube-system calico-node-8l5pp |grep nginx-ingress-controller-6664bc7855-8hxwv
2020-08-26 13:22:57.551 [INFO][22004] calc_graph.go 392: Local endpoint updated id=WorkloadEndpoint(node=k8s-worker-01.example.com, orchestrator=k8s, workload=eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv, name=eth0)
2020-08-26 13:22:57.555 [INFO][22004] int_dataplane.go 834: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv" endpoint_id:"eth0" > endpoint:<state:"active" name:"calicfa03c2c5b1" profile_ids:"kns.eai-selfservice-dev" profile_ids:"ksa.eai-selfservice-dev.eai-selfservice-dev-nginx-ingress-serviceaccount" ipv4_nets:"100.80.1.67/32" tiers:<name:"default" ingress_policies:"eai-selfservice-dev/knp.default.allow-ingress-access" ingress_policies:"eai-selfservice-dev/knp.default.allow-mssql" ingress_policies:"eai-selfservice-dev/knp.default.allow-own-namepace" ingress_policies:"eai-selfservice-dev/knp.default.allow-prometheus" ingress_policies:"eai-selfservice-dev/knp.default.allow-prometheus-springboot" ingress_policies:"eai-selfservice-dev/knp.default.default-deny-do-not-delete" egress_policies:"eai-selfservice-dev/knp.default.allow-active-directory" egress_policies:"eai-selfservice-dev/knp.default.allow-dns-access" egress_policies:"eai-selfservice-dev/knp.default.allow-mssql" egress_policies:"eai-selfservice-dev/knp.default.allow-nginx-apiserver-access" egress_policies:"eai-selfservice-dev/knp.default.allow-own-namepace" egress_policies:"eai-selfservice-dev/knp.default.allow-pki-cert" egress_policies:"eai-selfservice-dev/knp.default.allow-smtp" egress_policies:"eai-selfservice-dev/knp.default.allow-tibco-ems" egress_policies:"eai-selfservice-dev/knp.default.allow-tls" egress_policies:"eai-selfservice-dev/knp.default.default-deny-do-not-delete" > >
2020-08-26 13:22:57.555 [INFO][22004] endpoint_mgr.go 545: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv", EndpointId:"eth0"}
2020-08-26 13:22:57.556 [INFO][22004] endpoint_mgr.go 572: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv", EndpointId:"eth0"}
2020-08-26 13:22:57.556 [INFO][22004] endpoint_mgr.go 438: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv", EndpointId:"eth0"}
2020-08-26 13:22:57.557 [INFO][22004] status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv", EndpointId:"eth0"}
2020-08-26 13:22:57.635 [INFO][22004] status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv", EndpointId:"eth0"} ipVersion=0x4 status="up"
2020-08-26 13:22:57.635 [INFO][22004] status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"eai-selfservice-dev/nginx-ingress-controller-6664bc7855-8hxwv", EndpointId:"eth0"} status="up"

Context

We did redeploy calico before, because we needed to change the pod IP CIDR. For this reason we completely deleted and reapplied calico with all CRs and CRDs. All the pods on this node has been started after the pod IP CIDR change.

If I missed to provide some important information please let me know.

Your Environment

  • Calico version v3.11.3
  • Orchestrator version (e.g. kubernetes, mesos, rkt): k8s 1.17.6
  • Operating System and version: Flatcar Container Linux 2345.3.1

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 23 (9 by maintainers)

Most upvoted comments

HI @caseydavenport, I upgraded the first cluster to 3.17.0, but unfortunately I still see the same errors in calico-kube-controller. I increased the loglevel to debug:

2020-12-08 17:19:40.779 [INFO][1] ipam.go 45: Synchronizing IPAM data
2020-12-08 17:19:40.784 [DEBUG][1] k8s.go 579: Performing 'List' for {IPVersion:0} model.BlockListOptions
2020-12-08 17:19:40.784 [DEBUG][1] customresource.go 242: List Custom K8s Resource ListInterface=IPAMBlock Resource="IPAMBlocks"
2020-12-08 17:19:41.005 [DEBUG][1] ipam.go 107: Calico nodes found in IPAM: map[k8s-master-01.example.com:[{0xc0007ea970 map[namespace:kube-system node:k8s-master-01.example.com pod:kubernetes-dashboard-84dfb859c8-7kg8v]} {0xc0007ea9f0 map[namespace:kube-system node:k8s-master-01.example.com pod:default-http-backend-74c858b7df-6bbpt]} {0xc0007eaa70 map[namespace:kube-system node:k8s-master-01.example.com pod:calico-kube-controllers-648f4868b8-9c6lv]} {0xc000849250 map[namespace:kube-system node:k8s-master-01.example.com pod:default-http-backend-74c858b7df-8cq4k]} {0xc000849bf0 map[namespace:kube-system node:k8s-master-01.example.com pod:kubernetes-dashboard-84dfb859c8-dffm5]} {0xc0008627a0 map[namespace:kube-system node:k8s-master-01.example.com pod:default-http-backend-74c858b7df-4d4z8]} {0xc0008625c0 map[namespace:kube-system node:k8s-master-01.example.com pod:expire-check-master-9mj2v]} {0xc000862640 map[node:k8s-master-01.example.com type:ipipTunnelAddress]} {0xc0008626a0 map[namespace:monitoring node:k8s-master-01.example.com pod:ephemeral-demo-645f5f7f95-4hzwz]} {0xc000862720 map[namespace:kube-system node:k8s-master-01.example.com pod:expire-check-master-6djzs]} {0xc000861840 map[node:k8s-master-01.example.com type:ipipTunnelAddress]}] k8s-master-02.example.com:[{0xc0005cad30 map[namespace:kube-system node:k8s-master-02.example.com pod:expire-check-master-7knhw]} {0xc0005cadb0 map[namespace:kube-system node:k8s-master-02.example.com pod:kubernetes-dashboard-84dfb859c8-9ch6q]} {0xc0005caea0 map[namespace:kube-system node:k8s-master-02.example.com pod:calico-kube-controllers-648f4868b8-xvhpv]} {0xc0005caf20 map[namespace:kube-system node:k8s-master-02.example.com pod:default-http-backend-74c858b7df-b2zl5]} {0xc000849cf0 map[namespace:kube-system node:k8s-master-02.example.com pod:expire-check-master-dd5gr]} {0xc000849c70 map[namespace:kube-system node:k8s-master-02.example.com pod:expire-check-master-m5fxf]} {0xc000861660 map[node:k8s-master-02.example.com type:ipipTunnelAddress]} {0xc0008616b0 map[namespace:kube-system node:k8s-master-02.example.com pod:expire-check-master-2dkl8]} {0xc000861720 map[namespace:kube-system node:k8s-master-02.example.com pod:calico-kube-controllers-646cf4cccf-w6wjf]}] k8s-worker-00.example.com:[{0xc000849330 map[namespace:velero node:k8s-worker-00.example.com pod:restic-gbbnc]} {0xc0008493b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-discover-wsgck]} {0xc000849430 map[namespace:kube-system node:k8s-worker-00.example.com pod:expire-check-worker-xhvqh]} {0xc000849530 map[namespace:kube-system node:k8s-worker-00.example.com pod:expire-check-worker-vv5zl]} {0xc0008494b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-ceph-sbx-primary-backup-w79hr]} {0xc0008495b0 map[namespace:kube-system node:k8s-worker-00.example.com pod:coredns-5889fdf6fc-2t7pw]} {0xc000849630 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-ceph-sbx-primary-backup-m4lrg]} {0xc0008496b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-discover-cggmb]} {0xc000849730 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-ceph-mon-a-667b6dc754-r22t9]} {0xc000862d10 map[namespace:kube-system node:k8s-worker-00.example.com pod:expire-check-worker-jwjhw]} {0xc000862d90 map[namespace:kube-system node:k8s-worker-00.example.com pod:expire-check-worker-678v7]} {0xc000862e00 map[namespace:velero node:k8s-worker-00.example.com pod:restic-6fvhx]} {0xc000862e80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-discover-hk6df]} {0xc000862f00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-ceph-mon-a-85748f9d94-g8jw8]} {0xc000862f80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-ceph-sbx-primary-backup-k4bfh]} {0xc000863000 map[node:k8s-worker-00.example.com type:ipipTunnelAddress]} {0xc000863060 map[namespace:kube-system node:k8s-worker-00.example.com pod:expire-check-worker-ds5b7]} {0xc0008630e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-discover-ng8k8]} {0xc000863160 map[namespace:velero node:k8s-worker-00.example.com pod:restic-6p6l6]} {0xc0008631e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-ceph-sbx-primary-backup-w2pjf]} {0xc000863260 map[namespace:rook-ceph-sbx-primary node:k8s-worker-00.example.com pod:rook-ceph-mon-a-85748f9d94-p7zxm]}] k8s-worker-20.example.com:[{0xc0007eaf90 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-discover-vntp8]} {0xc0007eb1c0 map[namespace:velero node:k8s-worker-20.example.com pod:velero-sbx-776b9fd4f4-spqqz]} {0xc0007eb040 map[namespace:lxos-sbx node:k8s-worker-20.example.com pod:default-http-backend-6fd4978fb-j9r65]} {0xc0007eb0d0 map[namespace:kube-system node:k8s-worker-20.example.com pod:external-dns-top-zone-6bb48b5b77-wgncc]} {0xc0007eb140 map[namespace:kube-system node:k8s-worker-20.example.com pod:sealed-secrets-controller-65cf6f6fc7-m2hl7]} {0xc0007eab10 map[namespace:kube-system node:k8s-worker-20.example.com pod:external-dns-extern-top-zone-7cbccfcb8d-9976r]} {0xc0007eab90 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-ceph-tools-74c899d786-c5pcz]} {0xc0007eac20 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-prometheus-kube-state-metrics-6cc667cd78-7s5c9]} {0xc0007eace0 map[namespace:kube-system node:k8s-worker-20.example.com pod:metrics-server-64d955cbc6-nnj9t]} {0xc0007ead80 map[namespace:monitoring node:k8s-worker-20.example.com pod:deploy-547f6c9d8b-zd5vq]} {0xc0007eae00 map[namespace:catalog node:k8s-worker-20.example.com pod:catalog-catalog-controller-manager-6866cd78b6-b9xq8]} {0xc0007eae80 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-redis-master-0]} {0xc0007eaf10 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-redis-ha-server-1]} {0xc0007eb240 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-discover-cdfkk]} {0xc000833e20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-ceph-sbx-primary-backup-x5gbj]} {0xc000833e90 map[namespace:kube-system node:k8s-worker-20.example.com pod:expire-check-worker-hwmpp]} {0xc000833f00 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-redis-ha-server-1]} {0xc000848610 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-ceph-sbx-primary-backup-m8g5p]} {0xc000833fe0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-ceph-sbx-primary-backup-w7ggf]} {0xc000848050 map[namespace:kube-system node:k8s-worker-20.example.com pod:sealed-secrets-controller-65cf6f6fc7-t7bfc]} {0xc0008480c0 map[namespace:lxos-sbx node:k8s-worker-20.example.com pod:nginx-ingress-controller-855f55f857-5dn25]} {0xc000848130 map[namespace:kube-system node:k8s-worker-20.example.com pod:metrics-server-64d955cbc6-4679d]} {0xc000833f70 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-redis-ha-server-1]} {0xc0008481a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-ceph-operator-86dccf9fcc-5xm75]} {0xc000848220 map[namespace:metallb-system node:k8s-worker-20.example.com pod:metallb-controller-75cbdf4c77-zmqxh]} {0xc0008482a0 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-grafana-5b9ccf9bb-z9qws]} {0xc000848320 map[namespace:catalog node:k8s-worker-20.example.com pod:catalog-catalog-controller-manager-6866cd78b6-6dzfm]} {0xc000848390 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-redis-slave-0]} {0xc000848410 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-ceph-tools-74c899d786-95htx]} {0xc000848490 map[namespace:velero node:k8s-worker-20.example.com pod:velero-sbx-776b9fd4f4-lkjmm]} {0xc000848510 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-redis-master-0]} {0xc000848590 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-redis-ha-server-0]} {0xc000862990 map[namespace:kube-system node:k8s-worker-20.example.com pod:expire-check-worker-zkhsp]} {0xc000862a10 map[namespace:velero node:k8s-worker-20.example.com pod:restic-bsrvq]} {0xc000862a90 map[namespace:cert-manager node:k8s-worker-20.example.com pod:cert-manager-5c47f46f57-2hvkf]} {0xc000862b10 map[namespace:monitoring node:k8s-worker-20.example.com pod:nginx-ingress-controller-74c88944d7-rn6kl]} {0xc000862ba0 map[namespace:monitoring node:k8s-worker-20.example.com pod:mon-prometheus-server-1]} {0xc000862c20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-discover-xcqg2]} {0xc000862910 map[namespace:rook-ceph-sbx-primary node:k8s-worker-20.example.com pod:rook-ceph-sbx-primary-backup-g9k2t]} {0xc000862890 map[namespace:kube-system node:k8s-worker-20.example.com pod:expire-check-worker-ccdcr]}] k8s-worker-21.example.com:[{0xc0005cac40 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-redis-ha-server-1]} {0xc000832310 map[namespace:velero node:k8s-worker-21.example.com pod:restic-lkczk]} {0xc000832380 map[namespace:kube-system node:k8s-worker-21.example.com pod:expire-check-worker-zqqbb]} {0xc000832400 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-sbx-primary-backup-qfpjn]} {0xc000832480 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-mgr-a-7dfbc5ffd5-kzfff]} {0xc000832500 map[namespace:kube-system node:k8s-worker-21.example.com pod:expire-check-worker-hk2sk]} {0xc000832580 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-sbx-primary-backup-4x8tf]} {0xc000832980 map[namespace:kube-system node:k8s-worker-21.example.com pod:metrics-server-64d955cbc6-k7fm6]} {0xc000832a00 map[namespace:lxos-sbx node:k8s-worker-21.example.com pod:nginx-ingress-controller-855f55f857-zvsp2]} {0xc000832600 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-redis-ha-server-1]} {0xc000832680 map[namespace:monitoring node:k8s-worker-21.example.com pod:default-http-backend-6fd4978fb-2pcrk]} {0xc000832700 map[namespace:kube-system node:k8s-worker-21.example.com pod:external-dns-5d6686c558-8jqtr]} {0xc000832780 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-prometheus-kube-state-metrics-6cc667cd78-vv9f8]} {0xc000832800 map[namespace:lxos-sbx node:k8s-worker-21.example.com pod:default-http-backend-6fd4978fb-dq9xv]} {0xc000832880 map[namespace:catalog node:k8s-worker-21.example.com pod:catalog-catalog-apiserver-7764cdb499-z6njh]} {0xc000832900 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-prometheus-server-1]} {0xc000832a80 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-redis-ha-server-0]} {0xc000832b00 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-redis-slave-0]} {0xc000832b80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-mgr-a-7dfbc5ffd5-cvdgt]} {0xc000832c00 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-prometheus-server-1]} {0xc000832c80 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-prometheus-server-1]} {0xc000848690 map[namespace:aws-service-broker node:k8s-worker-21.example.com pod:aws-servicebroker-55dc94bb7-xx4gw]} {0xc000848710 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-redis-slave-0]} {0xc000848790 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-mgr-a-7dfbc5ffd5-pnt78]} {0xc000848810 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-mgr-a-7dfbc5ffd5-jwjdc]} {0xc000848890 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-mgr-a-7dfbc5ffd5-k5br8]} {0xc000848910 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-sbx-primary-backup-j4f9b]} {0xc000848990 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-discover-p4fwp]} {0xc000848a10 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-tools-6b94f5f84b-f8rll]} {0xc000848a90 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-mgr-a-dd5b9bb54-t2lkz]} {0xc000849e50 map[namespace:kube-system node:k8s-worker-21.example.com pod:k8s-toolbox-6bbdb4cb86-q9n5z]} {0xc000849ed0 map[namespace:kube-system node:k8s-worker-21.example.com pod:nginx-ingress-controller-7d7f565fb-6djw9]} {0xc000849f50 map[namespace:kube-system node:k8s-worker-21.example.com pod:prometheus-operator-58f5f5789d-98bk7]} {0xc000849fd0 map[namespace:kube-system node:k8s-worker-21.example.com pod:sealed-secrets-75756cb8b8-ztsv4]} {0xc000849dd0 map[namespace:kube-system node:k8s-worker-21.example.com pod:external-dns-extern-top-zone-7cbccfcb8d-shrp6]} {0xc000860dd0 map[namespace:kube-system node:k8s-worker-21.example.com pod:node-labeler-786896fb46-zgrd5]} {0xc000860e40 map[namespace:velero node:k8s-worker-21.example.com pod:restic-5bch9]} {0xc000860eb0 map[namespace:kube-system node:k8s-worker-21.example.com pod:expire-check-worker-qpwdb]} {0xc000861000 map[namespace:kube-system node:k8s-worker-21.example.com pod:expire-check-worker-k62xm]} {0xc000861070 map[namespace:velero node:k8s-worker-21.example.com pod:restic-n9hl2]} {0xc000861150 map[node:k8s-worker-21.example.com type:ipipTunnelAddress]} {0xc0008610e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-discover-5smnv]} {0xc000860f20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-mgr-a-7c5c47cff8-dw6qn]} {0xc000860f90 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-sbx-primary-backup-wdkls]} {0xc000861360 map[namespace:velero node:k8s-worker-21.example.com pod:restic-zgvqg]} {0xc000861590 map[namespace:monitoring node:k8s-worker-21.example.com pod:mon-prometheus-server-0]} {0xc0008613d0 map[namespace:monitoring node:k8s-worker-21.example.com pod:nginx-ingress-controller-74c88944d7-d266k]} {0xc000861520 map[namespace:opsview node:k8s-worker-21.example.com pod:kube-ops-view-6d99f87db9-2p92v]} {0xc000861440 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-tools-57644c8849-9gwbs]} {0xc0008614b0 map[namespace:monitoring node:k8s-worker-21.example.com pod:default-http-backend-6fd4978fb-fv8dd]} {0xc000861280 map[namespace:kube-system node:k8s-worker-21.example.com pod:expire-check-worker-77zjx]} {0xc000861210 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-discover-rznxz]} {0xc0008611a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-21.example.com pod:rook-ceph-sbx-primary-backup-tthtq]} {0xc0008612f0 map[namespace:kube-system node:k8s-worker-21.example.com pod:coredns-6fbcb44b7f-8vdr4]}] k8s-worker-22.example.com:[{0xc0005cb290 map[namespace:kube-system node:k8s-worker-22.example.com pod:expire-check-worker-cq5dx]} {0xc0005cb310 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-discover-ghmrx]} {0xc0005cb390 map[namespace:kube-system node:k8s-worker-22.example.com pod:node-labeler-6f9dcc7c49-zscxk]} {0xc0005cb410 map[namespace:kube-system node:k8s-worker-22.example.com pod:calico-kube-controllers-648f4868b8-k74v2]} {0xc0005cb490 map[namespace:aws-service-broker node:k8s-worker-22.example.com pod:aws-servicebroker-55dc94bb7-4t88n]} {0xc0005cb510 map[namespace:metallb-system node:k8s-worker-22.example.com pod:metallb-controller-75cbdf4c77-dxcfk]} {0xc0005cafa0 map[namespace:catalog node:k8s-worker-22.example.com pod:catalog-catalog-apiserver-7764cdb499-drdnr]} {0xc0005cb020 map[namespace:kube-system node:k8s-worker-22.example.com pod:kube-ops-view-58ddfb8d4f-7c8v7]} {0xc0005cb0a0 map[namespace:velero node:k8s-worker-22.example.com pod:velero-sbx-776b9fd4f4-pdtnq]} {0xc0005cb120 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-prometheus-pushgateway-779464cc74-8jq8b]} {0xc0005cb1a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:testmount-698dc786cc-7wxb8]} {0xc0005cb220 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-redis-slave-0]} {0xc0005cb590 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-redis-slave-0]} {0xc0005cb620 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-redis-master-0]} {0xc0005cb6a0 map[namespace:metallb-system node:k8s-worker-22.example.com pod:metallb-controller-7cc79d4b4b-gpmrp]} {0xc0005cb720 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-ceph-mgr-a-dd5b9bb54-vqnt2]} {0xc000848b70 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-redis-master-0]} {0xc000848f60 map[namespace:kube-system node:k8s-worker-22.example.com pod:expire-check-worker-7p44x]} {0xc000848bf0 map[namespace:kube-system node:k8s-worker-22.example.com pod:expire-check-worker-x2k86]} {0xc000848c70 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-ceph-sbx-primary-backup-gdxhs]} {0xc000848e60 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-prometheus-server-1]} {0xc000848ee0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-ceph-sbx-primary-backup-6ss7v]} {0xc000848cf0 map[namespace:aws-service-broker node:k8s-worker-22.example.com pod:aws-servicebroker-55dc94bb7-6pdrx]} {0xc000849160 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-prometheus-pushgateway-779464cc74-r7f5c]} {0xc000848fe0 map[namespace:kube-system node:k8s-worker-22.example.com pod:external-dns-extern-top-zone-7cbccfcb8d-p8p4q]} {0xc000849060 map[namespace:aws-service-broker node:k8s-worker-22.example.com pod:aws-servicebroker-55dc94bb7-m2nnf]} {0xc0008490e0 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-redis-master-0]} {0xc000848d70 map[namespace:velero node:k8s-worker-22.example.com pod:restic-szplm]} {0xc000848df0 map[namespace:kube-system node:k8s-worker-22.example.com pod:expire-check-worker-flw5p]} {0xc0008605c0 map[namespace:kube-system node:k8s-worker-22.example.com pod:expire-check-worker-dssb8]} {0xc000860260 map[namespace:kube-system node:k8s-worker-22.example.com pod:expire-check-worker-ftlhd]} {0xc000860a10 map[node:k8s-worker-22.example.com type:ipipTunnelAddress]} {0xc000860850 map[namespace:velero node:k8s-worker-22.example.com pod:restic-mkbff]} {0xc0008608c0 map[namespace:cert-manager node:k8s-worker-22.example.com pod:cert-manager-webhook-547567b88f-dvzz5]} {0xc000860930 map[namespace:kube-system node:k8s-worker-22.example.com pod:coredns-6fbcb44b7f-vlffn]} {0xc0008609a0 map[namespace:kube-system node:k8s-worker-22.example.com pod:kubernetes-dashboard-c584d7cc-brnmb]} {0xc0008602d0 map[namespace:kube-system node:k8s-worker-22.example.com pod:metrics-server-64d955cbc6-s5jdr]} {0xc000860340 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-discover-klrbt]} {0xc0008603c0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-ceph-tools-57644c8849-kjgwd]} {0xc000860bb0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-discover-4h7tx]} {0xc000860440 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:testmount-7757b9dc57-7x8v2]} {0xc0008604c0 map[namespace:monitoring node:k8s-worker-22.example.com pod:mon-grafana-5b9ccf9bb-zcxmh]} {0xc000860540 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-ceph-sbx-primary-backup-46tql]} {0xc000860b40 map[namespace:velero node:k8s-worker-22.example.com pod:restic-nmqqd]} {0xc000860c20 map[namespace:prometheus node:k8s-worker-22.example.com pod:master-kube-prometheus-sta-operator-7945d4fbb-s5ng2]} {0xc000860d00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-ceph-mgr-a-7c5c47cff8-bddcv]} {0xc000860c90 map[namespace:catalog node:k8s-worker-22.example.com pod:catalog-catalog-webhook-5d575f58cf-prqtc]} {0xc000860640 map[namespace:catalog node:k8s-worker-22.example.com pod:catalog-catalog-webhook-c9c7f9c6-l4r65]} {0xc0008606d0 map[namespace:catalog node:k8s-worker-22.example.com pod:catalog-catalog-controller-manager-75d98fbc4d-zd69p]} {0xc000860750 map[namespace:aws-service-broker node:k8s-worker-22.example.com pod:aws-servicebroker-b754458b7-lwt2h]} {0xc0008607d0 map[namespace:catalog node:k8s-worker-22.example.com pod:catalog-catalog-apiserver-946d9cf64-hmv2k]} {0xc000860a60 map[namespace:rook-ceph-sbx-primary node:k8s-worker-22.example.com pod:rook-ceph-sbx-primary-backup-99gv9]} {0xc000860ad0 map[namespace:kube-system node:k8s-worker-22.example.com pod:expire-check-worker-69qxn]}] k8s-worker-23.example.com:[{0xc0005cb810 map[namespace:velero node:k8s-worker-23.example.com pod:restic-77hrv]} {0xc0005cb890 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-discover-xzkvg]} {0xc0005cb910 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-slave-0]} {0xc0005cb990 map[namespace:kube-system node:k8s-worker-23.example.com pod:expire-check-worker-fhmcc]} {0xc0005cba10 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-ceph-sbx-primary-backup-l9grt]} {0xc0005cba90 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-prometheus-server-1]} {0xc0005cbca0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-discover-v8458]} {0xc0005cbd20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-ceph-sbx-primary-backup-75vg7]} {0xc0005cbb10 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-slave-1]} {0xc0005cbb90 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:testmount-698dc786cc-w8b8j]} {0xc0005cbc10 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-ceph-mgr-a-7dfbc5ffd5-xf5kw]} {0xc0005cbda0 map[namespace:kube-system node:k8s-worker-23.example.com pod:external-dns-5d6686c558-xgdqs]} {0xc0005cbe20 map[namespace:kube-system node:k8s-worker-23.example.com pod:sealed-secrets-controller-65cf6f6fc7-zs8sl]} {0xc0005cbea0 map[namespace:catalog node:k8s-worker-23.example.com pod:catalog-catalog-apiserver-7764cdb499-sfw9x]} {0xc0005cbf20 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-ha-server-1]} {0xc0005cbfa0 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-grafana-5b9ccf9bb-8f24c]} {0xc000832020 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-master-0]} {0xc0008320a0 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-slave-0]} {0xc000832120 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-ha-server-1]} {0xc0008321a0 map[namespace:velero node:k8s-worker-23.example.com pod:velero-sbx-776b9fd4f4-dr57t]} {0xc000832220 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-master-0]} {0xc0008623d0 map[namespace:velero node:k8s-worker-23.example.com pod:restic-75fqz]} {0xc000862450 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-discover-6mclj]} {0xc0008624d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-ceph-sbx-primary-backup-k98bj]} {0xc000862050 map[namespace:elastic-sbx node:k8s-worker-23.example.com pod:nginx-ingress-controller-76bff6c8b8-v5lnp]} {0xc0008620d0 map[namespace:elastic-sbx node:k8s-worker-23.example.com pod:nginx-ingress-controller-76bff6c8b8-crb7s]} {0xc000862150 map[namespace:monitoring node:k8s-worker-23.example.com pod:deploy-547f6c9d8b-96rms]} {0xc0008621d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-ceph-operator-6c656d4995-z6f48]} {0xc000862250 map[namespace:kube-system node:k8s-worker-23.example.com pod:nginx-ingress-controller-6f8f6bd84b-wvqhc]} {0xc0008622d0 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-redis-master-0]} {0xc000862350 map[namespace:kube-system node:k8s-worker-23.example.com pod:kube-ops-view-7cd88b99b7-x42z8]} {0xc000861c30 map[namespace:kube-system node:k8s-worker-23.example.com pod:expire-check-worker-7xkv8]} {0xc000861ca0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-discover-flpvf]} {0xc0008619b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-ceph-sbx-primary-backup-vxfpl]} {0xc000861940 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-discover-jzdtx]} {0xc000861d10 map[namespace:aws-service-broker node:k8s-worker-23.example.com pod:aws-service-broker-aws-servicebroker-5b8765c8f7-98m7w]} {0xc000861d80 map[namespace:monitoring node:k8s-worker-23.example.com pod:mon-prometheus-server-1]} {0xc000861a90 map[namespace:velero node:k8s-worker-23.example.com pod:restic-6wcfs]} {0xc000861b00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-23.example.com pod:rook-ceph-sbx-primary-backup-gmrvr]} {0xc000861b70 map[namespace:velero node:k8s-worker-23.example.com pod:restic-2g482]} {0xc000861be0 map[node:k8s-worker-23.example.com type:ipipTunnelAddress]} {0xc000861a20 map[namespace:kube-system node:k8s-worker-23.example.com pod:expire-check-worker-gkjmm]}] k8s-worker-50.example.com:[{0xc000832d70 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-discover-j87nh]} {0xc000832df0 map[namespace:velero node:k8s-worker-50.example.com pod:restic-4kthq]} {0xc000832e70 map[namespace:kube-system node:k8s-worker-50.example.com pod:expire-check-worker-8fmnw]} {0xc000832ef0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-sbx-primary-backup-2vr8w]} {0xc000832f70 map[namespace:elastic-sbx node:k8s-worker-50.example.com pod:default-http-backend-6fd4978fb-p9qlt]} {0xc000832ff0 map[namespace:elastic-sbx node:k8s-worker-50.example.com pod:nginx-ingress-controller-77bdb89dcc-xrzvn]} {0xc000833070 map[namespace:monitoring node:k8s-worker-50.example.com pod:nginx-ingress-controller-74c88944d7-gkzvv]} {0xc0008330f0 map[namespace:kube-system node:k8s-worker-50.example.com pod:prometheus-metrics-adapter-ddd7cfc7b-xxb87]} {0xc000833160 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-30-7966f694d7-89tl7]} {0xc0008331e0 map[namespace:kube-system node:k8s-worker-50.example.com pod:expire-check-worker-8nxx8]} {0xc000833260 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-31-697fbb75b8-jhrrs]} {0xc0008332e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-sbx-primary-backup-vbr8w]} {0xc000833360 map[namespace:velero node:k8s-worker-50.example.com pod:restic-rldcf]} {0xc0008333e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-mon-c-b798db7db-2jsj6]} {0xc000833460 map[namespace:kube-system node:k8s-worker-50.example.com pod:nginx-ingress-controller-85bdc89bb7-jvq6g]} {0xc0008334e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-32-5887bb46b-g47xx]} {0xc000833560 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-34-6968b94d79-wprjs]} {0xc0008335e0 map[namespace:kube-system node:k8s-worker-50.example.com pod:prometheus-operator-58f5f5789d-8p92h]} {0xc000833660 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-discover-hd7lg]} {0xc0008336e0 map[namespace:lxos-sbx node:k8s-worker-50.example.com pod:nginx-ingress-controller-855f55f857-29xk4]} {0xc000833760 map[namespace:kube-system node:k8s-worker-50.example.com pod:coredns-5889fdf6fc-wj8hd]} {0xc0008337e0 map[namespace:kube-system node:k8s-worker-50.example.com pod:coredns-5889fdf6fc-s9sfd]} {0xc000833850 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-sbx-primary-backup-pj77q]} {0xc0008338d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-discover-hpqwh]} {0xc000833940 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-mon-b-76d49f5f6f-j6jbx]} {0xc0008339c0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-21-78fcb8ff97-66l75]} {0xc000833a40 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-24-866646d5d8-csp46]} {0xc000833ac0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-25-66c757588f-pw44w]} {0xc000833b40 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-5-6984d5dc67-872js]} {0xc000833bc0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-9-576db48b79-jsrb5]} {0xc000833c40 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-13-7c645bfb6-bqvvj]} {0xc000833cc0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-1-6bc4f84785-wjlrp]} {0xc000833d40 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-19-6bc8cc584-gj4nv]} {0xc000863350 map[namespace:kube-system node:k8s-worker-50.example.com pod:expire-check-worker-lnszs]} {0xc0008633d0 map[namespace:velero node:k8s-worker-50.example.com pod:restic-j6znd]} {0xc000863450 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-discover-q5vkh]} {0xc000863fd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-tools-57644c8849-sgl44]} {0xc000863dd0 map[namespace:kube-system node:k8s-worker-50.example.com pod:external-dns-extern-78cdb9fbc6-jnbxj]} {0xc000863e50 map[namespace:kube-system node:k8s-worker-50.example.com pod:prometheus-operator-58f5f5789d-r5q25]} {0xc000863ed0 map[namespace:monitoring node:k8s-worker-50.example.com pod:mon-grafana-5b9ccf9bb-fhr6j]} {0xc0008634d0 map[namespace:metallb-system node:k8s-worker-50.example.com pod:metallb-controller-7cc79d4b4b-4mqfd]} {0xc000863550 map[namespace:kube-system node:k8s-worker-50.example.com pod:external-dns-extern-eai-677c4d6f7b-6pktm]} {0xc0008635d0 map[namespace:lxos-sbx node:k8s-worker-50.example.com pod:nginx-ingress-controller-855f55f857-4gw97]} {0xc000863650 map[namespace:velero node:k8s-worker-50.example.com pod:restic-7gxxw]} {0xc0008636d0 map[namespace:kube-system node:k8s-worker-50.example.com pod:expire-check-worker-szkst]} {0xc000863f50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:testmount-7757b9dc57-g8sz4]} {0xc000863750 map[namespace:monitoring node:k8s-worker-50.example.com pod:mon-redis-ha-server-1]} {0xc0008637d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-operator-6c656d4995-j29wr]} {0xc000863850 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-discover-s8cfc]} {0xc0008638d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-mon-b-7b994ff4dc-nlxrs]} {0xc0007f4820 map[namespace:monitoring node:k8s-worker-50.example.com pod:mon-redis-ha-server-1]} {0xc000863950 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-1-5fb7d4db4c-lnvp5]} {0xc0008639d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-13-76c5d7df9b-hrxmb]} {0xc000863a50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-17-85b9dfddd4-6bx6d]} {0xc000863ad0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-24-84bc56d9d-86zls]} {0xc000863b50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-20-6bd674f8d6-7szfp]} {0xc000863bd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-28-55c77cc8f4-5vw75]} {0xc000863c50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-5-7655865f69-tf9qr]} {0xc000863cd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-9-7c8cd46f97-87czb]} {0xc0007f47a0 map[namespace:monitoring node:k8s-worker-50.example.com pod:mon-redis-slave-1]} {0xc000863d50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-sbx-primary-backup-z2vfx]} {0xc0007f4050 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-24-84bc56d9d-k2cmr]} {0xc0007f40d0 map[node:k8s-worker-50.example.com type:ipipTunnelAddress]} {0xc0007f4120 map[namespace:velero node:k8s-worker-50.example.com pod:restic-vhwdt]} {0xc0007f41a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-20-6bd674f8d6-6qdkr]} {0xc0007f4220 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-9-7c8cd46f97-g5xnp]} {0xc0007f42b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-mon-b-7b994ff4dc-hxwmb]} {0xc0007f4320 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-sbx-primary-backup-p9kwd]} {0xc0007f43a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-discover-2rxmm]} {0xc0007f4420 map[namespace:kube-system node:k8s-worker-50.example.com pod:expire-check-worker-hxxkl]} {0xc0007f44a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-17-85b9dfddd4-bx9wn]} {0xc0007f4520 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-5-7655865f69-jrzz8]} {0xc0007f45a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-1-5fb7d4db4c-jvcdg]} {0xc0007f4620 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-28-55c77cc8f4-sk2nk]} {0xc0007f46a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-osd-13-76c5d7df9b-n725x]} {0xc0007f4720 map[namespace:rook-ceph-sbx-primary node:k8s-worker-50.example.com pod:rook-ceph-operator-6c656d4995-z5fsj]}] k8s-worker-51.example.com:[{0xc0007eb330 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-discover-7pvns]} {0xc0007eb3b0 map[namespace:velero node:k8s-worker-51.example.com pod:restic-7hrwb]} {0xc0007eb430 map[namespace:kube-system node:k8s-worker-51.example.com pod:expire-check-worker-4pbfh]} {0xc0007eb4b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-sbx-primary-backup-95vrl]} {0xc0007eb530 map[namespace:elastic-sbx node:k8s-worker-51.example.com pod:nginx-ingress-controller-77bdb89dcc-tdxjx]} {0xc0007eba60 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-23-5746f7f978-m2jpn]} {0xc0007ebae0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-22-df9d56954-g2wnp]} {0xc0007ebb60 map[namespace:kube-system node:k8s-worker-51.example.com pod:nginx-ingress-controller-85bdc89bb7-rm76j]} {0xc0007eb5c0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-mon-b-6fcdb55fb-bf6ts]} {0xc0007eb640 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-27-67dc99b5d6-jvzb4]} {0xc0007eb6c0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-discover-bqfvt]} {0xc0007eb740 map[namespace:velero node:k8s-worker-51.example.com pod:restic-q8ptb]} {0xc0007eb7e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-26-6f7c868bf9-p95cw]} {0xc0007eb860 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-25-556b7469b7-znslp]} {0xc0007eb8e0 map[namespace:monitoring node:k8s-worker-51.example.com pod:nginx-ingress-controller-74c88944d7-wwss4]} {0xc0007eb960 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-sbx-primary-backup-mrtsh]} {0xc0007eb9e0 map[namespace:kube-system node:k8s-worker-51.example.com pod:expire-check-worker-2kvhk]} {0xc0007ebc50 map[namespace:kube-system node:k8s-worker-51.example.com pod:external-dns-5d6686c558-qnzxg]} {0xc0007ebd00 map[namespace:kube-system node:k8s-worker-51.example.com pod:external-dns-extern-78cdb9fbc6-ggzrv]} {0xc0007ebbe0 map[namespace:monitoring node:k8s-worker-51.example.com pod:default-http-backend-6fd4978fb-xzbpj]} {0xc0007ebd80 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-grafana-5b9ccf9bb-964qk]} {0xc0007ebe50 map[namespace:lxos-sbx node:k8s-worker-51.example.com pod:default-http-backend-6fd4978fb-hnngf]} {0xc0007ebef0 map[namespace:kube-system node:k8s-worker-51.example.com pod:kube-ops-view-7cd88b99b7-hp6gz]} {0xc0007ebfc0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-sbx-primary-backup-gjhvg]} {0xc000840100 map[namespace:kube-system node:k8s-worker-51.example.com pod:prometheus-metrics-adapter-c74fd5b66-9wl5x]} {0xc000813320 map[namespace:kube-system node:k8s-worker-51.example.com pod:expire-check-worker-66k6z]} {0xc0008124c0 map[namespace:velero node:k8s-worker-51.example.com pod:restic-g2dgv]} {0xc000812540 map[namespace:kube-system node:k8s-worker-51.example.com pod:expire-check-worker-rr62q]} {0xc0008125c0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-discover-245ds]} {0xc000840200 map[namespace:metallb-system node:k8s-worker-51.example.com pod:metallb-controller-7cc79d4b4b-6mnpb]} {0xc000812f20 map[namespace:monitoring node:k8s-worker-51.example.com pod:default-http-backend-6fd4978fb-mlnqn]} {0xc000812640 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-redis-ha-server-2]} {0xc000812fa0 map[namespace:kube-system node:k8s-worker-51.example.com pod:coredns-6fbcb44b7f-7prvh]} {0xc000813020 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-prometheus-alertmanager-59b857f46-fn5pl]} {0xc000813e80 map[namespace:cert-manager node:k8s-worker-51.example.com pod:certmanager-cert-manager-webhook-7959675d59-6grvs]} {0xc0008126c0 map[namespace:kube-system node:k8s-worker-51.example.com pod:node-labeler-786896fb46-hncvf]} {0xc000812740 map[namespace:elastic-sbx node:k8s-worker-51.example.com pod:default-http-backend-6fd4978fb-s5q6f]} {0xc0008127c0 map[namespace:kube-system node:k8s-worker-51.example.com pod:prometheus-metrics-adapter-ddd7cfc7b-2sgrb]} {0xc000812840 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-redis-master-0]} {0xc0008128c0 map[namespace:kube-system node:k8s-worker-51.example.com pod:kube-ops-view-7cd88b99b7-bngfb]} {0xc0008130a0 map[namespace:kube-system node:k8s-worker-51.example.com pod:metrics-server-64d955cbc6-hk2fz]} {0xc000813120 map[namespace:catalog node:k8s-worker-51.example.com pod:catalog-catalog-webhook-c9c7f9c6-6wtzc]} {0xc000812940 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-discover-v8gq9]} {0xc0008131a0 map[namespace:catalog node:k8s-worker-51.example.com pod:catalog-catalog-apiserver-946d9cf64-4dz9l]} {0xc0008129b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-mon-c-84d69df88-6t27d]} {0xc000813220 map[namespace:kube-system node:k8s-worker-51.example.com pod:kubernetes-dashboard-c584d7cc-wx69w]} {0xc000812a30 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-redis-slave-1]} {0xc000812aa0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-23-5555c75797-78v4v]} {0xc000812b20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-27-76ccd4b77c-wrgnn]} {0xc000812ba0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-4-57fbc4d444-t2ppg]} {0xc000812c20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-0-7944f87978-kqs6h]} {0xc000812ca0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-8-78cb4b95b6-ptbpg]} {0xc000812d20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-12-b759bdf5c-m62gr]} {0xc000812da0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-19-5864484d66-pcbtt]} {0xc000812e20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-16-74c487b9-79b2t]} {0xc0008132a0 map[namespace:aws-service-broker node:k8s-worker-51.example.com pod:aws-servicebroker-b754458b7-4gj77]} {0xc000812ea0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-sbx-primary-backup-7vt7w]} {0xc0008134a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-discover-4c6dl]} {0xc000813780 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-19-5864484d66-9kvph]} {0xc000840280 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-redis-ha-server-0]} {0xc000813800 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-12-b759bdf5c-gmrt6]} {0xc000813880 map[namespace:catalog node:k8s-worker-51.example.com pod:catalog-catalog-controller-manager-959946d4d-pfs8x]} {0xc000813f80 map[namespace:external-dns node:k8s-worker-51.example.com pod:default-internal-external-dns-6bff4bd58d-4tgjm]} {0xc000813d00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-sbx-primary-backup-8fxdw]} {0xc000813c80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-4-57fbc4d444-bsm7t]} {0xc000813900 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-27-76ccd4b77c-nd9lg]} {0xc000813980 map[namespace:kube-system node:k8s-worker-51.example.com pod:expire-check-worker-s2vgk]} {0xc000813a00 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-prometheus-pushgateway-779464cc74-9qhkh]} {0xc000813a80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-8-78cb4b95b6-shmgf]} {0xc000813b00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-16-74c487b9-2f6n8]} {0xc000813f00 map[namespace:external-dns node:k8s-worker-51.example.com pod:default-external-external-dns-675d7dc758-77wb7]} {0xc000813e00 map[namespace:cert-manager node:k8s-worker-51.example.com pod:certmanager-cert-manager-84484dbc7d-vl2qq]} {0xc0008402f0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-mon-c-84d69df88-nw4xg]} {0xc000813d80 map[namespace:monitoring node:k8s-worker-51.example.com pod:mon-redis-slave-0]} {0xc000813420 map[namespace:velero node:k8s-worker-51.example.com pod:restic-w8wdm]} {0xc000813c00 map[namespace:kube-system node:k8s-worker-51.example.com pod:node-labeler-557b7c6596-5j4w5]} {0xc000840000 map[namespace:external-dns node:k8s-worker-51.example.com pod:top-zone-external-external-dns-c6b866d8-rqmdq]} {0xc000840080 map[namespace:external-dns node:k8s-worker-51.example.com pod:top-zone-internal-external-dns-5c97c49568-lgfgl]} {0xc000813b80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-discover-bptzl]} {0xc000813520 map[node:k8s-worker-51.example.com type:ipipTunnelAddress]} {0xc000813580 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-0-7944f87978-d9ll4]} {0xc0008133a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-sbx-primary-backup-mswvp]} {0xc000813600 map[namespace:velero node:k8s-worker-51.example.com pod:restic-4z886]} {0xc000840180 map[namespace:kube-system node:k8s-worker-51.example.com pod:coredns-6fbcb44b7f-d2r6x]} {0xc000813680 map[namespace:cert-manager node:k8s-worker-51.example.com pod:certmanager-cert-manager-cainjector-5d6c58bf66-kngvf]} {0xc000813700 map[namespace:rook-ceph-sbx-primary node:k8s-worker-51.example.com pod:rook-ceph-osd-23-5555c75797-788xn]}] k8s-worker-52.example.com:[{0xc0005ca0b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-sbx-primary-backup-z9n97]} {0xc0005ca130 map[namespace:kube-system node:k8s-worker-52.example.com pod:expire-check-worker-z4ftg]} {0xc0005ca1b0 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-redis-ha-server-2]} {0xc0005cab50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-discover-f4s4x]} {0xc0005ca950 map[namespace:velero node:k8s-worker-52.example.com pod:restic-6jvt8]} {0xc0005ca9d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-sbx-primary-backup-ws588]} {0xc0005caa50 map[namespace:monitoring node:k8s-worker-52.example.com pod:default-http-backend-6fd4978fb-lmmk8]} {0xc0005caad0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-mon-e-74cd84fd5d-qdlw4]} {0xc0005ca230 map[namespace:lxos-sbx node:k8s-worker-52.example.com pod:default-http-backend-6fd4978fb-vp24w]} {0xc0005ca2b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-10-5c79dc8c8f-gnpsd]} {0xc0005ca330 map[namespace:kube-system node:k8s-worker-52.example.com pod:expire-check-worker-464nd]} {0xc0005ca3b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-operator-86dccf9fcc-g4ccd]} {0xc0005ca430 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-19-b4b8b888f-6q2sv]} {0xc0005ca4c0 map[namespace:metallb-system node:k8s-worker-52.example.com pod:metallb-controller-75cbdf4c77-qxm77]} {0xc0005ca560 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-7-5cd8cfccfc-95tcb]} {0xc0005ca5e0 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-prometheus-kube-state-metrics-6cc667cd78-62rlb]} {0xc0005ca660 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-1-54b7dc9695-q5q89]} {0xc0005ca6e0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-18-89cccdbf7-s7bvh]} {0xc0005ca760 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-15-9db58595d-kxw6k]} {0xc0005ca7e0 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-redis-ha-server-2]} {0xc0005ca850 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-grafana-5b9ccf9bb-xr8vq]} {0xc0005ca8d0 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-prometheus-server-0]} {0xc000860010 map[namespace:monitoring node:k8s-worker-52.example.com pod:deploy-547f6c9d8b-tqmh7]} {0xc0008403e0 map[namespace:kube-system node:k8s-worker-52.example.com pod:expire-check-worker-ct86r]} {0xc000840460 map[namespace:velero node:k8s-worker-52.example.com pod:restic-mr6md]} {0xc0008404e0 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-redis-slave-1]} {0xc000840f50 map[namespace:kube-system node:k8s-worker-52.example.com pod:expire-check-worker-rxqt8]} {0xc000840fd0 map[namespace:kube-system node:k8s-worker-52.example.com pod:external-dns-5d6686c558-2ztcm]} {0xc000841050 map[namespace:velero node:k8s-worker-52.example.com pod:velero-sbx-776b9fd4f4-bksmp]} {0xc0008410d0 map[namespace:lxos-sbx node:k8s-worker-52.example.com pod:default-http-backend-6fd4978fb-hm7tg]} {0xc000841150 map[namespace:kube-system node:k8s-worker-52.example.com pod:k8s-toolbox-6fd9d7f698-k9hsg]} {0xc000840560 map[namespace:kube-system node:k8s-worker-52.example.com pod:external-dns-eai-5695f9ddd9-pnwpf]} {0xc0008405e0 map[namespace:kube-system node:k8s-worker-52.example.com pod:external-dns-extern-top-zone-5ff88f6bc7-vrfpn]} {0xc0008411d0 map[namespace:kube-system node:k8s-worker-52.example.com pod:expire-check-worker-vnqfv]} {0xc000840660 map[namespace:kube-system node:k8s-worker-52.example.com pod:coredns-6fbcb44b7f-vzzjr]} {0xc0008406e0 map[namespace:elastic-sbx node:k8s-worker-52.example.com pod:nginx-ingress-controller-76bff6c8b8-7fl5g]} {0xc000840760 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-prometheus-kube-state-metrics-6cc667cd78-mmvd8]} {0xc0008407e0 map[namespace:kube-system node:k8s-worker-52.example.com pod:external-dns-top-zone-79bbbbd96b-whf5t]} {0xc000841250 map[namespace:velero node:k8s-worker-52.example.com pod:restic-cr6hj]} {0xc000840860 map[namespace:kube-system node:k8s-worker-52.example.com pod:sealed-secrets-controller-65cf6f6fc7-srnhl]} {0xc000860090 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-prometheus-kube-state-metrics-6cc667cd78-79ssg]} {0xc000841f20 map[namespace:kube-system node:k8s-worker-52.example.com pod:pdb-controller-687f679c49-r6vr2]} {0xc000841ea0 map[namespace:prometheus node:k8s-worker-52.example.com pod:alertmanager-master-kube-prometheus-sta-alertmanager-1]} {0xc0008408e0 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-redis-ha-server-2]} {0xc000840950 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-discover-w9b8f]} {0xc0008409d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-mon-d-54f5f6bff-hb5gd]} {0xc000840a50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-25-86dbc999f7-slzgg]} {0xc000840ad0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-29-77bfbdfd85-cvltz]} {0xc000840b50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-3-7d99c48f47-wzcpj]} {0xc000840bd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-31-677b4dbcc8-gd49f]} {0xc000840c50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-7-58565bd855-qljz2]} {0xc000840cd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-11-7c7b47c5dd-ld4mf]} {0xc000840d50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-15-866588fb-gvnlk]} {0xc000840dd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-21-68d64777fc-przlj]} {0xc000841e20 map[namespace:prometheus node:k8s-worker-52.example.com pod:prometheus-master-kube-prometheus-sta-prometheus-0]} {0xc000840e50 map[namespace:monitoring node:k8s-worker-52.example.com pod:mon-prometheus-server-577c57785c-8lpwk]} {0xc000840ed0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-sbx-primary-backup-4ljs2]} {0xc000841f90 map[namespace:kube-system node:k8s-worker-52.example.com pod:sealed-secrets-controller-7ccb6648c5-gxdhz]} {0xc0008412d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-sbx-primary-backup-b7c8w]} {0xc000841d20 map[namespace:elastic-sbx node:k8s-worker-52.example.com pod:nginx-ingress-controller-76bff6c8b8-wjph7]} {0xc000841c20 map[namespace:kube-system node:k8s-worker-52.example.com pod:kubernetes-dashboard-c584d7cc-8qdw6]} {0xc000841350 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-discover-fb9t7]} {0xc000841910 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-21-68d64777fc-gdkdq]} {0xc000841ca0 map[namespace:kube-system node:k8s-worker-52.example.com pod:nginx-ingress-controller-6f8f6bd84b-vdf8c]} {0xc000841da0 map[namespace:kube-system node:k8s-worker-52.example.com pod:coredns-6fbcb44b7f-4rr66]} {0xc000860110 map[namespace:monitoring node:k8s-worker-52.example.com pod:nginx-ingress-controller-74c88944d7-ckn65]} {0xc000841ba0 map[namespace:velero node:k8s-worker-52.example.com pod:backup-velero-84f95868ff-lbwhl]} {0xc000860180 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-mon-d-54f5f6bff-qqpqs]} {0xc000841990 map[namespace:elastic-sbx node:k8s-worker-52.example.com pod:default-http-backend-6fd4978fb-jghc9]} {0xc000841a10 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-discover-gt7pg]} {0xc000841aa0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-25-86dbc999f7-vcgmc]} {0xc000841b20 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-11-7c7b47c5dd-x8n89]} {0xc0008413d0 map[node:k8s-worker-52.example.com type:ipipTunnelAddress]} {0xc000841430 map[namespace:velero node:k8s-worker-52.example.com pod:restic-wffzw]} {0xc0008414b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-31-677b4dbcc8-xxp7t]} {0xc000841530 map[namespace:prometheus node:k8s-worker-52.example.com pod:master-grafana-665d79dc5b-27f9t]} {0xc0008415a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-29-77bfbdfd85-qhvtj]} {0xc000841620 map[namespace:kube-system node:k8s-worker-52.example.com pod:dashboard-metrics-scraper-6fb9d4c667-7z8ft]} {0xc0008416a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-3-7d99c48f47-6467p]} {0xc000841720 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-7-58565bd855-r5vb5]} {0xc0008417a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-osd-15-866588fb-xnnkd]} {0xc000841820 map[namespace:kube-system node:k8s-worker-52.example.com pod:expire-check-worker-6pd2p]} {0xc0008418a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-52.example.com pod:rook-ceph-sbx-primary-backup-p2sms]}] k8s-worker-53.example.com:[{0xc000405990 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-sbx-primary-backup-nc744]} {0xc000405a00 map[namespace:kube-system node:k8s-worker-53.example.com pod:expire-check-worker-dlnxp]} {0xc0007ea830 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-sbx-primary-backup-cszgx]} {0xc000405e60 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-3-c46bcd56b-4crxh]} {0xc000405ed0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-14-85d9d97f58-nxtsm]} {0xc000405f40 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-mon-a-7ffdfb99d9-dvz5j]} {0xc000405fb0 map[namespace:kube-system node:k8s-worker-53.example.com pod:expire-check-worker-zglmt]} {0xc000405a70 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-13-84597f6fff-vfs2g]} {0xc000405ae0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-12-756c685d88-f4kfl]} {0xc000405b50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-8-99d59c7b7-lzh66]} {0xc000405bc0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-sbx-primary-backup-v7k22]} {0xc000405c30 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-discover-5qm9d]} {0xc000405ca0 map[namespace:kube-system node:k8s-worker-53.example.com pod:coredns-d5999b86b-k49bj]} {0xc000405d10 map[namespace:monitoring node:k8s-worker-53.example.com pod:mon-prometheus-alertmanager-59b857f46-hp925]} {0xc000405d80 map[namespace:velero node:k8s-worker-53.example.com pod:restic-q5g4q]} {0xc000405df0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-16-7d554b969c-x2z7r]} {0xc0007ea1a0 map[namespace:monitoring node:k8s-worker-53.example.com pod:mon-redis-ha-server-0]} {0xc0007ea020 map[namespace:monitoring node:k8s-worker-53.example.com pod:mon-redis-slave-1]} {0xc0007ea0a0 map[namespace:kube-system node:k8s-worker-53.example.com pod:external-dns-extern-78cdb9fbc6-d6pvk]} {0xc0007ea120 map[namespace:lxos-sbx node:k8s-worker-53.example.com pod:default-http-backend-6fd4978fb-dvpfd]} {0xc0007ea230 map[namespace:monitoring node:k8s-worker-53.example.com pod:mon-prometheus-server-1]} {0xc0007ea2b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-sbx-primary-backup-2f56j]} {0xc0007ea330 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-discover-kl4mn]} {0xc0007ea3b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-mon-e-7557749798-rvwgt]} {0xc0007ea420 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-17-cc578fbb4-6jqhx]} {0xc0007ea4a0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-18-775d7ccf-v5pd7]} {0xc0007ea520 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-16-6664f9cc77-sknnz]} {0xc0007ea5b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-6-5dbdc78988-n2zg5]} {0xc0007ea630 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-3-5f4f7445bc-kbxlx]} {0xc0007ea6b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-20-6fdc55798b-g6szp]} {0xc0007ea730 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-10-6f899f975f-tg6bn]} {0xc0007ea7b0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-0-6f7c44796b-lcqnf]} {0xc0007f4c80 map[namespace:cert-manager node:k8s-worker-53.example.com pod:cert-manager-cainjector-6659d6844d-qdtgf]} {0xc0007f4910 map[namespace:kube-system node:k8s-worker-53.example.com pod:expire-check-worker-fqg7x]} {0xc0007f5100 map[namespace:velero node:k8s-worker-53.example.com pod:restic-gs6vb]} {0xc0007f5180 map[namespace:kube-system node:k8s-worker-53.example.com pod:pdb-controller-687f679c49-l5wlj]} {0xc0007f5580 map[namespace:kube-system node:k8s-worker-53.example.com pod:expire-check-worker-4jmgw]} {0xc0007f5200 map[namespace:kube-system node:k8s-worker-53.example.com pod:expire-check-worker-wf9tt]} {0xc0007f5280 map[namespace:lxos-sbx node:k8s-worker-53.example.com pod:nginx-ingress-controller-855f55f857-lnz28]} {0xc0007f5300 map[namespace:kube-system node:k8s-worker-53.example.com pod:nginx-ingress-controller-6f8f6bd84b-w864p]} {0xc0007f4990 map[namespace:elastic-sbx node:k8s-worker-53.example.com pod:nginx-ingress-controller-76bff6c8b8-2jc8p]} {0xc0007f4a00 map[namespace:kube-system node:k8s-worker-53.example.com pod:dashboard-metrics-scraper-6fb9d4c667-xmc7q]} {0xc0007f4a80 map[namespace:kube-system node:k8s-worker-53.example.com pod:calico-kube-controllers-648f4868b8-nxqsp]} {0xc0008123d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-mon-e-596c86cc96-z5q8s]} {0xc0007f5380 map[namespace:cert-manager node:k8s-worker-53.example.com pod:cert-manager-webhook-547567b88f-mbflj]} {0xc0007f5400 map[namespace:monitoring node:k8s-worker-53.example.com pod:deploy-547f6c9d8b-vcp7f]} {0xc0007f4b00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-discover-45mk8]} {0xc0007f4b80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-mon-e-596c86cc96-k7tfm]} {0xc0007f5480 map[namespace:catalog node:k8s-worker-53.example.com pod:catalog-catalog-controller-manager-75d98fbc4d-jt8l8]} {0xc0007f4c00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-18-5b7bb4f6f-5bp5f]} {0xc0007f4d00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-2-84f4ccd5bb-d9msq]} {0xc0007f4d80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-26-59b9bbbb54-9cg64]} {0xc0007f4e00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-22-85ccd74f66-wtfsq]} {0xc0007f4e80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-30-bcf498ccf-9659q]} {0xc0007f4f00 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-10-5bf8f68557-5z6qf]} {0xc0007f4f80 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-6-6d4c4dbc44-nswmj]} {0xc0007f5000 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-14-786d68dbc7-lgq4l]} {0xc0007f54f0 map[namespace:kube-system node:k8s-worker-53.example.com pod:coredns-6fbcb44b7f-rd8df]} {0xc0007f5080 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-sbx-primary-backup-6sjr9]} {0xc000812250 map[namespace:external-dns node:k8s-worker-53.example.com pod:eai-internal-external-dns-8699f9d89-q52d5]} {0xc0007f5600 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-sbx-primary-backup-svflf]} {0xc000812350 map[namespace:prometheus node:k8s-worker-53.example.com pod:prometheus-master-kube-prometheus-sta-prometheus-1]} {0xc0008121d0 map[namespace:elastic-sbx node:k8s-worker-53.example.com pod:nginx-ingress-controller-76bff6c8b8-8l6lg]} {0xc0007f5680 map[namespace:velero node:k8s-worker-53.example.com pod:restic-6zb29]} {0xc0007f5700 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-discover-7m5dr]} {0xc0007f5f50 map[namespace:prometheus node:k8s-worker-53.example.com pod:master-kube-state-metrics-69cdbd8958-j84pd]} {0xc0008122d0 map[namespace:monitoring node:k8s-worker-53.example.com pod:mon-redis-ha-server-2]} {0xc0007f5fd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-30-bcf498ccf-9gb8w]} {0xc000812150 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-2-84f4ccd5bb-mz8xm]} {0xc000812050 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-18-5b7bb4f6f-kxcf8]} {0xc0008120d0 map[namespace:monitoring node:k8s-worker-53.example.com pod:mon-redis-master-0]} {0xc0007f5780 map[node:k8s-worker-53.example.com type:ipipTunnelAddress]} {0xc0007f57d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-6-6d4c4dbc44-5j4ft]} {0xc0007f5850 map[namespace:kube-system node:k8s-worker-53.example.com pod:nginx-ingress-controller-6f8f6bd84b-4n2q9]} {0xc0007f58d0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-discover-2ffbl]} {0xc0007f5950 map[namespace:velero node:k8s-worker-53.example.com pod:restic-p5lkm]} {0xc0007f59d0 map[namespace:external-dns node:k8s-worker-53.example.com pod:eai-external-external-dns-5b44d4c4d5-k7rss]} {0xc0007f5a50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-26-59b9bbbb54-hpf9k]} {0xc0007f5ad0 map[namespace:k8s-updates node:k8s-worker-53.example.com pod:k8s-updates-7d6f4b56f7-nhdj5]} {0xc0007f5b50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-14-786d68dbc7-jk29b]} {0xc0007f5bd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-10-5bf8f68557-fpjrl]} {0xc0007f5c50 map[namespace:kube-system node:k8s-worker-53.example.com pod:metrics-server-64d955cbc6-qsmgg]} {0xc0007f5cd0 map[namespace:kube-system node:k8s-worker-53.example.com pod:expire-check-worker-8jw5s]} {0xc0007f5d50 map[namespace:prometheus node:k8s-worker-53.example.com pod:alertmanager-master-kube-prometheus-sta-alertmanager-0]} {0xc0007f5dd0 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-osd-22-85ccd74f66-v2vwk]} {0xc0007f5e50 map[namespace:rook-ceph-sbx-primary node:k8s-worker-53.example.com pod:rook-ceph-sbx-primary-backup-qtdg5]} {0xc0007f5ed0 map[namespace:monitoring node:k8s-worker-53.example.com pod:mon-prometheus-alertmanager-59b857f46-x7j6t]}]]
2020-12-08 17:19:41.011 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-22.example.com" k8sNode="k8s-worker-22.example.com"
2020-12-08 17:19:41.015 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-50.example.com" k8sNode="k8s-worker-50.example.com"
2020-12-08 17:19:41.020 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-00.example.com" k8sNode="k8s-worker-00.example.com"
2020-12-08 17:19:41.026 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-53.example.com" k8sNode="k8s-worker-53.example.com"
2020-12-08 17:19:41.030 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-master-01.example.com" k8sNode="k8s-master-01.example.com"
2020-12-08 17:19:41.030 [DEBUG][1] k8s.go 564: Performing 'Get' for Node(k8s-worker-20.example.com)
2020-12-08 17:19:41.030 [DEBUG][1] node.go 114: Received Get request on Node type
2020-12-08 17:19:41.034 [INFO][1] ipam.go 282: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(k8s-worker-20.example.com) with error: nodes "k8s-worker-20.example.com" not found
2020-12-08 17:19:41.034 [INFO][1] ipam.go 137: Checking node calicoNode="k8s-worker-20.example.com" k8sNode=""
2020-12-08 17:19:42.314 [INFO][1] ipam.go 178: Cleaning up IPAM resources for deleted node calicoNode="k8s-worker-20.example.com" k8sNode=""

[…]

2020-12-08 17:19:48.772 [INFO][1] ipam.go 1325: Releasing all IPs with handle 'k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6'
2020-12-08 17:19:48.772 [DEBUG][1] k8s.go 564: Performing 'Get' for IPAMHandleKey(id=k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6)
2020-12-08 17:19:48.772 [DEBUG][1] customresource.go 208: Get custom Kubernetes resource Key=IPAMHandle(k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6) Resource="IPAMHandles" Revision=""
2020-12-08 17:19:48.772 [DEBUG][1] customresource.go 219: Get custom Kubernetes resource by name Key=IPAMHandle(k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6) Name="k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6" Namespace="" Resource="IPAMHandles" Revision=""
2020-12-08 17:19:48.974 [DEBUG][1] customresource.go 228: Error getting resource Key=IPAMHandle(k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6) Name="k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6" Namespace="" Resource="IPAMHandles" Revision="" error=ipamhandles.crd.projectcalico.org "k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6" not found
2020-12-08 17:19:48.974 [DEBUG][1] ipam.go 207: IP already released calicoNode="k8s-worker-20.example.com" handle="k8s-pod-network.ea59dfc594ff620a37a7936051d94f287dfe30fc34f0680212fe9f9548b64ff6\r\neth0"

2020-12-08 17:19:48.974 [INFO][1] ipam.go 1325: Releasing all IPs with handle 'k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb'
2020-12-08 17:19:48.974 [DEBUG][1] k8s.go 564: Performing 'Get' for IPAMHandleKey(id=k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb)
2020-12-08 17:19:48.974 [DEBUG][1] customresource.go 208: Get custom Kubernetes resource Key=IPAMHandle(k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb) Resource="IPAMHandles" Revision=""
2020-12-08 17:19:48.974 [DEBUG][1] customresource.go 219: Get custom Kubernetes resource by name Key=IPAMHandle(k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb) Name="k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb" Namespace="" Resource="IPAMHandles" Revision=""
2020-12-08 17:19:49.172 [DEBUG][1] customresource.go 228: Error getting resource Key=IPAMHandle(k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb) Name="k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb" Namespace="" Resource="IPAMHandles" Revision="" error=ipamhandles.crd.projectcalico.org "k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb" not found
2020-12-08 17:19:49.173 [DEBUG][1] ipam.go 207: IP already released calicoNode="k8s-worker-20.example.com" handle="k8s-pod-network.5f934f9128f834ad21de044bce97524e6ff595c95798d26b7dd5c5aa323575eb\r\neth0"

2020-12-08 17:19:49.173 [INFO][1] ipam.go 218: Couldn't release all IPs for stale node, schedule retry calicoNode="k8s-worker-20.example.com"
2020-12-08 17:19:49.173 [WARNING][1] ipam.go 181: Error cleaning up node calicoNode="k8s-worker-20.example.com" error=Couldn't release all IPs k8sNode=""
2020-12-08 17:19:49.178 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-51.example.com" k8sNode="k8s-worker-51.example.com"
2020-12-08 17:19:49.183 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-master-02.example.com" k8sNode="k8s-master-02.example.com"
2020-12-08 17:19:49.187 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-52.example.com" k8sNode="k8s-worker-52.example.com"
2020-12-08 17:19:49.191 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-21.example.com" k8sNode="k8s-worker-21.example.com"
2020-12-08 17:19:49.195 [DEBUG][1] ipam.go 133: Node still exists, continue calicoNode="k8s-worker-23.example.com" k8sNode="k8s-worker-23.example.com"

I split the logs in multiple code blocks since the “Releasing all IPs”-blocks are examples and logged multiple times for different ipamhandles. The “Node still exists, continue”-lines change every time it appears for different hosts and different host counts.

+1
TheKangaroo on Dec 8, 2020

Merged a cherry-pick to v3.17.0: https://github.com/projectcalico/libcalico-go/pull/1346

+1
caseydavenport on Nov 19, 2020

@TheKangaroo I’m very sorry for the delay on this one. I’ve taken a look at the logs and this is my understanding of the issue:

2020-10-23 14:46:32.021 [INFO][1] ipam.go 1298: Releasing all IPs with handle 'k8s-pod-network.7ecd07e742bb804bdf8fb37d7820985a5a3b0fd3bb098fc0fb4bd315f2152a05
eth0'

Yeah, it looks very much like this is a result of “bad” data still in the IPAM allocations from the previous bug: https://github.com/projectcalico/cni-plugin/issues/821

There appears to be a mismatch in the Handle resource names - which seem to be properly formatted - and the attributes in the IPAM blocks - which seem to have the invalid format k8s-pod-network.<id>\r\neth0

The controller tries to release the allocation using the invalid handle, finds that no Handle object exists which matches the bad name, and hits this block: https://github.com/projectcalico/kube-controllers/blob/master/pkg/controllers/node/ipam.go#L208

When it comes around again to retry, the allocations are obviously still there since we didn’t release anything, and it tries the whole thing over again. What I am not 100% on is how this is causing duplicate IP addresses to be assigned in the cluster, but I can imagine that it is related.

I think that a change like this would fix the endless retries that you are seeing by allowing the code to properly compare the correctly formatted handle IDs with the improperly formatted ones: https://github.com/projectcalico/libcalico-go/pull/1345

@fasaxc WDYT?

+1
caseydavenport on Nov 17, 2020

Actually we already drained, deleted, redeployed and rejoined every node in the cluster after reinstalling calico, so this should be fine.

@fasaxc assisted me in troubleshooting this issue in calico slack (https://calicousers.slack.com/archives/C0BCA117T/p1598449811033700) but I could no longer reproduce this issue. Maybe it is best to close this issue and I’ll come back with more information when I see this happen again.

+1
TheKangaroo on Sep 1, 2020
Read more comments on GitHub
← calico: Calico readiness and liveliness probe fails
calico: calicoctl convert does not successfully convert selector field →