calico: Memory leak in calico-kube-controllers - v3.20

Expected Behavior

Reasonable memory usage with no apparent memory leaks.

Current Behavior

calico-kube-controllers v3.20 leaks memory (e.g: 9GB on a 16GB host) running on EKS 1.19. Eventually this leads to pod evictions and alerts firing on the hosting k8s node until the controller pod is recycled.

After dumping all the strings in the process’ memory space we see what appears to be the pod specs for all current and formerly running pods are in memory. Why this might impact us especially hard is that we utilize Airflow to run and schedule many frequent jobs in our cluster.

For example, from the calico-kube-controllers memory dump we obtained, we see 304052 copies of the following pod spec snippet:

304052 {"f:metadata":{"f:labels":{".":{},"f:airflow-worker":{},"f:airflow_version":{},"f:app.kubernetes.io/name":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_executor":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{".":{},"f:nodeAffinity":{".":{},"f:requiredDuringSchedulingIgnoredDuringExecution":{".":{},"f:nodeSelectorTerms":{}}}},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:command":{},"f:env":{".":{},"k:{\"name\":\"AIRFLOW_HOME\"}"....

Note that while Airflow managed pods appear most frequently, we see stale pod specs in the memory dump for non-airflow pods as well.

Steps to Reproduce (for bugs)

Unknown. Per above, possibly by running creating and deleting pods frequently, particularly doing so via Airflow .

Your Environment

• Kubernetes version (kubectl version): v1.19.13
• Operating System and version: EKS Amazon Linux - v1.19.13-eks-8df270
• Manifest:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    bgp: Enabled
    hostPorts: Enabled
    ipPools:
    - blockSize: 26
      cidr: 100.64.0.0/10
      encapsulation: IPIP
      natOutgoing: Enabled
      nodeSelector: all()
    linuxDataplane: Iptables
    mtu: 1440
    multiInterfaceMode: None
    nodeAddressAutodetectionV4:
      canReach: 8.8.8.8
  cni:
    ipam:
      type: Calico
    type: Calico
  componentResources:
  - componentName: Node
    resourceRequirements:
      requests:
        cpu: 250m
  flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
  kubernetesProvider: EKS
  nodeUpdateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  variant: Calico

Context

If helpful, here’s the a recent dump of the last 100 lines of the controller logs, with minor redactions:

2021-09-07 16:47:12.723 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.fc90c4e5f824e2cf08399b4cdc80d927c21837b84b55abef918708b86eeb05c8" ip="100.127.36.253" node="ip-x-x-32-110.us-west-2.compute.internal" pod="data/xxx-3d283c8214c544d4bf21ea09f6b4aad3"
2021-09-07 16:47:13.047 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:47:13.074 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-110.us-west-2.compute.internal" subnet="100.127.36.192/26"
2021-09-07 16:47:13.083 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-110.us-west-2.compute.internal" error=block '100.127.36.192/26' is not empty
2021-09-07 16:47:13.083 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.127.36.192/26' is not empty node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:47:13.723 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 16:47:13.742 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-224.us-west-2.compute.internal" subnet="100.87.50.128/26"
2021-09-07 16:47:13.750 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-224.us-west-2.compute.internal" error=block '100.87.50.128/26' is not empty
2021-09-07 16:47:13.750 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.87.50.128/26' is not empty node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 16:47:15.030 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 16:47:15.051 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-36-15.us-west-2.compute.internal" subnet="100.80.22.128/26"
2021-09-07 16:47:15.063 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-36-15.us-west-2.compute.internal" error=block '100.80.22.128/26' is not empty
2021-09-07 16:47:15.063 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.80.22.128/26' is not empty node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 16:47:15.063 [WARNING][1] ipam.go 252: error syncing IPAM data error=block '100.80.22.128/26' is not empty
2021-09-07 16:47:15.103 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-224.us-west-2.compute.internal) with error: nodes "ip-x-x-32-224.us-west-2.compute.internal" not found
2021-09-07 16:47:15.123 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-36-15.us-west-2.compute.internal) with error: nodes "ip-x-x-36-15.us-west-2.compute.internal" not found
2021-09-07 16:47:15.472 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-110.us-west-2.compute.internal) with error: nodes "ip-x-x-32-110.us-west-2.compute.internal" not found
2021-09-07 16:47:16.869 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.ee250e358a497d80fcf7d5b712be367f9b21136d548b21438296178712017ddd" ip="100.80.22.186" node="ip-x-x-36-15.us-west-2.compute.internal" pod="data/xxx-50ccb6ff77e84009972b09c060d70309"
2021-09-07 16:47:17.069 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.fc90c4e5f824e2cf08399b4cdc80d927c21837b84b55abef918708b86eeb05c8" ip="100.127.36.253" node="ip-x-x-32-110.us-west-2.compute.internal" pod="data/xxx-3d283c8214c544d4bf21ea09f6b4aad3"
2021-09-07 16:47:17.267 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.42fa01cdc2d9dd1aceb1bf810d29b26b2427a373a1935a6804ea4ff202208725" ip="100.87.50.140" node="ip-x-x-32-224.us-west-2.compute.internal" pod="data/xxx-585b1906fcf747dd97502bd0c1c8d46b"
2021-09-07 16:47:19.837 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 16:47:19.853 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-224.us-west-2.compute.internal" subnet="100.87.50.128/26"
2021-09-07 16:47:19.862 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-224.us-west-2.compute.internal" error=block '100.87.50.128/26' is not empty
2021-09-07 16:47:19.862 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.87.50.128/26' is not empty node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 16:47:24.982 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 16:47:25.000 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-36-15.us-west-2.compute.internal" subnet="100.80.22.128/26"
2021-09-07 16:47:25.010 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-36-15.us-west-2.compute.internal" error=block '100.80.22.128/26' is not empty
2021-09-07 16:47:25.011 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.80.22.128/26' is not empty node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 16:47:35.251 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:47:35.268 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-110.us-west-2.compute.internal" subnet="100.127.36.192/26"
2021-09-07 16:47:35.277 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-110.us-west-2.compute.internal" error=block '100.127.36.192/26' is not empty
2021-09-07 16:47:35.277 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.127.36.192/26' is not empty node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:47:35.277 [WARNING][1] ipam.go 252: error syncing IPAM data error=block '100.127.36.192/26' is not empty
2021-09-07 16:47:35.287 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-110.us-west-2.compute.internal) with error: nodes "ip-x-x-32-110.us-west-2.compute.internal" not found
2021-09-07 16:47:35.302 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-36-15.us-west-2.compute.internal) with error: nodes "ip-x-x-36-15.us-west-2.compute.internal" not found
2021-09-07 16:47:35.319 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-224.us-west-2.compute.internal) with error: nodes "ip-x-x-32-224.us-west-2.compute.internal" not found
2021-09-07 16:47:37.086 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.fc90c4e5f824e2cf08399b4cdc80d927c21837b84b55abef918708b86eeb05c8" ip="100.127.36.253" node="ip-x-x-32-110.us-west-2.compute.internal" pod="data/xxx-3d283c8214c544d4bf21ea09f6b4aad3"
2021-09-07 16:47:37.281 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.42fa01cdc2d9dd1aceb1bf810d29b26b2427a373a1935a6804ea4ff202208725" ip="100.87.50.140" node="ip-x-x-32-224.us-west-2.compute.internal" pod="data/xxx-585b1906fcf747dd97502bd0c1c8d46b"
2021-09-07 16:47:37.483 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.ee250e358a497d80fcf7d5b712be367f9b21136d548b21438296178712017ddd" ip="100.80.22.186" node="ip-x-x-36-15.us-west-2.compute.internal" pod="data/xxx-50ccb6ff77e84009972b09c060d70309"
2021-09-07 16:47:57.969 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:47:57.985 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-110.us-west-2.compute.internal" subnet="100.127.36.192/26"
2021-09-07 16:47:57.995 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-110.us-west-2.compute.internal" error=block '100.127.36.192/26' is not empty
2021-09-07 16:47:57.995 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.127.36.192/26' is not empty node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:48:38.955 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 16:48:38.973 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-36-15.us-west-2.compute.internal" subnet="100.80.22.128/26"
2021-09-07 16:48:38.989 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-36-15.us-west-2.compute.internal" error=block '100.80.22.128/26' is not empty
2021-09-07 16:48:38.989 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.80.22.128/26' is not empty node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 16:50:00.909 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 16:50:00.928 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-224.us-west-2.compute.internal" subnet="100.87.50.128/26"
2021-09-07 16:50:00.938 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-224.us-west-2.compute.internal" error=block '100.87.50.128/26' is not empty
2021-09-07 16:50:00.938 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.87.50.128/26' is not empty node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 16:50:00.938 [WARNING][1] ipam.go 252: error syncing IPAM data error=block '100.87.50.128/26' is not empty
2021-09-07 16:50:00.942 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-110.us-west-2.compute.internal) with error: nodes "ip-x-x-32-110.us-west-2.compute.internal" not found
2021-09-07 16:50:01.353 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-224.us-west-2.compute.internal) with error: nodes "ip-x-x-32-224.us-west-2.compute.internal" not found
2021-09-07 16:50:01.757 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-36-15.us-west-2.compute.internal) with error: nodes "ip-x-x-36-15.us-west-2.compute.internal" not found
2021-09-07 16:50:02.749 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.fc90c4e5f824e2cf08399b4cdc80d927c21837b84b55abef918708b86eeb05c8" ip="100.127.36.253" node="ip-x-x-32-110.us-west-2.compute.internal" pod="data/xxx-3d283c8214c544d4bf21ea09f6b4aad3"
2021-09-07 16:50:02.947 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.42fa01cdc2d9dd1aceb1bf810d29b26b2427a373a1935a6804ea4ff202208725" ip="100.87.50.140" node="ip-x-x-32-224.us-west-2.compute.internal" pod="data/xxx-585b1906fcf747dd97502bd0c1c8d46b"
2021-09-07 16:50:03.146 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.ee250e358a497d80fcf7d5b712be367f9b21136d548b21438296178712017ddd" ip="100.80.22.186" node="ip-x-x-36-15.us-west-2.compute.internal" pod="data/xxx-50ccb6ff77e84009972b09c060d70309"
2021-09-07 16:52:46.994 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:52:47.013 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-110.us-west-2.compute.internal" subnet="100.127.36.192/26"
2021-09-07 16:52:47.022 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-110.us-west-2.compute.internal" error=block '100.127.36.192/26' is not empty
2021-09-07 16:52:47.022 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.127.36.192/26' is not empty node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 16:58:14.702 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 16:58:14.720 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-224.us-west-2.compute.internal" subnet="100.87.50.128/26"
2021-09-07 16:58:14.727 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-224.us-west-2.compute.internal" error=block '100.87.50.128/26' is not empty
2021-09-07 16:58:14.727 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.87.50.128/26' is not empty node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 17:09:10.087 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 17:09:10.106 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-36-15.us-west-2.compute.internal" subnet="100.80.22.128/26"
2021-09-07 17:09:10.114 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-36-15.us-west-2.compute.internal" error=block '100.80.22.128/26' is not empty
2021-09-07 17:09:10.114 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.80.22.128/26' is not empty node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 17:09:10.114 [WARNING][1] ipam.go 252: error syncing IPAM data error=block '100.80.22.128/26' is not empty
2021-09-07 17:09:10.441 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-110.us-west-2.compute.internal) with error: nodes "ip-x-x-32-110.us-west-2.compute.internal" not found
2021-09-07 17:09:10.524 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-224.us-west-2.compute.internal) with error: nodes "ip-x-x-32-224.us-west-2.compute.internal" not found
2021-09-07 17:09:11.125 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-36-15.us-west-2.compute.internal) with error: nodes "ip-x-x-36-15.us-west-2.compute.internal" not found
2021-09-07 17:09:11.921 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.fc90c4e5f824e2cf08399b4cdc80d927c21837b84b55abef918708b86eeb05c8" ip="100.127.36.253" node="ip-x-x-32-110.us-west-2.compute.internal" pod="data/xxx-3d283c8214c544d4bf21ea09f6b4aad3"
2021-09-07 17:09:12.126 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.42fa01cdc2d9dd1aceb1bf810d29b26b2427a373a1935a6804ea4ff202208725" ip="100.87.50.140" node="ip-x-x-32-224.us-west-2.compute.internal" pod="data/xxx-585b1906fcf747dd97502bd0c1c8d46b"
2021-09-07 17:09:12.319 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.ee250e358a497d80fcf7d5b712be367f9b21136d548b21438296178712017ddd" ip="100.80.22.186" node="ip-x-x-36-15.us-west-2.compute.internal" pod="data/xxx-50ccb6ff77e84009972b09c060d70309"
2021-09-07 17:25:52.324 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 17:25:52.342 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-110.us-west-2.compute.internal" subnet="100.127.36.192/26"
2021-09-07 17:25:52.351 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-110.us-west-2.compute.internal" error=block '100.127.36.192/26' is not empty
2021-09-07 17:25:52.351 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.127.36.192/26' is not empty node="ip-x-x-32-110.us-west-2.compute.internal"
2021-09-07 17:29:04.560 [INFO][1] resources.go 377: Terminating main client watcher loop
2021-09-07 17:29:04.574 [INFO][1] resources.go 349: Main client watcher loop
2021-09-07 17:34:37.742 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/ipam/v2/assignment/"
2021-09-07 17:42:32.351 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 17:42:32.372 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-32-224.us-west-2.compute.internal" subnet="100.87.50.128/26"
2021-09-07 17:42:32.380 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-32-224.us-west-2.compute.internal" error=block '100.87.50.128/26' is not empty
2021-09-07 17:42:32.380 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.87.50.128/26' is not empty node="ip-x-x-32-224.us-west-2.compute.internal"
2021-09-07 17:45:39.019 [INFO][1] watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2021-09-07 17:59:12.380 [INFO][1] ipam.go 871: Cleaning up IPAM resources for deleted node node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 17:59:12.397 [INFO][1] ipam_block_reader_writer.go 300: Block must be empty but is not empty, refusing to remove affinity. host="ip-x-x-36-15.us-west-2.compute.internal" subnet="100.80.22.128/26"
2021-09-07 17:59:12.407 [ERROR][1] ipam.go 930: Failed to release block affinities for node calicoNode="ip-x-x-36-15.us-west-2.compute.internal" error=block '100.80.22.128/26' is not empty
2021-09-07 17:59:12.407 [WARNING][1] ipam.go 874: Error cleaning up node error=block '100.80.22.128/26' is not empty node="ip-x-x-36-15.us-west-2.compute.internal"
2021-09-07 17:59:12.407 [WARNING][1] ipam.go 242: Periodic IPAM sync failed error=block '100.80.22.128/26' is not empty
2021-09-07 17:59:12.418 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-36-15.us-west-2.compute.internal) with error: nodes "ip-x-x-36-15.us-west-2.compute.internal" not found
2021-09-07 17:59:12.446 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-224.us-west-2.compute.internal) with error: nodes "ip-x-x-32-224.us-west-2.compute.internal" not found
2021-09-07 17:59:13.017 [INFO][1] ipam.go 966: Calico Node referenced in IPAM data does not exist error=resource does not exist: Node(ip-x-x-32-110.us-west-2.compute.internal) with error: nodes "ip-x-x-32-110.us-west-2.compute.internal" not found
2021-09-07 17:59:14.211 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.ee250e358a497d80fcf7d5b712be367f9b21136d548b21438296178712017ddd" ip="100.80.22.186" node="ip-x-x-36-15.us-west-2.compute.internal" pod="data/xxx-50ccb6ff77e84009972b09c060d70309"
2021-09-07 17:59:14.413 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.fc90c4e5f824e2cf08399b4cdc80d927c21837b84b55abef918708b86eeb05c8" ip="100.127.36.253" node="ip-x-x-32-110.us-west-2.compute.internal" pod="data/xxx-3d283c8214c544d4bf21ea09f6b4aad3"
2021-09-07 17:59:14.614 [INFO][1] ipam.go 902: Garbage collecting leaked IP address handle="k8s-pod-network.42fa01cdc2d9dd1aceb1bf810d29b26b2427a373a1935a6804ea4ff202208725" ip="100.87.50.140" node="ip-x-x-32-224.us-west-2.compute.internal" pod="data/xxx-585b1906fcf747dd97502bd0c1c8d46b"