calico: calico/node is not ready: BIRD is not ready: BGP not established (Calico 3.6 / k8s 1.14.1)

I’m configuring the k8s cluster with 2 nodes (ubuntu 2 vm on VirtualBox). Nodes are connected with NAT Network (10.0.0.0/24). master node : 10.0.0.10 / host-only adapter (192.168.56.101) worker node2 : 10.0.0.11

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

  1. Install kubeadm on both master and worker node respectively.
  2. Execute kubeadm init on the master node kubeadm init --pod-network-cidr=192.168.0.0/16
  3. Using project calico 3.6 tutorial, apply pod network. https://docs.projectcalico.org/v3.6/getting-started/kubernetes/ kubectl apply -f
    https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Here, weird thing is that calico-node pod is ready 1/1 --> it should be 2/2 by manual.

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5cbcccc885-xtxck   1/1     Running   0          96s
kube-system   calico-node-n4tcf                          1/1     Running   0          96s
kube-system   coredns-fb8b8dccf-x8thj                    1/1     Running   0          4m9s
kube-system   coredns-fb8b8dccf-zvmsp                    1/1     Running   0          4m9s
kube-system   etcd-k8s-master                            1/1     Running   0          3m33s
kube-system   kube-apiserver-k8s-master                  1/1     Running   0          3m18s
kube-system   kube-controller-manager-k8s-master         1/1     Running   0          3m29s
kube-system   kube-proxy-5ck6g                           1/1     Running   0          4m9s
kube-system   kube-scheduler-k8s-master                  1/1     Running   0          3m18s

Anyway, 1 pod status seems to be normal.

$ kubectl describe pods -n kube-system calico-node-n4tcf
...
(skipped)
...
Events:
  Type    Reason     Age    From                 Message
  ----    ------     ----   ----                 -------
  Normal  Scheduled  2m21s  default-scheduler    Successfully assigned kube-system/calico-node-n4tcf to k8s-master
  Normal  Pulled     2m20s  kubelet, k8s-master  Container image "calico/cni:v3.6.1" already present on machine
  Normal  Created    2m20s  kubelet, k8s-master  Created container upgrade-ipam
  Normal  Started    2m19s  kubelet, k8s-master  Started container upgrade-ipam
  Normal  Pulled     2m19s  kubelet, k8s-master  Container image "calico/cni:v3.6.1" already present on machine
  Normal  Created    2m19s  kubelet, k8s-master  Created container install-cni
  Normal  Started    2m18s  kubelet, k8s-master  Started container install-cni
  Normal  Pulling    2m18s  kubelet, k8s-master  Pulling image "calico/node:v3.6.1"
  Normal  Pulled     2m5s   kubelet, k8s-master  Successfully pulled image "calico/node:v3.6.1"
  Normal  Created    2m5s   kubelet, k8s-master  Created container calico-node
  Normal  Started    2m4s   kubelet, k8s-master  Started container calico-node
  1. After joining worker node with kubeadm join, another calico-node was created. But, in a while, two calico-node (for both master and worker) are crashed.
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5cbcccc885-xtxck   1/1     Running   0          40m
kube-system   calico-node-n4tcf                          0/1     Running   0          40m
kube-system   calico-node-sjqsr                          0/1     Running   0          36m
kube-system   coredns-fb8b8dccf-x8thj                    1/1     Running   0          43m
kube-system   coredns-fb8b8dccf-zvmsp                    1/1     Running   0          43m
kube-system   etcd-k8s-master                            1/1     Running   0          42m
kube-system   kube-apiserver-k8s-master                  1/1     Running   0          42m
kube-system   kube-controller-manager-k8s-master         1/1     Running   0          42m
kube-system   kube-proxy-5ck6g                           1/1     Running   0          43m
kube-system   kube-proxy-ds549                           1/1     Running   0          36m
kube-system   kube-scheduler-k8s-master                  1/1     Running   0          42m

calico-node-n4tcf (for master) pod was changed from ready (1/1) to (0/1). calico-node-sjqsr (newly created for worker) pod was not ready.

$ kubectl describe pod -n kube-system calico-node-n4tcf
...
(skipped)
...
Warning  Unhealthy  10s (x2 over 20s)  kubelet, k8s-master  (combined from similar events): Readiness probe failed: Threshold time for bird readiness check:  30s
calico/node is not ready: BIRD is not ready: BGP not established with 10.0.0.112019-04-18 16:59:27.462 [INFO][607] readiness.go 88: Number of node(s) with BGP peering established = 0

$ kubectl describe pod -n kube-system calico-node-sjqsr
...
(skipped)
...
Warning  Unhealthy  6s (x4 over 36s)  kubelet, k8s-worker1  (combined from similar events): Readiness probe failed: Threshold time for bird readiness check:  30s
calico/node is not ready: BIRD is not ready: BGP not established with 192.168.56.1012019-04-18 16:59:49.812 [INFO][300] readiness.go 88: Number of node(s) with BGP peering established = 0

Context

I don’t know why it doesn’t work well. When I applied flannel add-on, there was no problem. Repeatedly I reset and reinstalled, but result was not different.

Could you tell me what should I do for solving the problem?

Your Environment

  • Calico version : 3.6
  • Orchestrator version (e.g. kubernetes, mesos, rkt): k8s 1.14.1, docker 18.9.5
  • Operating System and version: Ubuntu 18.04.2
  • Link to your project (optional):

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 28
  • Comments: 35 (1 by maintainers)

Most upvoted comments

I got the solution :

While making the cluster setup in Kubernetes if you are using calico below will be the problem.

The first preference of ifconfig(in my case <br-3faae2641db0>) is that it will try to connect worker-nodes which is not the right ip.

Solution:

Change the calico.yaml file by overriding that ip to etho-ip by using the following steps.

 # Specify interface
            - name: IP_AUTODETECTION_METHOD
              value: "interface=eth1"

Needs to open port Calico networking (BGP) - 179

calico.yaml

---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the backend to use.
  calico_backend: "bird"

  # Configure the MTU to use
  veth_mtu: "1440"

  # The CNI network configuration to install on each node.  The special
  # values in this config will be automatically populated.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        }
      ]
    }

---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
   name: felixconfigurations.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: FelixConfiguration
    plural: felixconfigurations
    singular: felixconfiguration
---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ipamblocks.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: IPAMBlock
    plural: ipamblocks
    singular: ipamblock

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: blockaffinities.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: BlockAffinity
    plural: blockaffinities
    singular: blockaffinity

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ipamhandles.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: IPAMHandle
    plural: ipamhandles
    singular: ipamhandle

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ipamconfigs.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: IPAMConfig
    plural: ipamconfigs
    singular: ipamconfig

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: bgppeers.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: BGPPeer
    plural: bgppeers
    singular: bgppeer

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: bgpconfigurations.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: BGPConfiguration
    plural: bgpconfigurations
    singular: bgpconfiguration

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ippools.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: IPPool
    plural: ippools
    singular: ippool

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: hostendpoints.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: HostEndpoint
    plural: hostendpoints
    singular: hostendpoint

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: clusterinformations.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: ClusterInformation
    plural: clusterinformations
    singular: clusterinformation

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: globalnetworkpolicies.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: GlobalNetworkPolicy
    plural: globalnetworkpolicies
    singular: globalnetworkpolicy

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: globalnetworksets.crd.projectcalico.org
spec:
  scope: Cluster
  group: crd.projectcalico.org
  version: v1
  names:
    kind: GlobalNetworkSet
    plural: globalnetworksets
    singular: globalnetworkset

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: networkpolicies.crd.projectcalico.org
spec:
  scope: Namespaced
  group: crd.projectcalico.org
  version: v1
  names:
    kind: NetworkPolicy
    plural: networkpolicies
    singular: networkpolicy

---

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: networksets.crd.projectcalico.org
spec:
  scope: Namespaced
  group: crd.projectcalico.org
  version: v1
  names:
    kind: NetworkSet
    plural: networksets
    singular: networkset
---
# Source: calico/templates/rbac.yaml

# Include a clusterrole for the kube-controllers component,
# and bind it to the calico-kube-controllers serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Nodes are watched to monitor for deletions.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - watch
      - list
      - get
  # Pods are queried to check for existence.
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
  # IPAM resources are manipulated when nodes are deleted.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
    verbs:
      - list
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
    verbs:
      - get
      - list
      - create
      - update
      - delete
  # Needs access to update clusterinformations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - clusterinformations
    verbs:
      - get
      - create
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---
# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
      # Used to discover Typhas.
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch
      # Calico stores some configuration information in node annotations.
      - update
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
  # Used by Calico for policy information.
  - apiGroups: [""]
    resources:
      - pods
      - namespaces
      - serviceaccounts
    verbs:
      - list
      - watch
  # The CNI plugin patches pods/status.
  - apiGroups: [""]
    resources:
      - pods/status
    verbs:
      - patch
  # Calico monitors various CRDs for config.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - ipamblocks
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
      - clusterinformations
      - hostendpoints
    verbs:
      - get
      - list
      - watch
  # Calico must create and update some CRDs on startup.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
  # Calico stores some configuration information on the node.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  # These permissions are only requried for upgrade from v2.6, and can
  # be removed after upgrade or on fresh installations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - bgpconfigurations
      - bgppeers
    verbs:
      - create
      - update
  # These permissions are required for Calico CNI to perform IPAM allocations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
    verbs:
      - get
      - list
      - create
      - update
      - delete
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ipamconfigs
    verbs:
      - get
  # Block affinities must also be watchable by confd for route aggregation.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
    verbs:
      - watch
  # The Calico IPAM migration needs to get daemonsets. These permissions can be
  # removed if not upgrading from an installation using host-local IPAM.
  - apiGroups: ["apps"]
    resources:
      - daemonsets
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system

---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
      annotations:
        # This, along with the CriticalAddonsOnly toleration below,
        # marks the pod as a critical add-on, ensuring it gets
        # priority scheduling and that its resources are reserved
        # if it ever gets evicted.
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux
      hostNetwork: true
      tolerations:
        # Make sure calico-node gets scheduled on all nodes.
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      priorityClassName: system-node-critical
      initContainers:
        # This container performs upgrade from host-local IPAM to calico-ipam.
        # It can be deleted if this is a fresh installation, or if you have already
        # upgraded to use calico-ipam.
        - name: upgrade-ipam
          image: calico/cni:v3.8.2
          command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
          env:
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
          volumeMounts:
            - mountPath: /var/lib/cni/networks
              name: host-local-net-dir
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
        # This container installs the CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: calico/cni:v3.8.2
          command: ["/install-cni.sh"]
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            # Set the hostname based on the k8s node name.
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # CNI MTU Config variable
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Prevents the container from sleeping forever.
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
        # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
        # to communicate with Felix over the Policy Sync API.
        - name: flexvol-driver
          image: calico/pod2daemon-flexvol:v3.8.2
          volumeMounts:
          - name: flexvol-driver-host
            mountPath: /host/driver
      containers:
        # Runs calico-node container on each Kubernetes node.  This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: calico/node:v3.8.2
          env:
            # Use Kubernetes API as the backing datastore.
            - name: DATASTORE_TYPE
              value: "kubernetes"
            # Wait for the datastore.
            - name: WAIT_FOR_DATASTORE
              value: "true"
            # Set based on the k8s node name.
            - name: NODENAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Specify interface
            - name: IP_AUTODETECTION_METHOD
              value: "interface=eth1"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "192.168.0.0/16"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          livenessProbe:
            httpGet:
              path: /liveness
              port: 9099
              host: localhost
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -bird-ready
              - -felix-ready
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - name: policysync
              mountPath: /var/run/nodeagent
      volumes:
        # Used by calico-node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Mount in the directory for host-local IPAM allocations. This is
        # used when upgrading from host-local to calico-ipam, and can be removed
        # if not using the upgrade-ipam init container.
        - name: host-local-net-dir
          hostPath:
            path: /var/lib/cni/networks
        # Used to create per-pod Unix Domain Sockets
        - name: policysync
          hostPath:
            type: DirectoryOrCreate
            path: /var/run/nodeagent
        # Used to install Flex Volume Driver
        - name: flexvol-driver-host
          hostPath:
            type: DirectoryOrCreate
            path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system

---
# Source: calico/templates/calico-kube-controllers.yaml

# See https://github.com/projectcalico/kube-controllers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  # The controllers can only have a single active instance.
  replicas: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux
      tolerations:
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: calico-kube-controllers
      priorityClassName: system-cluster-critical
      containers:
        - name: calico-kube-controllers
          image: calico/kube-controllers:v3.8.2
          env:
            # Choose which controllers to run.
            - name: ENABLED_CONTROLLERS
              value: node
            - name: DATASTORE_TYPE
              value: kubernetes
          readinessProbe:
            exec:
              command:
              - /usr/bin/check-status
              - -r

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system
---
# Source: calico/templates/calico-etcd-secrets.yaml

---
# Source: calico/templates/calico-typha.yaml

---
# Source: calico/templates/configure-canal.yaml

calico.zip

Due to the fact that “eth.*” does not exist on some platforms, you may find an alternative setting to be more portable and useful. This is what I did:

            # Set auto-detect method to be publicly routable, whatever that interface may be
            - name: IP_AUTODETECTION_METHOD
              value: "can-reach=8.8.8.8"

This at least forced Calico to use a public-routing interface. If you wanted a private interface, stick the IP of one of a node on your internal network in there. Assuming your routing tables aren’t completely busted (and you have other problems), it’ll do the right thing more reliably.

FYI I had very similar problem (see also #2193).

My unrelated IP was 10.44.0.0:

calico/node is not ready: BIRD is not ready: BGP not established with 10.44.0.0

The problem was I previously tried Weave Net (and flannel also) which didn’t delete the network interface weave, and misled calico’s autodetect feature. 😊

# ifconfig
...

weave: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 65535
        inet 10.44.0.0  netmask 255.240.0.0  broadcast 10.47.255.255
        ether a6:5d:b9:0c:c5:e7  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5794  bytes 243348 (237.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

Resolving the issue was easy as deleting that weave network interface. I got the tip from kubernetes/kubernetes#70202, from the “uninstall flannel” step:

Remove interfaces related to docker and flannel: ip link For each interface for docker or flannel, do the following ifconfig <name of interface from ip link> down ip link delete <name of interface from ip link>

Actually, I have met this problem on 3.8.* in several different scenarios:

  • network interface: calico uses wrong interface; please use regex interface=en.+ or similar. Once calico uses previous flannel interfaces, which were not cleared properly.
  • Re-establish k8s after tearing down: please run iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X as suggested by kubeadm. Some system will reset your ssh connection (e.g. you ssh to one worker) when you are resetting iptables, so the four parts of the resetting commands may not finish. Please ensure this command is fully performed on each machine.

Quick response for new calico version: If you are using Tigers operator for manage and configure calico DaemonSet, you should change your Tigera config file like that:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    # Note: The ipPools section cannot be modified post-install.
    ipPools:
    - blockSize: 26
      cidr: 10.244.0.0/16       <---- don't forget to change this part for your own usage
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled
      nodeSelector: all()
    nodeAddressAutodetectionV4:
      interface: "ens*"    <---- don't forget to change this part for your own usage

You can see more options here : https://docs.projectcalico.org/reference/node/configuration

IP_AUTODETECTION_METHOD was also the same problem for me. The calico-node-* pods in the kube-system namespace wer in 0/0 ready state.

Run: kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=can-reach=www.google.com

Then the pods became ready within a few seconds.

Reference: https://docs.projectcalico.org/networking/ip-autodetection#change-the-autodetection-method

Check the calico logs definitely you will get some idea.

In my case, this problem still exists.

After uninstalling Calico 3.7, I installed Flannel, and then uninstalled Flannel to install Calico 3.10.1. I know it’s weird to do this, and now something strange is happening.

I have removed Flannel’s interface:

ifconfig does not display any interfaces of flannel.

Server real internal IP: master 172.17.106.122 node1 172.17.106.121 node2 172.17.106.120

My real network card eth0 was deployed using the yaml file of the official 3.10 calico.

My POD CIDR is 10.100.0.1/20, which proves correct.

Then any of my nodes will prompt this error:

Readiness probe failed: calico / node is not ready: BIRD is not ready: BGP not established with 172.17.106.121,172.17.106.1202019-12-11 06: 54: 11.577 [INFO] [127] health.go 156: Number of node (s) with BGP peering established = 0

The IPs that appear inside are the contents of the real internal IP of the server above.

It will not be another IP that does not exist.

This is incredible.

            - name: IP
              value: "autodetect"
            - name: IP_AUTODETECTION_METHOD
              value: "interface=eth.*"
            # value: "interface=eth0"
            # value: "can-reach=www.163.com"

While operating k8s using a virtual machine, I experience the same phenomenon, solve it, and leave a post. The host machine is using Centos7 and operated 3 K8s nodes through KVM. Activate the network interface used by calico-node through firewall-cmd. Looking at the log of calico-node, gatway could not be called, but this occurred because the virtual machine interface of the host is not in the internal zone and is not activated.

Updating this section in @MadAboutProgramming 's solution resolved my problem.

 # Specify interface
            - name: IP_AUTODETECTION_METHOD
              value: "interface=eth.*"

Due to the fact that “eth.*” does not exist on some platforms, you may find an alternative setting to be more portable and useful. This is what I did:

            # Set auto-detect method to be publicly routable, whatever that interface may be
            - name: IP_AUTODETECTION_METHOD
              value: "can-reach=8.8.8.8"

This at least forced Calico to use a public-routing interface. If you wanted a private interface, stick the IP of one of a node on your internal network in there. Assuming your routing tables aren’t completely busted (and you have other problems), it’ll do the right thing more reliably.

It worked for me. Thanks a lot

Updating this section in @MadAboutProgramming 's solution resolved my problem.

 # Specify interface
            - name: IP_AUTODETECTION_METHOD
              value: "interface=eth.*"

In my case the following setting did it!

             - name: IP_AUTODETECTION_METHOD
              value: "interface=enp.*"

Needs to open port Calico networking (BGP) - 179

In my case it’s all about adding TCP 179 to Nodes Security Group (for AWS installation).

for Centos i found a solution, it is about the IP_AUTODETECTION_METHOD

you have to find your connection device first with this code nmcli connection show

for my case it was nm-bond

then you can update calico env with this code kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface=nm-bond

validated the change: kubectl get daemonset/calico-node -n kube-system --output json | jq '.spec.template.spec.containers[].env[] | select(.name | startswith("IP"))'

While operating k8s using a virtual machine, I experience the same phenomenon, solve it, and leave a post. The host machine is using Centos7 and operated 3 K8s nodes through KVM. Activate the network interface used by calico-node through firewall-cmd. Looking at the log of calico-node, gatway could not be called, but this occurred because the virtual machine interface of the host is not in the internal zone and is not activated.

Experiencing same issue with Centos7 vm machine… can you share the solution?

I resolve my problem by using @MadAboutProgramming 's solution. Thanks a lot!