calico: Bird is adding a blackhole route to the service cluster CIDR that blocks access inside the cluster
(seemingly) randomly, we will see a blackhole route to our service cluster CIDR on the Kubernetes workers that prevents access to the whole network, when then prevents Calico from talking to the K8s API.
Removing the network fixes the problem, until it comes back.
Expected Behavior
Pods should be able to access service cluster IPs. Either the blackhole network should not exist or iptables/routing should provide an override to the specific destinations that should be accessible.
Current Behavior
We have a small 1.13.2 k8s cluster with two masters and 4 workers running Calico 3.5.2 (we upgraded from 3.4.0, and the problem existed then, too). We are using CALICO_ADVERTISE_CLUSTER_IPS=x.y.z.0/23 to advertise the cluster network to external BGP routers, and this particular cluster is set up with route reflectors internally on the masters.
Everything works, then all of a sudden Calico will start failing health checks and nothing inside the cluster can talk to the K8s API over the cluster network. If I look at ip route I see:
blackhole x.y.z.0/23 proto bird
Deleting that fixes the particular node.
Steps to Reproduce (for bugs)
I wish we could reproduce this on demand.
Context
Internal communication to cluster IPs breaks.
Your Environment
- Calico version - 3.5.2
- Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.13.2
- Operating System and version: CentOS 7 (3.10.0-862.3.2.el7.x86_64) on the masters, Ubuntu 18.10 (4.18.0-15-generic) on workers
The NAT table from a failing worker:
Chain PREROUTING (policy ACCEPT 2 packets, 138 bytes)
pkts bytes target prot opt in out source destination
9246K 555M cali-PREROUTING all -- any any anywhere anywhere /* cali:6gwbT8clXdHdC1b1 */
9246K 555M KUBE-SERVICES all -- any any anywhere anywhere /* kubernetes service portals */
80493 4796K DOCKER all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 2 packets, 138 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
556K 38M cali-OUTPUT all -- any any anywhere anywhere /* cali:tVnHkvAo15HuiPy0 */
556K 38M KUBE-SERVICES all -- any any anywhere anywhere /* kubernetes service portals */
0 0 DOCKER all -- any any anywhere !localhost/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
9700K 587M cali-POSTROUTING all -- any any anywhere anywhere /* cali:O3lYWMrLQYEMJtB5 */
559K 38M KUBE-POSTROUTING all -- any any anywhere anywhere /* kubernetes postrouting rules */
0 0 MASQUERADE all -- any !docker0 172.17.0.0/16 anywhere
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 any anywhere anywhere
Chain KUBE-MARK-DROP (0 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- any any anywhere anywhere MARK or 0x8000
Chain KUBE-MARK-MASQ (23 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- any any anywhere anywhere MARK or 0x4000
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- any any anywhere anywhere /* default/nginx-test: */ tcp dpt:32101
0 0 KUBE-SVC-QGCAAURWPBTRIF6C tcp -- any any anywhere anywhere /* default/nginx-test: */ tcp dpt:32101
Chain KUBE-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- any any anywhere anywhere /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
Chain KUBE-SEP-2BI3CGY3RBOXNHMR (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any master1.example.com anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:hostnetwork.70:6443
Chain KUBE-SEP-63SUW6MPSDSXTKGH (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.1.15 anywhere
0 0 DNAT udp -- any any anywhere anywhere udp to:192.168.1.15:53
Chain KUBE-SEP-75UHAM7TT664VORT (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.7.33 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.7.33:44134
Chain KUBE-SEP-7XJGCDZBVZGKF5UR (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.1.15 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.1.15:53
Chain KUBE-SEP-7ZRN5IHZBAQLK72Q (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.10.28 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.10.28:5044
Chain KUBE-SEP-BAUZZQOICB45SVJN (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.9.7 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.9.7:8080
Chain KUBE-SEP-EWIPOZ7CSGARYHVW (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.1.16 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.1.16:53
Chain KUBE-SEP-H4EJ4QVFG5N57ZS6 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.9.34 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.9.34:9091
Chain KUBE-SEP-HRUT4WCAXXNYPXP3 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.1.16 anywhere
0 0 DNAT udp -- any any anywhere anywhere udp to:192.168.1.16:53
Chain KUBE-SEP-PDOLTKTNRFEIGU3E (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any master2.example.com anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:hostnetwork.71:6443
Chain KUBE-SEP-VFXAJ4PXGQMVLQCH (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.9.8 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.9.8:80
Chain KUBE-SEP-YAXH47DHY2PSAE4P (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.7.34 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.7.34:5044
Chain KUBE-SEP-ZTHCW6C6KVPP4EGK (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- any any 192.168.10.30 anywhere
0 0 DNAT tcp -- any any anywhere anywhere tcp to:192.168.10.30:9090
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- any any !192.168.0.0/16 x.y.z.133 /* default/nginx-test: cluster IP */ tcp dpt:http
0 0 KUBE-SVC-QGCAAURWPBTRIF6C tcp -- any any anywhere x.y.z.133 /* default/nginx-test: cluster IP */ tcp dpt:http
0 0 KUBE-MARK-MASQ tcp -- any any !192.168.0.0/16 x.y.z.120 /* kube-system/tiller-deploy:tiller cluster IP */ tcp dpt:44134
0 0 KUBE-SVC-K7J76NXP7AUZVFGS tcp -- any any anywhere x.y.z.120 /* kube-system/tiller-deploy:tiller cluster IP */ tcp dpt:44134
0 0 KUBE-MARK-MASQ tcp -- any any !192.168.0.0/16 x.y.z.127 /* kube-system/prometheus-pushgateway:http cluster IP */ tcp dpt:9091
0 0 KUBE-SVC-YI5AWINSNTWJJY2C tcp -- any any anywhere x.y.z.127 /* kube-system/prometheus-pushgateway:http cluster IP */ tcp dpt:9091
0 0 KUBE-MARK-MASQ tcp -- any any !192.168.0.0/16 x.y.z.1 /* default/kubernetes:https cluster IP */ tcp dpt:https
0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- any any anywhere x.y.z.1 /* default/kubernetes:https cluster IP */ tcp dpt:https
0 0 KUBE-MARK-MASQ udp -- any any !192.168.0.0/16 x.y.z.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- any any anywhere x.y.z.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
0 0 KUBE-MARK-MASQ tcp -- any any !192.168.0.0/16 x.y.z.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- any any anywhere x.y.z.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
0 0 KUBE-MARK-MASQ tcp -- any any !192.168.0.0/16 x.y.z.169 /* kube-system/logstash:beats cluster IP */ tcp dpt:5044
0 0 KUBE-SVC-XH5XEKZMVFTAGFL7 tcp -- any any anywhere x.y.z.169 /* kube-system/logstash:beats cluster IP */ tcp dpt:5044
9 556 KUBE-NODEPORTS all -- any any anywhere anywhere /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-7XJGCDZBVZGKF5UR all -- any any anywhere anywhere statistic mode random probability 0.50000000000
0 0 KUBE-SEP-EWIPOZ7CSGARYHVW all -- any any anywhere anywhere
Chain KUBE-SVC-GIJ5JTPHH2LUSL4V (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-BAUZZQOICB45SVJN all -- any any anywhere anywhere
Chain KUBE-SVC-K7J76NXP7AUZVFGS (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-75UHAM7TT664VORT all -- any any anywhere anywhere
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-2BI3CGY3RBOXNHMR all -- any any anywhere anywhere statistic mode random probability 0.50000000000
0 0 KUBE-SEP-PDOLTKTNRFEIGU3E all -- any any anywhere anywhere
Chain KUBE-SVC-QGCAAURWPBTRIF6C (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-VFXAJ4PXGQMVLQCH all -- any any anywhere anywhere
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-63SUW6MPSDSXTKGH all -- any any anywhere anywhere statistic mode random probability 0.50000000000
0 0 KUBE-SEP-HRUT4WCAXXNYPXP3 all -- any any anywhere anywhere
Chain KUBE-SVC-WJKYX3YCFA6UPZA7 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-ZTHCW6C6KVPP4EGK all -- any any anywhere anywhere
Chain KUBE-SVC-XH5XEKZMVFTAGFL7 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-7ZRN5IHZBAQLK72Q all -- any any anywhere anywhere statistic mode random probability 0.50000000000
0 0 KUBE-SEP-YAXH47DHY2PSAE4P all -- any any anywhere anywhere
Chain KUBE-SVC-YI5AWINSNTWJJY2C (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-H4EJ4QVFG5N57ZS6 all -- any any anywhere anywhere
Chain cali-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
556K 38M cali-fip-dnat all -- any any anywhere anywhere /* cali:GBTAv2p5CwevEyJm */
Chain cali-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
9700K 587M cali-fip-snat all -- any any anywhere anywhere /* cali:Z-c7XtVd2Bq7s_hA */
9700K 587M cali-nat-outgoing all -- any any anywhere anywhere /* cali:nYKhEzDlr11Jccal */
0 0 MASQUERADE all -- any tunl0 anywhere anywhere /* cali:SXWvdsbh4Mw7wOln */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL
Chain cali-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
9246K 555M cali-fip-dnat all -- any any anywhere anywhere /* cali:r6XmIziWUJsdOK6Z */
Chain cali-fip-dnat (2 references)
pkts bytes target prot opt in out source destination
Chain cali-fip-snat (1 references)
pkts bytes target prot opt in out source destination
Chain cali-nat-outgoing (1 references)
pkts bytes target prot opt in out source destination
9141K 548M MASQUERADE all -- any any anywhere anywhere /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 30 (18 by maintainers)
I discovered a similar issue that when you add a
serviceExternalIPsinto BGPConfiguration but then delete it, there will be a blackhole left on the controller nodes, where the other nodes chooses the contorller nodes as the next hop. making it impossible to reach that from inside the worker nodes. However controller nodes aren’t affected by the blackhole. Not a big deal for me but surprising and confusing