calico: calico-node / felix health check fails: "aws-source-destination-check"

Hi, I’ve installed calico networking on EKS following the simple instructions here. Everything seems to work, but calico-node health check keeps failing.

Please advise, I’m not a pro in this area, and I couldn’t find anything relevant.

Expected Behavior

Health check passing

Current Behavior

Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503

The log from the calico-node pod:

[INFO][39] felix/int_dataplane.go 1300: Applying dataplane updates
[INFO][39] felix/ipsets.go 223: Asked to resync with the dataplane on next update. family="inet"
[INFO][39] felix/ipsets.go 306: Resyncing ipsets with dataplane. family="inet"
[INFO][39] felix/ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=1.308558ms
[INFO][39] felix/int_dataplane.go 1314: Finished applying updates to dataplane. msecToApply=1.636869
[INFO][38] monitor-addresses/startup.go 756: Using autodetected IPv4 address on interface eth0: 192.168.68.211/19
[WARNING][39] felix/health.go 188: Reporter is not ready. name="aws-source-destination-check"
[WARNING][39] felix/health.go 154: Health: not ready
[WARNING][39] felix/health.go 188: Reporter is not ready. name="aws-source-destination-check"

This source/dest check seems to be turned off on the ENI.

Your Environment

Cluster was just set up via eksctl, running k8s 1.18, also running cluster autoscaler and ingress-nginx

Thank you in advance

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 22 (4 by maintainers)

Most upvoted comments

Hello,

I hit the same issue while migrating from Calico 3.17.2 to Calico 3.18.1. I used the calico-vxlan.yaml file from the release and the calico-node pods were never ready:

2021-03-12 11:15:43.187 [WARNING][42] felix/health.go 154: Health: not ready
2021-03-12 11:15:43.721 [WARNING][42] felix/health.go 188: Reporter is not ready. name="aws-source-destination-check"
2021-03-12 11:15:53.204 [WARNING][42] felix/health.go 188: Reporter is not ready. name="aws-source-destination-check"

The default calico-vxlan.yaml file for 3.18.1 contains:

            # Disable AWS source-destination check on nodes.
            - name: FELIX_AWSSRCDSTCHECK
              value: Disable

I also had to remove this option in order to make calico-node start. So I think the issue is not solved yet.

mcorbin on Mar 12, 2021

Please create a new issue, it’s likely to be something different. If I had to guess, you’ve started your nodes without the required IAM permission for the AWS source dest check feature to work. This could be a change in EKS to no longer grant that permission by default, for example.

The permission I’d expect us to need is ModifyNetworkInterfaceAttribute (and possibly DescribeInstances and DescribeNetworkInterfaces).

fasaxc on Nov 18, 2021

This same problem came up in this discussion in the “install” Slack channel.

seh on Nov 11, 2020