calico: Cannot access local node services when using eBPF

I have one cluster and also install Prometheus in it. So, each node has a service (HostIP:9100) to export node information. However, if I run one Pod on Node1(Node1 Host IP: 172.21.149.119), I cannot access 172.21.149.119:9100 in the Pod. But I can access other services on the other nodes, such as 172.21.149.xx:9100 .

Expected Behavior

All local serivces on every node can be accessed in the Pod.

Current Behavior

Now, only local services run on different nodes with my Pod can be accessed in my Pod.

Possible Solution

https://github.com/projectcalico/calico/issues/6065

Steps to Reproduce (for bugs)

Install Calico v3.23.1 or v3.22.3
Install Prometheus
Run a Pod on any node
In the Pod, use curl to access HostIP:9100 (HostIP is the node IP where the Pod runs on.)

Context

I also can see the following log messages in calico-node Pod.

libbpf: prog 'calico_connect_v4': failed to attach to cgroup: Invalid argument
2023-01-30 12:01:50.409 [INFO][129] felix/connecttime.go 146: Loaded cgroup program cgroup="/run/calico/cgroup" program="calico_connect_v4"
libbpf: prog 'calico_sendmsg_v4': failed to attach to cgroup: Invalid argument
2023-01-30 12:01:50.499 [INFO][129] felix/connecttime.go 146: Loaded cgroup program cgroup="/run/calico/cgroup" program="calico_sendmsg_v4"
libbpf: prog 'calico_recvmsg_v4': failed to attach to cgroup: Invalid argument
2023-01-30 12:01:50.501 [INFO][129] felix/connecttime.go 146: Loaded cgroup program cgroup="/run/calico/cgroup" program="calico_recvmsg_v4"
libbpf: prog 'calico_sendmsg_v6': failed to attach to cgroup: Invalid argument
2023-01-30 12:01:50.502 [INFO][129] felix/connecttime.go 146: Loaded cgroup program cgroup="/run/calico/cgroup" program="calico_sendmsg_v6"
libbpf: prog 'calico_recvmsg_v6': failed to attach to cgroup: Invalid argument

I think this problem is very similar as #6065. However, the problem still exists after upgrading Calico to v3.23.1.

Your Environment

Calico version: v3.25.1
Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes 1.26.5
Operating System and version: Ubuntu 22.04 LTS

Someone can help me?

About this issue

Original URL
State: open
Created a year ago
Comments: 33 (16 by maintainers)

Commits related to this issue

[BPF] pods in nat-outgoing shoul dnot SNAT to local host When a pod is accessing a local host, it should not get SNATed as the host when it is in a nat-outgoing ippool. (a) it is unnecessary as the l... — committed to tomastigera/project-calico-calico by tomastigera 6 months ago

Most upvoted comments

What if you set config option defaultEndpointToHostAction to Accept https://docs.tigera.io/calico/latest/reference/resources/felixconfig Based on the logs, I do not think we hit that, however, let’s cover all bases.

I modified it and it still not works. Here is my Felix config

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  annotations:
    projectcalico.org/metadata: '{"uid":"2af5e1da-dfcf-46d2-94dd-7d2c1015214e","creationTimestamp":"2023-09-20T06:24:14Z"}'
  creationTimestamp: "2023-09-20T06:24:14Z"
  generation: 2
  name: default
  resourceVersion: "27168"
  uid: 3730d464-f3d2-40d5-9044-39a42b151afe
spec:
  bpfEnabled: true
  bpfExternalServiceMode: DSR
  bpfLogLevel: Debug
  defaultEndpointToHostAction: Accept
  floatingIPs: Disabled
  ipipEnabled: false
  logSeverityScreen: Info
  reportingInterval: 0s
  vxlanEnabled: true
  wireguardEnabled: false

Aslan-Liu on Sep 27, 2023

@tomastigera Here are the output from my environment

# calico-node -bpf policy dump calic440f455693 all
IfaceName: calic440f455693
Hook: tc egress
Error:
Policy Info:
start:
      bf16000000000000 Mov64 dst=R6 src=R1 off=0 imm=0x00000000/0
      b701000000000000 MovImm64 dst=R1 src=R0 off=0 imm=0x00000000/0
      631afcff00000000 StoreReg32 dst=R10 src=R1 off=-4 imm=0x00000000/0
      bfa2000000000000 Mov64 dst=R2 src=R10 off=0 imm=0x00000000/0
      07020000fcffffff AddImm64 dst=R2 src=R0 off=0 imm=0xfffffffc/-4
// Load packet metadata saved by previous program
      181100000c000000 LoadImm64 dst=R1 src=R1 off=0 imm=0x0000000c/12
      0000000000000000 LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0x00000000/0
      8500000001000000 Call dst=R0 src=R0 off=0 imm=0x00000001/1                                       call bpf_map_lookup_elem
      1500110000000000 JumpEqImm64 dst=R0 src=R0 off=17 imm=0x00000000/0                               goto exit
// Save state pointer in register R9
      bf09000000000000 Mov64 dst=R9 src=R0 off=0 imm=0x00000000/0
policy:
      7991980100000000 LoadReg64 dst=R1 src=R9 off=408 imm=0x00000000/0                                R1 = *(u64 *)(R9 + 408) /* state->flags */
      570100000c000000 AndImm64 dst=R1 src=R0 off=0 imm=0x0000000c/12
      5501010000000000 JumpNEImm64 dst=R1 src=R0 off=1 imm=0x00000000/0                                goto allowed_by_host_policy
      0500000000000000 JumpA dst=R0 src=R0 off=0 imm=0x00000000/0                                      goto allowed_by_host_policy
// Start of rule action:"allow" rule_id:"aBMQCbsUMESPKGRp"
// count = 0
allowed_by_host_policy:
      7191640000000000 LoadReg8 dst=R1 src=R9 off=100 imm=0x00000000/0                                 R1 = *(u8 *)(R9 + 100) /* state->rules_hit */
      35010c0020000000 JumpGEImm64 dst=R1 src=R0 off=12 imm=0x00000020/32                              goto allow
      bf12000000000000 Mov64 dst=R2 src=R1 off=0 imm=0x00000000/0
      0702000001000000 AddImm64 dst=R2 src=R0 off=0 imm=0x00000001/1
      7329640000000000 StoreReg8 dst=R9 src=R2 off=100 imm=0x00000000/0                                *(u8 *) (R9 + 100) /* state->rules_hit */ = R2
      6701000003000000 ShiftLImm64 dst=R1 src=R0 off=0 imm=0x00000003/3
      0701000068000000 AddImm64 dst=R1 src=R0 off=0 imm=0x00000068/104
      1802000037ab8758 LoadImm64 dst=R2 src=R0 off=0 imm=0x5887ab37/1485286199
      0000000080e3d4e6 LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0xe6d4e380/-422255744
      0f91000000000000 Add64 dst=R1 src=R9 off=0 imm=0x00000000/0
      7b21000000000000 StoreReg64 dst=R1 src=R2 off=0 imm=0x00000000/0                                 *(u64 *) (R1 + 0) /*  */ = R2
      0500020000000000 JumpA dst=R0 src=R0 off=2 imm=0x00000000/0                                      goto allow
// End of rule aBMQCbsUMESPKGRp
exit:
      b700000002000000 MovImm64 dst=R0 src=R0 off=0 imm=0x00000002/2
      9500000000000000 Exit dst=R0 src=R0 off=0 imm=0x00000000/0
allow:
      b401000001000000 MovImm32 dst=R1 src=R0 off=0 imm=0x00000001/1
      6319540000000000 StoreReg32 dst=R9 src=R1 off=84 imm=0x00000000/0                                *(u32 *) (R9 + 84) /* state->pol_rc */ = R1
      bf61000000000000 Mov64 dst=R1 src=R6 off=0 imm=0x00000000/0
      1812000046000000 LoadImm64 dst=R2 src=R1 off=0 imm=0x00000046/70
      0000000000000000 LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0x00000000/0
      b403000003000000 MovImm32 dst=R3 src=R0 off=0 imm=0x00000003/3
      850000000c000000 Call dst=R0 src=R0 off=0 imm=0x0000000c/12                                      call bpf_tail_call
      b40100000a000000 MovImm32 dst=R1 src=R0 off=0 imm=0x0000000a/10
      6319540000000000 StoreReg32 dst=R9 src=R1 off=84 imm=0x00000000/0                                *(u32 *) (R9 + 84) /* state->pol_rc */ = R1
      b700000002000000 MovImm64 dst=R0 src=R0 off=0 imm=0x00000002/2
      9500000000000000 Exit dst=R0 src=R0 off=0 imm=0x00000000/0
IfaceName: calic440f455693
Hook: tc ingress
Error:
Policy Info:
start:
      bf16000000000000 Mov64 dst=R6 src=R1 off=0 imm=0x00000000/0
      b701000000000000 MovImm64 dst=R1 src=R0 off=0 imm=0x00000000/0
      631afcff00000000 StoreReg32 dst=R10 src=R1 off=-4 imm=0x00000000/0
      bfa2000000000000 Mov64 dst=R2 src=R10 off=0 imm=0x00000000/0
      07020000fcffffff AddImm64 dst=R2 src=R0 off=0 imm=0xfffffffc/-4
// Load packet metadata saved by previous program
      181100000c000000 LoadImm64 dst=R1 src=R1 off=0 imm=0x0000000c/12
      0000000000000000 LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0x00000000/0
      8500000001000000 Call dst=R0 src=R0 off=0 imm=0x00000001/1                                       call bpf_map_lookup_elem
      1500240000000000 JumpEqImm64 dst=R0 src=R0 off=36 imm=0x00000000/0                               goto exit
// Save state pointer in register R9
      bf09000000000000 Mov64 dst=R9 src=R0 off=0 imm=0x00000000/0
policy:
      7991980100000000 LoadReg64 dst=R1 src=R9 off=408 imm=0x00000000/0                                R1 = *(u64 *)(R9 + 408) /* state->flags */
      570100000c000000 AndImm64 dst=R1 src=R0 off=0 imm=0x0000000c/12
      5501010000000000 JumpNEImm64 dst=R1 src=R0 off=1 imm=0x00000000/0                                goto to_or_from_host
      05000c0000000000 JumpA dst=R0 src=R0 off=12 imm=0x00000000/0                                     goto allowed_by_host_policy
to_or_from_host:
      7191640000000000 LoadReg8 dst=R1 src=R9 off=100 imm=0x00000000/0                                 R1 = *(u8 *)(R9 + 100) /* state->rules_hit */
      3501160020000000 JumpGEImm64 dst=R1 src=R0 off=22 imm=0x00000020/32                              goto deny
      bf12000000000000 Mov64 dst=R2 src=R1 off=0 imm=0x00000000/0
      0702000001000000 AddImm64 dst=R2 src=R0 off=0 imm=0x00000001/1
      7329640000000000 StoreReg8 dst=R9 src=R2 off=100 imm=0x00000000/0                                *(u8 *) (R9 + 100) /* state->rules_hit */ = R2
      6701000003000000 ShiftLImm64 dst=R1 src=R0 off=0 imm=0x00000003/3
      0701000068000000 AddImm64 dst=R1 src=R0 off=0 imm=0x00000068/104
      1802000000000000 LoadImm64 dst=R2 src=R0 off=0 imm=0x00000000/0
      0000000000000000 LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0x00000000/0
      0f91000000000000 Add64 dst=R1 src=R9 off=0 imm=0x00000000/0
      7b21000000000000 StoreReg64 dst=R1 src=R2 off=0 imm=0x00000000/0                                 *(u64 *) (R1 + 0) /*  */ = R2
      05000c0000000000 JumpA dst=R0 src=R0 off=12 imm=0x00000000/0                                     goto deny
// Start of rule action:"allow" rule_id:"8iYOzpfn3SU3eATK"
// count = 0
rule_0_no_match:
allowed_by_host_policy:
      7191640000000000 LoadReg8 dst=R1 src=R9 off=100 imm=0x00000000/0                                 R1 = *(u8 *)(R9 + 100) /* state->rules_hit */
      3501130020000000 JumpGEImm64 dst=R1 src=R0 off=19 imm=0x00000020/32                              goto allow
      bf12000000000000 Mov64 dst=R2 src=R1 off=0 imm=0x00000000/0
      0702000001000000 AddImm64 dst=R2 src=R0 off=0 imm=0x00000001/1
      7329640000000000 StoreReg8 dst=R9 src=R2 off=100 imm=0x00000000/0                                *(u8 *) (R9 + 100) /* state->rules_hit */ = R2
      6701000003000000 ShiftLImm64 dst=R1 src=R0 off=0 imm=0x00000003/3
      0701000068000000 AddImm64 dst=R1 src=R0 off=0 imm=0x00000068/104
      18020000d9e1f7ff LoadImm64 dst=R2 src=R0 off=0 imm=0xfff7e1d9/-532007
      0000000087e4c74d LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0x4dc7e487/1304945799
      0f91000000000000 Add64 dst=R1 src=R9 off=0 imm=0x00000000/0
      7b21000000000000 StoreReg64 dst=R1 src=R2 off=0 imm=0x00000000/0                                 *(u64 *) (R1 + 0) /*  */ = R2
      0500090000000000 JumpA dst=R0 src=R0 off=9 imm=0x00000000/0                                      goto allow
// End of rule 8iYOzpfn3SU3eATK
rule_2_no_match:
deny:
      b401000002000000 MovImm32 dst=R1 src=R0 off=0 imm=0x00000002/2
      6319540000000000 StoreReg32 dst=R9 src=R1 off=84 imm=0x00000000/0                                *(u32 *) (R9 + 84) /* state->pol_rc */ = R1
      bf61000000000000 Mov64 dst=R1 src=R6 off=0 imm=0x00000000/0
      1812000048000000 LoadImm64 dst=R2 src=R1 off=0 imm=0x00000048/72
      0000000000000000 LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0x00000000/0
      b403000005000000 MovImm32 dst=R3 src=R0 off=0 imm=0x00000005/5
      850000000c000000 Call dst=R0 src=R0 off=0 imm=0x0000000c/12                                      call bpf_tail_call
exit:
      b700000002000000 MovImm64 dst=R0 src=R0 off=0 imm=0x00000002/2
      9500000000000000 Exit dst=R0 src=R0 off=0 imm=0x00000000/0
allow:
      b401000001000000 MovImm32 dst=R1 src=R0 off=0 imm=0x00000001/1
      6319540000000000 StoreReg32 dst=R9 src=R1 off=84 imm=0x00000000/0                                *(u32 *) (R9 + 84) /* state->pol_rc */ = R1
      bf61000000000000 Mov64 dst=R1 src=R6 off=0 imm=0x00000000/0
      1812000048000000 LoadImm64 dst=R2 src=R1 off=0 imm=0x00000048/72
      0000000000000000 LoadImm64Pt2 dst=R0 src=R0 off=0 imm=0x00000000/0
      b403000003000000 MovImm32 dst=R3 src=R0 off=0 imm=0x00000003/3
      850000000c000000 Call dst=R0 src=R0 off=0 imm=0x0000000c/12                                      call bpf_tail_call
      b40100000a000000 MovImm32 dst=R1 src=R0 off=0 imm=0x0000000a/10
      6319540000000000 StoreReg32 dst=R9 src=R1 off=84 imm=0x00000000/0                                *(u32 *) (R9 + 84) /* state->pol_rc */ = R1
      b700000002000000 MovImm64 dst=R0 src=R0 off=0 imm=0x00000002/2
      9500000000000000 Exit dst=R0 src=R0 off=0 imm=0x00000000/0
2023-09-22 02:36:54.616 [ERROR][2259] confd/policy_debug.go 78: Failed to dump policy info. error=stat /var/run/calico/bpf/policy/calic440f455693_xdp_v4.json: no such file or directory

# ipset list
Name: cali40all-ipam-pools
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1048576 bucketsize 12 initval 0x1493c672
Size in memory: 504
References: 0
Number of entries: 1
Members:
240.0.0.0/12

Name: cali40masq-ipam-pools
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1048576 bucketsize 12 initval 0xc8f7e12b
Size in memory: 504
References: 0
Number of entries: 1
Members:
240.0.0.0/12

Name: cali40all-vxlan-net
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1048576 bucketsize 12 initval 0x15dc28ab
Size in memory: 696
References: 0
Number of entries: 5
Members:
172.21.147.151
172.21.147.152
172.21.147.153
172.21.147.154
172.21.147.155

Above information are dumped from calico-node pod which is running on Host 172.21.147.156. And I run a nginx pod on Host 172.21.147.156, too. Last, my http server is running on Host 172.21.147.156 and listening Port 9999 natively. (not pod)

Could you see anything wrong here?

Aslan-Liu on Sep 22, 2023

@tomastigera Upstream kubernetes (1.27.5) installed with kubespray. Will try to gather some logs according to your guide and get back to you.

spantazi on Sep 19, 2023