cilium: Intermittent 504 error from service when using ingress network policy.

Bug report

General Information

  • Cilium 1.8.2 (Installed via Helm)
  • EKS 1.17, v1.17.9-eks-4c6976
  • Kernel version : 5.4.58-27.104.amzn2.x86_64
  • Orchestration system version in use kubectl version: v1.17.4

How to reproduce the issue

  1. Apply the following Ingress network policy:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: service-1
  labels:
    app.kubernetes.io/name: service-1
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: service-1
  ingress:
  - fromEndpoints:
    - matchLabels:
        app.kubernetes.io/name: service-2
  - fromEndpoints:
    - matchLabels:
        app.kubernetes.io/name: service-3
  - fromEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": system
        "k8s:app": nginx-ingress
  - fromEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": ambassador
        "k8s:service": ambassador
  1. Curl service-1 through the ingress controller
curl "https://service-1.example.com"
# The curl intermittently fails with 504 from nginx-ingress
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>

Additional Logs

  1. The logs on the endpoint show that it is continuously regenerate about every minute or so
$ kubectl -n kube-system exec -ti cilium-bg6rc -- cilium endpoint log 1677
2020-09-21T16:59:05Z   OK       ready                   Successfully regenerated endpoint program (Reason: one or more identities created or deleted)
2020-09-21T16:59:05Z   OK       ready                   Completed endpoint regeneration with no pending regeneration requests[
2020-09-21T16:59:05Z   OK       regenerating            Regenerating endpoint: one or more identities created or deleted
2020-09-21T16:59:05Z   OK       waiting-to-regenerate   Triggering endpoint regeneration due to one or more identities created or deleted
2020-09-21T16:58:00Z   OK       ready                   Successfully regenerated endpoint program (Reason: one or more identities created or deleted)
2020-09-21T16:58:00Z   OK       ready                   Completed endpoint regeneration with no pending regeneration requestst
  1. Logs from the endpoint where it intermittently denies traffic.
kubectl -n kube-system exec -ti cilium-bg6rc -- cilium monitor --related-to 1677
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> stack flow 0xa20dcc25 identity 4043->1 state new ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52530 -> 10.20.170.97:8126 tcp SYN
-> endpoint 1677 flow 0x4ef1a2 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52530 tcp SYN, ACK
-> stack flow 0xa20dcc25 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52530 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0xa20dcc25 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52530 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0x4ef1a2 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52530 tcp ACK, FIN
-> stack flow 0xa20dcc25 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52530 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0x4ef1a2 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52530 tcp ACK
-> stack flow 0xa20dcc25 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52530 -> 10.20.170.97:8126 tcp ACK, FIN
Policy verdict log: flow 0x462b02c9 local EP ID 1677, remote ID 1, dst port 8080, proto 6,ingress true, action allow, match L3-Only, 10.20.170.97:36570 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0x462b02c9 identity 1->4043 state new ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36570 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0xea0489c2 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36570 tcp SYN, ACK
-> endpoint 1677 flow 0x462b02c9 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36570 -> 10.20.162.79:8080 tcp ACK
-> endpoint 1677 flow 0x462b02c9 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36570 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0xea0489c2 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36570 tcp ACK
-> endpoint 1677 flow 0x462b02c9 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36570 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0xea0489c2 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36570 tcp ACK, FIN
-> endpoint 1677 flow 0x462b02c9 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36570 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0x4cfd8ebd identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.166.94:36752 tcp ACK, FIN
-> endpoint 1677 flow 0x0 identity 3070->4043 state established ifindex 0 orig-ip 10.20.166.94: 10.20.166.94:36752 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0x22a8c2c1 identity 4043->1 state new ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52566 -> 10.20.170.97:8126 tcp SYN
-> endpoint 1677 flow 0x2efe2165 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52566 tcp SYN, ACK
-> stack flow 0x22a8c2c1 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52566 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0x22a8c2c1 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52566 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0x2efe2165 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52566 tcp ACK
-> stack flow 0x22a8c2c1 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52566 -> 10.20.170.97:8126 tcp ACK, FIN
-> endpoint 1677 flow 0x2efe2165 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52566 tcp ACK, FIN
-> stack flow 0x22a8c2c1 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52566 -> 10.20.170.97:8126 tcp ACK
Policy verdict log: flow 0xd7609e01 local EP ID 1677, remote ID 1, dst port 8080, proto 6,ingress true, action allow, match L3-Only, 10.20.170.97:36580 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0xd7609e01 identity 1->4043 state new ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36580 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0xcf997519 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36580 tcp SYN, ACK
-> endpoint 1677 flow 0xd7609e01 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36580 -> 10.20.162.79:8080 tcp ACK
-> endpoint 1677 flow 0xd7609e01 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36580 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0xcf997519 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36580 tcp ACK
-> endpoint 1677 flow 0xd7609e01 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36580 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0xcf997519 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36580 tcp ACK, FIN
-> endpoint 1677 flow 0xd7609e01 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36580 -> 10.20.162.79:8080 tcp ACK
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 3070, dst port 8080, proto 6, ingress true, action allow, match L3-Only, 10.20.155.107:39038 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0x0 identity 3070->4043 state new ifindex 0 orig-ip 10.20.155.107: 10.20.155.107:39038 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0x4fcf178c identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.155.107:39038 tcp SYN, ACK
-> endpoint 1677 flow 0x0 identity 3070->4043 state established ifindex 0 orig-ip 10.20.155.107: 10.20.155.107:39038 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0x4fcf178c identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.155.107:39038 tcp ACK
-> stack flow 0x4bfcc551 identity 4043->1 state new ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52578 -> 10.20.170.97:8126 tcp SYN
-> endpoint 1677 flow 0xa237e2f identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52578 tcp SYN, ACK
-> stack flow 0x4bfcc551 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52578 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0x4bfcc551 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52578 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0xa237e2f identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52578 tcp ACK
-> stack flow 0x4bfcc551 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52578 -> 10.20.170.97:8126 tcp ACK, FIN
-> endpoint 1677 flow 0xa237e2f identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52578 tcp ACK, FIN
-> stack flow 0x4bfcc551 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52578 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0x530e8847 identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.156.189:57990 tcp ACK, FIN
-> endpoint 1677 flow 0x0 identity 3070->4043 state established ifindex 0 orig-ip 10.20.156.189: 10.20.156.189:57990 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0x4fcf178c identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.155.107:39038 tcp ACK, FIN
-> endpoint 1677 flow 0x0 identity 3070->4043 state established ifindex 0 orig-ip 10.20.155.107: 10.20.155.107:39038 -> 10.20.162.79:8080 tcp ACK, FIN
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 3070, dst port 8080, proto 6, ingress true, action allow, match L3-Only, 10.20.130.149:52026 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0x0 identity 3070->4043 state new ifindex 0 orig-ip 10.20.130.149: 10.20.130.149:52026 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0x6bb90b5c identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.130.149:52026 tcp SYN, ACK
-> endpoint 1677 flow 0x0 identity 3070->4043 state established ifindex 0 orig-ip 10.20.130.149: 10.20.130.149:52026 -> 10.20.162.79:8080 tcp ACK
-> endpoint 1677 flow 0x0 identity 3070->4043 state established ifindex 0 orig-ip 10.20.130.149: 10.20.130.149:52026 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0xacc19236 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:48943 -> 10.20.170.97:8125 udp
-> stack flow 0x6bb90b5c identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.130.149:52026 tcp ACK
-> stack flow 0xc8928840 identity 4043->1 state new ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52658 -> 10.20.170.97:8126 tcp SYN
-> endpoint 1677 flow 0xcf6d307b identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52658 tcp SYN, ACK
-> stack flow 0xc8928840 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52658 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0xc8928840 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52658 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0xcf6d307b identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52658 tcp ACK
-> stack flow 0xc8928840 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52658 -> 10.20.170.97:8126 tcp ACK, FIN
-> endpoint 1677 flow 0xcf6d307b identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52658 tcp ACK, FIN
-> stack flow 0xc8928840 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52658 -> 10.20.170.97:8126 tcp ACK
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 2, dst port 8080, proto 6, ingress true, action deny, match none, 10.20.136.209:49656 -> 10.20.162.79:8080 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 1677, identity 2->4043: 10.20.136.209:49656 -> 10.20.162.79:8080 tcp SYN
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 2, dst port 8080, proto 6, ingress true, action deny, match none, 10.20.136.209:49656 -> 10.20.162.79:8080 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 1677, identity 2->4043: 10.20.136.209:49656 -> 10.20.162.79:8080 tcp SYN
Policy verdict log: flow 0xebfe50de local EP ID 1677, remote ID 1, dst port 8080, proto 6,ingress true, action allow, match L3-Only, 10.20.170.97:36702 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0xebfe50de identity 1->4043 state new ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36702 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0xc52ba1d3 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36702 tcp SYN, ACK
-> endpoint 1677 flow 0xebfe50de identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36702 -> 10.20.162.79:8080 tcp ACK
-> endpoint 1677 flow 0xebfe50de identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36702 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0xc52ba1d3 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36702 tcp ACK
-> endpoint 1677 flow 0xebfe50de identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36702 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0xc52ba1d3 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36702 tcp ACK, FIN
-> endpoint 1677 flow 0xebfe50de identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36702 -> 10.20.162.79:8080 tcp ACK
Policy verdict log: flow 0x4b224429 local EP ID 1677, remote ID 1, dst port 8080, proto 6,ingress true, action allow, match L3-Only, 10.20.170.97:36716 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0x4b224429 identity 1->4043 state new ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36716 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0xdf3dcf59 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36716 tcp SYN, ACK
-> endpoint 1677 flow 0x4b224429 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36716 -> 10.20.162.79:8080 tcp ACK
-> endpoint 1677 flow 0x4b224429 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36716 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0xc4617dcf identity 4043->1 state new ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52706 -> 10.20.170.97:8126 tcp SYN
-> endpoint 1677 flow 0xca199e63 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52706 tcp SYN, ACK
-> stack flow 0xc4617dcf identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52706 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0xdf3dcf59 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36716 tcp ACK
-> stack flow 0xc4617dcf identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52706 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0xca199e63 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52706 tcp ACK
-> endpoint 1677 flow 0x4b224429 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36716 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0xc4617dcf identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52706 -> 10.20.170.97:8126 tcp ACK, FIN
-> endpoint 1677 flow 0xca199e63 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52706 tcp ACK, FIN
-> stack flow 0xc4617dcf identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52706 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0xdf3dcf59 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36716 tcp ACK, FIN
-> endpoint 1677 flow 0x4b224429 identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36716 -> 10.20.162.79:8080 tcp ACK
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 2, dst port 8080, proto 6, ingress true, action deny, match none, 10.20.136.209:49656 -> 10.20.162.79:8080 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 1677, identity 2->4043: 10.20.136.209:49656 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0x6bb90b5c identity 4043->3070 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.130.149:52026 tcp ACK, FIN
-> endpoint 1677 flow 0x0 identity 3070->4043 state established ifindex 0 orig-ip 10.20.130.149: 10.20.130.149:52026 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0xfdfc1bcc identity 4043->1 state new ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52714 -> 10.20.170.97:8126 tcp SYN
-> endpoint 1677 flow 0x46a9eb8e identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52714 tcp SYN, ACK
-> stack flow 0xfdfc1bcc identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52714 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0xfdfc1bcc identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52714 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0x46a9eb8e identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52714 tcp ACK
-> stack flow 0xfdfc1bcc identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52714 -> 10.20.170.97:8126 tcp ACK, FIN
-> endpoint 1677 flow 0x46a9eb8e identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52714 tcp ACK, FIN
-> stack flow 0xfdfc1bcc identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52714 -> 10.20.170.97:8126 tcp ACK
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 2, dst port 8080, proto 6, ingress true, action deny, match none, 10.20.136.209:49756 -> 10.20.162.79:8080 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 1677, identity 2->4043: 10.20.136.209:49756 -> 10.20.162.79:8080 tcp SYN
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 2, dst port 8080, proto 6, ingress true, action deny, match none, 10.20.136.209:49756 -> 10.20.162.79:8080 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 1677, identity 2->4043: 10.20.136.209:49756 -> 10.20.162.79:8080 tcp SYN
Policy verdict log: flow 0xb2081fac local EP ID 1677, remote ID 1, dst port 8080, proto 6,ingress true, action allow, match L3-Only, 10.20.170.97:36844 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0xb2081fac identity 1->4043 state new ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36844 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0xc9572f21 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36844 tcp SYN, ACK
-> endpoint 1677 flow 0xb2081fac identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36844 -> 10.20.162.79:8080 tcp ACK
-> endpoint 1677 flow 0xb2081fac identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36844 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0xacc19236 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:48943 -> 10.20.170.97:8125 udp
-> stack flow 0xc9572f21 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36844 tcp ACK
-> endpoint 1677 flow 0xb2081fac identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36844 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0xc9572f21 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36844 tcp ACK, FIN
-> endpoint 1677 flow 0xb2081fac identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36844 -> 10.20.162.79:8080 tcp ACK
Policy verdict log: flow 0x5c410bdd local EP ID 1677, remote ID 1, dst port 8080, proto 6,ingress true, action allow, match L3-Only, 10.20.170.97:36854 -> 10.20.162.79:8080 tcp SYN
-> endpoint 1677 flow 0x5c410bdd identity 1->4043 state new ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36854 -> 10.20.162.79:8080 tcp SYN
-> stack flow 0x79d6f617 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36854 tcp SYN, ACK
-> endpoint 1677 flow 0x5c410bdd identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36854 -> 10.20.162.79:8080 tcp ACK
-> endpoint 1677 flow 0x5c410bdd identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36854 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0x79d6f617 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36854 tcp ACK
-> endpoint 1677 flow 0x5c410bdd identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36854 -> 10.20.162.79:8080 tcp ACK, FIN
-> stack flow 0x79d6f617 identity 4043->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:8080 -> 10.20.170.97:36854 tcp ACK, FIN
-> endpoint 1677 flow 0x5c410bdd identity 1->4043 state established ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:36854 -> 10.20.162.79:8080 tcp ACK
-> stack flow 0x21c0cd14 identity 4043->1 state new ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52844 -> 10.20.170.97:8126 tcp SYN
-> endpoint 1677 flow 0x28621b06 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52844 tcp SYN, ACK
-> stack flow 0x21c0cd14 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52844 -> 10.20.170.97:8126 tcp ACK
-> stack flow 0x21c0cd14 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52844 -> 10.20.170.97:8126 tcp ACK
-> endpoint 1677 flow 0x28621b06 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52844 tcp ACK
-> stack flow 0x21c0cd14 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52844 -> 10.20.170.97:8126 tcp ACK, FIN
-> endpoint 1677 flow 0x28621b06 identity 2->4043 state reply ifindex 0 orig-ip 10.20.170.97: 10.20.170.97:8126 -> 10.20.162.79:52844 tcp ACK, FIN
-> stack flow 0x21c0cd14 identity 4043->1 state established ifindex 0 orig-ip 0.0.0.0: 10.20.162.79:52844 -> 10.20.170.97:8126 tcp ACK
Policy verdict log: flow 0x0 local EP ID 1677, remote ID 2, dst port 8080, proto 6, ingress true, action deny, match none, 10.20.136.209:49756 -> 10.20.162.79:8080 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 1677, identity 2->4043: 10.20.136.209:49756 -> 10.20.162.79:8080 tcp SYN

Notice: that ip 10.20.136.209 gets denied while the other ips are allowed. This ip is the pod ip for one of the nginx-ingress controller pods. I have compared the endpoints for the ningx-ingress pods that are allowed and this one and found that their identities are exactly the same.

k get -n system cep
NAME                                             ENDPOINT ID   IDENTITY ID   INGRESS ENFORCEMENT   EGRESS ENFORCEMENT   VISIBILITY POLICY   ENDPOINT STATE   IPV4            IPV6
nginx-ingress-controller-955456df-27cdr          626           3070                                                                         ready            10.20.166.63
nginx-ingress-controller-955456df-2zssb          862           3070                                                                         ready            10.20.166.94
nginx-ingress-controller-955456df-8nsfx          391           3070                                                                         ready            10.20.136.209
nginx-ingress-controller-955456df-ghrm2          330           3070                                                                         ready            10.20.156.189
nginx-ingress-controller-955456df-sbrrf          1227          3070                                                                         ready            10.20.155.107
nginx-ingress-controller-955456df-wsh2x          370           3070                                                                         ready            10.20.130.149
nginx-ingress-default-backend-5b48879958-ljv96   2069          12618                                                                        ready            10.20.134.252

Here is the identity of the endpoint getting denied:

k get -n system cep nginx-ingress-controller-955456df-8nsfx -ojson | jq -rM ".status.identity"
{
  "id": 3070,
  "labels": [
    "k8s:app.kubernetes.io/component=controller",
    "k8s:app=nginx-ingress",
    "k8s:component=controller",
    "k8s:io.cilium.k8s.policy.cluster=default",
    "k8s:io.cilium.k8s.policy.serviceaccount=nginx-ingress",
    "k8s:io.kubernetes.pod.namespace=system",
    "k8s:release=nginx-ingress"
  ]
}

and here is the identity of one of the endpoint that are getting allowed

k get -n system cep nginx-ingress-controller-955456df-sbrrf  -ojson | jq -rM ".status.identity"
{
  "id": 3070,
  "labels": [
    "k8s:app.kubernetes.io/component=controller",
    "k8s:app=nginx-ingress",
    "k8s:component=controller",
    "k8s:io.cilium.k8s.policy.cluster=default",
    "k8s:io.cilium.k8s.policy.serviceaccount=nginx-ingress",
    "k8s:io.kubernetes.pod.namespace=system",
    "k8s:release=nginx-ingress"
  ]
}

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 16 (12 by maintainers)

Most upvoted comments

@YesemKebede this PR will fix the issue https://github.com/cilium/cilium/pull/13592. Thanks for submitting the bug report.

+2
aanm on Oct 15, 2020

@alex1989hu can you open a new GH issue with a sysdump? It might be a different issue.

+1
aanm on Dec 4, 2020

This is fixed.

I can verify that this is still an issue with v1.9.0. I see in hubble that a Pod called argocd-redis considered as world/no category from multiple pods within the same namespace. Pod deletion or kill also does not help it. I don’t know what triggers to get into this situtation, but everything goes to green if I remove the CNP.

Timestamp 2020-12-04T16:50:17.057Z Verdict dropped Drop reason Policy denied Traffic direction egress TCP flags SYN Source pod argocd-repo-server-75ff7d8b4c-l5k2v Source identity 18696 Source labels name=argocd-repo-server io.cilium.k8s.policy.cluster=default io.cilium.k8s.policy.serviceaccount=default namespace=argocd Source IP 10.0.4.163 Destination pod argocd-redis-678b9dd489-xlzw9 Destination identity 52549 Destination IP 10.0.0.237 Destination port 6379

$ kubectl -n argocd describe pod argocd-redis-678b9dd489-xlzw9
Name:         argocd-redis-678b9dd489-xlzw9
Namespace:    argocd
Priority:     0
Node:         bucaw250-9/10.135.250.9
Start Time:   Fri, 04 Dec 2020 16:39:58 +0000
Labels:       app.kubernetes.io/name=argocd-redis
              pod-template-hash=678b9dd489
Annotations:  kubernetes.io/psp: privileged
Status:       Running
IP:           10.0.0.237
IPs:
  IP:           10.0.0.237
Controlled By:  ReplicaSet/argocd-redis-678b9dd489
...

ciliumv190world

+1
alex1989hu on Dec 4, 2020

@YesemKebede no, it will be part of v1.8.5. However, you can already test this if you use the image docker.io/cilium/cilium-operator:v1.8 since it only affects the operator and that docker tag already contains the changes.

+1
aanm on Oct 21, 2020

@YesemKebede can you take a system dump for both nodes that have the nginx and the one that ran the job? https://docs.cilium.io/en/v1.8/operations/troubleshooting/#automatic-log-state-collection

You can send it to me over slack.

+1
aanm on Oct 15, 2020

@YesemKebede sorry, I thought I had left a message here. Can you try 1.8.4 to see if you hit this issue? Thanks!

+1
aanm on Oct 8, 2020
Read more comments on GitHub
← cilium: Ingress: incompatible with cert-manager ACME HTTP-01
cilium: Intermittent report of dropped logs from hubble. →