cilium: Cilium in DSR mode: Can't reach nodeport service outside cluster from remote node in self managed AWS cluster

Is there an existing issue for this?

I have searched the existing issues

What happened?

I cannot access a nodePort nginx service outside the cluster through nodePublicIpAddress:nodePort from any of the nodes that does not have the pod running locally in it. The connection always times out when the backend pod is on a remote node.

I am running kubernetes 1.27.3 with Cilium v1.14.0-snapshot.4 in a self managed dual stack k8s cluster installed using kubeadm in AWS. There are two ec2 worker nodes, with 1 control plane node, all in the same AWS region and availability zone (us-east-1a).

I deployed Cilium in strict kube-proxy replacement mode and also disabled ip source/destination checks in AWS for all 3 nodes

Here is the output of kubectl -n kube-system exec ds/cilium -- cilium status --verbose. It shows that all nodes and endpoints are reachable.

[ec2-user@ip-172-31-15-25 ~]$ kubectl -n kube-system exec ds/cilium -- cilium status --verbose
Defaulted container "cilium-agent" out of: cilium-agent, cilium-monitor, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.27 (v1.27.3) [linux/arm64]
Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [ens5 ipv4 ipv6 (Direct Routing)]
Host firewall:          Disabled
CNI Chaining:           none
Cilium:                 Ok   1.14.0-snapshot.4 (v1.14.0-snapshot.4-6c8db759)
NodeMonitor:            Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok   
IPAM:                   IPv4: 3/254 allocated from 10.10.0.0/24, IPv6: 3/18446744073709551614 allocated from fd10:800b:444f:2b00::/64
Allocated addresses:
  10.10.0.184 (router)
  10.10.0.187 (health)
  10.10.0.73 (kube-system/coredns-6fccb86bcb-5n4bj)
  fd10:800b:444f:2b00::26e3 (kube-system/coredns-6fccb86bcb-5n4bj)
  fd10:800b:444f:2b00::50be (health)
  fd10:800b:444f:2b00::f003 (router)
IPv4 BIG TCP:           Disabled
IPv6 BIG TCP:           Disabled
BandwidthManager:       Disabled
Host Routing:           BPF
Masquerading:           BPF       [ens5]   10.10.0.0/16 [IPv4: Enabled, IPv6: Enabled]
Clock Source for BPF:   jiffies   [100 Hz]
Controller Status:      24/24 healthy
  Name                                  Last success   Last error   Count   Message
  cilium-health-ep                      45s ago        never        0       no error   
  dns-garbage-collector-job             49s ago        never        0       no error   
  endpoint-1837-regeneration-recovery   never          never        0       no error   
  endpoint-572-regeneration-recovery    never          never        0       no error   
  endpoint-891-regeneration-recovery    never          never        0       no error   
  endpoint-gc                           49s ago        never        0       no error   
  ipcache-inject-labels                 46s ago        10m48s ago   0       no error   
  k8s-heartbeat                         19s ago        never        0       no error   
  link-cache                            1s ago         never        0       no error   
  metricsmap-bpf-prom-sync              4s ago         never        0       no error   
  resolve-identity-1837                 46s ago        never        0       no error   
  resolve-identity-572                  30s ago        never        0       no error   
  resolve-identity-891                  45s ago        never        0       no error   
  sync-host-ips                         46s ago        never        0       no error   
  sync-lb-maps-with-k8s-services        10m46s ago     never        0       no error   
  sync-policymap-1837                   10s ago        never        0       no error   
  sync-policymap-572                    10s ago        never        0       no error   
  sync-policymap-891                    10s ago        never        0       no error   
  sync-to-k8s-ciliumendpoint (1837)     6s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (572)      10s ago        never        0       no error   
  sync-to-k8s-ciliumendpoint (891)      5s ago         never        0       no error   
  sync-utime                            46s ago        never        0       no error   
  template-dir-watcher                  never          never        0       no error   
  write-cni-file                        10m49s ago     never        0       no error   
Proxy Status:            OK, ip 10.10.0.184, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range:   min 256, max 65535
Hubble:                  Ok   Current/Max Flows: 3911/4095 (95.51%), Flows/s: 6.01   Metrics: Disabled
KubeProxyReplacement Details:
  Status:                 Strict
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Socket LB Coverage:     Full
  Devices:                ens5 ipv4 ipv6 (Direct Routing)
  Mode:                   DSR
  Backend Selection:      Maglev (Table Size: 65521)
  Session Affinity:       Enabled
  Graceful Termination:   Enabled
  NAT46/64 Support:       Disabled
  XDP Acceleration:       Disabled
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767) 
  - LoadBalancer:   Enabled 
  - externalIPs:    Enabled 
  - HostPort:       Enabled
BPF Maps:   dynamic sizing: on (ratio: 0.002500)
  Name                          Size
  Auth                          524288
  Non-TCP connection tracking   65536
  TCP connection tracking       131072
  Endpoint policy               65535
  IP cache                      512000
  IPv4 masquerading agent       16384
  IPv6 masquerading agent       16384
  IPv4 fragmentation            8192
  IPv4 service                  65536
  IPv6 service                  65536
  IPv4 service backend          65536
  IPv6 service backend          65536
  IPv4 service reverse NAT      65536
  IPv6 service reverse NAT      65536
  Metrics                       1024
  NAT                           131072
  Neighbor table                131072
  Global policy                 16384
  Session affinity              65536
  Sock reverse NAT              65536
  Tunnel                        65536
Encryption:                                  Disabled        
Cluster health:                              3/3 reachable   (2023-06-21T17:25:31Z)
  Name                                       IP              Node        Endpoints
  master-node (localhost)   internalIPV4   reachable   reachable
  worker-1               internalIPIV4    reachable   reachable
  worker-2               internalIpV4    reachable   reachable

However, i can curl the nginx service from any of the 3 nodes internally using each node’s ip address and port

# this works from any node internally
curl nodeInternalIp:nodePort

Cilium Version

v1.14.0-snapshot.4

Kernel Version

6.1.29-50.88.amzn2023.aarch64

Kubernetes Version

1.27.3

Sysdump

cilium-sysdump-20230621-174034.zip

Relevant log output

No response

Anything else?

Here is the deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80

here is the command that i used to expose the service

kubectl expose deployment my-nginx --type=NodePort --port=80