ovn-kubernetes: NodePort service is not accessible

I have create a NodePort service “nginx-test” listening on port 80, see the service list:

# kubectl get svc --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP         4d
default       nginx-test   NodePort    10.105.116.76   <none>        80:80/TCP       2d1h
kube-system   kube-dns     ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP   4d

Kubernetes cluster rough info:

NODE            PHYSICAL IP
master1        172.16.126.202
master2        172.16.126.203
master3        172.16.126.204
node001       172.16.126.208
node002       172.16.126.209
...                   ...
node008       172.16.126.215

I can access it through the cluster ip -> http://10.105.116.76:80:

# curl -i http://10.105.116.76:80
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Sat, 02 Mar 2019 18:53:44 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Thu, 31 Jan 2019 23:37:45 GMT
Connection: keep-alive
ETag: "5c5386c9-264"
Accept-Ranges: bytes
...

But cannot access it through any physical ip of minion node (from outside the respective machine):

# curl -i http://172.16.126.209:80
curl: (7) Failed connect to 172.16.126.209:80; Connection timed out

The pod of service “nginx-test” is scheduled on node002 which ip address is 172.16.126.209.

Port listening on every node:

# netstat -tpln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4223/sshd           
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      3679/kubelet        
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      5543/kube-proxy     
tcp        0      0 127.0.0.1:43177         0.0.0.0:*               LISTEN      3679/kubelet        
tcp6       0      0 :::22                   :::*                    LISTEN      4223/sshd           
tcp6       0      0 :::10250                :::*                    LISTEN      3679/kubelet        
tcp6       0      0 :::80                   :::*                    LISTEN      5543/kube-proxy     
tcp6       0      0 :::10256                :::*                    LISTEN      5543/kube-proxy

The parameters of ovnkube on every minion node, node001 as an example: /usr/bin/ovnkube --init-node=node001 --init-gateways --nodeport --cluster-subnet=10.112.0.0/12 --service-cluster-ip-range=10.96.0.0/12 --config-file=/etc/openvswitch/ovn_k8s.conf

Load Balancers:

# ovn-nbctl lb-list
UUID                                    LB                  PROTO      VIP                 IPs
f5315290-f4e2-40e2-866d-015ca1f99cbd                        udp        10.96.0.10:53       10.112.0.3:53,10.112.1.3:53,10.112.2.3:53
5eb463e8-cc95-445c-a5e0-d78fad2a9be3                        tcp        10.105.116.76:80    10.112.4.4:80
                                                            tcp        10.96.0.10:53       10.112.0.3:53,10.112.1.3:53,10.112.2.3:53
                                                            tcp        10.96.0.1:443       172.16.126.202:6443,172.16.126.203:6443,172.16.126.204:6443

I have enabled ipvs for kube-proxy, see the service/real-server mapping of ipvs below, the pod ip is 10.112.4.4, scheduled on minion node which physical ip is 172.16.126.209:

# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.126.209:80 rr
  -> 10.112.4.4:80                Masq    1      0          0         
TCP  10.96.0.1:443 rr
  -> 172.16.126.202:6443          Masq    1      0          0         
  -> 172.16.126.203:6443          Masq    1      0          0         
  -> 172.16.126.204:6443          Masq    1      0          0         
TCP  10.96.0.10:53 rr
  -> 10.112.0.3:53                Masq    1      0          0         
  -> 10.112.1.3:53                Masq    1      0          0         
  -> 10.112.2.3:53                Masq    1      0          0         
TCP  10.105.116.76:80 rr
  -> 10.112.4.4:80                Masq    1      0          0         
TCP  10.112.4.2:80 rr
  -> 10.112.4.4:80                Masq    1      0          0         
TCP  127.0.0.1:80 rr
  -> 10.112.4.4:80                Masq    1      0          0         
UDP  10.96.0.10:53 rr
  -> 10.112.0.3:53                Masq    1      0          0         
  -> 10.112.1.3:53                Masq    1      0          0         
  -> 10.112.2.3:53                Masq    1      0          0

ipvs mapping on other minion nodes (172.16.126.208 as example):

# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.126.208:80 rr
TCP  10.96.0.1:443 rr
  -> 172.16.126.202:6443          Masq    1      0          0         
  -> 172.16.126.203:6443          Masq    1      0          0         
  -> 172.16.126.204:6443          Masq    1      0          0         
TCP  10.96.0.10:53 rr
  -> 10.112.0.3:53                Masq    1      0          0         
  -> 10.112.1.3:53                Masq    1      0          0         
  -> 10.112.2.3:53                Masq    1      0          0         
TCP  10.105.116.76:80 rr
  -> 10.112.4.4:80                Masq    1      0          0         
TCP  10.112.3.2:80 rr
TCP  127.0.0.1:80 rr
UDP  10.96.0.10:53 rr
  -> 10.112.0.3:53                Masq    1      0          0         
  -> 10.112.1.3:53                Masq    1      0          0         
  -> 10.112.2.3:53                Masq    1      0          0

dump-flows it seems correct:

# ovs-ofctl dump-flows breth0
 cookie=0x0, duration=7616.030s, table=0, n_packets=0, n_bytes=0, priority=100,ip,in_port="k8s-patch-breth" actions=ct(commit,zone=64000),output:eth0
 cookie=0x0, duration=7616.010s, table=0, n_packets=197515, n_bytes=59340278, priority=50,ip,in_port=eth0 actions=ct(table=1,zone=64000)
 cookie=0x0, duration=7615.871s, table=0, n_packets=5036, n_bytes=372664, priority=100,tcp,in_port=eth0,tp_dst=80 actions=output:"k8s-patch-breth"
 cookie=0x0, duration=7621.075s, table=0, n_packets=321302, n_bytes=26439356, priority=0 actions=NORMAL
 cookie=0x0, duration=7615.994s, table=1, n_packets=0, n_bytes=0, priority=100,ct_state=+est+trk actions=output:"k8s-patch-breth"
 cookie=0x0, duration=7615.968s, table=1, n_packets=0, n_bytes=0, priority=100,ct_state=+rel+trk actions=output:"k8s-patch-breth"
 cookie=0x0, duration=7615.954s, table=1, n_packets=197508, n_bytes=59246320, priority=0 actions=LOCAL

module openvswitch is loadded

# lsmod | grep openvswitch
openvswitch           131072  7 vport_geneve
nsh                    16384  1 openvswitch
nf_nat_ipv6            20480  1 openvswitch
nf_nat_ipv4            16384  3 ipt_MASQUERADE,openvswitch,iptable_nat
nf_conncount           24576  1 openvswitch
nf_nat                 36864  3 nf_nat_ipv6,nf_nat_ipv4,openvswitch
nf_conntrack          143360  9 xt_conntrack,nf_nat,nf_nat_ipv6,ipt_MASQUERADE,nf_nat_ipv4,openvswitch,nf_conntrack_netlink,nf_conncount,ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,openvswitch
libcrc32c              16384  5 nf_conntrack,nf_nat,openvswitch,xfs,ip_vs

# modinfo openvswitch
filename:       /lib/modules/4.20.5-1.el7.elrepo.x86_64/kernel/net/openvswitch/openvswitch.ko
alias:          net-pf-16-proto-16-family-ovs_ct_limit
alias:          net-pf-16-proto-16-family-ovs_meter
alias:          net-pf-16-proto-16-family-ovs_packet
alias:          net-pf-16-proto-16-family-ovs_flow
alias:          net-pf-16-proto-16-family-ovs_vport
alias:          net-pf-16-proto-16-family-ovs_datapath
license:        GPL
description:    Open vSwitch switching datapath
srcversion:     23AB11CC772ECA25E9A68AE
depends:        nf_conntrack,nf_nat,nf_conncount,libcrc32c,nf_nat_ipv6,nf_nat_ipv4,nf_defrag_ipv6,nsh
retpoline:      Y
intree:         Y
name:           openvswitch
vermagic:       4.20.5-1.el7.elrepo.x86_64 SMP mod_unload modversions 
# uname -a
Linux node001 4.20.5-1.el7.elrepo.x86_64 #1 SMP Sat Jan 26 10:55:51 EST 2019 x86_64 x86_64 x86_64 GNU/Linux

Version of Open vSwitch:

# ovs-appctl version
ovs-vswitchd (Open vSwitch) 2.10.1

Northbound database:

# ovn-nbctl show
switch 1bd13c40-e7ec-4844-aaf7-53ec0be482fa (ext_master2)
    port etor-GR_master2
        type: router
        addresses: ["32:56:c1:1e:fd:44"]
        router-port: rtoe-GR_master2
    port br-localnet_master2
        addresses: ["unknown"]
switch 99f97b71-5545-4411-be33-ff772ea7c251 (node003)
    port stor-node003
        type: router
        addresses: ["00:00:00:A6:08:27"]
        router-port: rtos-node003
    port k8s-node003
        addresses: ["76:07:89:4d:93:49 10.112.5.2"]
    port default_busybox-g8l9n
        addresses: ["0a:00:00:00:00:19 10.112.5.3"]
switch 4d19b592-7716-4d0a-8515-8af393b3ab6a (node005)
    port stor-node005
        type: router
        addresses: ["00:00:00:09:7A:04"]
        router-port: rtos-node005
    port default_busybox-kp7nt
        addresses: ["0a:00:00:00:00:1b 10.112.7.3"]
    port k8s-node005
        addresses: ["92:f9:d4:b0:da:4b 10.112.7.2"]
switch 635f0dc2-1412-4558-a6e7-57ab8d5b3703 (ext_node004)
    port etor-GR_node004
        type: router
        addresses: ["00:1d:d8:b7:1c:0e"]
        router-port: rtoe-GR_node004
    port breth0_node004
        addresses: ["unknown"]
switch d87fc4c2-41ef-4195-ac16-998962ebfc93 (master1)
    port stor-master1
        type: router
        addresses: ["00:00:00:BF:24:EE"]
        router-port: rtos-master1
    port kube-system_coredns-s5fxx
        addresses: ["0a:00:00:00:00:10 10.112.0.3"]
    port k8s-master1
        addresses: ["f6:94:d6:5c:65:01 10.112.0.2"]
    port default_busybox-fqvpj
        addresses: ["0a:00:00:00:00:1f 10.112.0.4"]
switch e1fb1473-f3aa-4220-ae45-14757c99bc36 (node002)
    port stor-node002
        type: router
        addresses: ["00:00:00:0D:A7:8C"]
        router-port: rtos-node002
    port default_busybox-fp82w
        addresses: ["0a:00:00:00:00:1e 10.112.4.3"]
    port k8s-node002
        addresses: ["22:9b:bf:19:94:fc 10.112.4.2"]
    port default_nginx-test-6489dfd864-sqcvl
        addresses: ["0a:00:00:00:00:0e 10.112.4.4"]
switch ef1b3093-199e-4350-a701-024a4e46df6c (ext_node006)
    port etor-GR_node006
        type: router
        addresses: ["00:1d:d8:b7:1c:10"]
        router-port: rtoe-GR_node006
    port breth0_node006
        addresses: ["unknown"]
switch dfac86c3-661f-4a14-89e4-c64e0d5d263a (node001)
    port k8s-node001
        addresses: ["3e:ee:84:10:ed:ab 10.112.3.2"]
    port stor-node001
        type: router
        addresses: ["00:00:00:AC:D7:4C"]
        router-port: rtos-node001
    port default_busybox-fjbjj
        addresses: ["0a:00:00:00:00:20 10.112.3.3"]
switch 4555adc4-6990-4806-876f-b9c24567e527 (ext_node007)
    port breth0_node007
        addresses: ["unknown"]
    port etor-GR_node007
        type: router
        addresses: ["00:1d:d8:b7:1c:11"]
        router-port: rtoe-GR_node007
switch 6d35cfb4-5152-404b-804a-03c00788e6e1 (node006)
    port k8s-node006
        addresses: ["82:98:25:84:12:25 10.112.8.2"]
    port default_busybox-hbxp9
        addresses: ["0a:00:00:00:00:1c 10.112.8.3"]
    port stor-node006
        type: router
        addresses: ["00:00:00:AD:FE:E7"]
        router-port: rtos-node006
switch d9f5f7f5-a982-4a82-a8bd-da6ec5a01c18 (ext_master3)
    port br-localnet_master3
        addresses: ["unknown"]
    port etor-GR_master3
        type: router
        addresses: ["3a:d6:a2:d6:b9:42"]
        router-port: rtoe-GR_master3
switch c590c876-767c-41cf-a35e-ca5aa183bfe0 (ext_node003)
    port etor-GR_node003
        type: router
        addresses: ["00:1d:d8:b7:1c:0d"]
        router-port: rtoe-GR_node003
    port breth0_node003
        addresses: ["unknown"]
switch a53504e6-0704-4221-a2fb-85ffccd34537 (join)
    port jtor-GR_node006
        type: router
        addresses: ["00:00:00:1C:7D:27"]
        router-port: rtoj-GR_node006
    port jtor-GR_node005
        type: router
        addresses: ["00:00:00:76:EB:3F"]
        router-port: rtoj-GR_node005
    port jtor-GR_node007
        type: router
        addresses: ["00:00:00:B8:72:11"]
        router-port: rtoj-GR_node007
    port jtor-GR_master1
        type: router
        addresses: ["00:00:00:FE:FB:9F"]
        router-port: rtoj-GR_master1
    port jtor-GR_master3
        type: router
        addresses: ["00:00:00:BB:2E:13"]
        router-port: rtoj-GR_master3
    port jtor-ovn_cluster_router
        type: router
        addresses: ["00:00:00:3D:3B:6B"]
        router-port: rtoj-ovn_cluster_router
    port jtor-GR_node004
        type: router
        addresses: ["00:00:00:BB:AE:54"]
        router-port: rtoj-GR_node004
    port jtor-GR_node001
        type: router
        addresses: ["00:00:00:AC:F2:C7"]
        router-port: rtoj-GR_node001
    port jtor-GR_node002
        type: router
        addresses: ["00:00:00:8F:28:28"]
        router-port: rtoj-GR_node002
    port jtor-GR_node003
        type: router
        addresses: ["00:00:00:BB:50:0F"]
        router-port: rtoj-GR_node003
    port jtor-GR_master2
        type: router
        addresses: ["00:00:00:16:67:B9"]
        router-port: rtoj-GR_master2
    port jtor-GR_node008
        type: router
        addresses: ["00:00:00:88:56:AD"]
        router-port: rtoj-GR_node008
switch 6904ae77-492a-4ba9-967c-743e94fc4c51 (ext_node002)
    port etor-GR_node002
        type: router
        addresses: ["00:1d:d8:b7:1c:0c"]
        router-port: rtoe-GR_node002
    port breth0_node002
        addresses: ["unknown"]
switch 8a7ec017-1e5c-497c-84a5-87fbe280b5cf (master3)
    port kube-system_coredns-52zkg
        addresses: ["0a:00:00:00:00:15 10.112.2.3"]
    port default_busybox-9tcsr
        addresses: ["0a:00:00:00:00:16 10.112.2.4"]
    port stor-master3
        type: router
        addresses: ["00:00:00:70:63:94"]
        router-port: rtos-master3
    port k8s-master3
        addresses: ["62:bc:ad:44:2a:ab 10.112.2.2"]
switch 506aad59-f535-49ed-9315-d3ab74086ff2 (node007)
    port default_busybox-hcfb7
        addresses: ["0a:00:00:00:00:1d 10.112.9.3"]
    port k8s-node007
        addresses: ["82:79:77:24:85:de 10.112.9.2"]
    port stor-node007
        type: router
        addresses: ["00:00:00:69:92:FA"]
        router-port: rtos-node007
switch 0e4d5919-fffa-4f63-ab5b-04c8cad37526 (ext_master1)
    port br-localnet_master1
        addresses: ["unknown"]
    port etor-GR_master1
        type: router
        addresses: ["e2:54:54:a3:36:41"]
        router-port: rtoe-GR_master1
switch d243c3a9-219d-4a49-9b5e-d9e6ae3f00d7 (node004)
    port stor-node004
        type: router
        addresses: ["00:00:00:16:F2:4B"]
        router-port: rtos-node004
    port k8s-node004
        addresses: ["7e:20:db:01:10:4d 10.112.6.2"]
    port default_busybox-ts5p8
        addresses: ["0a:00:00:00:00:18 10.112.6.3"]
switch d76f2b7c-366c-4066-80ab-aa7e112a2652 (node008)
    port default_busybox-wlpf2
        addresses: ["0a:00:00:00:00:1a 10.112.10.3"]
    port stor-node008
        type: router
        addresses: ["00:00:00:1B:25:DC"]
        router-port: rtos-node008
    port k8s-node008
        addresses: ["02:65:bb:66:c4:3b 10.112.10.2"]
switch 1bd18f1c-4285-4104-bd2b-b864870cc48b (ext_node001)
    port etor-GR_node001
        type: router
        addresses: ["00:1d:d8:b7:1c:0b"]
        router-port: rtoe-GR_node001
    port breth0_node001
        addresses: ["unknown"]
switch 8573c13e-4366-4712-8b74-4c36b65ab6a9 (ext_node005)
    port etor-GR_node005
        type: router
        addresses: ["00:1d:d8:b7:1c:0f"]
        router-port: rtoe-GR_node005
    port breth0_node005
        addresses: ["unknown"]
switch 48d3525e-c3d7-4932-bd31-6bd494b98ed3 (master2)
    port default_busybox-6s2p8
        addresses: ["0a:00:00:00:00:02 10.112.1.4"]
    port k8s-master2
        addresses: ["26:22:6b:1d:30:68 10.112.1.2"]
    port stor-master2
        type: router
        addresses: ["00:00:00:59:7C:A4"]
        router-port: rtos-master2
    port kube-system_coredns-shl6x
        addresses: ["0a:00:00:00:00:12 10.112.1.3"]
switch 7d1f8ccf-4014-4b2a-92e9-c29ea9e2f36c (ext_node008)
    port breth0_node008
        addresses: ["unknown"]
    port etor-GR_node008
        type: router
        addresses: ["00:1d:d8:b7:1c:12"]
        router-port: rtoe-GR_node008
router 22e18eb0-a4b2-4d53-aea7-0729fefe5287 (GR_node008)
    port rtoj-GR_node008
        mac: "00:00:00:88:56:AD"
        networks: ["100.64.1.6/24"]
    port rtoe-GR_node008
        mac: "00:1d:d8:b7:1c:12"
        networks: ["172.16.126.215/24"]
    nat 97b3c687-7fd6-4a98-87a6-f1aaae52936d
        external ip: "172.16.126.215"
        logical ip: "10.112.0.0/12"
        type: "snat"
router a06f75e1-9615-43f8-8a84-64f227d31d4a (GR_node007)
    port rtoj-GR_node007
        mac: "00:00:00:B8:72:11"
        networks: ["100.64.1.11/24"]
    port rtoe-GR_node007
        mac: "00:1d:d8:b7:1c:11"
        networks: ["172.16.126.214/24"]
    nat 98230eb7-f772-4cd9-8197-fd478028f688
        external ip: "172.16.126.214"
        logical ip: "10.112.0.0/12"
        type: "snat"
router 03721222-6562-4dc3-98fb-ba83c8cdc433 (GR_node004)
    port rtoj-GR_node004
        mac: "00:00:00:BB:AE:54"
        networks: ["100.64.1.8/24"]
    port rtoe-GR_node004
        mac: "00:1d:d8:b7:1c:0e"
        networks: ["172.16.126.211/24"]
    nat 06da05e0-056f-4e01-8299-19545a35d7b6
        external ip: "172.16.126.211"
        logical ip: "10.112.0.0/12"
        type: "snat"
router 435a4d1a-bc36-483e-935e-9d24488ffaf8 (GR_node005)
    port rtoe-GR_node005
        mac: "00:1d:d8:b7:1c:0f"
        networks: ["172.16.126.212/24"]
    port rtoj-GR_node005
        mac: "00:00:00:76:EB:3F"
        networks: ["100.64.1.9/24"]
    nat e447866d-d296-4987-ab4d-a35139963034
        external ip: "172.16.126.212"
        logical ip: "10.112.0.0/12"
        type: "snat"
router 5dc98cfc-d348-4ae7-98b5-b91733d51f0e (ovn_cluster_router)
    port rtos-master1
        mac: "00:00:00:BF:24:EE"
        networks: ["10.112.0.1/24"]
    port rtos-node002
        mac: "00:00:00:0D:A7:8C"
        networks: ["10.112.4.1/24"]
    port rtos-master2
        mac: "00:00:00:59:7C:A4"
        networks: ["10.112.1.1/24"]
    port rtos-node008
        mac: "00:00:00:1B:25:DC"
        networks: ["10.112.10.1/24"]
    port rtos-master3
        mac: "00:00:00:70:63:94"
        networks: ["10.112.2.1/24"]
    port rtos-node005
        mac: "00:00:00:09:7A:04"
        networks: ["10.112.7.1/24"]
    port rtos-node001
        mac: "00:00:00:AC:D7:4C"
        networks: ["10.112.3.1/24"]
    port rtoj-ovn_cluster_router
        mac: "00:00:00:3D:3B:6B"
        networks: ["100.64.1.1/24"]
    port rtos-node007
        mac: "00:00:00:69:92:FA"
        networks: ["10.112.9.1/24"]
    port rtos-node006
        mac: "00:00:00:AD:FE:E7"
        networks: ["10.112.8.1/24"]
    port rtos-node004
        mac: "00:00:00:16:F2:4B"
        networks: ["10.112.6.1/24"]
    port rtos-node003
        mac: "00:00:00:A6:08:27"
        networks: ["10.112.5.1/24"]
router dfc3c545-42a3-45c2-8190-bb3e115536ce (GR_node002)
    port rtoe-GR_node002
        mac: "00:1d:d8:b7:1c:0c"
        networks: ["172.16.126.209/24"]
    port rtoj-GR_node002
        mac: "00:00:00:8F:28:28"
        networks: ["100.64.1.5/24"]
    nat b68869c3-5bac-4fd9-a352-0199f0f4709f
        external ip: "172.16.126.209"
        logical ip: "10.112.0.0/12"
        type: "snat"
router aa04bf0e-9462-414f-98aa-2a0049246778 (GR_master1)
    port rtoe-GR_master1
        mac: "e2:54:54:a3:36:41"
        networks: ["169.254.33.2/24"]
    port rtoj-GR_master1
        mac: "00:00:00:FE:FB:9F"
        networks: ["100.64.1.2/24"]
    nat 67fe3714-11ff-4cd0-8b6a-dd70853a0fc7
        external ip: "169.254.33.2"
        logical ip: "10.112.0.0/12"
        type: "snat"
router 936ea4ce-08fd-4c1c-89ab-00406698f9db (GR_node001)
    port rtoe-GR_node001
        mac: "00:1d:d8:b7:1c:0b"
        networks: ["172.16.126.208/24"]
    port rtoj-GR_node001
        mac: "00:00:00:AC:F2:C7"
        networks: ["100.64.1.4/24"]
    nat f83d01d5-bb8f-4571-ba5b-5325f4f4ccde
        external ip: "172.16.126.208"
        logical ip: "10.112.0.0/12"
        type: "snat"
router 665ac94e-d920-422f-b5e9-97a7910caa4e (GR_master3)
    port rtoe-GR_master3
        mac: "3a:d6:a2:d6:b9:42"
        networks: ["169.254.33.2/24"]
    port rtoj-GR_master3
        mac: "00:00:00:BB:2E:13"
        networks: ["100.64.1.12/24"]
    nat 6a437615-0e4a-464c-a672-772dcf34699a
        external ip: "169.254.33.2"
        logical ip: "10.112.0.0/12"
        type: "snat"
router c4b0216f-6b1a-4e5f-b3dd-ecdc45cd049c (GR_node006)
    port rtoj-GR_node006
        mac: "00:00:00:1C:7D:27"
        networks: ["100.64.1.10/24"]
    port rtoe-GR_node006
        mac: "00:1d:d8:b7:1c:10"
        networks: ["172.16.126.213/24"]
    nat f12911f3-1baa-4c91-8eda-0d1e6615a113
        external ip: "172.16.126.213"
        logical ip: "10.112.0.0/12"
        type: "snat"
router 635f9bdf-7aa7-497e-bb17-2df0f6dbc683 (GR_master2)
    port rtoj-GR_master2
        mac: "00:00:00:16:67:B9"
        networks: ["100.64.1.3/24"]
    port rtoe-GR_master2
        mac: "32:56:c1:1e:fd:44"
        networks: ["169.254.33.2/24"]
    nat 67804c78-1c93-4441-ad40-997fdb953dd6
        external ip: "169.254.33.2"
        logical ip: "10.112.0.0/12"
        type: "snat"
router 4e18b9d7-f6fb-4deb-890b-f49621053203 (GR_node003)
    port rtoj-GR_node003
        mac: "00:00:00:BB:50:0F"
        networks: ["100.64.1.7/24"]
    port rtoe-GR_node003
        mac: "00:1d:d8:b7:1c:0d"
        networks: ["172.16.126.210/24"]
    nat 42de066b-a272-48b2-a780-07fe5469ca4f
        external ip: "172.16.126.210"
        logical ip: "10.112.0.0/12"
        type: "snat"

I have followed @shettyg 's instruction that mentioned int the thread #405 to delete the addtional ports from logical switch “ext_XXXX”, it still not work, and the addional port with unknown addresses will be created back after ovnkube restart

ovn-nbctl lsp-del breth0_node001 && \
ovn-nbctl lsp-del breth0_node002 && \
ovn-nbctl lsp-del breth0_node003 && \
ovn-nbctl lsp-del breth0_node004 && \
ovn-nbctl lsp-del breth0_node005 && \
ovn-nbctl lsp-del breth0_node006 && \
ovn-nbctl lsp-del breth0_node007 && \
ovn-nbctl lsp-del breth0_node008

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (6 by maintainers)

Commits related to this issue

Most upvoted comments

Like @shettyg said, we don’t need kube-proxy for any of the gateway modes we support. So, I would remove kube-proxy like below

kubectl delete ds -n kube-system kube-proxy

Is your kube-proxy running? If so, that may be adding them. ovn-kubernetes will only add openflow flows if you use br-localnet for N/S traffic.