ingress-nginx: tcp-services configMap add one configuration , Other services that are accessed through tcp cannot be accessed properly

kubectl version 1.23.6, nginx-ingress-controller image version v1.3.0

What happened: I added a piece of data to the tcp-services configMap,The ipv6 dual-stack nginx service is enabled,then ran the curl nodeIp:port command to get the normal service return data. However, when I added another piece of data to the configMap of tcp-services(ipv6 dual stack is also enabled for nginx services), I looped through shell.sh to access the previously added services and got an abnormal return,It took about 30 seconds to get the normal service result.

What you expected to happen: When I add configuration data to the configMap of tcp-services, I do not affect access to other services that are also configured through the configMap of tcp-services

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller Release: v1.3.0 Build: 2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.19.10

Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.6”, GitCommit:“ad3338546da947756e8a88aa6822e9c11e7eac22”, GitTreeState:“clean”, BuildDate:“2022-04-14T08:49:13Z”, GoVersion:“go1.17.9”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.6”, GitCommit:“ad3338546da947756e8a88aa6822e9c11e7eac22”, GitTreeState:“clean”, BuildDate:“2022-04-14T08:43:11Z”, GoVersion:“go1.17.9”, Compiler:“gc”, Platform:“linux/amd64”}

Environment: NAME=“CentOS Linux” VERSION=“7 (Core)” ID=“centos” ID_LIKE=“rhel fedora” VERSION_ID=“7” PRETTY_NAME=“CentOS Linux 7 (Core)” ANSI_COLOR=“0;31” CPE_NAME=“cpe:/o:centos:centos:7” HOME_URL=“https://www.centos.org/” BUG_REPORT_URL=“https://bugs.centos.org/

CENTOS_MANTISBT_PROJECT=“CentOS-7” CENTOS_MANTISBT_PROJECT_VERSION=“7” REDHAT_SUPPORT_PRODUCT=“centos” REDHAT_SUPPORT_PRODUCT_VERSION=“7”

Linux 10.19.38.223 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

  • How was the ingress-nginx-controller installed:
    • If helm was used then please show output of helm ls -A | grep -i ingress
    • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
    • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
    • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

[root@10 sh]# kubectl describe ingressclasses Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.3.0 Annotations: <none> Controller: k8s.io/ingress-nginx Events: <none> [root@10 sh]# [root@10 sh]# kubectl get po -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-629dv 0/1 Completed 0 62d ingress-nginx-admission-patch-4rlht 0/1 Completed 0 62d ingress-nginx-controller-4bfc6 1/1 Running 1 (43d ago) 50d ingress-nginx-controller-cghx9 1/1 Running 1 (43d ago) 43d

shell script content: #!/bin/bash for I in {1…50}; do
curl -v http://x.x.x.x:31687 let SUM=$I echo “$SUM” sleep 1 done

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 22 (11 by maintainers)

Most upvoted comments

Thanks for your update.

But please format the contents, you can refer to this documentation https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#fenced-code-blocks

ok ,I’ve formatted the content

I re-used two pod of mysql and the corresponding service with ipv6 enabled for testing. The testing procedure and test results are as follows

step1. i create two deployment and service (with ipv6), the mirror is MySQL , below is the resource object

[root@10 ingress-ipv6-svc]# kubectl get all |grep mysql
pod/ssx-mysql-dm-2-d8977d7cd-gs566        1/1     Running            0                 2m3s
pod/ssx-mysql-dm-789c4dd484-rc9dv         1/1     Running            2 (4d23h ago)     4d23h
service/ssx-mysql-sv      NodePort    2001:db8:40:1::acc7   <none>        9000:31090/TCP   4d23h
service/ssx-mysql-sv-2    NodePort    2001:db8:40:1::ac8f   <none>        9001:31091/TCP   5h3m
deployment.apps/ssx-mysql-dm         1/1     1            1           4d23h
deployment.apps/ssx-mysql-dm-2       1/1     1            1           5h3m
replicaset.apps/ssx-mysql-dm-2-d8977d7cd        1         1         1       5h3m
replicaset.apps/ssx-mysql-dm-789c4dd484         1         1         1       4d23h

step2. add two pieces of data to the configMap of tcp-services

[root@10 ingress-ipv6-svc]# kubectl get cm -n ingress-nginx tcp-services -o yaml
apiVersion: v1
data:
  "9090": monitoring/prometheus-k8s:9090
  "32001": default/ssx-mysql-sv:9000
  "32002": default/ssx-mysql-sv-2:9001
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"8888":"a/b:9090","9090":"monitoring/prometheus-k8s:9090"},"kind":"ConfigMap","metadata":{"annotations":{},"creationTimestamp":"2023-03-08T10:01:30Z","labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"tcp-services","namespace":"ingress-nginx","resourceVersion":"48942888","selfLink":"/api/v1/namespaces/ingress-nginx/configmaps/tcp-services","uid":"106b64b3-f596-4e2c-8014-56c3ac43fa02"}}
  creationTimestamp: "2023-03-08T10:01:30Z"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: tcp-services
  namespace: ingress-nginx
  resourceVersion: "72771677"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/tcp-services
  uid: 106b64b3-f596-4e2c-8014-56c3ac43fa02
[root@10 ingress-ipv6-svc]#

step3. use IP of the node where the pod of nginx ingress controller is located,with the configured tcp host port,to access the created service . I was able to log into MySQL successfully and log out safely

[root@10 ingress-ipv6-svc]# mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 737
Server version: 5.7.20 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [sys]> exit
Bye
[root@10 ingress-ipv6-svc]#

step4. I tried to log in and out of mysql multiple times using a shell script, using the host ip and port,And execute ./test-1-6-mysql-demo.sh .the script content and execution result are as follows

[root@10 ingress-ipv6-svc]# cat test-1-6-mysql-demo.sh
#!/usr/bin/expect
 for {set i 0} {$i < 10} {incr i} {
        puts " $i"
        spawn  mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
        expect "MySQL"
        send "exit"
        sleep 2
    }
[root@10 ingress-ipv6-svc]# ./test-1-6-mysql-demo.sh
 0
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 872
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [sys]>  1
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 873
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

 2
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 874
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 3
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 875
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [sys]>  4
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 876
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

 5
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 877
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [sys]>  6
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 878
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

 7
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 879
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 8
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 880
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 9
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 881
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

[root@10 ingress-ipv6-svc]#

step5. I tried to delete another mysql pod created during the execution of the shell script ,and found many errors in the shell script execution results

[root@10 ingress-ipv6-svc]# ./test-1-6-mysql-demo.sh
 0
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 882
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [sys]>  1
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 883
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [sys]>  2
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
 3
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
 4
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
 5
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
 6
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
 7
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
 8
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
 9
spawn mysql -h fd69:1:10:f980::a13:26e0 -P32001 -uroot -p1 sys
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
[root@10 ingress-ipv6-svc]#


Viewing the ingress-nginx pod also has a lot of error messages

W0601 03:05:56.828406       7 controller.go:456] Service "default/ssx-mysql-sv-2" does not have any active Endpoint for TCP port 9001
W0601 03:05:56.828580       7 controller.go:423] Error getting Service "foot/nginx-ns": no object matching key "foot/nginx-ns" in local store
I0601 03:05:56.829402       7 controller.go:167] "Configuration changes detected, backend reload required"
I0601 03:05:56.987781       7 controller.go:184] "Backend successfully reloaded"
I0601 03:05:56.989090       7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-48nw7", UID:"69febc3b-9dc7-4523-9fd0-9012b6a3ee6f", APIVersion:"v1", ResourceVersion:"72961152", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:05:57 +0000] TCP 502 0 0 0.000
2023/06/01 03:05:57 [error] 712#712: *25823 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:05:57 [crit] 712#712: *25823 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:05:59 +0000] TCP 502 0 0 0.000
2023/06/01 03:05:59 [error] 712#712: *25859 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:05:59 [crit] 712#712: *25859 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
W0601 03:06:00.162047       7 controller.go:423] Error getting Service "foot/nginx-ns": no object matching key "foot/nginx-ns" in local store
I0601 03:06:00.162280       7 controller.go:167] "Configuration changes detected, backend reload required"
I0601 03:06:00.291845       7 controller.go:184] "Backend successfully reloaded"
I0601 03:06:00.292568       7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-48nw7", UID:"69febc3b-9dc7-4523-9fd0-9012b6a3ee6f", APIVersion:"v1", ResourceVersion:"72961152", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2023/06/01 03:06:01 [error] 847#847: *25909 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:06:01 [crit] 847#847: *25909 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:06:01 +0000] TCP 502 0 0 0.000
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:06:03 +0000] TCP 502 0 0 0.000
2023/06/01 03:06:03 [error] 847#847: *25955 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:06:03 [crit] 847#847: *25955 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:06:05 [error] 847#847: *25989 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:06:05 [crit] 847#847: *25989 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:06:05 +0000] TCP 502 0 0 0.000
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:06:07 +0000] TCP 502 0 0 0.000
2023/06/01 03:06:07 [error] 847#847: *26023 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:06:07 [crit] 847#847: *26023 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:06:09 +0000] TCP 502 0 0 0.000
2023/06/01 03:06:09 [error] 847#847: *26059 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:06:09 [crit] 847#847: *26059 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0
[fd69:1:10:f980::a13:26df] [01/Jun/2023:03:06:11 +0000] TCP 502 0 0 0.000
2023/06/01 03:06:11 [error] 847#847: *26093 stream [lua] tcp_udp_balancer.lua:196: balance(): error while setting current upstream peer 2001:db8:42:83:318a:6059:3e82:d1d6:3306: invalid port while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, bytes from/to client:0/0, bytes from/to upstream:0/0
2023/06/01 03:06:11 [crit] 847#847: *26093 connect() to 0.0.0.1:1234 failed (22: Invalid argument) while connecting to upstream, client: fd69:1:10:f980::a13:26df, server: [::]:32001, upstream: "0.0.0.1:1234", bytes from/to client:0/0, bytes from/to upstream:0/0

@longwuyuan @tao12345666333