kubernetes: NodePort will not listen on external IP if same IP is used as loadBalancerIP

What happened?

Hey, after upgrading kubernetes from 1.22 to 1.23, I’m experiencing strange service provisioning behavior using nodePort. I’m using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn’t listen to NodePort on this interface.
Unfortunately, I can’t find any changes that could be causing this, any ideas? Kubernetes 1.23

apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/loadBalancerIPs: externalip1
  name: service
spec:
  ports:
  - name: client
    port: 31010
    protocol: TCP
    targetPort: client
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: service2
spec:
  ports:
  - appProtocol: http
    name: http
    nodePort: 32080
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 32443
    port: 443
    protocol: TCP
    targetPort: https
  type: NodePort

ipvsadm -Ln | grep externalip1
node1:
    TCP  externalip1:31010 lc
    TCP  externalip1:32010 lc
node2:
    TCP  externalip1:31010 lc
    TCP  externalip1:32010 lc
node3:
    TCP  externalip1:31010 lc
    TCP  externalip1:32010 lc

Missing externalip1 here:

ipvsadm -L -n | grep 32443
node1:
    TCP  172.17.0.1:32443 lc
    TCP  10.20.10.13:32443 lc
    TCP  10.50.135.0:32443 lc
node2:
    TCP  externalip2:32443 lc
    TCP  172.17.0.1:32443 lc
    TCP  10.20.10.12:32443 lc
    TCP  10.58.88.192:32443 lc
node3:
    TCP  externalip3:32443 lc
    TCP  172.17.0.1:32443 lc
    TCP  172.26.0.1:32443 lc
    TCP  10.20.10.10:32443 lc
    TCP  10.20.10.11:32443 lc
    TCP  10.63.32.192:32443 lc

What did you expect to happen?

Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service.

Kubernetes 1.22

apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/loadBalancerIPs: externalip1
  name: service
spec:
  ports:
  - name: client
    port: 31010
    protocol: TCP
    targetPort: client
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: service2
spec:
  ports:
  - appProtocol: http
    name: http
    nodePort: 32080
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 32443
    port: 443
    protocol: TCP
    targetPort: https
  type: NodePort

ipvsadm -Ln | grep externalip1
node1:
    TCP  externalip1:31010 lc
    TCP  externalip1:32010 lc
node2:
    TCP  externalip1:31010 lc
    TCP  externalip1:32010 lc
node3:
    TCP  externalip1:31010 lc
    TCP  externalip1:32010 lc

ipvsadm -Ln | grep 32443
node1:
    TCP  172.17.0.1:32443 lc
    TCP  externalip1:32443 lc
    TCP  10.10.10.1:32443 lc
    TCP  10.36.174.0:32443 lc
node2:
    TCP  172.17.0.1:32443 lc
    TCP  externalip2:32443 lc
    TCP  10.10.10.4:32443 lc
    TCP  10.45.192.0:32443 lc
node3:
    TCP  172.17.0.1:32443 lc
    TCP  externalip3:32443 lc
    TCP  10.10.10.2:32443 lc
    TCP  10.45.11.192:32443 lc

How can we reproduce it (as minimally and precisely as possible)?

Upgrade k8s from 1.22.15 to 1.23.15

Anything else we need to know?

No response

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux D001-FSN1DC17 6.0.8-060008-generic #202211110629-Ubuntu SMP PREEMPT_DYNAMIC Fri Nov 11 06:36:01 UTC x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)

    ipvs:
      excludeCIDRs: null
      scheduler: lc
      strictARP: true