kubernetes: Service cannot be accessed via nodeport

What happened: A NodePort service can be accessed via the ClusterIP, but not via the NodePort (the request hangs)

What you expected to happen: Request works the same as called via cluster IP.

How to reproduce it (as minimally and precisely as possible): On the master node the following works:

curl http://10.96.111.117:8080/

but the following hangs:

curl://localhost:32585/

The service yaml is attached service.txt

Anything else we need to know?: I enabled traces for the packet in the iptables chains and noticed, that the first SYN packet does not get an ACK within one second and is repeated (the cluster consists of 3 VMs on a slow box - so this may be normal). The repetition packet takes a different path through the chains and is therefore not MASQUERADED.

iptable rules: rules.txt

trace log, original SYN packet has timestamp 10:10:04, repetition has timestamp 10:10:05: trace.txt

Environment:

  • Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.0”, GitCommit:“70132b0f130acc0bed193d9ba59dd186f0e634cf”, GitTreeState:“clean”, BuildDate:“2019-12-07T21:20:10Z”, GoVersion:“go1.13.4”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.0”, GitCommit:“70132b0f130acc0bed193d9ba59dd186f0e634cf”, GitTreeState:“clean”, BuildDate:“2019-12-07T21:12:17Z”, GoVersion:“go1.13.4”, Compiler:“gc”, Platform:“linux/amd64”}
  • Cloud provider or hardware configuration: 3 VMs on a Linux box
  • OS (e.g: cat /etc/os-release): NAME=“Ubuntu” VERSION=“18.04.3 LTS (Bionic Beaver)” ID=ubuntu ID_LIKE=debian PRETTY_NAME=“Ubuntu 18.04.3 LTS” VERSION_ID=“18.04” HOME_URL=“https://www.ubuntu.com/” SUPPORT_URL=“https://help.ubuntu.com/” BUG_REPORT_URL=“https://bugs.launchpad.net/ubuntu/” PRIVACY_POLICY_URL=“https://www.ubuntu.com/legal/terms-and-policies/privacy-policy” VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic
  • Kernel (e.g. uname -a): Linux dockerswarm0 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: kubeadm 1.17.0
  • Network plugin and version (if this is a network-related bug): https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 34 (13 by maintainers)

Most upvoted comments

@ghhenry I experienced the same issue, I was using Flannel as the Kubernetes CNI. I replaced Flannel with “Weave Net” and this resolved my issue. You can refer the following link to do so :- https://jackiechen.org/2019/11/01/replace-flannel-with-weave-net-in-kubernetes/

@alexanderConstantinescu yes, seems to be related. I disabled offloading with sudo ethtool --offload flannel.1 rx off tx off and this indeed helped.