kind: port-forwarding broken in kindest/node:v1.18.2 but works on kindest/node:v1.18.0

What happened:

port-forwarding to a Kind cluster doesn’t appear to work after 1.18.2 (works OK on 1.18.0)

How to reproduce it (as minimally and precisely as possible):

  1. Run:
kind create cluster --image=kindest/node:v1.18.2
  1. Apply this YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-html
  labels:
    app: example-html
spec:
  selector:
    matchLabels:
      app: example-html
  template:
    metadata:
      labels:
        app: example-html
    spec:
      containers:
      - name: example-html
        image: busybox
        command: ["busybox", "httpd", "-f", "-p", "8000"]
        ports:
        - containerPort: 8000
  1. Wait until the pod is ready

  2. Run

kubectl port-forward deployment/example-html 8000:8000
  1. Visit http://localhost:8000/

Expected result: a 404 response

Actual result: Connection hangs indefinitely

In the kubectl port-forward terminal, I see:

E0710 19:56:07.104125 2250228 portforward.go:400] an error occurred forwarding 8000 -> 8000: error forwarding port 8000 to pod 7e6be7fb9a3b95c5c80f206f931b339d955f4fd9423a393f7ee072c477d9f370, uid : failed to execute portforward in network namespace “/var/run/netns/cni-1b6beba4-efb4-40fc-741e-fcce3506a3d3”: socat command returns error: exit status 1, stderr: “2020/07/10 23:56:07 socat[1814] E connect(5, AF=2 92.242.140.21:8000, 16): Connection timed out\n”

Anything else we need to know?:

If I start Kind with kind create cluster --image=kindest/node:v1.18.0, port-forwarding works fine

I am so curious what the problem is here, and would love tips on how to debug this further. I’ve been banging my head against the wall for awhile

Environment:

  • kind version: (use kind version): kind v0.8.1 go1.14.3 linux/amd64
  • Kubernetes version: (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-07T14:04:52Z", GoVersion:"go1.13.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"
  • Docker version: (use docker info): Server Version: 19.03.11

  • OS (e.g. from /etc/os-release): NAME=“Ubuntu” VERSION=“20.04 LTS (Focal Fossa)”

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 35 (35 by maintainers)

Most upvoted comments

yes, sorry!! To be clear, port-forwarding is broken with the default image. I was using the --image= flag to try to diagnose why it’s broken, didn’t mean to imply that I expected this to be a good way to operate kind 😂

scratch my systemd-resolved comment, is disabled inside the container and overcomplicate everything

I tested it locally and seems to work using files first, I submitted a PR to test in the CI https://github.com/kubernetes-sigs/kind/pull/1731 if it works I think we should make the change

TIL 😕 Question about that IP https://askubuntu.com/q/587895 and its answer https://askubuntu.com/a/587954

This is weird because there is an e2e covering this. @nicks how is it trying to dial to a public ip? 92.242.140.21:8000

Is there something “special” in your KIND environment?

(also, totally not urgent, good luck fixing all kubernetes PRs 😬)