kubernetes: kubectl port-forward doesn't properly recover from interrupted connection errors

Same error when port-forwarding to AWS EKS pod:

E0806 17:04:44.805492 78962 portforward.go:385] error copying from local connection to remote stream: read tcp6 [::1]:3000->[::1]:61216: read: connection reset by peer

kubectl version:

Client Version: version.Info{Major:“1”, Minor:“15”, GitVersion:“v1.15.2”, GitCommit:“f6278300bebbb750328ac16ee6dd3aa7d3549568”, GitTreeState:“clean”, BuildDate:“2019-08-05T16:54:35Z”, GoVersion:“go1.12.7”, Compiler:“gc”, Platform:“darwin/amd64”}

Does anybody have a solid workaround for this?

I’m giving up on this

Not a true solution, but helps in the meantime

while true; do <port-forward command> done

Does anybody have a solid workaround for this?

I tried @moewill’s loop solution, but most of the time, kubectl wouldn’t exit when the error occurred, leaving it stranded in a failed state until I manually hit CTRL+C (and the loop relaunched it).

To address the problem, I cloned the kubectl source and modified it to exit whenever an error occurs. This appears to be a pretty good workaround for my use case since when the loop relaunches it, everything starts working again. Maybe this workaround would help folks out until a proper fix is merged. I’ll try to put together a pull request.

duplicate of https://github.com/kubernetes/kubectl/issues/686

While this is no solution for the port-forward recovering properly, I did find it solved my use-case.

When trying to tunnel to a mysql box we couldn’t keep a connection alive for more than ~60 seconds, completely ruining the experience for anyone running long-running queries.

By creating a new connection regularly we seemed to be able to keep the port-forward tunnel open.

• Start your port-forward command
• Write something that can ping the pod every 20 seconds on the target port, for our usecase we just have mysqladmin ping -h 127.0.0.1 running in a loop
• Run your intended long-running command alongside this ping script.

Here’s (only) my case: all of the usual networking options normally involve exposing the service to the internet.

• From the above comments, many people use a basic version of grafana, probably not configured for internet exposure (or sometimes company policy prevents it anyway). I used this example because grafana is exceptionally easy to break through port-forward (it breaks within minutes, requiring a restart of the forwarding).
• we use it mostly for developers who already have access to their namespaces for easy access to backend services that should also not be exposed to the internet. They already have their service account and RBAC set up. It’s useful to be able to push or pull a database dump. It’s much harder when your connection tunnel is unstable.

On the tools side, many tools (minikube for example) seem to rely on port-forward too.

On the dev-side (using minikube or other dev-cluster tools that don’t provide a stable IP for your Nodeports/Loadbalancers) it’s just very convenient to just know that you can use localhost with a known port in your scripts instead of looking up the IP every time (and maybe changing your hosts file).

port-forward is the only tunneling approach (that I know) that doesn’t need elevation (or root on linux).

Since it looks like this isn’t likely to get fixed any time soon:

Here’s a rather ugly hack/workaround Bash script to automatically restart the port forwarding when it crashes.

Change the value variable to the appropriate port forwarding command.

It will run port forwarding, then parse the output stderr for errors; if the errors contain “portforward.go” (which these kind of errors do), then it will restart; if the port forwarding exits for any other reason (such as the process being killed, or a legitimate error, then it will exit instead.

This could probably be written nicer, but it works for my purposes.

value='kubectl port-forward svc/grafana 3000:80'
while true; do
      echo "--> $value"
      $value 2>&1 >/dev/null |
      while IFS= read -r line
      do
            echo "### $line"
            if [[ "$line" == *"portforward.go"* ]]; then
                  echo "Retarting port forwarding $value"
                  exit 1
            else
                  exit 0
            fi
      done
      if [ $? -eq 0 ]; then
            break;
      fi
done

Thanks @atinsinghal97. This does seem to help. For my particular issue I believe the fault is with Grafana (older builds have no issues) however this may be useful for testing that I’m doing sending OpenTelemetry data forward from my local machine.

this is not fixed yet, come on, random problems are the worst