kubernetes: Cluster dns stops functioning after 1.6 -> 1.5 downgrade

I have been able to reproduce this twice.

Steps to reproduce:

  • Download kubernetes 1.5.5 tar, extract and cd into the directory
  • run cluster/get-kube-binaries.sh
  • run cluster/kube-up.sh
  • Run the steps defined here here
    • You don’t need to wait for the HPA to react, just verify that the script says “OK!” a bunch of times to ensure it can find the servier
  • Download the v1.6.0-rc.1 release tar, extract, and cd into the directory
  • run cluster/get-kube-binaries.sh
  • Warning: Set the following env or you will not be able to downgrade later
export TARGET_STORAGE=etcd3
export ETCD_IMAGE=3.0.17
export TARGET_VERSION=3.0.17
export STORAGE_MEDIA_TYPE=application/json
  • run ./cluster/gce/upgrade.sh -M v1.6.0-rc.1
  • Run the steps defined here here to verify the Pod can see the service
  • run ./cluster/gce/upgrade.sh -N -o v1.6.0-rc.1 to upgrade the nodes
  • Run the steps defined here here to verify the Pod can see the service
  • run ./cluster/gce/upgrade.sh -N -o v1.5.5 to downgrade the nodes
  • Run the steps defined here here to verify the Pod can see the service
  • Warning: Set the following env to downgrade
export TARGET_STORAGE=etcd2
export ETCD_IMAGE=3.0.17
export TARGET_VERSION=2.2.1
export STORAGE_MEDIA_TYPE=application/json
  • run ./cluster/gce/upgrade.sh -M v1.5.5 to downgrade the master
  • Run the steps defined here here to verify the Pod can see the service
    • This is where things fail for me. I am not longer able to connect to the service through the DNS address.
wget php-apache.default.svc.cluster.local -q -O -
wget: bad address 'php-apache.default.svc.cluster.local'
  • I am able to connect to the service using its IP address
kubectl get services
NAME         CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   10.0.0.1      <none>        443/TCP   1h
php-apache   10.0.52.190   <none>        80/TCP    1h
wget -q -O - 10.0.52.190
OK!

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 27 (27 by maintainers)

Commits related to this issue

Most upvoted comments

fixed in #43676

successfully upgraded v1.5.5 -> ci/latest-1.6 -> ci/latest-1.5 and ensured kube-proxy and kube-dns were functioning correctly after upgrade and downgrade, and tokens file contained correct control plane users after each step