calico: Worker node or pod cannot reach kube-dns service IP

Expected Behavior

On a worker node or on a pod running on a worker node, kube-dns serviceIP should be reachable. It works on the master node:

$ nslookup kubernetes.default.svc.cluster.local 10.96.0.10
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

Current Behavior

The IP is not reachable:

$ docker exec -it <POD> sh
/ # nslookup kubernetes.default.svc.cluster.local 10.96.0.10
Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default.svc.cluster.local'

Steps to Reproduce (for bugs)

  1. Setup a 2-node cluster with kubeadm following the steps on kubernetes.io
  2. Deploy a job with the following configuration below
  3. Attach to the pod running on the worker node and try to nslookup using kube-dns’s IP
---
apiVersion: batch/v1
kind: Job
metadata:
  name: n1
spec:
  template:
    spec:
      containers:
      - name: node
        image: busybox
        imagePullPolicy: Always
        command: ["sh", "-c", "sleep 3600"]
      restartPolicy: Never
  backoffLimit: 1
...

Context

That seems pretty basic so I must be doing something wrong.

Your Environment

  • Calico version: 1.7
  • Orchestrator version (e.g. kubernetes, mesos, rkt): kube 1.9.4
  • Operating System and version: RHEL 7.4

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 15 (6 by maintainers)

Most upvoted comments

A few things:

  • You said Calico version: 1.7, that is quite old, and since this is a new cluster you are setting up, you should use version v2.6.8 or ideally v3.0.
  • Calico does not setup service IPs but kube-proxy does, you should look at kube-proxy on your nodes and ensure they are not reporting errors.
  • I’d suggest taking a look at https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/kubeadm/ and ensure you have met the requirements stated there.