kubernetes: DNS is not working for external domains like google.com

I have installed Kubernetes on Bare-metal/Ubuntu. I am on 6b649d7f9f2b09ca8b0dd8c0d3e14dcb255432d1 commit in git. I used cd kubernetes/cluster; KUBERNETES_PROVIDER=ubuntu ./kube-up.sh followed by cd kubernetes/cluster/ubuntu; ./deployAddons.sh to start the cluster. Everything went fine and the cluster got up.

My /ubuntu/config-default.sh is as follows:

    # Define all your cluster nodes, MASTER node comes first"
    # And separated with blank space like <user_1@ip_1> <user_2@ip_2> <user_3@ip_3> 
    export nodes=${nodes:-"root@192.168.48.170 root@192.168.48.180"}

    # Define all your nodes role: a(master) or i(minion) or ai(both master and minion), must be the order same 
    role=${role:-"ai i"}
    # If it practically impossible to set an array as an environment variable
    # from a script, so assume variable is a string then convert it to an array
    export roles=($role)

    # Define minion numbers
    export NUM_NODES=${NUM_NODES:-2}
    # define the IP range used for service cluster IPs.
    # according to rfc 1918 ref: https://tools.ietf.org/html/rfc1918 choose a private ip range here.
    export SERVICE_CLUSTER_IP_RANGE=${SERVICE_CLUSTER_IP_RANGE:-192.168.3.0/24}  # formerly PORTAL_NET
    # define the IP range used for flannel overlay network, should not conflict with above SERVICE_CLUSTER_IP_RANGE
    export FLANNEL_NET=${FLANNEL_NET:-172.16.0.0/16}

    # Optionally add other contents to the Flannel configuration JSON
    # object normally stored in etcd as /coreos.com/network/config.  Use
    # JSON syntax suitable for insertion into a JSON object constructor
    # after other field name:value pairs.  For example:
    # FLANNEL_OTHER_NET_CONFIG=', "SubnetMin": "172.16.10.0", "SubnetMax": "172.16.90.0"'

    export FLANNEL_OTHER_NET_CONFIG
    FLANNEL_OTHER_NET_CONFIG=''

    # Admission Controllers to invoke prior to persisting objects in cluster
    export ADMISSION_CONTROL=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,SecurityContextDeny

    # Path to the config file or directory of files of kubelet
    export KUBELET_CONFIG=${KUBELET_CONFIG:-""}

    # A port range to reserve for services with NodePort visibility
    SERVICE_NODE_PORT_RANGE=${SERVICE_NODE_PORT_RANGE:-"30000-32767"}

    # Optional: Enable node logging.
    ENABLE_NODE_LOGGING=false
    LOGGING_DESTINATION=${LOGGING_DESTINATION:-elasticsearch}

    # Optional: When set to true, Elasticsearch and Kibana will be setup as part of the cluster bring up.
    ENABLE_CLUSTER_LOGGING=false
    ELASTICSEARCH_LOGGING_REPLICAS=${ELASTICSEARCH_LOGGING_REPLICAS:-1}

    # Optional: When set to true, heapster, Influxdb and Grafana will be setup as part of the cluster bring up.
    ENABLE_CLUSTER_MONITORING="${KUBE_ENABLE_CLUSTER_MONITORING:-true}"

    # Extra options to set on the Docker command line.  This is useful for setting
    # --insecure-registry for local registries.
    DOCKER_OPTS=${DOCKER_OPTS:-""}

    # Extra options to set on the kube-proxy command line.  This is useful
    # for selecting the iptables proxy-mode, for example.
    KUBE_PROXY_EXTRA_OPTS=${KUBE_PROXY_EXTRA_OPTS:-""}

    # Optional: Install cluster DNS.
    ENABLE_CLUSTER_DNS="${KUBE_ENABLE_CLUSTER_DNS:-true}"
    # DNS_SERVER_IP must be a IP in SERVICE_CLUSTER_IP_RANGE
    DNS_SERVER_IP=${DNS_SERVER_IP:-"192.168.3.10"}
    DNS_DOMAIN=${DNS_DOMAIN:-"cluster.local"}
    DNS_REPLICAS=${DNS_REPLICAS:-1}

    # Optional: Install Kubernetes UI
    ENABLE_CLUSTER_UI="${KUBE_ENABLE_CLUSTER_UI:-true}"

    # Optional: Enable setting flags for kube-apiserver to turn on behavior in active-dev
    RUNTIME_CONFIG="--basic-auth-file=password.csv"

    # Optional: Add http or https proxy when download easy-rsa.
    # Add envitonment variable separated with blank space like "http_proxy=http://10.x.x.x:8080 https_proxy=https://10.x.x.x:8443"
    PROXY_SETTING=${PROXY_SETTING:-""}

    DEBUG=${DEBUG:-"false"}

Then, I created a pod using the following yml file:

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

And a service using the following yml:

    apiVersion: v1
    kind: Service
    metadata:
      name: nginx-service
    spec:
      ports:
      - port: 8000
        targetPort: 80
        protocol: TCP
      selector:
        app: nginx
      type: NodePort

Then, I got into the started container terminal using docker exec -it [CONTAINER_ID] bash. The problem is that I cannot ping external domains like google.com, but I can ping external IPs like 8.8.8.8. So the container has internet access.

The host’s /etc/resolve.conf file is as follows:

    nameserver 8.8.8.8
    nameserver 127.0.1.1

The container’s /etc/resolve.conf file is as follows:

    search default.svc.cluster.local svc.cluster.local cluster.local
    nameserver 192.168.3.10
    nameserver 8.8.8.8
    nameserver 127.0.1.1
    options ndots:5

I think it could be related to either SkyDNS nameservers misconfigurarion or a custom configuration that I have to do but I am not aware of.

Any workarounds?

I have also asked this question in stackoverflow, but no one has answered yet.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 34 (19 by maintainers)

Commits related to this issue

Most upvoted comments

Hi, I have the same issue resolving external domain name in pods. I have a workaround by adding extra external name server into kube-dnsmasq pod parameters. ex.

      - name: dnsmasq
        image: gcr.io/google_containers/kube-dnsmasq-amd64:1.4
        livenessProbe:
          httpGet:
            path: /healthz-dnsmasq
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --cache-size=1000
        - --no-resolv
        - --server=127.0.0.1#10053
+        - --server=8.8.8.8
+        - --all-servers
        - --log-facility=-

I don’t understand what is confusing for you here. the problem is that kube-dns resolves internal services and pod names but it cannot resolve external domains like google.com. So, for some unknown reason the DNS forwarding feature is not working. I am not sure if it is still happening in the current version because I didn’t find time to re-produce the problem with the current version. However, as some ppl reported here and in the stackoverflow question the above workaround which was working for the previous version is not working anymore, I opened the issue.

Also, notice that DNS resolution is working fine at the individual nodes that host the containers and specifically ones that host SkyDNS containers. However, inside the containers, the DNS has been configured by kubernetes to use kube-dns/SkyDNS and it doesnt resolve external domains.