kubernetes: Storage accounts plugins cannot resolve internal DNS

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:“1”, Minor:“5+”, GitVersion:“v1.5.3-beta.0”, GitCommit:“b5f9d56cab78ccaad2b726223ba8be5802026f0b”, GitTreeState:“clean”, BuildDate:“2017-01-21T11:31:55Z”, GoVersion:“go1.7.4”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“5”, GitVersion:“v1.5.1”, GitCommit:“82450d03cb057bab0950214ef122b67c83fb11df”, GitTreeState:“clean”, BuildDate:“2016-12-14T00:52:01Z”, GoVersion:“go1.7.4”, Compiler:“gc”, Platform:“linux/amd64”}

Environment:

Cloud provider or hardware configuration: Azure
OS (e.g. from /etc/os-release): Ubuntu 16.04
Kernel (e.g. uname -a): Linux k8s-master-50E6FAC2-0 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Install tools: azure acs

What happened:

I tried to setup glusterfs cluster with heketi and point glusterfs storage class to it with dns entry, but it seems that storageclass is not able to resolve dns address of service.

Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  55s		5s		5	{persistentvolume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "glusterfs": glusterfs: create volume err: error creating volume Post http://heketi.heketi.svc.cluster.local:8080/volumes: dial tcp: lookup heketi.heketi.svc.cluster.local: no such host.

What you expected to happen:

glusterfs storage class is able to resolve dns address to heketi service

How to reproduce it (as minimally and precisely as possible):

Setup glusterfs cluster inside kubernetes using heketi at heketi namespace
Create storage class as follows"

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: glusterfs
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://heketi.heketi.svc.cluster.local:8080"

try to create any stateful app with this storageclass

Anything else we need to know:

DNS is resolvable from busybox container:

wget -qO- http://heketi.heketi.svc.cluster.local:8080/volumes
{"volumes":["85516baa0869f946ca492cc941ca2b76"]}

After I’ve change fqdn name to ip address in storage configration, it started working (but I’d prefer to use dns name).

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 18
Comments: 39 (4 by maintainers)

Most upvoted comments

@innocode-devops I’m experiencing this in a baremetal 1.8.2 k8s cluster

rporres on Jan 2, 2018

@jenciso What do you mean with k8s IP service? It currently works with a nodeport but i’m looking for a better solution

Mikey032 on Oct 13, 2017

Appears that pod with hostNetwork should be using “dnsPolicy: ClusterFirstWithHostNet” according to the K8S docs referenced in the issue mentioned above.

To quote the other issue: According to the documentation (https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) the use of “dnsPolicy: ClusterFirst” is not valid when “hostNetwork: true” is set, instead “dnsPolicy: ClusterFirstWithHostNet” should be used.

howels on Sep 21, 2018

Same problem, really the storageclasses should be able to use internal DNS and it’s not intuitive.

howels on Aug 9, 2018

If we run kube-controller-manager as a pod and with dnsPolicy is ClusterFirstWithHostNet, then it’s possible to resolve internal k8s service like Heketi.

canhnt on Jun 1, 2018

FWIW, it looks to me like the problem is in the kube-controller-manager, as it doesn’t appear to be able to resolve dns entries within the cluster:

I0528 16:16:27.126597       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"kube-system", Name:"monitoring-prometheus-alertmanager", UID:"78ebea70-6292-11e8-a7e0-525400c7a637", APIVersion:"v1", ResourceVersion:"9507", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "glusterfs-storage": create volume error: error creating volume Post http://heketi.kube-system.svc.cluster.local:8080/volumes: dial tcp: lookup heketi.kube-system.cluster.local: no such host

and here’s an extract from a shell on the controller:

root@k8s-01:/# nslookup $_
Server:         192.168.252.253
Address:        192.168.252.253#53

** server can't find heketi.kube-system.svc.cluster.local: NXDOMAIN

root@k8s-01:/# cat /etc/resolv.conf 
nameserver 192.168.252.253
nameserver 192.168.121.1

marksutherland on May 29, 2018

Just a side note here, and maybe it can make the life easier, you can use https://rook.github.io, I used it and it made it easy to handle those issues, it is free and it will orchestrate some storage types like Ceph, Minio and CockrouchDB, and in my case I used it with Ceph and it’s working

arkanmgerges on Aug 9, 2018

@arkanmgerges: You can take a look on the examples here and here. In this case, kube-controller-manager is run as a pod rather than a systemd service and bootstrapped by the kubelet.

canhnt on Jun 29, 2018

I just tested this on K8S 1.10; the IP works but the svc name does not. I tested name resolution with busybox and that works well, but the GlusterFS plugin does not resolve the name. In my case I am working with a bare-metal installation on-premise.

patotech on May 9, 2018

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot on Apr 9, 2018

Ok, thx sheerun. For now I’m exposing heketi service on a NodePort to avoid using a specific node IP.

clementnero on Apr 7, 2017

I didn’t. It’s still broken

sheerun on Apr 7, 2017