kind: Cannot talk to cluster inside dind container

I managed to get it working with the following:

$ docker run --rm -t -i --privileged ubuntu:artful

$ apt-get update
$ apt-get install --yes golang-go git curl unzip wget apt-transport-https curl ca-certificates

$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list

$ curl -s https://download.docker.com/linux/ubuntu/gpg | apt-key add -
$ echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" > /etc/apt/sources.list.d/docker.list

$ apt-get update
$ apt-get install --yes kubectl docker-ce

$ echo '{"storage-driver": "vfs"}' > /etc/docker/daemon.json
$ service docker start

$ export GOPATH=/usr/local/go
$ export PATH="${GOPATH}/bin:${PATH}"

$ go get sigs.k8s.io/kind

$ kind create

$ export KUBECONFIG="/root/.kube/kind-config-1"
$ kubectl cluster-info

So instead of using host’s Docker socket, I do simply a proper dind inside my container.

FYI for future folks finding this issue we now have a contrib repo https://kind.sigs.k8s.io/docs/user/resources/#using-kind-in-ci that documents CI setups such as this.

And thanks for all this work and thank you for all the help.

OK. So I do not know about gitlab.com CI, but on our private GitLab instance I discovered that it seems I am given docker.sock mounted into my container from host, I guess. (I have some thoughts about such setup and security of it, but I will not complain at the moment.) So I can simply have one Docker container inside which I do everything. I do the following (in an image with Go and Docker client already installed):

go get sigs.k8s.io/kind
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt-get update -q -q
apt-get install --yes kubectl
kind create
export KUBECONFIG="/root/.kube/kind-config-1"
kubectl cluster-info || true
docker ps
docker exec kind-1-control-plane ps

So I just install kind and kubectl and then test it out. Sadly, it still does not work but I think this is closer. The output of final commands is is as follows:

$ export KUBECONFIG="/root/.kube/kind-config-1"
$ kubectl cluster-info || true

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server localhost:32768 was refused - did you specify the right host or port?
$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                     NAMES
4aaf4aaa8acf        kindest/node:v1.11.3   "/usr/local/bin/entr…"   2 minutes ago       Up About a minute   0.0.0.0:32768->6443/tcp   kind-1-control-plane
83494fdd744b        a37a729110ce           "sh -c 'if [ -x /usr…"   3 minutes ago       Up 3 minutes                                  runner-b938861c-project-880-concurrent-0-build
$ docker exec kind-1-control-plane ps
  PID TTY          TIME CMD
    1 ?        00:00:01 systemd
   52 ?        00:00:00 systemd-journal
   66 ?        00:00:38 dockerd
   97 ?        00:00:00 docker-containe
  814 ?        00:00:06 kubelet
  922 ?        00:00:00 docker-containe
  923 ?        00:00:00 docker-containe
  925 ?        00:00:00 docker-containe
  926 ?        00:00:00 docker-containe
  992 ?        00:00:00 pause
  995 ?        00:00:00 pause
 1003 ?        00:00:00 pause
 1005 ?        00:00:00 pause
 1069 ?        00:00:00 docker-containe
 1088 ?        00:00:01 kube-scheduler
 1097 ?        00:00:00 docker-containe
 1118 ?        00:00:32 kube-apiserver
 1119 ?        00:00:00 docker-containe
 1135 ?        00:00:00 docker-containe
 1168 ?        00:00:06 etcd
 1180 ?        00:00:03 kube-controller
 1412 ?        00:00:00 docker-containe
 1431 ?        00:00:00 pause
 1453 ?        00:00:00 docker-containe
 1470 ?        00:00:00 kube-proxy
 1555 ?        00:00:00 docker-containe
 1591 ?        00:00:00 pause
 1695 ?        00:00:00 exe
 1707 ?        00:00:00 ps

So you see that docker ps now shows my CI container alongside kind-1-control-plane container. I think this is better because it means I should be able to connect directly to stuff in kind-1-control-plane container. But I am not yet able. Any suggestions here?

Have you done it while talking to a networked service running over in the dind container before?

Oh, I remember. I think I had issues with that in the past. The issue was that from outside, I can see only the dind container, and not any network behind it. So I had to publish ports in Docker containers running through dind, so that they got available on the dind container. So dind container is like host, and you do not have access directly to containers behind.

Which might be also additional problem for me because I want to run then pods on the cluster, which again might not be available from my testing container, because it would again be behind dind container/host.

When you run in the docker executor, are you running everything with the dind container, or are you running the dind container alongside another container there as well?

So currently I run dind as a service container, and then I have my main container in which I would like to run my tests on the kubernetes cluster. I have done this setup in the past for using regular Docker images/containers and it works well.

See a bit about this here as well.

So I am using --link just for debugging purposes to simulate what I believe GitLab CI is doing. Otherwise I am targeting Docker executor with docker-in-docker. They have a concept of services and then you connect to those services.