origin: oc cluster up: Error: timed out waiting for the condition

oc cluster up result into an error

Error: timed out waiting for the condition
Version
Mac OS: 10.14.1

docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     false

oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth

docker images
REPOSITORY                                     TAG                 IMAGE ID            CREATED             SIZE
openshift/origin-node                          v3.11               48cb61702ea2        29 hours ago        1.16GB
openshift/origin-control-plane                 v3.11               fb643d7aeb7a        29 hours ago        818MB
openshift/origin-hypershift                    v3.11               92fdbc9dd636        29 hours ago        541MB
openshift/origin-hyperkube                     v3.11               6c60787c0d3e        29 hours ago        501MB
openshift/origin-cli                           v3.11               782540b81ca5        29 hours ago        376MB
openshift/origin-pod                           v3.10               61b86934f357        46 hours ago        224MB
openshift/origin-service-serving-cert-signer   v3.11               47dadf9d43b6        2 weeks ago         276MB
Steps To Reproduce
  1. oc cluster up
Current Result
oc cluster up
Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Pulling image openshift/origin-control-plane:v3.11
Pulled 1/5 layers, 23% complete
Pulled 2/5 layers, 61% complete
Pulled 3/5 layers, 77% complete
Pulled 4/5 layers, 83% complete
Pulled 5/5 layers, 100% complete
Extracting
Image pull complete
Pulling image openshift/origin-cli:v3.11
Image pull complete
Pulling image openshift/origin-node:v3.11
Pulled 5/6 layers, 88% complete
Pulled 6/6 layers, 100% complete
Extracting
Image pull complete
Creating shared mount directory on the remote host ...
Determining server IP ...
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking prerequisites for port forwarding ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Starting OpenShift using openshift/origin-control-plane:v3.11 ...
I1103 16:06:53.568319   17438 flags.go:30] Running "create-kubelet-flags"
I1103 16:06:54.749134   17438 run_kubelet.go:49] Running "start-kubelet"
I1103 16:06:55.072466   17438 run_self_hosted.go:181] Waiting for the kube-apiserver to be ready ...
I1103 16:07:34.343218   17438 interface.go:26] Installing "kube-proxy" ...
I1103 16:07:34.343258   17438 interface.go:26] Installing "kube-dns" ...
I1103 16:07:34.343279   17438 interface.go:26] Installing "openshift-service-cert-signer-operator" ...
I1103 16:07:34.343293   17438 interface.go:26] Installing "openshift-apiserver" ...
I1103 16:07:34.343328   17438 apply_template.go:81] Installing "openshift-apiserver"
I1103 16:07:34.343352   17438 apply_template.go:81] Installing "kube-proxy"
I1103 16:07:34.343398   17438 apply_template.go:81] Installing "kube-dns"
I1103 16:07:34.343437   17438 apply_template.go:81] Installing "openshift-service-cert-signer-operator"
I1103 16:08:08.205618   17438 interface.go:41] Finished installing "kube-proxy" "kube-dns" "openshift-service-cert-signer-operator" "openshift-apiserver"
Error: timed out waiting for the condition
Expected Result

Local up and running cluster

Additional Information
docker ps
CONTAINER ID        IMAGE                                          COMMAND                  CREATED              STATUS              PORTS               NAMES
1ae0cac4567b        openshift/origin-service-serving-cert-signer   "service-serving-cer…"   About a minute ago   Up About a minute                       k8s_service-serving-cert-signer-controller_service-serving-cert-signer-668c45d5f-pxwqx_openshift-service-cert-signer_2a529539-cf8d-11e8-b78b-025000000001_0
e2e9ac3dae11        openshift/origin-service-serving-cert-signer   "service-serving-cer…"   About a minute ago   Up About a minute                       k8s_operator_openshift-service-cert-signer-operator-6d477f986b-qqdzd_openshift-core-operators_0db6c342-cf8d-11e8-b78b-025000000001_0
511631535c11        openshift/origin-hypershift                    "hypershift openshif…"   About a minute ago   Up About a minute                       k8s_apiserver_openshift-apiserver-j2r72_openshift-apiserver_d10429eb-d3db-11e8-b8e9-025000000001_0
e2fb6a36000a        openshift/origin-control-plane                 "openshift start net…"   About a minute ago   Up About a minute                       k8s_kube-proxy_kube-proxy-rg5mq_kube-proxy_0dc7cb92-cf8d-11e8-b78b-025000000001_0
61b476fdb48b        openshift/origin-service-serving-cert-signer   "service-serving-cer…"   About a minute ago   Up About a minute                       k8s_apiservice-cabundle-injector-controller_apiservice-cabundle-injector-8ffbbb6dc-6scp8_openshift-service-cert-signer_2b9fce4e-cf8d-11e8-b78b-025000000001_0
be52142fb105        openshift/origin-control-plane                 "openshift start net…"   About a minute ago   Up About a minute                       k8s_kube-dns_kube-dns-vdrf8_kube-dns_0dd32d8a-cf8d-11e8-b78b-025000000001_0
8d989e9845fb        openshift/origin-node:v3.11                    "hyperkube kubelet -…"   2 minutes ago        Up 2 minutes                            origin

Logs

docker logs origin > origin.log 2>&1
docker logs be52142fb105 > kube-dns.log 2>&1
docker logs 61b476fdb48b > apiservice-cabundle-injector.log 2>&1
docker logs e2fb6a36000a > kube-proxy.log 2>&1
docker logs 511631535c11 > openshift-apiserver.log 2>&1
docker logs e2e9ac3dae11 > openshift-service-cert-signer-operator.log 2>&1
docker logs 1ae0cac4567b > service-serving-cert-signer-controller.log 2>&1

https://gist.github.com/jkroepke/a3d2994b94e78c5956df7856662291e3

Workaround

Running oc cluster up with an base-dir

mkdir -p "$HOME/.occluster"
oc cluster up --base-dir="$HOME/.occluster"

Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Creating shared mount directory on the remote host ...
Determining server IP ...
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking prerequisites for port forwarding ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Starting OpenShift using openshift/origin-control-plane:v3.11 ...
I1103 16:23:02.811666   18115 config.go:40] Running "create-master-config"
I1103 16:23:05.446271   18115 config.go:46] Running "create-node-config"
I1103 16:23:07.135914   18115 flags.go:30] Running "create-kubelet-flags"
I1103 16:23:08.341419   18115 run_kubelet.go:49] Running "start-kubelet"
I1103 16:23:08.645223   18115 run_self_hosted.go:181] Waiting for the kube-apiserver to be ready ...
I1103 16:23:35.895717   18115 interface.go:26] Installing "kube-proxy" ...
I1103 16:23:35.895738   18115 interface.go:26] Installing "kube-dns" ...
I1103 16:23:35.895744   18115 interface.go:26] Installing "openshift-service-cert-signer-operator" ...
I1103 16:23:35.895750   18115 interface.go:26] Installing "openshift-apiserver" ...
I1103 16:23:35.895777   18115 apply_template.go:81] Installing "kube-proxy"
I1103 16:23:35.895778   18115 apply_template.go:81] Installing "openshift-apiserver"
I1103 16:23:35.895786   18115 apply_template.go:81] Installing "openshift-service-cert-signer-operator"
I1103 16:23:35.895777   18115 apply_template.go:81] Installing "kube-dns"
I1103 16:23:38.830911   18115 interface.go:41] Finished installing "kube-proxy" "kube-dns" "openshift-service-cert-signer-operator" "openshift-apiserver"
I1103 16:26:08.908169   18115 run_self_hosted.go:242] openshift-apiserver available
I1103 16:26:08.908200   18115 interface.go:26] Installing "openshift-controller-manager" ...
I1103 16:26:08.908218   18115 apply_template.go:81] Installing "openshift-controller-manager"
I1103 16:26:11.618352   18115 interface.go:41] Finished installing "openshift-controller-manager"
Adding default OAuthClient redirect URIs ...
Adding centos-imagestreams ...
Adding registry ...
Adding router ...
Adding sample-templates ...
Adding persistent-volumes ...
Adding web-console ...
I1103 16:26:11.641602   18115 interface.go:26] Installing "centos-imagestreams" ...
I1103 16:26:11.641623   18115 interface.go:26] Installing "openshift-image-registry" ...
I1103 16:26:11.641630   18115 interface.go:26] Installing "openshift-router" ...
I1103 16:26:11.641635   18115 interface.go:26] Installing "sample-templates" ...
I1103 16:26:11.641640   18115 interface.go:26] Installing "persistent-volumes" ...
I1103 16:26:11.641650   18115 interface.go:26] Installing "openshift-web-console-operator" ...
I1103 16:26:11.641778   18115 apply_list.go:67] Installing "centos-imagestreams"
I1103 16:26:11.641835   18115 interface.go:26] Installing "sample-templates/nodejs quickstart" ...
I1103 16:26:11.641854   18115 interface.go:26] Installing "sample-templates/rails quickstart" ...
I1103 16:26:11.641877   18115 interface.go:26] Installing "sample-templates/jenkins pipeline ephemeral" ...
I1103 16:26:11.641890   18115 interface.go:26] Installing "sample-templates/mongodb" ...
I1103 16:26:11.641902   18115 interface.go:26] Installing "sample-templates/mariadb" ...
I1103 16:26:11.641910   18115 interface.go:26] Installing "sample-templates/postgresql" ...
I1103 16:26:11.641919   18115 interface.go:26] Installing "sample-templates/cakephp quickstart" ...
I1103 16:26:11.641927   18115 interface.go:26] Installing "sample-templates/django quickstart" ...
I1103 16:26:11.641936   18115 interface.go:26] Installing "sample-templates/mysql" ...
I1103 16:26:11.641977   18115 apply_list.go:67] Installing "sample-templates/rails quickstart"
I1103 16:26:11.641987   18115 apply_list.go:67] Installing "sample-templates/nodejs quickstart"
I1103 16:26:11.642036   18115 apply_list.go:67] Installing "sample-templates/jenkins pipeline ephemeral"
I1103 16:26:11.642068   18115 apply_list.go:67] Installing "sample-templates/mysql"
I1103 16:26:11.642000   18115 interface.go:26] Installing "sample-templates/dancer quickstart" ...
I1103 16:26:11.642161   18115 interface.go:26] Installing "sample-templates/sample pipeline" ...
I1103 16:26:11.642249   18115 apply_list.go:67] Installing "sample-templates/sample pipeline"
I1103 16:26:11.642283   18115 apply_list.go:67] Installing "sample-templates/postgresql"
I1103 16:26:11.642381   18115 apply_list.go:67] Installing "sample-templates/cakephp quickstart"
I1103 16:26:11.642402   18115 apply_template.go:81] Installing "openshift-web-console-operator"
I1103 16:26:11.642439   18115 apply_list.go:67] Installing "sample-templates/dancer quickstart"
I1103 16:26:11.642460   18115 apply_list.go:67] Installing "sample-templates/mariadb"
I1103 16:26:11.642539   18115 apply_list.go:67] Installing "sample-templates/django quickstart"
I1103 16:26:11.643068   18115 apply_list.go:67] Installing "sample-templates/mongodb"
I1103 16:26:19.810920   18115 interface.go:41] Finished installing "sample-templates/nodejs quickstart" "sample-templates/rails quickstart" "sample-templates/jenkins pipeline ephemeral" "sample-templates/mongodb" "sample-templates/mariadb" "sample-templates/postgresql" "sample-templates/cakephp quickstart" "sample-templates/django quickstart" "sample-templates/mysql" "sample-templates/dancer quickstart" "sample-templates/sample pipeline"
I1103 16:26:53.420432   18115 interface.go:41] Finished installing "centos-imagestreams" "openshift-image-registry" "openshift-router" "sample-templates" "persistent-volumes" "openshift-web-console-operator"
Login to server ...
Creating initial project "myproject" ...
Server Information ...
OpenShift server started.

The server is accessible via web console at:
    https://127.0.0.1:8443

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

WARNING: An HTTP proxy (gateway.docker.internal:3128) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: An HTTPS proxy (gateway.docker.internal:3129) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: A proxy is configured for Docker, however 172.30.1.1 is not included in its NO_PROXY list.
   172.30.1.1 needs to be included in the Docker daemon's NO_PROXY environment variable so pushes to the local OpenShift registry can succeed.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 12
  • Comments: 45 (4 by maintainers)

Commits related to this issue

Most upvoted comments

I had the same issue on CentOS 7 and oc tools 3.11 but after adding a new firewalld zone as per documentation (https://github.com/openshift/origin/blob/release-3.11/docs/cluster_up_down.md#linux), everything worked fine.

firewall-cmd --permanent --new-zone dockerc
firewall-cmd --permanent --zone dockerc --add-source 172.17.0.0/16
firewall-cmd --permanent --zone dockerc --add-port 8443/tcp
firewall-cmd --permanent --zone dockerc --add-port 53/udp
firewall-cmd --permanent --zone dockerc --add-port 8053/udp
firewall-cmd --reload
+38
raven4ever on Feb 5, 2019

I’ve solved this issue by performing the following steps:

  1. Reset docker to its factory state (Preferences > Reset > Reset to factory defaults).
  2. Add OC insecure IP (172.30.0.0/16) to docker (Preferences > Daemon).
  3. Add docker filesystem access to the openshift folder: docker run -it --rm --privileged --pid=host justincormack/nsenter1 /bin/sh -c 'mkdir -p /var/lib/kubelet/device-plugins'
  4. Start oc cluster passing a base directory as parameter, ensuring a fresh installation: oc cluster up --loglevel=5 --base-dir new-openshift
+7
davi-romero on Aug 2, 2019

I have the same issue on ubunto 18.04 bionic with openshift 3.11 😦 Also tried starting the cluster up as root

oc cluster up --public-hostname=‘’ E0426 07:55:33.909661 20849 run_self_hosted.go:571] API server error: Get https://127.0.0.1:8443/healthz?timeout=32s: dial tcp 127.0.0.1:8443: connect: connection refused () Error: timed out waiting for the condition

oc cluster up --public-hostname=xx.xxx.xx.xxx where xx.xxx.xx.xxx is my public ip(returned by dig +short myip.opendns.com @resolver1.opendns.com) . This yields network unreachable

E0426 07:37:36.025451 19048 run_self_hosted.go:571] API server error: Get https://xx.xxx.xx.xxx:8443/healthz?timeout=32s: dial tcp xx.xxx.xx.xxx:8443: connect: network is unreachable () Error: timed out waiting for the condition

+6
aarthipa on Apr 26, 2019

I’ve solved this issue reducing the amount of memory for the minishift Virtualbox Machine. By default minishift uses 4GB of RAM, I opened Virtualbox and go to Machine configuration and set it to 2GB and now it works correctly.

My host is 6GB memory.

+4
lemol on Aug 5, 2019

Same here. Workaround is not working instead.

+4
xeviknal on Nov 29, 2018

believeit or not, I solved this by add some mem to my vm from 8G to 10G …

+3
R1310328554 on Aug 1, 2019

Workaround which has worked for me :

  1. Using hyperv + oc 3.11 + minishift v1.34.1+c2ff9cb.
  2. Problem only on not so good internet connections.
  3. Once minishift start command fails for the first time, ssh into minishift vm. Run oc cluster down followed by oc cluster up (copy this from the logs). Repeat this down/up step till your cluster is up. It could be 10-20 times or more or less depending on the network maybe.
  4. Once the cluster is up, run minishift stop/ start and then you will be ready. Hope this helps
+3
niteshthakur on Jul 31, 2019

I had the same issue on CentOS 7 and oc tools 3.11 but after adding a new firewalld zone as per documentation (https://github.com/openshift/origin/blob/release-3.11/docs/cluster_up_down.md#linux), everything worked fine.

firewall-cmd --permanent --new-zone dockerc
firewall-cmd --permanent --zone dockerc --add-source 172.17.0.0/16
firewall-cmd --permanent --zone dockerc --add-port 8443/tcp
firewall-cmd --permanent --zone dockerc --add-port 53/udp
firewall-cmd --permanent --zone dockerc --add-port 8053/udp
firewall-cmd --reload

after I added it, but still have error.

Finished installing “kube-proxy” “kube-dns” “openshift-service-cert-signer-operator” “openshift-apiserver” Error: timed out waiting for the condition

+2
DatHT on Nov 2, 2019

Tried the base-dir, the firewall settings, nothing is working for me. It started the first time and now not anymore after I tried to configure the domain name for the UI.

+2
dwamara on Aug 15, 2019

Same here, host is centos-7: http://logs.openstack.org/67/570667/32/check/nodepool-functional-openshift/1bba9e7/ara-report/result/6dfc7855-60bd-4cf7-b3a4-95784e6b7f3c/

+2
TristanCacqueray on Dec 3, 2018

So for Ubuntu 18 bionic weaver, along with the firewall setup, I had to disable firewall with sudo ufw disable, deleted all log files and started up with a fresh install. Starting the cluster as root did not work. I am now able to start the cluster as non-root and cluster available at https://127.0.0.1:8443

+1
aarthipa on Apr 26, 2019
Read more comments on GitHub
← origin: oc cluster up broken in rc1 with Docker for OSX
origin: oc cluster up fails on mac with "error creating node config" →