minikube: Cannot start ingress addon: waiting for pod

Steps to reproduce the issue:

  1. minikube start --vm-driver hyperv --cpus 2 --memory=8192
  2. minikube addons enable ingress --alsologtostderr

Full output of failed command:

❯ minikube addons enable ingress --alsologtostderr
I0220 16:57:51.899458   28192 out.go:229] Setting OutFile to fd 88 ...
I0220 16:57:51.900398   28192 out.go:276] TERM=,COLORTERM=, which probably does not support color
I0220 16:57:51.900398   28192 out.go:242] Setting ErrFile to fd 92...
I0220 16:57:51.901397   28192 out.go:276] TERM=,COLORTERM=, which probably does not support color
I0220 16:57:51.921175   28192 addons.go:55] Setting ingress=true in profile "minikube"
I0220 16:57:51.921826   28192 addons.go:131] Setting addon ingress=true in "minikube"
I0220 16:57:51.922369   28192 host.go:66] Checking if "minikube" exists ...
I0220 16:57:51.922904   28192 main.go:119] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0220 16:57:52.389395   28192 main.go:119] libmachine: [stdout =====>] : Running

I0220 16:57:52.390040   28192 main.go:119] libmachine: [stderr =====>] :
I0220 16:57:52.390040   28192 addons.go:247] installing /etc/kubernetes/addons/ingress-configmap.yaml
I0220 16:57:52.390734   28192 ssh_runner.go:310] scp memory --> /etc/kubernetes/addons/ingress-configmap.yaml (1251 bytes)
I0220 16:57:52.390734   28192 main.go:119] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0220 16:57:52.848442   28192 main.go:119] libmachine: [stdout =====>] : Running

I0220 16:57:52.850487   28192 main.go:119] libmachine: [stderr =====>] :
I0220 16:57:52.851015   28192 main.go:119] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Hyper-V\Get-VM minikube ).networkadapters[0]).ipaddresses[0]
I0220 16:57:53.576395   28192 main.go:119] libmachine: [stdout =====>] : 172.23.0.69

I0220 16:57:53.576923   28192 main.go:119] libmachine: [stderr =====>] :
I0220 16:57:53.576923   28192 sshutil.go:48] new ssh client: &{IP:172.23.0.69 Port:22 SSHKeyPath:C:\Users\XX\.minikube\machines\minikube\id_rsa Username:docker}
I0220 16:57:53.675885   28192 addons.go:247] installing /etc/kubernetes/addons/ingress-rbac.yaml
I0220 16:57:53.675885   28192 ssh_runner.go:310] scp memory --> /etc/kubernetes/addons/ingress-rbac.yaml (4828 bytes)
I0220 16:57:53.682452   28192 addons.go:247] installing /etc/kubernetes/addons/ingress-dp.yaml
I0220 16:57:53.682452   28192 ssh_runner.go:310] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (8533 bytes)
I0220 16:57:53.717473   28192 ssh_runner.go:149] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml
I0220 16:57:53.859260   28192 addons.go:346] Verifying addon ingress=true in "minikube"
I0220 16:57:53.861262   28192 out.go:119] * Verifying ingress addon...
* Verifying ingress addon...
I0220 16:57:53.869261   28192 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "kube-system" ...
I0220 16:57:53.884052   28192 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0220 16:57:53.884052   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:54.391870   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:54.889169   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:55.400154   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:55.900645   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:56.403320   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:56.904065   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:57.400503   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:57.902159   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:58.391056   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:58.891532   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]

Windows 10 Docker version 20.10.2, build 2291f61 minikube version: v1.17.1 commit: 043bdca07e54ab6e4fc0457e3064048f34133d7e

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 52 (11 by maintainers)

Most upvoted comments

I’m seeing this same issue when enabling the ingress addon. Let me know what other information would be helpful here.

$ uname -a
Linux awx 4.19.0-14-arm64 #1 SMP Debian 4.19.171-2 (2021-01-30) aarch64 GNU/Linux
$ minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5
$ minikube addons enable ingress -v=5 --alsologtostderr
I0411 08:10:33.701931   21466 out.go:278] Setting OutFile to fd 1 ...
I0411 08:10:33.702134   21466 out.go:330] isatty.IsTerminal(1) = true
I0411 08:10:33.702167   21466 out.go:291] Setting ErrFile to fd 2...
I0411 08:10:33.702191   21466 out.go:330] isatty.IsTerminal(2) = true
I0411 08:10:33.702383   21466 root.go:317] Updating PATH: /home/nate/.minikube/bin
W0411 08:10:33.702615   21466 root.go:292] Error reading config file at /home/nate/.minikube/config/config.json: open /home/nate/.minikube/config/config.json: no such file or directory
I0411 08:10:33.703183   21466 addons.go:55] Setting ingress=true in profile "minikube"
I0411 08:10:33.703226   21466 addons.go:131] Setting addon ingress=true in "minikube"
I0411 08:10:33.703292   21466 host.go:66] Checking if "minikube" exists ...
I0411 08:10:33.704276   21466 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0411 08:10:33.775282   21466 out.go:157]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
I0411 08:10:33.776992   21466 out.go:157]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0411 08:10:33.778497   21466 out.go:157]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0411 08:10:33.778595   21466 addons.go:261] installing /etc/kubernetes/addons/ingress-configmap.yaml
I0411 08:10:33.778638   21466 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-configmap.yaml (1865 bytes)
I0411 08:10:33.778744   21466 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0411 08:10:33.837673   21466 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49157 SSHKeyPath:/home/nate/.minikube/machines/minikube/id_rsa Username:docker}
I0411 08:10:33.957576   21466 addons.go:261] installing /etc/kubernetes/addons/ingress-rbac.yaml
I0411 08:10:33.957619   21466 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-rbac.yaml (6005 bytes)
I0411 08:10:33.977878   21466 addons.go:261] installing /etc/kubernetes/addons/ingress-dp.yaml
I0411 08:10:33.977918   21466 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (9125 bytes)
I0411 08:10:33.998092   21466 ssh_runner.go:149] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml
I0411 08:10:34.351846   21466 addons.go:299] Verifying addon ingress=true in "minikube"
I0411 08:10:34.354113   21466 out.go:157] 🔎  Verifying ingress addon...
🔎  Verifying ingress addon...
I0411 08:10:34.357070   21466 kapi.go:59] client config for minikube: &rest.Config{Host:"https://192.168.49.2:8443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/home/nate/.minikube/profiles/minikube/client.crt", KeyFile:"/home/nate/.minikube/profiles/minikube/client.key", CAFile:"/home/nate/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0x132efd0), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}
I0411 08:10:34.357853   21466 cert_rotation.go:137] Starting client certificate rotation controller
I0411 08:10:34.361097   21466 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0411 08:10:34.387895   21466 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0411 08:10:34.387929   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:34.893599   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:35.392547   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:35.892783   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:36.392584   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:36.893074   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
+3
natesales on Apr 11, 2021

I think I know why, it because the China’s great GFW build by you know who.

In the minikube , here is a --image-mirror-country , and unfortunately I should set it to cn ,to use aliyun’s registry.

But in some reason, it not available. so I get that error.

If other dude in China, not use --image-mirror-country, it will use gcr and will get same thing for good reason by GFW.

solution

In minikube ssh pull image from docker hub :

  1. jettech/kube-webhook-certgen:v1.5.1
  2. willdockerhub/ingress-nginx-controller:v0.44.0

The second is a personal image just mirror ingress-nginx/controller for gcr

And then rename tag to what you need, and then minikube addons enable ingress

I think the registry of aliyun can use for his own build maybe

+2
yangyaofei on Aug 27, 2021

@adamf Can you do a fresh setup and send the logs?

$ minikube delete --purge --all
$ minikube start --addons=ingress --alsologtostderr

I’m having trouble recreating the issue. Also add any driver settings and make sure to paste them here.

+2
andriyDev on Jun 9, 2021

In my case, apache o ubunto was using port 80 sudo /etc/init.d/apache2 stop solved it

+2
zbeedatm on May 27, 2021

I’m in china and here is my way to solve this problem: minikube version: v1.23.2

I found no matter how I specify the --image="IngressController=***", It still goes to k8s.gcr.io So I looked into source code and found this

"ingress": NewAddon([]*BinAsset{
		MustBinAsset(addons.IngressAssets,
			"ingress/ingress-deploy.yaml.tmpl",
			vmpath.GuestAddonsDir,
			"ingress-deploy.yaml",
			"0640"),
	}, false, "ingress", "", map[string]string{
		// https://github.com/kubernetes/ingress-nginx/blob/557604f4ef526f7755d36089b617bc7686c389f9/deploy/static/provider/kind/deploy.yaml#L323
		"IngressController": "ingress-nginx/controller:v1.0.0-beta.3@sha256:44a7a06b71187a4529b0a9edee5cc22bdf71b414470eff696c3869ea8d90a695",
		// https://github.com/kubernetes/ingress-nginx/blob/557604f4ef526f7755d36089b617bc7686c389f9/deploy/static/provider/kind/deploy.yaml#L612
		"KubeWebhookCertgenCreate": "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068",
		// https://github.com/kubernetes/ingress-nginx/blob/557604f4ef526f7755d36089b617bc7686c389f9/deploy/static/provider/kind/deploy.yaml#L660
		"KubeWebhookCertgenPatch": "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068",
	}, map[string]string{
		"IngressController": "k8s.gcr.io",
	}),

func NewAddon(assets []*BinAsset, enabled bool, addonName string, maintainer string, images map[string]string, registries map[string]string) *Addon {
	a := &Addon{
		Assets:     assets,
		enabled:    enabled,
		addonName:  addonName,
		Maintainer: maintainer,
		Images:     images,
		Registries: registries,
	}
	return a
}

So I use minikube addons enable ingress --registries="IngressController=docker.io" --images="IngressController=wangshun1024/ingress-nginx-controller:v1.0.0-beta.3,KubeWebhookCertgenCreate=liangjw/kube-webhook-certgen:v1.0,KubeWebhookCertgenPatch=liangjw/kube-webhook-certgen:v1.0" and it works.

For others: You can found a replacement ingress and kube-webhook-certgen images for your version of minikube and use the same command as I but with different --images and --registries parameters.

Some suggestions: Maybe remove the Registries parameter for IngressController and move into Images directly is a better idea? just like
KubeWebhookCertgenCreate and KubeWebhookCertgenPatch.

+1
kevinw66 on Oct 24, 2021

I had the same issue with ‘minikube addons enable ingress’ command. Hence tried the following solution: Tried to pull the ingress image directly using ‘docker pull’ from command line Got the error - Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp 192.168.10.1:443: connect: connection refused Changed the network to another ISP and reissued the ‘docker pull’ command. This time the 2 images (k8s.gcr.io/ingress-nginx/controller:v0.44.0 , docker.io/jettech/kube-webhook-certgen:v1.5.1 ) downloaded properly Post that, issued the minikube addons enable ingress command and it worked successfully

I faced this image downloading issue during other scenarios also and had to use the same - switch ISP solution. So bottomline - one possibility to check is

  1. Change ISP
  2. Download images using normal ‘docker pull <supply-image-name-here>’
  3. Try ‘minikube addons enable ingress’ command again
+1
pshendre1 on Sep 16, 2021
Read more comments on GitHub
← minikube: Cannot find provisioner "k8s.io/minikube-hostpath" for hostPath-driven volumes
minikube: Can't pull images from an insecure registry in Minikube VM →