minikube: Can't pull images from an insecure registry in Minikube VM

This a BUG REPORT

Minikube version v0.9.0:

Environment:

OS Mac OS X
VM Driver xhyve
Docker version 1.12.1
Install tools:
Others:

What happened:

We currently host Docker images on a private, insecure registry hosted behind our firewall. (I use a VPN client (openconnect) in order to access the private registry.)

After adding the URL of the private registry in the “Advanced” tab of the Docker.app UI, I was able to use the Docker CLI pull and run images from the private registry without any problem.

I figured that I should be able to do the same thing after starting Minikube and specifying the same registry hostname:port combination that I did in the Docker UI, but using the --insecure-registry flag instead. However, when I tried to run the same image as above, the pod didn’t start up, and I saw errors like the following:

Failed to pull image "<image-name>": Error response from daemon: Get https://<registry-hostname>:5000/v1/_ping: EOF

I tried minikube ssh-ing into the VM and noticed that there was no evidence of the --insecure-registry flag being passed to the docker daemon command. I expected to see it included in EXTRA_ARGS in /var/lib/boot2docker/profile. I tried adding the flag to that file and restarting the Docker daemon – and was then able to pull and run images from the private registry as expected. Of course, when I stopped and restarted the Minikube VM, my changes were lost and I was no longer able to pull from the private registry.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 1
Comments: 19 (4 by maintainers)

Links to this issue

macos - How can I allow a private insecure registry to be used inside a minikube node? - Stack Overflow

Most upvoted comments

I just tried this with minikube v0.10.0 and --insecure-registry='docker-registry.example.com:443' was correctly copied into /var/lib/boot2docker/profile in the new xhyve VM. However, as @dlorenc hinted, the --insecure-registry flag is ignored if the machine already existed (even if it is stopped). You must first minikube delete if you want new flags to be respected (see #523 ).

+113

aecolley on Sep 17, 2016

@gabencui @aecolley You don’t need to delete and re-create the VM just to pass --insecure-registry.

There is a JSON file created after the minikube VM is created (I used virtualbox on linux) - $HOME/.minikube/machines/minikube/config.json - that seems to contain a lot of config for the VM.

Here is mine: https://gist.github.com/kunalg/015aacb58d18bd110844922da7329c22

In this, you can see some interesting config under ‘HostOptions’ -> ‘EngineOptions’ - including the ‘InsecureRegistry’.

You can edit this JSON array while your minikube VM is stopped. And, once restarted, it seems to take effect - without deleting the VM 😌 😁

This is the output from my current minikube vm (minikube ssh):

$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─10-machine.conf
   Active: active (running) since Fri 2017-06-16 17:18:44 UTC; 2 days ago
     Docs: https://docs.docker.com
           https://docs.docker.com
 Main PID: 3061 (docker)
    Tasks: 184
   Memory: 615.6M
      CPU: 31min 50.245s
   CGroup: /system.slice/docker.service
           ├─2809 docker-containerd-shim 6a777123b7d6e104e9648384ac22010aa3f1057b45e69e13938660e769d3a26f /var/run/docker/libcontainerd/6a777123b7d6e104e9648384ac22010aa3f1057b45e69e13938660e769d3a26f docker-runc
           ├─3061 /usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 192.168.99.1:5000
           ├─3088 docker-containerd -l /var/run/docker/libcontainerd/docker-containerd.sock --runtime docker-runc
           ├─3454 docker-containerd-shim 7847cef9dd3121421ed08ef36339f65630f46d9edb4ed3608f781aa9c886ad78 /var/run/docker/libcontainerd/7847cef9dd3121421ed08ef36339f65630f46d9edb4ed3608f781aa9c886ad78 docker-runc
           ├─3506 docker-containerd-shim fbd5cbe70c1baa64801e57a1a02aef4bc7f394b0023f7e5239e64d28bf88410d /var/run/docker/libcontainerd/fbd5cbe70c1baa64801e57a1a02aef4bc7f394b0023f7e5239e64d28bf88410d docker-runc
           ├─3586 docker-containerd-shim 7a8c730b42e59f49ad8f3e48955e7802db0c4399714e241d89a762654f31f236 /var/run/docker/libcontainerd/7a8c730b42e59f49ad8f3e48955e7802db0c4399714e241d89a762654f31f236 docker-runc
           ├─3762 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 18080 -container-ip 172.17.0.4 -container-port 18080
           ├─3795 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.17.0.4 -container-port 443
           ├─3809 docker-containerd-shim 17ba3c733bb58e2caf51af8d062c961864d815620da0d92a23ece4f9d811332a /var/run/docker/libcontainerd/17ba3c733bb58e2caf51af8d062c961864d815620da0d92a23ece4f9d811332a docker-runc
           ├─3814 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.4 -container-port 80
           ├─3857 docker-containerd-shim 6af25cb29ccfb82c902b201a45fa2b7d795a9a86182ca9443da43250e7528018 /var/run/docker/libcontainerd/6af25cb29ccfb82c902b201a45fa2b7d795a9a86182ca9443da43250e7528018 docker-runc
           ├─3859 docker-containerd-shim 5b14db4a4bf5d57ed04aa34cfc7baf0fed521fedb71fae60ddc717ccb6a1b35d /var/run/docker/libcontainerd/5b14db4a4bf5d57ed04aa34cfc7baf0fed521fedb71fae60ddc717ccb6a1b35d docker-runc
           ├─4000 docker-containerd-shim 249aa4921f43af0f2e7ee4bd9f8df87b7b975f3b15828221753eb5373ff6ef6c /var/run/docker/libcontainerd/249aa4921f43af0f2e7ee4bd9f8df87b7b975f3b15828221753eb5373ff6ef6c docker-runc
           ├─4065 docker-containerd-shim 0bda4419c09c5c1a6d0cd29bf5b6d25f0c13636888eba6067b3afda79f89dd64 /var/run/docker/libcontainerd/0bda4419c09c5c1a6d0cd29bf5b6d25f0c13636888eba6067b3afda79f89dd64 docker-runc
           ├─4116 docker-containerd-shim 2b8ebcdb17451604f237eaa9c92d580dc613ef19d4026d15313a515291b6292f /var/run/docker/libcontainerd/2b8ebcdb17451604f237eaa9c92d580dc613ef19d4026d15313a515291b6292f docker-runc
           ├─4198 docker-containerd-shim b3344fc0a219142a84792fc9a1f0c03aacf259eb7d582f49315cdff074378427 /var/run/docker/libcontainerd/b3344fc0a219142a84792fc9a1f0c03aacf259eb7d582f49315cdff074378427 docker-runc
           ├─4470 docker-containerd-shim 974f5751887d394e41b2817a613805690c0928155777bb6c80949bc813420e19 /var/run/docker/libcontainerd/974f5751887d394e41b2817a613805690c0928155777bb6c80949bc813420e19 docker-runc
           ├─4884 docker-containerd-shim bf40c06301a4916a7cbea5083d06a809bfc6876012cb2d0ad0564092ee312b07 /var/run/docker/libcontainerd/bf40c06301a4916a7cbea5083d06a809bfc6876012cb2d0ad0564092ee312b07 docker-runc
           └─4936 docker-containerd-shim 0c162ec316408c4f1766322d12dc101e642b3db4d813d14735b62d74fc4bd46d /var/run/docker/libcontainerd/0c162ec316408c4f1766322d12dc101e642b3db4d813d14735b62d74fc4bd46d docker-runc

If you look at process id 3061, it carries the --insecure-registry flag.

Now, following output is after stopping the VM, editing the config.json file to remove the insecure registry entry and restarting the vm:

$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─10-machine.conf
   Active: active (running) since Sun 2017-06-18 18:38:26 UTC; 59s ago
     Docs: https://docs.docker.com
           https://docs.docker.com
 Main PID: 3049 (docker)
    Tasks: 175
   Memory: 60.2M
      CPU: 2.953s
   CGroup: /system.slice/docker.service
           ├─3049 /usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox
           ├─3079 docker-containerd -l /var/run/docker/libcontainerd/docker-containerd.sock --runtime docker-runc
           ├─3470 docker-containerd-shim 86c2d37aedcffd56f6d02d807f93cb685ae1fcbb74eedd1cc52925751d59d455 /var/run/docker/libcontainerd/86c2d37aedcffd56f6d02d807f93cb685ae1fcbb74eedd1cc52925751d59d455 docker-runc
           ├─3525 docker-containerd-shim 2a32423f150c4e5efbc3f4a981be95532b26861be528947dc32cc97bff2dbf6f /var/run/docker/libcontainerd/2a32423f150c4e5efbc3f4a981be95532b26861be528947dc32cc97bff2dbf6f docker-runc
           ├─3609 docker-containerd-shim a866be39ee460111f8b5cbfdae48e5751a7af3cde5f3d26357ab6867c54dcb5a /var/run/docker/libcontainerd/a866be39ee460111f8b5cbfdae48e5751a7af3cde5f3d26357ab6867c54dcb5a docker-runc
           ├─3673 docker-containerd-shim 2298dcc0318f3a495669ebadd60240794b814c9693b2310478b56f62f4a2670f /var/run/docker/libcontainerd/2298dcc0318f3a495669ebadd60240794b814c9693b2310478b56f62f4a2670f docker-runc
           ├─3750 docker-containerd-shim 06d507f573f4dd3c38d912e26114900f71873db40727292d49eb8d8e7174c02e /var/run/docker/libcontainerd/06d507f573f4dd3c38d912e26114900f71873db40727292d49eb8d8e7174c02e docker-runc
           ├─3754 docker-containerd-shim b1efc9febec0eb283e5e527a87ce284150bf515897670e06b92e0dbb662e6d25 /var/run/docker/libcontainerd/b1efc9febec0eb283e5e527a87ce284150bf515897670e06b92e0dbb662e6d25 docker-runc
           ├─3871 docker-containerd-shim 4a8b286e3cc6d96316bfc1f364766a320e7dffdc4dcef74da09c68d79bbf5061 /var/run/docker/libcontainerd/4a8b286e3cc6d96316bfc1f364766a320e7dffdc4dcef74da09c68d79bbf5061 docker-runc
           ├─3926 docker-containerd-shim ac969b1a2c9ae674a83ee172b10dc60f4339b30409ff6a1b22e1b529be6af37f /var/run/docker/libcontainerd/ac969b1a2c9ae674a83ee172b10dc60f4339b30409ff6a1b22e1b529be6af37f docker-runc
           ├─3962 docker-containerd-shim d89dc294e265020b696b008aa5bbdf3656aa5a61d7f4ef68b3482778c91d7e1a /var/run/docker/libcontainerd/d89dc294e265020b696b008aa5bbdf3656aa5a61d7f4ef68b3482778c91d7e1a docker-runc
           ├─4011 docker-containerd-shim 515be31ac9b6342c597d618498731a6772fe0ffc7063a52b24186f10f0a46f2a /var/run/docker/libcontainerd/515be31ac9b6342c597d618498731a6772fe0ffc7063a52b24186f10f0a46f2a docker-runc
           ├─4060 docker-containerd-shim 12ca7b9024a5e108bb65e30ab46e1531fb59c68c0d022d47f36cb46e361c7d53 /var/run/docker/libcontainerd/12ca7b9024a5e108bb65e30ab46e1531fb59c68c0d022d47f36cb46e361c7d53 docker-runc
           ├─4156 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 18080 -container-ip 172.17.0.6 -container-port 18080
           ├─4165 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.17.0.6 -container-port 443
           ├─4173 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.6 -container-port 80
           ├─4187 docker-containerd-shim 9c65c289e1f90f5d022207edf8f86412661937fb20576652fdc7033ddffdf759 /var/run/docker/libcontainerd/9c65c289e1f90f5d022207edf8f86412661937fb20576652fdc7033ddffdf759 docker-runc
           ├─4236 docker-containerd-shim 9ffc6e01199d967257f9f8ddaf615136ae0aaa693f3a6d3073680898f3f0083a /var/run/docker/libcontainerd/9ffc6e01199d967257f9f8ddaf615136ae0aaa693f3a6d3073680898f3f0083a docker-runc
           └─4304 docker-containerd-shim 993bf7a64a3a731e436cb82afcc4307227142db582cc0543fbdc47fcfe515a68 /var/run/docker/libcontainerd/993bf7a64a3a731e436cb82afcc4307227142db582cc0543fbdc47fcfe515a68 docker-runc

Now, if you see process id 3049 in the above output, there is no --insecure-registry passed.

This is after adding the entry again:

$ systemctl status docker                                 
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─10-machine.conf
   Active: active (running) since Sun 2017-06-18 18:42:38 UTC; 1min 26s ago
     Docs: https://docs.docker.com
           https://docs.docker.com
 Main PID: 2922 (docker)
    Tasks: 182
   Memory: 63.4M
      CPU: 3.138s
   CGroup: /system.slice/docker.service
           ├─2922 /usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 192.168.99.1:5000
           ├─2950 docker-containerd -l /var/run/docker/libcontainerd/docker-containerd.sock --runtime docker-runc
           ├─3365 docker-containerd-shim 62b5787414c2f247c71562716732f235ef72d5cb3128d23650b2c5995bb19834 /var/run/docker/libcontainerd/62b5787414c2f247c71562716732f235ef72d5cb3128d23650b2c5995bb19834 docker-runc
           ├─3417 docker-containerd-shim a2466c5d3e47cd6fd6961b0a54987527e2f1b61dc802074922a50e56b8fbd025 /var/run/docker/libcontainerd/a2466c5d3e47cd6fd6961b0a54987527e2f1b61dc802074922a50e56b8fbd025 docker-runc
           ├─3519 docker-containerd-shim 9983df8d102b0102c21b53f5dc1ad77447777bdaae908712103632acb9ff8758 /var/run/docker/libcontainerd/9983df8d102b0102c21b53f5dc1ad77447777bdaae908712103632acb9ff8758 docker-runc
           ├─3566 docker-containerd-shim 59d3a76bf96fe06b7532315843ce362c0453f14f2f997d0fcf1786d564fc352a /var/run/docker/libcontainerd/59d3a76bf96fe06b7532315843ce362c0453f14f2f997d0fcf1786d564fc352a docker-runc
           ├─3640 docker-containerd-shim 2455788a4bfaee69170aa4b7f76f496c05ed1c53fffaf222f4b45c5b5e6ebf1d /var/run/docker/libcontainerd/2455788a4bfaee69170aa4b7f76f496c05ed1c53fffaf222f4b45c5b5e6ebf1d docker-runc
           ├─3671 docker-containerd-shim a9553273627afd202b5eb226a57c390d23f7b91362dd595e7f144a1068b1b509 /var/run/docker/libcontainerd/a9553273627afd202b5eb226a57c390d23f7b91362dd595e7f144a1068b1b509 docker-runc
           ├─3752 docker-containerd-shim c7163f71ed87525a94de87ac5aab151860684bbecc51fa7cf7bbd30ed3c33552 /var/run/docker/libcontainerd/c7163f71ed87525a94de87ac5aab151860684bbecc51fa7cf7bbd30ed3c33552 docker-runc
           ├─3819 docker-containerd-shim 4c2b0c19d2929737cfc53367dfd0e20a0945136098e0c47823b4c0c2dd75ef3f /var/run/docker/libcontainerd/4c2b0c19d2929737cfc53367dfd0e20a0945136098e0c47823b4c0c2dd75ef3f docker-runc
           ├─3906 docker-containerd-shim fd68f54489ec320dc66980677f9cd37cf759b4a28018a8658b5df83ff6990d3e /var/run/docker/libcontainerd/fd68f54489ec320dc66980677f9cd37cf759b4a28018a8658b5df83ff6990d3e docker-runc
           ├─3966 docker-containerd-shim 0d916d0bc26bf71582743fff9210bb6d1b2b16f33595385acf906b8a28315379 /var/run/docker/libcontainerd/0d916d0bc26bf71582743fff9210bb6d1b2b16f33595385acf906b8a28315379 docker-runc
           ├─4025 docker-containerd-shim 491fc9f75d10bf5aa0103ea587318c24aee0ec453806da01ecf21231bba77439 /var/run/docker/libcontainerd/491fc9f75d10bf5aa0103ea587318c24aee0ec453806da01ecf21231bba77439 docker-runc
           ├─4064 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 18080 -container-ip 172.17.0.6 -container-port 18080
           ├─4094 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.17.0.6 -container-port 443
           ├─4106 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.6 -container-port 80
           ├─4112 docker-containerd-shim 4f2af8bb6d39dbcce9b45e8c82484421bab09128fde91736110459314c354db6 /var/run/docker/libcontainerd/4f2af8bb6d39dbcce9b45e8c82484421bab09128fde91736110459314c354db6 docker-runc
           ├─4187 docker-containerd-shim 3cefdfc9f9c6aa1c0d8412fc06d1fa6d3034b51c7f6f355ecd4f622d886c0174 /var/run/docker/libcontainerd/3cefdfc9f9c6aa1c0d8412fc06d1fa6d3034b51c7f6f355ecd4f622d886c0174 docker-runc
           └─4284 docker-containerd-shim e9494013e94f6c30d0109203a1a4027c7613bfe99733b27193b0c63ef6246c1c /var/run/docker/libcontainerd/e9494013e94f6c30d0109203a1a4027c7613bfe99733b27193b0c63ef6246c1c docker-runc

And voilà, --insecure-registry flag back again in process id 2922. 😉

+27

kunalg on Jun 18, 2017

Sure, I passed it as a flag to minikube start:

minikube start --vm-driver="xhyve" --insecure-registry="<registry-hostname>:5000"

+21

dbriones on Sep 19, 2016

@aecolley thank you very match! “the --insecure-registry flag is ignored if the machine already existed (even if it is stopped). You must first minikube delete if you want new flags to be respected” is correct.

gabencui on May 24, 2017

Just wanted to add a note that as of minikube 0.19.0 (and maybe earlier, but I only got this now) minikube uses systemd, hence:

minikube ssh
sudo systemctl restart docker

aisrael on May 24, 2017

Finally I found the solution without deleting the VM here: http://stackoverflow.com/a/39698096/5408030

Also, I show you the content of my profile archive EXTRA_ARGS=’ –label provider=xhyve –insecure-registry=192.168.64.2:30873 ’ CACERT=/var/lib/boot2docker/ca.pem DOCKER_HOST=‘-H tcp://0.0.0.0:2376’ DOCKER_STORAGE=aufs DOCKER_TLS=auto SERVERKEY=/var/lib/boot2docker/server-key.pem SERVERCERT=/var/lib/boot2docker/server.pem

I copy the post below

For an http registry this steps works for me:

minikube ssh
edit /var/lib/boot2docker/profile and add to $EXTRA_ARGS --insecure-registry yourdomain.com:5000
restart the docker daemon sudo /etc/init.d/docker restart

cesargomezvela on Jan 16, 2017