minishift: Openshift cannot pull the image from the internal registry

Type of issue

/kind bug

Steps to reproduce

$ eval $(minishift docker-env)
$ oc login -u developer -p developer
Login successful.

You have access to the following projects and can switch between them with 'oc project <projectname>':

    myproject
  * ticker

Using project "ticker".


$ oc new-project proj
Now using project "proj" on server "https://192.168.42.2:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.


$ docker pull centos/httpd
Using default tag: latest
latest: Pulling from centos/httpd
d9aaf4d82f24: Pull complete 
17695ed7ebda: Pull complete 
603fea4e6d47: Pull complete 
1923be8fd3f1: Pull complete 
Digest: sha256:79eacf66ec2d0eb930b0693dc524c729ea6fa04cb6ca825c0479231e85dc18f8
Status: Downloaded newer image for centos/httpd:latest


$ docker tag centos/httpd docker-registry-default.192.168.42.2.nip.io:443/proj/myhttpd
$ docker login -u developer -p `oc whoami -t` docker-registry-default.192.168.42.2.nip.io:443
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

$ docker push docker-registry-default.192.168.42.2.nip.io:443/proj/myhttpd
The push refers to a repository [docker-registry-default.192.168.42.2.nip.io:443/proj/myhttpd]
ffdbc94377b2: Pushed 
95847010d10a: Pushed 
2a9e8fbc6354: Pushed 
cf516324493c: Pushed 
latest: digest: sha256:17d46753675e6014df5170d44ff5c748e19ab053afb7be00c79ed99c4b3e0831 size: 1155


$ oc run web --image docker-registry-default.192.168.42.2.nip.io:443/proj/myhttpd
deploymentconfig "web" created

$ oc get pods
NAME           READY     STATUS             RESTARTS   AGE
web-1-8g06l    0/1       ImagePullBackOff   0          4m
web-1-deploy   1/1       Running            0          4m

$ oc get events
LASTSEEN   FIRSTSEEN   COUNT     NAME           KIND                    SUBOBJECT                     TYPE      REASON              SOURCE                        MESSAGE
33s        33s         1         web-1-8g06l    Pod                                                   Normal    Scheduled           default-scheduler             Successfully assigned web-1-8g06l to localhost
17s        32s         2         web-1-8g06l    Pod                     spec.containers{web}          Normal    Pulling             kubelet, localhost            pulling image "docker-registry-default.192.168.42.2.nip.io:443/proj/myhttpd"
16s        32s         2         web-1-8g06l    Pod                     spec.containers{web}          Warning   Failed              kubelet, localhost            Failed to pull image "docker-registry-default.192.168.42.2.nip.io:443/proj/myhttpd": rpc error: code = 2 desc = Error: image proj/myhttpd:latest not found
6s         32s         4         web-1-8g06l    Pod                                                   Warning   FailedSync          kubelet, localhost            Error syncing pod
6s         31s         2         web-1-8g06l    Pod                     spec.containers{web}          Normal    BackOff             kubelet, localhost            Back-off pulling image "docker-registry-default.192.168.42.2.nip.io:443/proj/myhttpd"
35s        35s         1         web-1-deploy   Pod                                                   Normal    Scheduled           default-scheduler             Successfully assigned web-1-deploy to localhost
34s        34s         1         web-1-deploy   Pod                     spec.containers{deployment}   Normal    Pulled              kubelet, localhost            Container image "openshift/origin-deployer:v3.6.0" already present on machine
33s        33s         1         web-1-deploy   Pod                     spec.containers{deployment}   Normal    Created             kubelet, localhost            Created container
33s        33s         1         web-1-deploy   Pod                     spec.containers{deployment}   Normal    Started             kubelet, localhost            Started container
33s        33s         1         web-1          ReplicationController                                 Normal    SuccessfulCreate    replication-controller        Created pod: web-1-8g06l
35s        35s         1         web            DeploymentConfig                                      Normal    DeploymentCreated   deploymentconfig-controller   Created new replication controller "web-1" for version 1

Other info

Minishift start logs

$ minishift start --cpus 4 --memory 6144 
-- Checking if KVM driver is installed ... 
   Driver is available at /usr/local/bin/docker-machine-driver-kvm ... 
   Checking driver binary is executable ... OK
-- Checking if Libvirt is installed ... OK
-- Checking if Libvirt default network is present and active ... OK
-- Starting local OpenShift cluster using 'kvm' hypervisor ...
-- Minishift VM will be configured with ...
   Memory:    6 GB
   vCPUs :    4
   Disk size: 20 GB
-- Starting Minishift VM ....................................................... OK
-- Checking for IP address ... OK
-- Checking if external host is reachable from the Minishift VM ... 
   Pinging 8.8.8.8 ... OK
-- Checking HTTP connectivity from the VM ... 
   Retrieving http://minishift.io/index.html ... OK
-- Checking if persistent storage volume is mounted ... OK
-- Checking available disk space ... 0% used OK
-- Downloading OpenShift binary 'oc' version 'v3.6.0'
 3.92 MiB / 34.72 MiB [=================>--------------------------------------------------------------------------------------------------------------------------------------]  11.28% 8m52s 34.72 MiB / 34.72 MiB [==========================================================================================================================================================] 100.00% 0s-- Downloading OpenShift v3.6.0 checksums ... OK
-- OpenShift cluster will be configured with ...
   Version: v3.6.0
-- Checking `oc` support for startup flags ... 
   host-config-dir ... OK
   host-data-dir ... OK
   host-pv-dir ... OK
   host-volumes-dir ... OK
   routing-suffix ... OK
Starting OpenShift using openshift/origin:v3.6.0 ...
Pulling image openshift/origin:v3.6.0
Pulled 1/4 layers, 26% complete
Pulled 2/4 layers, 54% complete
Pulled 3/4 layers, 90% complete
Pulled 4/4 layers, 100% complete
Extracting
Image pull complete
OpenShift server started.

The server is accessible via web console at:
    https://192.168.42.2:8443

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

-- Waiting for persistent volumes to be created ... OK

-- Applying addon 'admin-user':..

-- Applying addon 'anyuid':.
 Add-on 'anyuid' changed the default security context constraints to allow pods to run as any user.
 Per default OpenShift runs containers using an arbitrarily assigned user ID.
 Refer to https://docs.openshift.org/latest/architecture/additional_concepts/authorization.html#security-context-constraints and
 https://docs.openshift.org/latest/creating_images/guidelines.html#openshift-origin-specific-guidelines for more information.

-- Applying addon 'registry-route':........
Add-on 'registry-route' created docker-registry route. Please run following commands to login to the OpenShift docker registry:
$ eval $(minishift docker-env)
$ docker login -u developer -p `oc whoami -t` docker-registry-default.192.168.42.2.nip.io:443

versions:

$ minishift version
minishift v1.7.0+1549135

$ oc version
oc v3.7.0-alpha.1+fdbd3dc
kubernetes v1.7.0+695f48a16f
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.42.2:8443
openshift v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 22 (11 by maintainers)

Commits related to this issue

Issue #1615 Document update for internal registry image pull - oc run web --image internal_expose_registry/project/myweb fails to pull images from exposed internal registry. — committed to praveenkumar/minishift by praveenkumar 6 years ago
Issue #1615 Document update for internal registry image pull - oc run web --image internal_expose_registry/project/myweb fails to pull images from exposed internal registry. — committed to minishift/minishift by praveenkumar 6 years ago

Most upvoted comments

@praveenkumar. I this is kinda expected behavior. Idea is that registry ,from openshift point of view, is an external registry. It’s being treated as one. And by default it requires authentication. I don’t think it’s valid point to implement this for oc cluster up as registry expose was not implemented there too. It sounds more as minishift feature.

The solution would be:

docker login -u developer -p oc whoami -t docker-registry-default.127.0.0.1.nip.io
oc secrets new internal-reg .dockerconfigjson=/root/.docker/config.json #or alternatilvy pull flag based secret creation syntax oc secrets new-dockercfg myseret --docker-email=test --docker-password=$(oc whoami -t) --docker-username=admin --docker-server=docker-registry-default.127.0.0.1.nip.io
oc secrets link default internal-reg or add it to deployment configs. This will enable DC to pull images using this secret from exposed registry.

But because it’s per project basis you might want to think how to do this…

At the same time as soon as you push image into registry is is being created, and if you use internal is reference, pull should be handled using default secrets.

oc run web --image=172.30.1.1:5000/myproject/myhttpd

mjudeikis on Dec 7, 2017

@anjannath : i tried to deploy Jenkins image from webconsole. add to project -> browse catalog -> Jenkins (Ephemeral)

This itself errors out when i try to deploy a project with Jenkins (Ephemeral) image

deepikathimmegowda on Nov 8, 2018