che: Unable to pull container images from the OpenShift internal registry

Describe the bug

Within a running workspace, podman is able to pull images from external registries like quay.io but is unable to pull images from the internal OpenShift Registry.

The internal registry certificate is not trusted, and the workspace user is not authorized.

Che version

7.63@latest

Steps to reproduce

Create a workspace with https://github.com/eclipse-che-demo-app/che-demo-app.git
Start a terminal

Execute:

podman pull quay.io/sclorg/postgresql-15-c9s:c9s

Observe success:

WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
Trying to pull quay.io/sclorg/postgresql-15-c9s:c9s...
Getting image source signatures
Copying blob 204a508c7797 done  
Copying blob 1a4b50973163 done  
Copying blob eae77addda82 done  
Copying config 034374e72d done  
Writing manifest to image destination
Storing signatures
034374e72d2c12dacc9e3557f8752d3407b1b69ca9aed1e7ea709a31922f180c

Execute:

podman pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest

Observe error:

Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest...
Error: initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift/cli:latest: pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": x509: certificate signed by unknown authority

Ignore TLS:

podman pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest --tls-verify=false

Observe authentication error:

Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest...
Error: initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift/cli:latest: reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift/cli: unauthorized: authentication required

Expected behavior

Expect podman configuration to be setup for interacting with the internal image registry.

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

macOS

Eclipse Che Logs

No response

Additional context

OpenShift is OKD 4.12

About this issue

Original URL
State: closed
Created a year ago
Comments: 20 (18 by maintainers)

Most upvoted comments

$HOME/.config/containers/certs.d is another path if it’s easier to get permissions there

benoitf on May 11, 2023

PR has been sent - https://github.com/eclipse-che/che-dashboard/pull/851

ibuziuk on Jul 4, 2023

@ibuziuk you are linking the external routes certs CA (ca.crt) but you need to link the internal services certs CA (service-ca.crt) to make it work.

If I run the following commands on developer sandbox

export CERTS_SRC="/var/run/secrets/kubernetes.io/serviceaccount"
export CERTS_DEST="$HOME/.config/containers/certs.d/image-registry.openshift-image-registry.svc:5000"

mkdir -p ${CERTS_DEST} && \
ln -s ${CERTS_SRC}/service-ca.crt ${CERTS_DEST}/service-ca.crt && \
podman pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest

I get unauthorized: authentication required (which is normal I guess) rather than x509: certificate signed by unknown authority

In general we should link both certificates.

l0rd on May 31, 2023

It is even better, the certificate is already mounted into a container:

podman login --cert-dir /var/run/secrets/kubernetes.io/serviceaccount -u $(oc whoami) -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000
podman pull --cert-dir /var/run/secrets/kubernetes.io/serviceaccount image-registry.openshift-image-registry.svc:5000/openshift/cli:latest

tolusha on Apr 11, 2023