che: [OCP4] Che deployment fails due to permission denied in Postgres

Describe the bug

I installed a basic OCP 4 cluster on AWS. The default aws-ebs storage is used. I tried to install Che from the OperatorHub marketplace and the install failed because Postgres entered a CrashLoopBackOff state.

The Postgres container’s logs show the following error:

johns-mbp-3:.odo johncollier$ oc logs postgres-cc6b567f-fc9hj
mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

Che version

latest
nightly
other: please specify

Steps to reproduce

Deploy OpenShift 4 on AWS
Make sure aws-ebs is used for default strorage (should be for new installs on AWS)
Install the Che operator from OperatorHub
Create a CheCluster custom resource to install Che
The Che deployment will fail because Postgres will report the following error:

Expected behavior

Runtime

kubernetes (include output of kubectl version)
Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minishift (include output of minishift version and oc version)
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

johns-mbp-3:.odo johncollier$ oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth

Server https://<url>:6443
kubernetes v1.14.0+573d946

Screenshots

Installation method

chectl
che-operator 7.0.0
minishift-addon
I don’t know

Environment

my computer
- Windows
- Linux
- macOS
Cloud
- Amazon
- Azure
- GCE
- other (please specify)
other: please specify

Additional context

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 25 (24 by maintainers)

Most upvoted comments

Comparing Che installed in the che namespace to Che installed in the default namespace reveals that the securityContext gets set to {} in default, and is properly set in the che namespace:

  securityContext:
    seLinuxOptions:
      level: 's0:c22,c9'
    fsGroup: 1000480000

tomgeorge on Oct 29, 2019

As I’m investigating this more, it seems like default does not respect any of the security context constraints by default that are present in openshift… If I make a new user and give them create access to pods and deployments, and they run a pod, it will run as root/the default UID present in that images Dockerfile. When I run the same pod in another namespace it is runs in a security context. default seems to have annotations regarding security contexts, but does not respect them:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c6,c5
    openshift.io/sa.scc.supplemental-groups: 1000040000/10000
    openshift.io/sa.scc.uid-range: 1000040000/10000
  creationTimestamp: "2019-10-28T20:25:55Z"
  name: default
  resourceVersion: "7335"
  selfLink: /api/v1/namespaces/default
  uid: 2515d6e5-f9c1-11e9-9124-028754979780
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c6,c5
    openshift.io/sa.scc.supplemental-groups: 1000040000/10000
    openshift.io/sa.scc.uid-range: 1000040000/10000
  creationTimestamp: "2019-10-28T20:25:55Z"
  name: default
  resourceVersion: "7335"
  selfLink: /apis/project.openshift.io/v1/projects/default
  uid: 2515d6e5-f9c1-11e9-9124-028754979780
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

I’m starting to think this is just a documentation issue that we will need to point out as @l0rd and @davidfestal have suggested. I reached out on the aos-devel slack channel but haven’t had a reply yet.

tomgeorge on Oct 30, 2019