postgres-operator: Minimal postgresql manifest example does not start correctly: "could not connect to PostgreSQL database" by operator

Hi,

I am doing my first steps with postgres-operator, so I’m not sure if it’s a mistake by my side or what’s exactly going wrong here.

I installed postgres-operator by using the helm chart, without having any problems. I found the following problem: I’m using the minimal example from the documentation (https://postgres-operator.readthedocs.io/en/latest/user/#create-a-manifest-for-a-new-postgresql-cluster) to set up a PostgreSQL cluster. Pods get started, everything looked good so far. Then I tried to connect to the master service - without success. I discovered, that the master service does not have any selector and the operator logs are also full of connection errors. What did I wrong?

The template’s questions:

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.5.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Kubernetes on Hetzner Cloud
• Are you running Postgres Operator in production? not yet
• Type of issue? Question, maybe bug.

kubectl get all -o wide:

NAME                         READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
pod/acid-minimal-cluster-0   1/1     Running   0          10m   10.244.68.220   worker-fsn1-2   <none>           <none>
pod/acid-minimal-cluster-1   1/1     Running   0          10m   10.244.67.58    worker-nbg1-1   <none>           <none>

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE     SELECTOR
service/acid-minimal-cluster          ClusterIP   10.244.179.108   <none>        5432/TCP   10m     <none>
service/acid-minimal-cluster-config   ClusterIP   None             <none>        <none>     10m     <none>
service/acid-minimal-cluster-repl     ClusterIP   10.244.147.195   <none>        5432/TCP   10m     application=spilo,cluster-name=acid-minimal-cluster,spilo-role=replica
service/kubernetes                    ClusterIP   10.244.128.1     <none>        443/TCP    3d14h   <none>

NAME                                    READY   AGE   CONTAINERS   IMAGES
statefulset.apps/acid-minimal-cluster   2/2     10m   postgres     registry.opensource.zalan.do/acid/spilo-12:1.6-p3

Operator logs:

time="2020-11-06T08:55:58Z" level=info msg="\"ADD\" event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=1
time="2020-11-06T08:55:58Z" level=info msg="creation of the cluster started" cluster-name=default/acid-minimal-cluster pkg=controller worker=1
time="2020-11-06T08:55:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1550124\", FieldPath:\"\"}): type: 'Normal' reason: 'Create' Started creation of new cluster resources"
time="2020-11-06T08:55:58Z" level=info msg="endpoint \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1550124\", FieldPath:\"\"}): type: 'Normal' reason: 'Endpoints' Endpoint \"default/acid-minimal-cluster\" has been successfully created"
time="2020-11-06T08:55:58Z" level=info msg="master service \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:58Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1550124\", FieldPath:\"\"}): type: 'Normal' reason: 'Services' The service \"default/acid-minimal-cluster\" for role master has been successfully created"
time="2020-11-06T08:55:58Z" level=info msg="replica service \"default/acid-minimal-cluster-repl\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1550124\", FieldPath:\"\"}): type: 'Normal' reason: 'Services' The service \"default/acid-minimal-cluster-repl\" for role replica has been successfully created"
time="2020-11-06T08:55:58Z" level=debug msg="team API is disabled, returning empty list of members for team \"acid\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:58Z" level=info msg="users have been initialized" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=debug msg="secret \"default/standby.acid-minimal-cluster.credentials.postgresql.acid.zalan.do\" already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=debug msg="secret \"default/postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do\" already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=debug msg="created new secret \"default/zalando.acid-minimal-cluster.credentials.postgresql.acid.zalan.do\", uid: \"6ae7dd7f-1f39-45b0-a49d-78f3ea8b745c\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=debug msg="created new secret \"default/foo-user.acid-minimal-cluster.credentials.postgresql.acid.zalan.do\", uid: \"6d833134-f94b-426b-8611-6dab51d4981b\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1550124\", FieldPath:\"\"}): type: 'Normal' reason: 'Secrets' The secrets have been successfully created"
time="2020-11-06T08:55:59Z" level=info msg="pod disruption budget \"default/postgres-acid-minimal-cluster-pdb\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE acid-minimal-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/12/bin\"},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=debug msg="created new statefulset \"default/acid-minimal-cluster\", uid: \"43668cb0-5667-4436-bc2a-0c45c9b3c1dc\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=info msg="statefulset \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:55:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1550124\", FieldPath:\"\"}): type: 'Normal' reason: 'StatefulSet' Statefulset \"default/acid-minimal-cluster\" has been successfully created"
time="2020-11-06T08:56:17Z" level=debug msg="Waiting for 2 pods to become ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:56:38Z" level=info msg="pods are ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:56:38Z" level=info msg="Create roles" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:56:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1550124\", FieldPath:\"\"}): type: 'Normal' reason: 'StatefulSet' Pods are ready"
time="2020-11-06T08:56:53Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:57:08Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:57:23Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:57:38Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:57:53Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:58:08Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:58:23Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:58:38Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.244.179.108:5432: i/o timeout" cluster-name=default/acid-minimal-cluster pkg=cluster worker=1
time="2020-11-06T08:58:38Z" level=error msg="could not create cluster: could not create users: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster pkg=controller worker=1
time="2020-11-06T08:58:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"postgresql\", Namespace:\"default\", Name:\"acid-minimal-cluster\", UID:\"c3b1d195-81be-4a01-8a06-1966b1f289f1\", APIVersion:\"acid.zalan.do/v1\", ResourceVersion:\"1551012\", FieldPath:\"\"}): type: 'Warning' reason: 'Create' could not create cluster: could not create users: could not init db connection: could not init db connection: still failing after 8 retries"