longhorn: [BUG] OpenShift 4.15.3 - Lonhorn 1.6.1 - longhorn-ui nginx (13: Permission denied)

Question

How to run 1.6.1 on OpenShift 4.15.3 ? 1.4.2 worked on OpenShift 4.15.3

Environment

• Longhorn version: 1.6.1
• Kubernetes version: v1.28.7+6e2789b
• Node config
  • OS type and version: Red Hat Enterprise Linux CoreOS 415.92.202403080220-0 (Plow)
  • Kernel version: 5.14.0-284.55.1.el9_2.x86_64
  • CPU per node: 4
  • Memory per node: 8 GB worker / 16 GB controlplane
  • Disk type: Virtual Disk
  • Network bandwidth and latency between the nodes:
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): VMWare

Additional context

[core@osdemo1 ~]$ oc get nodes -o wide
NAME                 STATUS     ROLES                  AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
osdemo1.ourdomain.tld   Ready      control-plane,master   15d   v1.28.7+6e2789b   192.168.7.221   <none>        Red Hat Enterprise Linux CoreOS 415.92.202403080220-0 (Plow)   5.14.0-284.55.1.el9_2.x86_64   cri-o://1.28.4-4.rhaos4.15.git92d1839.el9
osdemo2.ourdomain.tld   Ready      control-plane,master   15d   v1.28.7+6e2789b   192.168.7.222   <none>        Red Hat Enterprise Linux CoreOS 415.92.202403080220-0 (Plow)   5.14.0-284.55.1.el9_2.x86_64   cri-o://1.28.4-4.rhaos4.15.git92d1839.el9
osdemo3.ourdomain.tld   Ready      control-plane,master   15d   v1.28.7+6e2789b   192.168.7.223   <none>        Red Hat Enterprise Linux CoreOS 415.92.202403080220-0 (Plow)   5.14.0-284.55.1.el9_2.x86_64   cri-o://1.28.4-4.rhaos4.15.git92d1839.el9
osdemo4.ourdomain.tld   Ready      worker                 15d   v1.28.7+6e2789b   192.168.7.224   <none>        Red Hat Enterprise Linux CoreOS 415.92.202403080220-0 (Plow)   5.14.0-284.55.1.el9_2.x86_64   cri-o://1.28.4-4.rhaos4.15.git92d1839.el9
osdemo5.ourdomain.tld   Ready      worker                 15d   v1.28.7+6e2789b   192.168.7.225   <none>        Red Hat Enterprise Linux CoreOS 415.92.202403080220-0 (Plow)   5.14.0-284.55.1.el9_2.x86_64   cri-o://1.28.4-4.rhaos4.15.git92d1839.el9
osdemo6.ourdomain.tld   Ready      worker                 15d   v1.28.7+6e2789b   192.168.7.226   <none>        Red Hat Enterprise Linux CoreOS 415.92.202403080220-0 (Plow)   5.14.0-284.55.1.el9_2.x86_64   cri-o://1.28.4-4.rhaos4.15.git92d1839.el9

wget https://raw.githubusercontent.com/longhorn/longhorn/v1.6.1/deploy/longhorn.yaml copy to controlplane oc apply -f longhorn.yaml oc adm policy add-scc-to-user anyuid -z default -n longhorn-system oc adm policy add-scc-to-user privileged -z longhorn-service-account -n longhorn-system

oc apply -f longhorn-route.yaml

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: longhorn
  namespace: longhorn-system
  labels:
    app: longhorn-ui
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/version: v1.6.1
spec:
  host: longhorn.apps.osc.ourdomain.tld
  to:
    kind: Service
    name: longhorn-frontend
    weight: 100
  port:
    targetPort: http
  wildcardPolicy: None

kubectl get clusterroles longhorn-role -o yaml > longhorn-role.yaml

add

- apiGroups:
  - longhorn.io
  resources:
  - engineimages/finalizers
  - instancemanagers/finalizers
  - sharemanagers/finalizers
  - backingimagemanagers/finalizer
  - nodes/finalizers
  verbs:
  - '*'

oc apply -f longhorn-role.yaml

oc get pods -n longhorn-system

[core@osdemo1 ~]$ oc get pods -n longhorn-system
NAME                                                READY   STATUS             RESTARTS        AGE
csi-attacher-5c4bfdcf59-2rssp                       1/1     Running            0               15m
csi-attacher-5c4bfdcf59-964n4                       1/1     Running            0               15m
csi-attacher-5c4bfdcf59-wmhpn                       1/1     Running            0               15m
csi-provisioner-667796df57-9mklv                    1/1     Running            0               15m
csi-provisioner-667796df57-m9n99                    1/1     Running            0               15m
csi-provisioner-667796df57-v5wlz                    1/1     Running            0               15m
csi-resizer-694f8f5f64-7mc5x                        1/1     Running            0               15m
csi-resizer-694f8f5f64-g7nbj                        1/1     Running            0               15m
csi-resizer-694f8f5f64-zdhdf                        1/1     Running            0               15m
csi-snapshotter-959b69d4b-dgxxq                     1/1     Running            0               15m
csi-snapshotter-959b69d4b-hvjks                     1/1     Running            0               15m
csi-snapshotter-959b69d4b-vjdps                     1/1     Running            0               15m
engine-image-ei-5cefaf2b-fbfp5                      1/1     Running            0               17m
engine-image-ei-5cefaf2b-jthk6                      1/1     Running            0               17m
engine-image-ei-5cefaf2b-k5tgc                      1/1     Running            0               17m
instance-manager-26d3e3e1b04c9e675a720b194c633a25   1/1     Running            0               16m
instance-manager-b422ca0f562b542741091e59c7e72dd6   1/1     Running            0               16m
instance-manager-f7cf196dfd5ab2261108a0131d2ec443   1/1     Running            0               16m
longhorn-csi-plugin-9fphh                           3/3     Running            0               15m
longhorn-csi-plugin-d4spq                           3/3     Running            0               15m
longhorn-csi-plugin-gbl22                           3/3     Running            0               15m
longhorn-driver-deployer-7449f56699-zz7md           1/1     Running            1 (16m ago)     17m
longhorn-manager-4ntcz                              1/1     Running            0               17m
longhorn-manager-k4srr                              1/1     Running            0               17m
longhorn-manager-kzlsc                              1/1     Running            0               17m
longhorn-ui-6c8c4fcbb8-8dhfl                        0/1     CrashLoopBackOff   6 (4m37s ago)   10m
longhorn-ui-6c8c4fcbb8-mrkhc                        0/1     CrashLoopBackOff   6 (4m52s ago)   10m

[core@osdemo1 ~]$ oc logs -n longhorn-system longhorn-ui-6c8c4fcbb8-mrkhc
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2024/04/03 11:29:18 [emerg] 10#10: mkdir() "/var/lib/nginx/tmp/" failed (13: Permission denied)
[core@osdemo1 ~]$ 

can not go inside the pod

[core@osdemo1 ~]$ kubectl exec -it longhorn-ui-6c8c4fcbb8-mrkhc -- sh
Error from server (NotFound): pods "longhorn-ui-6c8c4fcbb8-mrkhc" not found

What did I wrong ?

[core@osdemo1 ~]$ oc describe pod longhorn-ui-6c8c4fcbb8-mrkhc -n longhorn-system
Name:                 longhorn-ui-6c8c4fcbb8-mrkhc
Namespace:            longhorn-system
Priority:             1000000000
Priority Class Name:  longhorn-critical
Service Account:      longhorn-ui-service-account
Node:                 osdemo6.ourdomain.tld/192.168.7.226
Start Time:           Wed, 03 Apr 2024 11:18:26 +0000
Labels:               app=longhorn-ui
                      app.kubernetes.io/instance=longhorn
                      app.kubernetes.io/name=longhorn
                      app.kubernetes.io/version=v1.6.1
                      pod-template-hash=6c8c4fcbb8
Annotations:          k8s.ovn.org/pod-networks:
                        {"default":{"ip_addresses":["10.199.8.22/23"],"mac_address":"0a:58:0a:c7:08:16","gateway_ips":["10.199.8.1"],"routes":[{"dest":"10.199.0.0...
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ovn-kubernetes",
                            "interface": "eth0",
                            "ips": [
                                "10.199.8.22"
                            ],
                            "mac": "0a:58:0a:c7:08:16",
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: restricted-v2
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:               Running
SeccompProfile:       RuntimeDefault
IP:                   10.199.8.22
IPs:
  IP:           10.199.8.22
Controlled By:  ReplicaSet/longhorn-ui-6c8c4fcbb8
Containers:
  longhorn-ui:
    Container ID:   cri-o://5a6ede9de805b040fdef71c23c62d7a8e8062affebfb9cb100df6350322ca38f
    Image:          longhornio/longhorn-ui:v1.6.1
    Image ID:       docker.io/longhornio/longhorn-ui@sha256:4e8ca245c26260892c23a995410c0b2b51fb37405f305e2591730947ad24f26b
    Port:           8000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 03 Apr 2024 12:09:56 +0000
      Finished:     Wed, 03 Apr 2024 12:09:56 +0000
    Ready:          False
    Restart Count:  15
    Environment:
      LONGHORN_MANAGER_IP:  http://longhorn-backend:9500
      LONGHORN_UI_PORT:     8000
    Mounts:
      /var/cache/nginx/ from nginx-cache (rw)
      /var/config/nginx/ from nginx-config (rw)
      /var/run/ from var-run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vz52q (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nginx-cache:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  nginx-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  var-run:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-vz52q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       52m                    default-scheduler  Successfully assigned longhorn-system/longhorn-ui-6c8c4fcbb8-mrkhc to osdemo6.ourdomain.tld
  Normal   AddedInterface  52m                    multus             Add eth0 [10.199.8.22/23] from ovn-kubernetes
  Normal   Pulling         52m                    kubelet            Pulling image "longhornio/longhorn-ui:v1.6.1"
  Normal   Pulled          52m                    kubelet            Successfully pulled image "longhornio/longhorn-ui:v1.6.1" in 7.096s (7.096s including waiting)
  Normal   Created         50m (x5 over 52m)      kubelet            Created container longhorn-ui
  Normal   Started         50m (x5 over 52m)      kubelet            Started container longhorn-ui
  Normal   Pulled          50m (x4 over 52m)      kubelet            Container image "longhornio/longhorn-ui:v1.6.1" already present on machine
  Warning  BackOff         2m12s (x236 over 52m)  kubelet            Back-off restarting failed container longhorn-ui in pod longhorn-ui-6c8c4fcbb8-mrkhc_longhorn-system(dcd44fba-0fd2-4e1a-9891-1c1e308f6d75)