rook: 500 internal error to access dashboard in v0.9.1

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

The dashboard failed after create a 3 mon 3 osd cluster with the cluster configuration:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v13.2.2-20181023
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  # set the amount of mons to be started
  mon:
    count: 3
    allowMultiplePerNode: true
  # enable the ceph dashboard for viewing cluster status
  dashboard:
    enabled: true
    hostNetwork: false
  rbdMirroring:
    workers: 0
  resources:
  storage: # cluster level storage configuration and selection
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    location:
    config:
      databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: "1024"  # this value can be removed for environments with normal sized disks (20 GB or larger)
      osdsPerDevice: "1" # this value can be overridden at the node or device level
    nodes:
    - name: xxx.compute.internal
      devices:
      - name: "xvdg"
    - name: xxxx.compute.internal
      devices:
      - name: "xvdg"
    - name: xxxxx.compute.internal
      devices:
      - name: "xvdg"

Expected behavior:

How to reproduce it (minimal and precise):

Create a cluster with the docs. And

kubectl port-forward svc/rook-ceph-mgr-dashboard 8443:8443 -n rook-ceph

Then open https://localhost:8443 got error, I got logs from mgr:

[15/Jan/2019:10:13:03] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1837, in start
    self.tick()
  File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1902, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/ssl_builtin.py", line 52, in wrap
    keyfile=self.private_key, ssl_version=ssl.PROTOCOL_SSLv23)
  File "/usr/lib64/python2.7/ssl.py", line 934, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib64/python2.7/ssl.py", line 609, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:579)

Environment:

  • OS (e.g. from /etc/os-release): ubuntu 16.04
  • Kernel (e.g. uname -a): Linux xxxx 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:40:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Cloud provider or hardware configuration: aws
  • Rook version (use rook version inside of a Rook Pod): v0.9.0
  • Kubernetes version (use kubectl version): v1.13.2
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): bare metal
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_WARN mon c is low on available space

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 5
  • Comments: 35 (17 by maintainers)

Commits related to this issue

Most upvoted comments

Setting ceph config set mgr mgr/dashboard/server_addr 0.0.0.0 and then restarting the ceph-mgr pod enables the port-forward approach and the NodePort approach still works too.

Is there any particular reason to bind to the pod IP if all the access is through one of these two methods?

cc #2526

Hello.

I tried to install rook v0.9.2 to freshly installed k8s cluster:

$ helm repo add rook-stable https://charts.rook.io/stable
$ helm install --namespace rook-ceph-system rook-stable/rook-ceph

$ helm ls
NAME            REVISION        UPDATED                         STATUS          CHART                   APP VERSION     NAMESPACE
wobbly-parrot   1               Sat Jan 26 10:46:18 2019        DEPLOYED        rook-ceph-v0.9.2                        rook-ceph-system

$ kubectl create -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/cluster.yaml

Then I tried the port redirection to rook-ceph-mgr:

$ kubectl -n rook-ceph port-forward $(kubectl -n rook-ceph get pod -l app=rook-ceph-mgr -o jsonpath='{.items[0].metadata.name}') 8443:8443

but I’m still getting errors when trying to access dashboard on https://localhost:8443:

E0126 11:14:37.732730    7221 portforward.go:331] an error occurred forwarding 8443 -> 8443: error forwarding port 8443 to pod 751f952e3c9fd53c574c322fb3bdf55a41c8e56212e15a9e5ae40ec5dfe746b1, uid : exit status 1: 2019/01/26 10:14:37 socat[19752] E connect(5, AF=2 127.0.0.1:8443, 16): Connection refused

Was this issue really fixed in v0.9.2 ? Is it working for anybody who is using v0.9.2 ?

I believe this is the same root cause as #2404. The fix is in master and I’d like to have it in a v0.9.2 release by tomorrow.

I’m able to access it via Ingress if I set the nginx.ingress.kubernetes.io/ssl-passthrough annotation to “true”.

Reviewing the docs, I now noticed the following text:

The dashboard may be served without SSL (useful for when you deploy the dashboard behind a proxy already served using SSL) by setting the ssl option to be false. Note that the ssl setting will be ignored in Luminous as well as Mimic 13.2.2 or older where it is not supported https://github.com/rook/rook/blob/master/Documentation/ceph-dashboard.md

In this case, I suppose this is the expected behavior, after all.