kubeflow: ERROR 1049 (42000): Unknown database 'katib'

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.] I have install kubeflow via kfctl_k8s_istio.v1.0.1.yaml on existed kubernetes cluster v1.14.4. all pod were running properly except two: katib-mysql and katib-db-manager

[vimer@xushangming-dev ~]$ kubectl -n kubeflow describe pod katib-mysql-57884cb488-6gg98 Name: katib-mysql-57884cb488-6gg98 Namespace: kubeflow Priority: 0 PriorityClassName: <none> Node: k8s-slave02/192.168.10.186 Start Time: Mon, 16 Mar 2020 14:25:31 +0800 Labels: app=katib app.kubernetes.io/component=katib app.kubernetes.io/instance=katib-controller-0.8.0 app.kubernetes.io/managed-by=kfctl app.kubernetes.io/name=katib-controller app.kubernetes.io/part-of=kubeflow app.kubernetes.io/version=0.8.0 component=mysql pod-template-hash=57884cb488 Annotations: cni.projectcalico.org/podIP: 10.244.2.123/32 sidecar.istio.io/inject: false Status: Running IP: 10.244.2.123 Controlled By: ReplicaSet/katib-mysql-57884cb488 Containers: katib-mysql: Container ID: docker://e2bb6b7bc9baa2402dfac6ee7c14b4bc67f2ef084590a81d8c3c0303128e888f Image: mysql:8 Image ID: docker-pullable://mysql@sha256:4a30434ce03d2fa396d0414f075ad9ca9b0b578f14ea5685e24dcbf789450a2c Port: 3306/TCP Host Port: 0/TCP Args: –datadir /var/lib/mysql/datadir State: Running Started: Mon, 16 Mar 2020 14:25:34 +0800 Ready: False Restart Count: 0 Liveness: exec [/bin/bash -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=30s timeout=5s period=10s #success=1 #failure=3 Readiness: exec [/bin/bash -c mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e ‘SELECT 1’] delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: MYSQL_ROOT_PASSWORD: <set to the key ‘MYSQL_ROOT_PASSWORD’ in secret ‘katib-mysql-secrets’> Optional: false MYSQL_ALLOW_EMPTY_PASSWORD: true MYSQL_DATABASE: katib Mounts: /var/lib/mysql from katib-mysql (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-rhqvt (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: katib-mysql: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: katib-mysql ReadOnly: false default-token-rhqvt: Type: Secret (a volume populated by a Secret) SecretName: default-token-rhqvt Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message

---

Normal Scheduled 37m default-scheduler Successfully assigned kubeflow/katib-mysql-57884cb488-6gg98 to k8s-slave02 Normal Pulled 37m kubelet, k8s-slave02 Container image “mysql:8” already present on machine Normal Created 37m kubelet, k8s-slave02 Created container katib-mysql Normal Started 37m kubelet, k8s-slave02 Started container katib-mysql Warning Unhealthy 2m19s (x211 over 37m) kubelet, k8s-slave02 Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 1045 (28000): Access denied for user ‘root’@‘localhost’ (using password: YES)

[vimer@xushangming-dev kubeflow]$ kubectl logs katib-db-manager-64f548b47c-p76vb -n kubeflow E0316 07:13:33.545869 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:38.559161 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:43.560064 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:48.560665 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:53.561246 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:58.563278 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:03.565287 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:08.566754 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:13.567753 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:18.568420 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:23.569090 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout F0316 07:14:23.569212 1 main.go:83] Failed to open db connection: DB open failed: Timeout waiting for DB conn successfully opened. goroutine 1 [running]: github.com/kubeflow/katib/vendor/k8s.io/klog.stacks(0xc000238200, 0xc000250000, 0x89, 0xc0) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:830 +0xb8 github.com/kubeflow/katib/vendor/k8s.io/klog.(*loggingT).output(0xdf1ca0, 0xc000000003, 0xc00024e000, 0xd93a76, 0x7, 0x53, 0x0) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:781 +0x2d0 github.com/kubeflow/katib/vendor/k8s.io/klog.(*loggingT).printf(0xdf1ca0, 0x3, 0x9b448c, 0x20, 0xc0001b5f20, 0x1, 0x1) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:678 +0x14b github.com/kubeflow/katib/vendor/k8s.io/klog.Fatalf(…) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:1209 main.main() /go/src/github.com/kubeflow/katib/cmd/db-manager/v1alpha3/main.go:83 +0x165 [vimer@xushangming-dev kubeflow]$ [vimer@xushangming-dev kubeflow]$ kubectl logs -n kubeflow katib-mysql-57884cb488-6gg98 2020-03-16 06:25:34+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.19-1debian10 started. 2020-03-16 06:25:34+00:00 [Note] [Entrypoint]: Switching to dedicated user ‘mysql’ 2020-03-16 06:25:34+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.19-1debian10 started. 2020-03-16T06:25:34.763839Z 0 [Warning] [MY-011070] [Server] ‘Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it’ is deprecated and will be removed in a future release. 2020-03-16T06:25:34.765274Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.19) starting as process 1 2020-03-16T06:25:36.204511Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2020-03-16T06:25:36.239657Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location ‘/var/run/mysqld’ in the path is accessible to all OS users. Consider choosing a different directory. 2020-03-16T06:25:36.283798Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: ‘8.0.19’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 MySQL Community Server - GPL. 2020-03-16T06:25:36.428562Z 0 [ERROR] [MY-011294] [Server] Plugin mysqlx reported: ‘Unable to use user mysql.session account when connecting the server for internal plugin requests.’ 2020-03-16T06:25:36.428752Z 0 [ERROR] [MY-011301] [Server] Plugin mysqlx reported: ‘Unable to switch context to user mysql.session’ [vimer@xushangming-dev kubeflow]$

What did you expect to happen: [vimer@xushangming-dev kubeflow]$ kubectl logs katib-db-manager-64f548b47c-p76vb -n kubeflow E0316 07:13:33.545869 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:38.559161 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:43.560064 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:48.560665 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:53.561246 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:13:58.563278 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:03.565287 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:08.566754 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:13.567753 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:18.568420 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout E0316 07:14:23.569090 1 mysql.go:62] Ping to Katib db failed: dial tcp 10.97.244.59:3306: i/o timeout F0316 07:14:23.569212 1 main.go:83] Failed to open db connection: DB open failed: Timeout waiting for DB conn successfully opened. goroutine 1 [running]: github.com/kubeflow/katib/vendor/k8s.io/klog.stacks(0xc000238200, 0xc000250000, 0x89, 0xc0) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:830 +0xb8 github.com/kubeflow/katib/vendor/k8s.io/klog.(*loggingT).output(0xdf1ca0, 0xc000000003, 0xc00024e000, 0xd93a76, 0x7, 0x53, 0x0) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:781 +0x2d0 github.com/kubeflow/katib/vendor/k8s.io/klog.(*loggingT).printf(0xdf1ca0, 0x3, 0x9b448c, 0x20, 0xc0001b5f20, 0x1, 0x1) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:678 +0x14b github.com/kubeflow/katib/vendor/k8s.io/klog.Fatalf(…) /go/src/github.com/kubeflow/katib/vendor/k8s.io/klog/klog.go:1209 main.main() /go/src/github.com/kubeflow/katib/cmd/db-manager/v1alpha3/main.go:83 +0x165

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):
• kfctl version: (use kfctl version): kfctl v1.0.1-0-gf3edb9b
• Kubernetes platform: (e.g. minikube) docker
• Kubernetes version: (use kubectl version): v1.14.4
• OS (e.g. from /etc/os-release): CentOS Linux release 7.7.1908 (Core)