katib: katib-mysql: Access denied for user 'root'@'localhost' (using password: YES)
/kind bug
What steps did you take and what happened: Installed Kubeflow on an onpremise kubenetes cluster. Multiple Pods stuck in CrashLoopBackOff katib-db-manager katib-mysql
What did you expect to happen: Expect that the pods start normal
Anything else you would like to add: As mentioned in #1156 I increased liveness probe and readiness probe. I think, that the katib-db-manager also works if the katib-mysql is running.
kubectl -nkubeflow describe pod katib-mysql-98566c454-2kg4t
Name: katib-mysql-98566c454-2kg4t
Namespace: kubeflow
Priority: 0
Node:
Start Time: Fri, 08 Jan 2021 08:45:35 +0000
Labels: app=katib
app.kubernetes.io/component=katib
app.kubernetes.io/name=katib-controller
component=mysql
pod-template-hash=98566c454
Annotations: sidecar.istio.io/inject: false
Status: Running
IP:
IPs:
IP:
Controlled By: ReplicaSet/katib-mysql-98566c454
Containers:
katib-mysql:
Container ID: docker://8c14869f8a0cecdd6df1e6df2829838e11922167b66db1c6b5582d4ddbfb0818
Image: mysql:8
Image ID: docker-pullable://mysql@sha256:365e891b22abd3336d65baefc475b4a9a1e29a01a7b6b5be04367fcc9f373bb7
Port: 3306/TCP
Host Port: 0/TCP
Args:
--datadir
/var/lib/mysql/datadir
State: Running
Started: Fri, 08 Jan 2021 08:45:37 +0000
Ready: False
Restart Count: 0
Liveness: exec [/bin/bash -c mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}] delay=30s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [/bin/bash -c mysql -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=6
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-mysql-secrets'> Optional: false
MYSQL_ALLOW_EMPTY_PASSWORD: true
MYSQL_DATABASE: katib
Mounts:
/var/lib/mysql from katib-mysql (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5vvg8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
katib-mysql:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: katib-mysql
ReadOnly: false
default-token-5vvg8:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5vvg8
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 49s default-scheduler Successfully assigned kubeflow/katib-mysql-98566c454-2kg4t to
Normal Pulled 47s kubelet Container image "mysql:8" already present on machine
Normal Created 47s kubelet Created container katib-mysql
Normal Started 47s kubelet Started container katib-mysql
Warning Unhealthy 3s (x4 over 33s) kubelet Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
kubectl -nkubeflow logs katib-mysql-98566c454-2kg4t
2021-01-08 08:45:37+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.
2021-01-08 08:45:38+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2021-01-08 08:45:38+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.
2021-01-08T08:45:38.569814Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.22) starting as process 1
2021-01-08T08:45:38.676491Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-01-08T08:45:40.429366Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-01-08T08:45:40.636986Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-01-08T08:45:41.164747Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-01-08T08:45:41.164913Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-01-08T08:45:41.169798Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2021-01-08T08:45:41.188249Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
kubectl -nkubeflow exec -it katib-mysql-98566c454-2kg4t -- /bin/bash
root@katib-mysql-98566c454-2kg4t:/# echo $MYSQL_ROOT_PASSWORD
test
root@katib-mysql-98566c454-2kg4t:/# echo $MYSQL_DATABASE
katib
root@katib-mysql-98566c454-2kg4t:/# mysqladmin ping -u root -p$MYSQL_ROOT_PASSWORD
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
root@katib-mysql-98566c454-2kg4t:/# mysql -u root -e 'SELECT 1'
+---+
| 1 |
+---+
| 1 |
+---+
root@katib-mysql-98566c454-2kg4t:/# mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 41
Server version: 8.0.22 MySQL Community Server - GPL
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.01 sec)
Environment:
- Kubeflow version: kfctl v1.2.0-0-gbc038f9
- OnPremise Kubernetes Cluster
- Kubernetes version: v1.17.0
- OS: Ubuntu 18.04.5 LTS
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 37 (15 by maintainers)
@chinmay22897 Yes.
It works after I exec to the mysql pod, and execute below commands:
Thank you @anneum your fix worked for me. This is what I did:
kubectl edit deployment katib-mysql -n kubeflow, then somevim-fu to update the delays.pvc-25dc2b81-9873-430f-8bc4-365fe5ff0357_kubeflow_katib-mysql.kubectl delete pod -n kubeflow katib-mysql-...Indeed, it took 2 minutes to initialize the DB.
@andreyvelich Thank you very much for your help. Now it seems to work 👍 I changed the
katib-mysqlpod:readinessProbetoinitialDelaySeconds: 50livenessProbetoinitialDelaySeconds: 300As you can see in the logs, the
katib-mysqlpod takes 2,5 minutes to initialize the database files. Thekatib-db-managerpod will reboot twice in the meantime, but that’s not a problem.In summary, what has been changed from the original katib repository:
katib-mysqldeployment pod:readinessProbetoinitialDelaySeconds: 50livenessProbetoinitialDelaySeconds: 300Same issue here when install kubeflow in minikube, following https://www.kubeflow.org/docs/started/workstation/minikube-linux/
Basically password of root is “” rather than $MYSQL_ROOT_PASSWORD, which fail the redinessProbe