pipelines: [backend] metadata-grpc-service mysql_real_connect failed (SSL enabled)

Environment

How did you deploy Kubeflow Pipelines (KFP)? Kubeflow Pipelines Standalone
KFP version: 1.7.0-alpha.1
KFP SDK version: 1.6.2

Steps to reproduce

Enable SSL on external MySQL configuration (Azure MySQL Server)
Restart metadata-grpc-deployment pod

Observe error:

2021-10-08 19:32:42.461171: F ml_metadata/metadata_store/metadata_store_server_main.cc:226] Non-OK-status: status status: Internal: mysql_real_connect failed: errno: 0, error: MetadataStore cannot be created with the given connection config.

Expected result

metadata-grpc-deployment should be initialized (not sure if SSL parameter is expected here in deployment config).

Materials and Reference

Running gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0.

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 12
Comments: 17 (1 by maintainers)

Most upvoted comments

I have a similar problem with kubeflow. MySQL pod is not running and so grpc and ml-pipeline pods can’t connect to the database and are stuck in CrashLoopBackOff. Did you find a solution?

dashanji on Jul 26, 2023

I don’t know the cause of this permission denial unfortunately. That’s different from my case, I had only the mysql pod missing so redeploying got it fixed.

mohamedFaris47 on Jul 26, 2023

I am not very familiar with SQL connection in Azure, the article I can find is https://docs.microsoft.com/en-us/azure/azure-sql/database/connectivity-architecture.

@berndverst for more help on Azure.

zijianjoy on Oct 17, 2021