mlflow: [BUG] compatibility with protobuf 3.15?

Willingness to contribute

No. I cannot contribute a bug fix at this time.

System information

  • Have I written custom code (as opposed to using a stock example script provided in MLflow):
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.1 LTS
  • MLflow installed from (source or binary): pip
  • MLflow version (run mlflow --version): 1.26.1
  • Python version: 3.8.8
  • yarn version, if running the dev UI:

Describe the problem

integration tests failed when using mlflow 1.26.1 (worked fine with 1.26.0)

[2022-05-28T01:55:47.732Z] ======================================================================
[2022-05-28T01:55:47.732Z] ERROR: test_metrics_track (tests.test_handler_mlflow.TestHandlerMLFlow)
[2022-05-28T01:55:47.732Z] ----------------------------------------------------------------------
[2022-05-28T01:55:47.732Z] Traceback (most recent call last):
[2022-05-28T01:55:47.732Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_handler_mlflow.py", line 42, in test_metrics_track
[2022-05-28T01:55:47.732Z]     handler = MLFlowHandler(
[2022-05-28T01:55:47.732Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/handlers/mlflow_handler.py", line 91, in __init__
[2022-05-28T01:55:47.733Z]     mlflow.set_tracking_uri(tracking_uri)
[2022-05-28T01:55:47.733Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/utils/module.py", line 378, in __getattr__
[2022-05-28T01:55:47.733Z]     raise self._exception
[2022-05-28T01:55:47.733Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/utils/module.py", line 337, in optional_import
[2022-05-28T01:55:47.733Z]     pkg = __import__(module)  # top level module
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/__init__.py", line 32, in <module>
[2022-05-28T01:55:47.733Z]     import mlflow.tracking._model_registry.fluent
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/tracking/__init__.py", line 8, in <module>
[2022-05-28T01:55:47.733Z]     from mlflow.tracking.client import MlflowClient
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/tracking/client.py", line 16, in <module>
[2022-05-28T01:55:47.733Z]     from mlflow.entities import Experiment, Run, RunInfo, Param, Metric, RunTag, FileInfo, ViewType
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/entities/__init__.py", line 6, in <module>
[2022-05-28T01:55:47.733Z]     from mlflow.entities.experiment import Experiment
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/entities/experiment.py", line 2, in <module>
[2022-05-28T01:55:47.733Z]     from mlflow.entities.experiment_tag import ExperimentTag
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/entities/experiment_tag.py", line 2, in <module>
[2022-05-28T01:55:47.733Z]     from mlflow.protos.service_pb2 import ExperimentTag as ProtoExperimentTag
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/protos/service_pb2.py", line 18, in <module>
[2022-05-28T01:55:47.733Z]     from .scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
[2022-05-28T01:55:47.733Z]   File "/opt/conda/lib/python3.8/site-packages/mlflow/protos/scalapb/scalapb_pb2.py", line 22, in <module>
[2022-05-28T01:55:47.733Z]     options = DESCRIPTOR.extensions_by_name['options']
[2022-05-28T01:55:47.733Z] monai.utils.module.OptionalImportError: import mlflow ('NoneType' object has no attribute 'extensions_by_name').
[2022-05-28T01:55:47.733Z] 
[2022-05-28T01:55:47.733Z] For details about installing the optional dependencies, please visit:
[2022-05-28T01:55:47.733Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-05-28T01:55:47.733Z] 
[2022-05-28T01:55:47.733Z] ----------------------------------------------------------------------

Tracking information

mlflow 1.26.1, protobuf 3.15.6

Code to reproduce issue

import mlflow
mlflow.set_tracking_uri(Path(test_path).as_uri())

Other info / logs

No response

What component(s) does this bug affect?

  • area/artifacts: Artifact stores and artifact logging
  • area/build: Build and test infrastructure for MLflow
  • area/docs: MLflow documentation pages
  • area/examples: Example code
  • area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
  • area/models: MLmodel format, model serialization/deserialization, flavors
  • area/projects: MLproject format, project running backends
  • area/scoring: MLflow Model server, model deployment tools, Spark UDFs
  • area/server-infra: MLflow Tracking server backend
  • area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

  • area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
  • area/docker: Docker use across MLflow’s components, such as MLflow Projects and MLflow Models
  • area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
  • area/windows: Windows support

What language(s) does this bug affect?

  • language/r: R APIs and clients
  • language/java: Java APIs and clients
  • language/new: Proposals for new client languages

What integration(s) does this bug affect?

  • integrations/azure: Azure and Azure ML integrations
  • integrations/sagemaker: SageMaker integrations
  • integrations/databricks: Databricks integrations

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (10 by maintainers)

Most upvoted comments

https://github.com/mlflow/mlflow/pull/5970 indicates protobuf>= 3.19.4 works with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python. Can you upgrade protobuf used in your repo?

docker run --rm -e PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python:3.7 bash -c "pip install mlflow protobuf==3.15.6 && python -c 'from mlflow.tracking.client import MlflowClient'"

can reproduce the issue.