mlflow: databricks.proto error when importing MLflow after tensorflow

When I try to import MLflow in my project, I get the following error:

TypeError                                 Traceback (most recent call last)
<ipython-input-1-d5dc4544d486> in <module>()
      9 from keras.optimizers import SGD
     10 from keras.preprocessing.image import ImageDataGenerator
---> 11 import mlflow
     12 
     13 import numpy as np

/usr/local/lib/python3.5/dist-packages/mlflow/__init__.py in <module>()
      6 
      7 # pylint: disable=wrong-import-position
----> 8 import mlflow.projects as projects # noqa
      9 import mlflow.tracking as tracking  # noqa
     10 

/usr/local/lib/python3.5/dist-packages/mlflow/projects.py in <module>()
     16 
     17 from mlflow.entities.source_type import SourceType
---> 18 from mlflow.entities.param import Param
     19 from mlflow import data
     20 import mlflow.tracking as tracking

/usr/local/lib/python3.5/dist-packages/mlflow/entities/param.py in <module>()
      1 from mlflow.entities._mlflow_object import _MLflowObject
----> 2 from mlflow.protos.service_pb2 import Param as ProtoParam
      3 
      4 
      5 class Param(_MLflowObject):

/usr/local/lib/python3.5/dist-packages/mlflow/protos/service_pb2.py in <module>()
     18 
     19 from mlflow.protos.scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
---> 20 import mlflow.protos.databricks_pb2 as databricks__pb2
     21 
     22 

/usr/local/lib/python3.5/dist-packages/mlflow/protos/databricks_pb2.py in <module>()
     25   serialized_pb=_b('\n\x10\x64\x61tabricks.proto\x1a google/protobuf/descriptor.proto\x1a\x15scalapb/scalapb.proto\"\x9a\x01\n\x14\x44\x61tabricksRpcOptions\x12 \n\tendpoints\x18\x01 \x03(\x0b\x32\r.HttpEndpoint\x12\x1f\n\nvisibility\x18\x02 \x01(\x0e\x32\x0b.Visibility\x12\x1f\n\x0b\x65rror_codes\x18\x03 \x03(\x0e\x32\n.ErrorCode\x12\x1e\n\nrate_limit\x18\x04 \x01(\x0b\x32\n.RateLimit\"N\n\x0cHttpEndpoint\x12\x14\n\x06method\x18\x01 \x01(\t:\x04POST\x12\x0c\n\x04path\x18\x02 \x01(\t\x12\x1a\n\x05since\x18\x03 \x01(\x0b\x32\x0b.ApiVersion\"*\n\nApiVersion\x12\r\n\x05major\x18\x01 \x01(\x05\x12\r\n\x05minor\x18\x02 \x01(\x05\"@\n\tRateLimit\x12\x11\n\tmax_burst\x18\x01 \x01(\x03\x12 \n\x18max_sustained_per_second\x18\x02 \x01(\x03\"\x8c\x01\n\x15\x44ocumentationMetadata\x12\x11\n\tdocstring\x18\x01 \x01(\t\x12\x10\n\x08lead_doc\x18\x02 \x01(\t\x12\x1f\n\nvisibility\x18\x03 \x01(\x0e\x32\x0b.Visibility\x12\x1b\n\x13original_proto_path\x18\x04 \x03(\t\x12\x10\n\x08position\x18\x05 \x01(\x05\"g\n\x1f\x44\x61tabricksServiceExceptionProto\x12\x1e\n\nerror_code\x18\x01 \x01(\x0e\x32\n.ErrorCode\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x13\n\x0bstack_trace\x18\x03 \x01(\t*?\n\nVisibility\x12\n\n\x06PUBLIC\x10\x01\x12\x0c\n\x08INTERNAL\x10\x02\x12\x17\n\x13PUBLIC_UNDOCUMENTED\x10\x03*\xf6\x04\n\tErrorCode\x12\x12\n\x0eINTERNAL_ERROR\x10\x01\x12\x1b\n\x17TEMPORARILY_UNAVAILABLE\x10\x02\x12\x0c\n\x08IO_ERROR\x10\x03\x12\x0f\n\x0b\x42\x41\x44_REQUEST\x10\x04\x12\x1c\n\x17INVALID_PARAMETER_VALUE\x10\xe8\x07\x12\x17\n\x12\x45NDPOINT_NOT_FOUND\x10\xe9\x07\x12\x16\n\x11MALFORMED_REQUEST\x10\xea\x07\x12\x12\n\rINVALID_STATE\x10\xeb\x07\x12\x16\n\x11PERMISSION_DENIED\x10\xec\x07\x12\x15\n\x10\x46\x45\x41TURE_DISABLED\x10\xed\x07\x12\x1a\n\x15\x43USTOMER_UNAUTHORIZED\x10\xee\x07\x12\x1b\n\x16REQUEST_LIMIT_EXCEEDED\x10\xef\x07\x12\x1d\n\x18INVALID_STATE_TRANSITION\x10\xd1\x0f\x12\x1b\n\x16\x43OULD_NOT_ACQUIRE_LOCK\x10\xd2\x0f\x12\x1c\n\x17RESOURCE_ALREADY_EXISTS\x10\xb9\x17\x12\x1c\n\x17RESOURCE_DOES_NOT_EXIST\x10\xba\x17\x12\x13\n\x0eQUOTA_EXCEEDED\x10\xa1\x1f\x12\x1c\n\x17MAX_BLOCK_SIZE_EXCEEDED\x10\xa2\x1f\x12\x1b\n\x16MAX_READ_SIZE_EXCEEDED\x10\xa3\x1f\x12\x13\n\x0e\x44RY_RUN_FAILED\x10\x89\'\x12\x1c\n\x17RESOURCE_LIMIT_EXCEEDED\x10\x8a\'\x12\x18\n\x13\x44IRECTORY_NOT_EMPTY\x10\xf1.\x12\x18\n\x13\x44IRECTORY_PROTECTED\x10\xf2.\x12\x1f\n\x1aMAX_NOTEBOOK_SIZE_EXCEEDED\x10\xf3.:@\n\nvisibility\x12\x1d.google.protobuf.FieldOptions\x18\xd0\x86\x03 \x01(\x0e\x32\x0b.Visibility::\n\x11validate_required\x12\x1d.google.protobuf.FieldOptions\x18\xd1\x86\x03 \x01(\x08:4\n\x0bjson_inline\x12\x1d.google.protobuf.FieldOptions\x18\xd2\x86\x03 \x01(\x08:1\n\x08json_map\x12\x1d.google.protobuf.FieldOptions\x18\xd3\x86\x03 \x01(\x08:J\n\tfield_doc\x12\x1d.google.protobuf.FieldOptions\x18\xd4\x86\x03 \x03(\x0b\x32\x16.DocumentationMetadata:D\n\x03rpc\x12\x1e.google.protobuf.MethodOptions\x18\xd0\x86\x03 \x01(\x0b\x32\x15.DatabricksRpcOptions:L\n\nmethod_doc\x12\x1e.google.protobuf.MethodOptions\x18\xd4\x86\x03 \x03(\x0b\x32\x16.DocumentationMetadata:N\n\x0bmessage_doc\x12\x1f.google.protobuf.MessageOptions\x18\xd4\x86\x03 \x03(\x0b\x32\x16.DocumentationMetadata:N\n\x0bservice_doc\x12\x1f.google.protobuf.ServiceOptions\x18\xd4\x86\x03 \x03(\x0b\x32\x16.DocumentationMetadata:H\n\x08\x65num_doc\x12\x1c.google.protobuf.EnumOptions\x18\xd4\x86\x03 \x03(\x0b\x32\x16.DocumentationMetadata:O\n\x15\x65num_value_visibility\x12!.google.protobuf.EnumValueOptions\x18\xd0\x86\x03 \x01(\x0e\x32\x0b.Visibility:S\n\x0e\x65num_value_doc\x12!.google.protobuf.EnumValueOptions\x18\xd4\x86\x03 \x03(\x0b\x32\x16.DocumentationMetadataB*\n#com.databricks.api.proto.databricks\xe2?\x02\x10\x01')
     26   ,
---> 27   dependencies=[google_dot_protobuf_dot_descriptor__pb2.DESCRIPTOR,scalapb_dot_scalapb__pb2.DESCRIPTOR,])
     28 
     29 _VISIBILITY = _descriptor.EnumDescriptor(

/usr/local/lib/python3.5/dist-packages/google/protobuf/descriptor.py in __new__(cls, name, package, options, serialized_pb, dependencies, public_dependencies, syntax, pool)
    827         # TODO(amauryfa): use the pool passed as argument. This will work only
    828         # for C++-implemented DescriptorPools.
--> 829         return _message.default_pool.AddSerializedFile(serialized_pb)
    830       else:
    831         return super(FileDescriptor, cls).__new__(cls)

TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "databricks.proto":
  databricks.proto: Import "scalapb/scalapb.proto" has not been loaded.

Any idea where the issue may come from? Thank you!

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 23 (11 by maintainers)

Commits related to this issue

Most upvoted comments

The root cause here is that tensorflow is imported before mlflow. When this happens, the protobuf module is first initialized by tensorflow with the C implementation whereas mlflow expects protobuf to use the python implementation.

The fix is to make mlflow work with the C implementation by dropping in the original .proto files.