MLServer: Dependency updates & compatibility errors (re-training needed?)

I was running a model trained with mlflow 2.6.0 on mlserver 1.6.1. After upgrading mlserver to 1.7.0, I get compatibility errors as outlined below. I’m not sure where the root of issue is exactly, so this issue contains a lot of information from different angles. I apologize for the scattered thoughts and hope you can answer my questions regarding compatibility below. Thank you!

In mlserver v1.7.0, specifically this PR (https://github.com/SeldonIO/MLServer/pull/1951), mlserver has increased the version their proto gencode is built with from 4.25.1 to 5.27.2. This is a breaking change; the current version is not backwards compatible with any protobuf runtime <5.27.2. The protobuf cross-version runtime guarantee states that New Gencode + Old Runtime = Never Allowed.
This breaking change is causing issues with older protobuf runtimes as already reported: https://github.com/SeldonIO/MLServer/issues/2130

As outlined in that issue, I think the protobuf requirements should be changed/restricted from “*” to “>=5.27.2”.

This also implies an increase of the minimum supported mlflow version. Mlflow 2.6.0 has a requirement of protobuf “<5”, so it will pull the latest 4.x.x runtime. This runtime cannot work with the newer proto gencode of mlserver v1.7.0, where the proto version was 5.27.2. This issue persists up to mlflow v2.15.0, when they rebuild their own proto gencode and relax the constraint to “<6” (see PR https://github.com/mlflow/mlflow/pull/12360). (As you can see in their PR, they don’t have to increase the minimum proto version, because they build their protos in such a way, that it switches the version based on the currently running/imported protobuf version. mlserver does not do this resulting in a hard increase of the minimum protobuf version.)

Therefore, I think the mlflow dependency should also be changed/restricted from “*” to ">=2.15.0. Thanks to another compatibility bug found with mlflow (https://github.com/SeldonIO/MLServer/issues/2113), this has already been changed to “>=2.19.0”, but not released.

I am running models trained with mlflow v2.6.0 and want to increase the mlserver version to 1.7. I expected this to work without a problem because the environments should be separated and dependencies should not infere. However, I get the protobuf error as described in #1951. It seems I would need to re-train the model using a newer mlflow version (specifically >=2.19.0) in order for it to run on mlserver 1.7.0.

This situation leads me to some questions:

  1. In general, is it supported to run models trained with older mlflow versions on newer mlserver versions (given no major version changes in either)?
  2. Is it correct that models trained with mlflow <2.19.0 need to be re-trained with >=2.19.0 to run with mlserver 1.7.0?
  3. Are there any compatibility guarantees within a major like 1.x.x (only minor bumps) regarding trained/existing mlflow models?
  4. Why/how do the dependencies of the outside mlserver environment interact with the isolated environment of the mlflow model? Is this expected? (see below)

Some additional info because I fear that I’m chasing the wrong thing:

I noticed in the stacktrace of the protobuf error that it starts inside of the isolated mlflow environment but ends in the outer mlserver environment. It looks like a runpy call is jumping the model (mlflow) environment border and ending up in the mlserver environment, and I’m quite confused by that. Is this something that is expected? If this “sandbox escape” was not there and the runpy call would only use the dependencies in the model (mlflow) environment, it would not be able to import the new mlserver proto gencode (the one built with proto 5.27.2) where it currently fails. I’m not sure what it would import, because mlserver is only specified as an extra dependency and has no upper limit in 2.6.0 (source), so mlserver wouldn’t be there (?) unfortunately, I don’t understand the interaction of mlflow and mlserver in this case well enough.

Here’s that stacktrace (see env info below):

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/app-root/.conda/envs/env/bin/mlserver", line 5, in <module>
    from mlserver.cli import main
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/__init__.py", line 2, in <module>
    from .server import MLServer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/server.py", line 17, in <module>
    from .grpc import GRPCServer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/__init__.py", line 1, in <module>
    from .server import GRPCServer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/server.py", line 9, in <module>
    from .servicers import InferenceServicer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/servicers.py", line 3, in <module>
    from . import dataplane_pb2 as pb
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/dataplane_pb2.py", line 9, in <module>
    from google.protobuf import runtime_version as _runtime_version
ImportError: cannot import name 'runtime_version' from 'google.protobuf' (/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/site-packages/google/protobuf/__init__.py)

Environment info:

mlserver env (/opt/app-root/.conda/envs/env/)

Created manually with conda inside a container which then runs mlserver.

  • python3.11
  • mlflow==2.22.0
  • mlserver==1.7.0 -> contains gencode that requires protobuf runtime >=5.27.2

model (mlflow) env (/mnt/environments/3014daa3513)

Created/managed by mlserver or mlflow (?) according to the conda.yaml file in the mlflow artifact

  • python3.11
  • mlflow==2.6.0
  • implicitly protobuf<5

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

It is indeed related to #1892.

I also understand why you wouldn’t want to modify the model environment after training (I suspected that’s one of the causes you’re not choosing that path, but I thought to mention it nonetheless).

We’ll keep this issue open. As mentioned before, we welcome contributions on this from both yourself and the wider MLServer community. We’ll also post a note here when we are able to add this item on our roadmap, together with the targeted MLServer release.

So am I correct in saying that the core issue is that mlserver-mlflow runtime is not compatible with mlflow <2.19? Generally, we didn’t want the serving engine (mlserver) inside of the model environment, but in this case it seems that it’s conceptually impossible as it will interact, even if it’s not installed in the model env.

I believe you are correct. Conceptually, it’s not the “serving engine” that you need inside the model environment, but code that knows how to take the mlflow model artifact and “run it” within MLServer (the runtime). It’s just that the way the current runtimes are written, they import code from mlserver rather than both the runtime and mlserver importing a thin package with the required interfaces (which would be the solution to the problems you’re seeing). If you would want to contribute a PR to this before we get to it, we’d certainly welcome it!

The mlserver-mlflow runtime is tested with mlflow versions >=2.19. However, from what we have tried, model artifacts trained with versions <2.19 should still load/run on mlflow versions >=2.19 (i.e if you were to update the version installed in your environment)

Also, regarding this part:

In this case, I believe that the code of the mlflow runtime ends up trying to access the protobuf gencode via a chain of import paths (I haven’t dug deep into exactly how that ends up being the case, but as a matter of principle that code is not strictly needed from a runtime perspective).

I suspect the same thing. I even tried to fix it by moving imports around but I guess I would need to remove them instead of just moving them.

The runtime imports mlserver.X packages, but the __init__ of MLServer imports things like .server that bring in more dependencies than strictly needed.

In the short term, I believe that you should still be able to use MLServer by writing a simple custom runtime that loads your model as a way around this. Ultimately, it means implementing just a couple of methods like “load”, “predict” and “unload”.

In this case, we would need to re-implement the mlserver-mlflow package, right?

Yes, and for now you would need to copy the MLServer datatypes (mlserver/model.py and mlserver/settings.py and dependencies (mlserver/codecs/, mlserver/types/) rather than importing them from MLServer.

The mlflow-runtime is written fairly generally, but minimally (i.e for a PoC) you would need to implement/copy from the existing runtime just the the load and predict functions. You could also choose to write this specific to one model (i.e load a specific mlflow model artifact from the same directory) Further docs here.

We can also take a look in minimising the dependencies that end up being loaded by our existing runtimes, because they create inadvertent, implicit dependencies in the custom environments, which as you have noticed may lead to package conflicts.

That would be great! The current compatibility restrictions make life-cycling models hard.

I agree this is harder than it needs to be when using custom MLServer images and environments but without writing your own custom runtimes.