mlflow: [BUG]

System information

  • Have I written custom code (as opposed to using a stock example script provided in MLflow): custom model training script
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04 in Docker container
  • MLflow installed from (source or binary): source (mlflow repository, master branch)
  • MLflow version (run mlflow --version): 1.3.1.dev0
  • Python version: 3.7.3
  • npm version, if running the dev UI: 3.10.10
  • Exact command to reproduce: docker-compose up

Describe the problem

  1. On open some run in mlflow ui:
2019/10/22 10:20:08 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/model-versions/search [GET]
 Traceback (most recent call last):
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 2446, in wsgi_app
     response = self.full_dispatch_request()
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1951, in full_dispatch_request
     rv = self.handle_user_exception(e)
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1820, in handle_user_exception
     reraise(exc_type, exc_value, tb)
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/_compat.py", line 39, in reraise
     raise value
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1949, in full_dispatch_request
     rv = self.dispatch_request()
   File "/usr/local/lib/python3.7/site-packages/Flask-1.1.1-py3.7.egg/flask/app.py", line 1935, in dispatch_request
     return self.view_functions[rule.endpoint](**req.view_args)
   File "/usr/local/lib/python3.7/site-packages/mlflow-1.3.1.dev0-py3.7.egg/mlflow/server/handlers.py", line 137, in wrapper
     return func(*args, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/mlflow-1.3.1.dev0-py3.7.egg/mlflow/server/handlers.py", line 581, in _search_model_versions
     model_versions_detailed = _get_model_registry_store().search_model_versions(
 AttributeError: 'NoneType' object has no attribute 'search_model_versions'
  1. On click Register Model:

mlflow_model_registry_error

Code to reproduce issue

Project sources mlflow_model_registry.zip

Archive content:

I. Dockerfile

FROM python
WORKDIR /home

# RUN pip install sklearn git+https://github.com/mparkhe/mlflow.git@ssue-19
RUN apt-get update && \
    curl -sL https://deb.nodesource.com/setup_6.x | bash - && \
    apt-get update && \
    apt-get install -y nodejs npm

RUN pip install sklearn

RUN git clone https://github.com/mlflow/mlflow && \
    cd mlflow && python setup.py install && \
    cd mlflow/server/js && npm install

ENV MLFLOW_TRACKING_URI=/home/mlruns

CMD mlflow server --host 0.0.0.0 --port 5000 & (cd mlflow/mlflow/server/js && npm start)

II. docker-compose.yml

version: '3'

services:
  mlflow_model_registry:
    build:
      context: .

    image: mlflow_model_registry:latest

    container_name: mlflow_model_registry_1

    ports:
      - 5000:5000
      - 3000:3000

    volumes:
      - ./main.py:/home/main.py
      - ./mlruns:/home/mlruns

III. main.py

import joblib
import mlflow
from mlflow.sklearn import log_model
from sklearn import svm
from sklearn import datasets


clf = svm.SVC(gamma='scale')
iris = datasets.load_iris()
X, y = iris.data, iris.target
clf.fit(X, y)

joblib.dump(clf, 'iris.model')

log_model('iris.model', 'model')

Run all:

  1. build image and run container
docker-compose build
docker-compose up
  1. enter docker container in new terminal window
docker exec -ti mlflow_model_registry_1 /bin/bash
  1. run training script
python main.py
  1. open dev UI

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 18 (7 by maintainers)

Most upvoted comments

@MeisterUrian I ran into the exact same issue. This is because Model Registry does not support file based stores yet. (https://github.com/mlflow/mlflow/blob/master/mlflow/server/handlers.py#L60)

I used my local PostgreSQL as a tracking server and it worked.

@harupy When does Model Registry support file based stores? And MLflow Model Registry doesn’t mention about this. It is necessary to tell the truth for new users that Model Registry doesn’t support file based stores.

@harupy Should model registry work if the tracking server is on an ec2 instance (with artifacts being logged to s3)? Because I’m running into the same issue, it seems.

@MeisterUrian I ran into the exact same issue. This is because Model Registry does not support file based stores yet. (https://github.com/mlflow/mlflow/blob/master/mlflow/server/handlers.py#L60)

I used my local PostgreSQL as a tracking server and it worked.

@harupy Thank you very much! I will try it some later.

UPD: it works! I have tried it with SQLite and it solved my problem.

Thanks again!