mlflow: [BUG] MLflow tried to browse the wrong folder in FTP

Issues Policy acknowledgement

• I have read and agree to submit bug reports in accordance with the issues policy

Willingness to contribute

No. I cannot contribute a bug fix at this time.

MLflow version

pip install mlflow==2.2.2

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MasOs
• Python version: 3.9
• yarn version, if running the dev UI:

Describe the problem

I am trying to follow scenario 5 in https://mlflow.org/docs/latest/tracking.html, with a FTP server (artifacts) and local SQLite (store), but I run issues over issues.

I run a FTP server using:

docker run --rm \
  --name ftpd_server -p 21:21 -p 30000-30009:30000-30009 \
  -e "PUBLICHOST=localhost" \
  -e FTP_USER_NAME=admin -e FTP_USER_PASS=123 \
  -e FTP_USER_HOME=/home/admin \
  -v/abs/path_to/artifacts:/home/admin \
  stilliard/pure-ftpd

I tested different Docker images with different configurations, and I always get a similar issue…

I do see migrations of database when I run the server, so that is working correctly.

It feels like the MLflow server tried to browse the file system incorrectly (see stacktrace below), as a non absolute path is used. Could someone please provide some help? I have followed all the steps described in the documentation: what have I missed? I am fully stuck

Tracking information

Result of mlflow doctor:

MLflow version: 2.2.2
Tracking URI: http://localhost:3000
Artifact URI: mlflow-artifacts:/2/18cf9305894c4dd3851149c8b5d5064c/artifacts
System information: Darwin Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
Python version: 3.9.11
MLflow version: 2.2.2
MLflow module location: /Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/__init__.py
Tracking URI: http://localhost:3000
Registry URI: http://localhost:3000
Active experiment ID: 2
Active run ID: 18cf9305894c4dd3851149c8b5d5064c
Active run artifact URI: mlflow-artifacts:/2/18cf9305894c4dd3851149c8b5d5064c/artifacts
MLflow dependencies: 
  Flask: 2.2.3
  Jinja2: 3.1.2
  alembic: 1.10.2
  click: 8.1.3
  cloudpickle: 2.2.1
  databricks-cli: 0.17.5
  docker: 6.0.1
  entrypoints: 0.4
  gitpython: 3.1.31
  gunicorn: 20.1.0
  importlib-metadata: 6.0.0
  markdown: 3.4.1
  matplotlib: 3.7.1
  numpy: 1.23.5
  packaging: 23.0
  pandas: 1.5.3
  protobuf: 4.22.1
  pyarrow: 11.0.0
  pytz: 2022.7.1
  pyyaml: 6.0
  querystring-parser: 1.2.4
  requests: 2.28.2
  scikit-learn: 1.2.2
  scipy: 1.10.1
  shap: 0.41.0
  sqlalchemy: 2.0.6
  sqlparse: 0.4.3

Code that runs the mlflow tracking server:

	mlflow server \
	--backend-store-uri sqlite:///data.db \
	--artifacts-destination ftp://admin:123@localhost:21 \
	--serve-artifacts \
	--host 127.0.0.1 \
	--port 3000

Code to reproduce issue

import mlflow.pyfunc
import tempfile
import os
import time
from typing import Tuple
import numpy as np
import pandas as pd


class JudeLin(mlflow.pyfunc.PythonModel):
    def __init__(self, coefficients: Tuple[float, float]):
        a, b = coefficients
        self._coefficients: np.array = np.array([a,b], dtype='float64')

    def predict(self, context, model_input: pd.DataFrame):
        return self.predict_numpy(
            nparray_nx2=model_input.to_numpy()
        )

    def predict_numpy(self, nparray_nx2: np.ndarray) -> np.array:
        return np.matmul(nparray_nx2, self._coefficients)


def test_mlflow_log(experiment_id):
    model = JudeLin(coefficients=(2, -1))

    mlflow.set_tracking_uri('http://localhost:3000')
    mlflow.set_experiment("experiment_random_202")

    with tempfile.TemporaryDirectory("w+") as name:
        file_name = os.path.join(name, 'model.json')
        with open(file_name, "w") as f:
            f.write("[{}, {}]".format(*model._coefficients))

        with mlflow.start_run(run_name="my-run-{}".format(time.time())):

            print(mlflow.get_artifact_uri())

            mlflow.log_param("A", model._coefficients[0])
            mlflow.log_param("B", model._coefficients[1])
            mlflow.pyfunc.log_model(
                artifact_path="idk",
                python_model=model
            )

Please note that the name of the experiment is changed each time I run the code.

Stack trace

The print gives me

mlflow-artifacts:/2/efb7e67ea019498abb3aee831db54172/artifacts

which is correct.

I see in my folder (the one controlled by the FTP server) that there is indeed a

mlflow-artifacts:/2/efb7e67ea019498abb3aee831db54172/artifacts/python_env.yaml

file that is created. However, I then run into the following error:

2023/03/19 21:25:27 ERROR mlflow.server: Exception on /api/2.0/mlflow-artifacts/artifacts/2/efb7e67ea019498abb3aee831db54172/artifacts/idk/requirements.txt [PUT]
Traceback (most recent call last):
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/server/handlers.py", line 473, in wrapper
    return func(*args, **kwargs)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/server/handlers.py", line 539, in wrapper
    return func(*args, **kwargs)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/server/handlers.py", line 1639, in _upload_artifact
    artifact_repo.log_artifact(tmp_path, artifact_path=head or None)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/store/artifact/ftp_artifact_repo.py", line 77, in log_artifact
    ftp.cwd(artifact_dir)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 625, in cwd
    return self.voidcmd(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 286, in voidcmd
    return self.voidresp()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 259, in voidresp
    resp = self.getresp()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 550 Can't change directory to 2/efb7e67ea019498abb3aee831db54172/artifacts/idk: No such file or directory
2023/03/19 21:25:43 ERROR mlflow.server: Exception on /api/2.0/mlflow-artifacts/artifacts/2/efb7e67ea019498abb3aee831db54172/artifacts/idk/requirements.txt [PUT]
Traceback (most recent call last):
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/server/handlers.py", line 473, in wrapper
    return func(*args, **kwargs)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/server/handlers.py", line 539, in wrapper
    return func(*args, **kwargs)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/server/handlers.py", line 1639, in _upload_artifact
    artifact_repo.log_artifact(tmp_path, artifact_path=head or None)
  File "/Users/justin.dekeyser/Documents/explore/virtual_assistant/poc_1/ignored/venv39/lib/python3.9/site-packages/mlflow/store/artifact/ftp_artifact_repo.py", line 77, in log_artifact
    ftp.cwd(artifact_dir)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 625, in cwd
    return self.voidcmd(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 286, in voidcmd
    return self.voidresp()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 259, in voidresp
    resp = self.getresp()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 550 Can't change directory to 2/efb7e67ea019498abb3aee831db54172/artifacts/idk: No such file or directory

Other info / logs

No response

What component(s) does this bug affect?

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow’s components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

What language(s) does this bug affect?

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations