azure-sdk-for-python: Managed endpoint doesn't use Azure ML connections (private pip) when building image

• Package Name: azureml-mlflow
• Package Version: 1.37.0
• Operating System: Windows
• Python Version: 3.8.11

Describe the bug When using private pip feeds it is possible to set up a connection (token) which is shared across the Azure ML Studio. This works for creating an environment for training the model. If the model is saved in the MLFLOW format it also works when using an Azure Container Instance for creating a realtime endpoint very easily. But when trying to use the managed endpoint the token is not used.

To Reproduce

When using private pip feeds it is possible to set up a connection (token) which is shared across the Azure ML Studio by running 1:

ws.set_connection(
    name="private-pip",
    category= "PythonFeed",
    authType="PAT",
    target = "https://@pkgs.dev.azure.com/MY-ORG",
    value = "1234abcd"
)

2: This will then be used if you construct a environment were a private pip is referred and replace the package myprivatepackage with something in you feed:

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies

env = Environment("test_env")
cd = CondaDependencies.create(
    python_version="3.8.12",
    conda_packages=["pip"],
    pip_packages=["scikit-learn", "pandas","azureml-mlflow","pyyaml","myprivatepackage"]
    )
cd.set_pip_option("--index-url https://pkgs.dev.azure.com/MY-ORG/_packaging/MY-FEED/pypi/simple/")
env.python.conda_dependencies = cd
env.register(ws)
env.build(ws)

3: If you train a model you can deploy it where it connects using the token when building the image too.

First create a training file:

## ml_project/train.py
from sklearn.linear_model import LinearRegression
import mlflow
import mlflow.sklearn
from azureml.core import Run
import yaml

from sklearn.datasets import load_iris
X, y = load_iris(as_frame=True, return_X_y=True)

run = Run.get_context()
ws = run.experiment.workspace
env = run.get_environment()
conda_env = yaml.safe_load(env.python.conda_dependencies.serialize_to_string())

with mlflow.start_run():
    X, y = load_iris(as_frame=True, return_X_y=True)
    model = LinearRegression()
    model.fit(X, y)
    mlflow.sklearn.log_model(model, "model", conda_env=conda_env)

Then then train the model

from azureml.core import Experiment, ScriptRunConfig
from azureml.core.compute import ComputeTarget

cpu_cluster = ComputeTarget(workspace=ws, name="test-cluster")
exp = Experiment(workspace=ws, name="test")

src = ScriptRunConfig(
    source_directory= "mlproject",
    script='train.py',
    compute_target=cpu_cluster,
    environment=env
)
run = exp.submit(src)

Then register the model:

import mlflow
mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())
model_info = mlflow.register_model(f"runs:/{run.id}/model", "test_model")

4: You can deploy it using Azure Container Instance which works by grabbing the token. It deploys after some time.:

from mlflow.deployments import get_deploy_client
mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())
client = get_deploy_client(mlflow.get_tracking_uri())
client.create_deployment(model_uri=f'models:/test_model/1', name="mlflow-test-aci")

5: Go to the UI for Azure Machine Learning Studio and go to the wizard for setting up a managed endpoint. As the model is of type MLFLOW it will create enviorment and scoring script for you. When the image builds it will fail as it is expecting the user to put in a password:

Instance status:
SystemSetup: Succeeded
UserContainerImagePull: Succeeded
ModelDownload: Succeeded
UserContainerStart: InProgress

Container logs:
2022-01-11T12:29:04,946980587+00:00 - rsyslog/run 
2022-01-11T12:29:04,951881106+00:00 - gunicorn/run 

2022-01-11T12:29:04,953826713+00:00 - nginx/run 
AzureML image information: mlflow-ubuntu18.04-py37-cpu-inference:20210930.v4


PATH environment variable: /opt/miniconda/envs/amlenv/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH environment variable: 

Pip Dependencies
---------------
applicationinsights==0.11.10
azureml-inference-server-http==0.3.2
certifi==2021.5.30
charset-normalizer==2.0.6
click==8.0.1
cloudpickle==2.0.0
databricks-cli==0.15.0
entrypoints==0.3
Flask==1.0.3
gitdb==4.0.7
GitPython==3.1.24
gunicorn==20.1.0
idna==3.2
importlib-metadata==4.8.1
inference-schema==1.3.0
itsdangerous==2.0.1
Jinja2==3.0.1
MarkupSafe==2.0.1
mlflow-skinny==1.17.0
numpy==1.21.2
pandas==1.2.4
protobuf==3.18.0
python-dateutil==2.8.2
pytz==2021.1
PyYAML==5.4.1
requests==2.26.0
six==1.16.0
smmap==4.0.0
tabulate==0.8.9
typing-extensions==3.10.0.2
urllib3==1.26.7
Werkzeug==2.0.1
wrapt==1.12.1
zipp==3.6.0


Entry script directory: /var/mlflow_resources/.

Updating conda environment from /var/azureml-app/azureml-models/test_model/1/model/conda.yaml !
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

Downloading and Extracting Packages

ca-certificates-2020 | 128 KB    |            |   0% 
ca-certificates-2020 | 128 KB    | #2         |  13% 
ca-certificates-2020 | 128 KB    | ########## | 100% 

_openmp_mutex-4.5    | 22 KB     |            |   0% 
_openmp_mutex-4.5    | 22 KB     | ########## | 100% 

libgcc-ng-11.2.0     | 887 KB    |            |   0% 
libgcc-ng-11.2.0     | 887 KB    | ########## | 100% 
libgcc-ng-11.2.0     | 887 KB    | ########## | 100% 

libzlib-1.2.11       | 59 KB     |            |   0% 
libzlib-1.2.11       | 59 KB     | ########## | 100% 

certifi-2020.6.20    | 160 KB    |            |   0% 
certifi-2020.6.20    | 160 KB    | ########## | 100% 

ld_impl_linux-64-2.3 | 667 KB    |            |   0% 
ld_impl_linux-64-2.3 | 667 KB    | ########## | 100% 
ld_impl_linux-64-2.3 | 667 KB    | ########## | 100% 

libnsl-2.0.0         | 31 KB     |            |   0% 
libnsl-2.0.0         | 31 KB     | ########## | 100% 

sqlite-3.37.0        | 1.5 MB    |            |   0% 
sqlite-3.37.0        | 1.5 MB    | ########## | 100% 
sqlite-3.37.0        | 1.5 MB    | ########## | 100% 

xz-5.2.5             | 438 KB    |            |   0% 
xz-5.2.5             | 438 KB    | ########## | 100% 
xz-5.2.5             | 438 KB    | ########## | 100% 

libstdcxx-ng-11.2.0  | 4.2 MB    |            |   0% 
libstdcxx-ng-11.2.0  | 4.2 MB    | ########## | 100% 
libstdcxx-ng-11.2.0  | 4.2 MB    | ########## | 100% 

pip-20.2.4           | 2.0 MB    |            |   0% 
pip-20.2.4           | 2.0 MB    | ########## | 100% 
pip-20.2.4           | 2.0 MB    | ########## | 100% 

readline-8.1         | 295 KB    |            |   0% 
readline-8.1         | 295 KB    | ########## | 100% 

libgomp-11.2.0       | 427 KB    |            |   0% 
libgomp-11.2.0       | 427 KB    | ########## | 100% 
libgomp-11.2.0       | 427 KB    | ########## | 100% 

ncurses-6.2          | 1.1 MB    |            |   0% 
ncurses-6.2          | 1.1 MB    | ########## | 100% 
ncurses-6.2          | 1.1 MB    | ########## | 100% 

python-3.8.12        | 26.2 MB   |            |   0% 
python-3.8.12        | 26.2 MB   | 5          |   6% 
python-3.8.12        | 26.2 MB   | ##9        |  30% 
python-3.8.12        | 26.2 MB   | ####6      |  46% 
python-3.8.12        | 26.2 MB   | ######9    |  70% 
python-3.8.12        | 26.2 MB   | ########## | 100% 
python-3.8.12        | 26.2 MB   | ########## | 100% 

wheel-0.35.1         | 36 KB     |            |   0% 
wheel-0.35.1         | 36 KB     | ########## | 100% 

_libgcc_mutex-0.1    | 3 KB      |            |   0% 
_libgcc_mutex-0.1    | 3 KB      | ########## | 100% 

libffi-3.4.2         | 57 KB     |            |   0% 
libffi-3.4.2         | 57 KB     | ########## | 100% 

zlib-1.2.11          | 86 KB     |            |   0% 
zlib-1.2.11          | 86 KB     | ########## | 100% 

openssl-3.0.0        | 2.9 MB    |            |   0% 
openssl-3.0.0        | 2.9 MB    | ########## | 100% 
openssl-3.0.0        | 2.9 MB    | ########## | 100% 

tk-8.6.11            | 3.3 MB    |            |   0% 
tk-8.6.11            | 3.3 MB    | ########## | 100% 
tk-8.6.11            | 3.3 MB    | ########## | 100% 

setuptools-50.3.0    | 902 KB    |            |   0% 
setuptools-50.3.0    | 902 KB    | ########## | 100% 
setuptools-50.3.0    | 902 KB    | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.11.0

Please update conda by running

    $ conda update -n base -c defaults conda


Pip subprocess error:
ERROR: Exception:
Traceback (most recent call last):
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 339, in _get_abstract_dist_for
    self._populate_link(req)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 305, in _populate_link
    req.link = self._find_requirement_link(req)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 270, in _find_requirement_link
    best_candidate = self.finder.find_requirement(req, upgrade)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 900, in find_requirement
    best_candidate_result = self.find_best_candidate(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 883, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 827, in find_all_candidates
    package_links = self.process_project_url(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 791, in process_project_url
    html_page = self._link_collector.fetch_page(project_url)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 629, in fetch_page
    return _get_html_page(location, session=self.session)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 441, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 138, in _get_html_response
    resp = session.get(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/network/session.py", line 421, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 650, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 256, in handle_401
    username, password, save = self._prompt_for_password(parsed.netloc)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 226, in _prompt_for_password
    username = ask_input("User for {}: ".format(netloc))
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 259, in ask_input
    return input(message)
EOFError: EOF when reading a line


CondaEnvException: Pip failed

Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/miniconda/envs/userenv/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.i5j0upr4.requirements.txt']
Pip subprocess output:
Looking in indexes: https://pkgs.dev.azure.com/MY-ORG/_packaging/MY-FEED/pypi/simple/
User for pkgs.dev.azure.com: 
failed

6: I assume the same errro happens on managed Batch endpoint but I haven’t tried it out.

Expected behavior I expect the managed endpoint to work. I also expect it to work for batch endpoint.

Suggestion A good solution to this would be to allow for setting a variable in Azure ML studio which was always loaded (at build time) as an env var (PIP_INDEX_URL) when building an environment. That way we don’t have to specify the url in the environment/conda file and it can easily be changed if the pip feed changes it address.