azure-cli-extensions: az ml online-deployment error "MissingDriverProgram"

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name az ml online-deployment create Extension Name: ml. Version: 2.10.0.

Errors:

(BadRequest) The request is invalid.
Code: BadRequest
Message: The request is invalid.
Exception Details:	(MissingDriverProgram) Could not find driver program in the request.
	Code: MissingDriverProgram
	Message: Could not find driver program in the request.
Additional Information:Type: ComponentName
Info: {
    "value": "managementfrontend"
}Type: Correlation
Info: {
    "value": {
        "operation": "86de0d2d85d2d0ee19f82e28b518c7ac",
        "request": "06d9fb8f6b1aebfc"
    }
}Type: Environment
Info: {
    "value": "northeurope"
}Type: Location
Info: {
    "value": "northeurope"
}Type: Time
Info: {
    "value": "2022-11-25T13:49:53.1312542+00:00"
}

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

docker image built and pushed to ACR.azurecr.io/inference:v1
file online-deployment.yml

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: model-deployment
endpoint_name: model-endpoint
egress_public_network_access: disabled
model:
  path: ../model
  type: mlflow_model
environment_variables:
  MLFLOW_MODEL_FOLDER: ""
instance_type: Standard_DS3_v2
instance_count: 1
environment: 
  image: ACR.azurecr.io/inference:v1

az ml online-deployment create --all-traffic -n {} -f {}

Expected Behavior

Environment Summary

Linux-5.15.0-1017-azure-x86_64-with-glibc2.10, Ubuntu 20.04.5 LTS
Python 3.8.5
Installer: PIP

azure-cli 2.40.0 *

Extensions:
ml 2.10.0

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

Testing a vnet scenario, where the ACR and storage account are behind the vnet. Private endpoints are created accordingly. Local deployment can be generated, but same deployment configuration and command, removing the --local parameter does not work out of the box. It’s not possible to reference the model using azureml:model syntax because I’ve got an error, so I need to use path format. Using --local I need to reference a single PKL file, but removing the --local parameter, I must use a folder (otherwise the CLI returns an explicit error requesting file / folder) Could you please provide any insight about the “MissingDriverProgram” error?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (3 by maintainers)

Most upvoted comments

@franperezlopez , I created an az cli v2 yaml example, I will share a link tomorrow (2023-01-19)

rebremer on Jan 18, 2023

Confirming that this is an issue. I think that using MLFlow with private endpoints for AML is not an exotic requirement in a corporate context. It would be very helpful @SaurabhSharma-MSFT if you could share the workaround until there is a proper fix.

sh0rtcircuit on Jan 16, 2023

@SaurabhSharma-MSFT I am facing the same issue. Could you please share the steps for Mlflow based deployment?

socullen on Jan 11, 2023

@franperezlopez One day late, but here you can find an example of az ml cli v2, see https://github.com/rebremer/databricks-mlflow-azureml-moe-vnet and for the yaml file to deploy: https://github.com/rebremer/databricks-mlflow-azureml-moe-vnet/blob/main/MoE/acr-gboost-deployment.yaml

rebremer on Jan 20, 2023

@franperezlopez sorry for the inconvenience. Please refer to the sample code file modified for this scenario.

And please find the details below:

Need to build an image in ACR using Azure CLI. Followed below steps: • Create a Dockerfile like this one: Dockerfile

• Include in the same folder the conda environment of your model. • az login • az acr login -n <acr-instance>. • az acr build -t my-model-mlflow:latest -r <acr-instance> <docker-folder-path> • Then, create a deployment by indicate an inline environment with the property image=“my-model-mlflow”, (Please refer the attached file for more details)

For above you can use Compute Instance terminal from Azure Machine learning to run the “az command”

Some changes in online deployment code(Please refer attached code): • Adding environment variable for MLFLOW_MODEL_FOLDER and assign the relative path of the model • Specify environment to link to your ACR path • Also specify environment for inference_config for liveness_route, readiness_route, scoring_route

#create a blue deployment model = Model(name=“your-model”, version=“1”, path=“model”)

env = Environment( image=“…azurecr.io/my-model-mlflow:latest”, inference_config={ “liveness_route”: {“port”: 5001, “path”: “/”}, “readiness_route”: {“port”: 5001, “path”: “/”}, “scoring_route”: {“port”: 5001, “path”: “/score”}, }, )

blue_deployment = ManagedOnlineDeployment( name=“blue”, endpoint_name=online_endpoint_name, model=model, environment=env, environment_variables={ “MLFLOW_MODEL_FOLDER”: “model” }, instance_type=“Standard_F2s_v2”, instance_count=1, )

Please check and let me know if any questions.

SaurabhSharma-MSFT on Jan 18, 2023

I am having the same issue, @SaurabhSharma-MSFT can you please open the ticket and share the steps for MLflow based deployment for Azure ML with public access disabled?

rebremer on Jan 15, 2023