MachineLearningNotebooks: ML Pipelines DatabricksStep doesn't support Run.get_context()

@fahdkmsft It’s unclear that this is the intended usage given that the only documentation related to these additional variables is under DatabricksStep -> python_script_name and makes no mention of setting them in order to get a Run through Run.get_context().

I apologize if I missed some docs that explains this but if I have can you please post a link to them?

I’d also like to point out that no one has answered my question about what the intended use of the AzureMLTokenAuthentication is? A DatabricksStep provides all of the information to instantiate this class but you can’t authenticate a Workspace with it.

Hi all, is there any update on this issue? I followed the suggestion but it does not work from Databricks job cluster launched by a DatabrickStep and running a local python script.

@kamakay or @RothNRK, did you guys find any option to use the Run class from the python script in Databricks?

Thanks in advanced. BR.

@RothNRK Thank you for pointing this out. I have created a work item to investigate the issue. We will update you shortly.

@rastala , please check and create ICM for pipelines team if needed. Thanks!

@rastala Thank you. That is what I expected but while in a DatabricksStep it doesn’t seem to simply work.

Run.get_context(allow_offline=False) produces:

KeyError                                  Traceback (most recent call last)
/databricks/python/lib/python3.7/site-packages/azureml/core/run.py in _load_scope(cls)
    215             # Load authentication scope environment variables
--> 216             subscription_id = os.environ['AZUREML_ARM_SUBSCRIPTION']
    217             run_id = os.environ["AZUREML_RUN_ID"]

/local_disk0/pythonVirtualEnvDirs/virtualEnv-6c0c6c97-5068-4f34-91f1-ded12db18057/lib/python3.7/os.py in __getitem__(self, key)
    677             # raise KeyError with the original key value
--> 678             raise KeyError(key) from None
    679         return self.decodevalue(value)

KeyError: 'AZUREML_ARM_SUBSCRIPTION'

The above exception was the direct cause of the following exception:

RunEnvironmentException                   Traceback (most recent call last)
/databricks/python/lib/python3.7/site-packages/azureml/core/run.py in get_context(cls, allow_offline, used_for_context_manager, **kwargs)
    291         try:
--> 292             experiment, run_id = cls._load_scope()
    293 

/databricks/python/lib/python3.7/site-packages/azureml/core/run.py in _load_scope(cls)
    232         except KeyError as key_error:
--> 233             raise_from(RunEnvironmentException(), key_error)
    234         else:

/databricks/python/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

RunEnvironmentException: RunEnvironmentException:
	Message: Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run.
	InnerException None
	ErrorResponse 
{
    "error": {
        "message": "Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run."
    }
}

During handling of the above exception, another exception occurred:

RunEnvironmentException                   Traceback (most recent call last)
<command--1> in <module>
     13 
     14 with open(filename, "rb") as f:
---> 15   exec(f.read())
     16 

<string> in <module>

<string> in main()

<string> in _main(args)

<string> in _get_run_context(args)

<string> in _get_run()

/databricks/python/lib/python3.7/site-packages/azureml/core/run.py in get_context(cls, allow_offline, used_for_context_manager, **kwargs)
    303             else:
    304                 module_logger.debug("Could not load the run context and allow_offline set to False")
--> 305                 raise RunEnvironmentException(inner_exception=ex)
    306 
    307     @classmethod

RunEnvironmentException: RunEnvironmentException:
	Message: Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run.
	InnerException RunEnvironmentException:
	Message: Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run.
	InnerException None
	ErrorResponse 
{
    "error": {
        "message": "Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run."
    }
}
	ErrorResponse 
{
    "error": {
        "message": "Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run."
    }
}

So it looks like it’s looking for environment variables that aren’t set. If I run it with Run.get_context() I get an error related to the _OfflineRun not working (which makes sense since I need a Run not an _OfflineRun).