azure-sdk-for-python: ImportError: cannot import name 'SerializationError' from 'azure.core.exceptions'

"that started happening out of the blue, everything was working fine for quite some time. “

Completely unreliable. How on earth do things work fine for quite some time … then WITHOUT USER CODE CHANGES, behind the scenes, you start screwing up everything??

Pinning the version is not the fix. Tell us what you changed for transparency sake. Stop treating this as if the dev screwed up. Obviously YOU changed something. I had the same experience. No code change. Then suddenly one day redeploying the same code with only a minor config update, everything is broke and now requires version pinning?? Running stuff on your platform is a real crap-shoot.

Thank goodness most of my customers are on AWS.

We are also seeing this issue pop up and we’re on azure-core==1.14.0.

We didn’t have msrest in dependencies and so we were upgraded to 0.7.0 which released yesterday (June 7, 2022). Pinning to 0.6.21 did the trick.

I uh… respectfully agree with @odoo-ce-modules that we need to be told the dependency as the library is imported.

msrest 0.7.0 requires azure-core 1.24.0. If anyone pins azure-core, but not msrest, you may end up in a scenario were pip will upgrade only msrest, but not azure-core. This is a pip update problem, as we didn’t do a mistake in our setup.py of msrest

The recommended approach is to either pin to msrest 0.6.21 and any version of azure-core, or to ping to msrest 0.7.0 + azure-core 1.24.0. The last one is the prefered one, as you get more updated packages.

Problem Details

I’m getting the same error. We’re running Databricks jobs from a Data Factory pipeline and the pipeline works for days and weeks, and then out of the blue we get this error. Then it starts running successfully again.

Notebook Details

The notebook is running sklearn + Azure AutoML models within a Spark pipeline by wrapping the inference work within a Pandas UDF.

Error / Program Details

Note that I can’t share the code because it’s company proprietary information. But also note again that the notebook/code works for days and weeks, and then out of the blue we get this error.

Driver: Standard_DS3_v2 Workers: Standard_DS3_v2 · 1-8 workers Databricks Runtime: 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12) Notebook Dependencies: `dbutils.library.installPyPI(“numpy”, version=“1.16.2”) dbutils.library.installPyPI(“pandas”, version=“0.23.4”) dbutils.library.installPyPI(“scikit-learn”, version=“0.20.3”) dbutils.library.installPyPI(“xgboost”, version=“0.90”)

dbutils.library.installPyPI(“azure-common”, version=“1.1.25”) dbutils.library.installPyPI(“azure-core”, version=“1.8.1”) dbutils.library.installPyPI(“azure-graphrbac”, version=“0.61.1”) dbutils.library.installPyPI(“azure-identity”, version=“1.2.0”) dbutils.library.installPyPI(“azure-mgmt-authorization”, version=“0.61.0”) dbutils.library.installPyPI(“azure-mgmt-containerregistry”, version=“2.8.0”) dbutils.library.installPyPI(“azure-mgmt-core”, version=“1.2.0”) dbutils.library.installPyPI(“azure-mgmt-keyvault”, version=“7.0.0”) dbutils.library.installPyPI(“azure-mgmt-resource”, version=“10.2.0”) dbutils.library.installPyPI(“azure-mgmt-storage”, version=“16.0.0”) dbutils.library.installPyPI(“azureml-automl-core”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-automl-runtime”, version=“1.12.0.post1”) dbutils.library.installPyPI(“azureml-contrib-notebook”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-core”, version=“1.12.0.post2”) dbutils.library.installPyPI(“azureml-dataprep”, version=“2.0.10”) dbutils.library.installPyPI(“azureml-dataprep-native”, version=“20.1.1”) dbutils.library.installPyPI(“azureml-dataset-runtime”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-defaults”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-explain-model”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-interpret”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-model-management-sdk”, version=“1.0.1b6.post1”) dbutils.library.installPyPI(“azureml-pipeline”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-pipeline-core”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-pipeline-steps”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-sdk”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-telemetry”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-train”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-train-automl”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-train-automl-client”, version=“1.12.0.post2”) dbutils.library.installPyPI(“azureml-train-automl-runtime”, version=“1.12.0.post1”) dbutils.library.installPyPI(“azureml-train-core”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-train-restclients-hyperdrive”, version=“1.12.0”) dbutils.library.installPyPI(“azureml-widgets”, version=“1.12.0”)

dbutils.library.restartPython() # Removes Python state, but some libraries might not work without calling this function`

Error: Caused by: org.apache.spark.api.python.PythonException: 'ImportError: cannot import name 'SerializationError' from 'azure.core.exceptions'

Side Comments

“Completely unreliable. How on earth do things work fine for quite some time … then WITHOUT USER CODE CHANGES, behind the scenes” things stop working. - I completely agree with these statements. It makes it hard to recommend Azure AutoML when the libraries it depends on are completely unreliable.

@yeamusic21 Thanks for your feedback.

@luigiw FYI.

@xiangyan99 Thanks! A new virtual environment does help me fix the problem.👍

@Eddielin0926 Thanks for sharing. It looks to me your environment is messed up. E.g. azure library is already deprecated and is not fully compatible with other libraries.

Could you try creating a new virtual environment and install those packages again?

My azure-core version is 1.25.0 and msrest version is 0.7.1, but I still got the error. Is there any possible solution that I can follow?