azure-sdk-for-java: [BUG] Jackson issues with Azure SDK when run on Azure Synapse Analytics

Describe the bug

We have a spark application that runs on Azure Synapse Spark Pools. Synapse brings a list of dependencies with it, which are by default available on class-path at runtime: https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/Apache-Spark-3-runtime

In the list is Jackson 2.10.

Our project uses PlayJson 2.9.2 that relies on Jackson 2.10. And latest version of Azure SDK (that brings in Jackson 2.12). When run against the Synapse Spark 3 runtime, we get the following exception.

If we remove the azure sdk dependency, then all the code works fine.

Exception or Stack Trace

When run on Synapse, we get the following exception:

21/11/29 17:38:00 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.LinkageError: Package versions: jackson-annotations=2.10.0, jackson-core=2.10.0, jackson-databind=2.10.0, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.12.5, azure-core=1.19.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
	at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:73)
	at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:81)
	at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:58)
	at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.<clinit>(JacksonAdapter.java:113)
	at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122)
	at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:100)
	at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:139)
	at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:70)
	at com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:129)
	at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:123)com.microsoft.aiops.anomaly.events.AnomalyEventGridSender.$anonfun$sendEvents$1$adapted(AnomalyEventGridSender.scala:25)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.sendEvents(xxxxxx.scala:25)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.runner.xxxxxx.xxxxxx(xxxxxx.scala:82)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx(xxxxxx.scala:61)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.$anonfun$start$2(xxxxxx.scala:39)
	at scala.collection.TraversableLike$WithFilter.$anonfun$map$2(TraversableLike.scala:827)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:826)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.x.xxxxx.xxxxxx.start(xxxxxx.scala:36)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx$.main(xxxxxx.scala:29)
	at xxxxxx.xxxxxx.aiops.xxxxxx.xxxxxx.main(xxxxxx.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732)
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig;
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:176)
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:145)
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:127)
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218)
	at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:84)
	at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:70)
	... 45 more

To Reproduce Run a synapse spark job with azure-messaging-eventgrid and azure-storage-file-datalake dependency

Code Snippet

Expected behavior

Screenshots tree-2.txt Attached is the dependency tree.

Note: As mentioned above, synapse introduces its own set of dependencies at runtime. And we have no control over it.

Setup (please complete the following information):

• OS: WIndows
• IDE: Intellij
• Library/Libraries: Dependency tree attached
• Java version: 8
• App Server/Environment: Synapse
• Frameworks: PlayJson, Spark