azure-sdk-for-java: [BUG] Reactor/Netty dependencies issues when using KeyVault and Service Bus libraries

Describe the bug

Simple java function that get messages from a service bus queue and fetch a certificate from the keyvault. During the keyvault initialization a NoSuchMethod error occur

Exception or Stack Trace

Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure\nException: NoSuchMethodError: io.netty.handler.logging.LoggingHandler.<init>(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V\nStack: java.lang.reflect.InvocationTargetException\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodExecutorImpl.execute(JavaMethodExecutorImpl.java:54)\n\tat com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:57)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)\n\tat com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)\n\tat com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoSuchMethodError: io.netty.handler.logging.LoggingHandler.<init>(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V\n\tat reactor.netty.transport.logging.ReactorNettyLoggingHandler.<init>(ReactorNettyLoggingHandler.java:63)\n\tat reactor.netty.transport.logging.AdvancedByteBufFormat.toLoggingHandler(AdvancedByteBufFormat.java:109)\n\tat reactor.netty.http.client.HttpClientConfig.<clinit>(HttpClientConfig.java:635)\n\tat reactor.netty.http.client.HttpClientConnect.<init>(HttpClientConnect.java:84)\n\tat reactor.netty.http.client.HttpClient.create(HttpClient.java:391)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)\n\tat com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)\n\tat com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildAsyncClient(CertificateClientBuilder.java:167)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildClient(CertificateClientBuilder.java:109)

To Reproduce

create a skeleton java function application using a @ServiceBusQueueTrigger create a keyvault with the function list/get permissions try to fetch a certificate from the keyvault from within the function

Code Snippet

        certificateClient = new CertificateClientBuilder()
                .vaultUrl(keyVaultUrl)
                .credential(new ManagedIdentityCredentialBuilder().build())
                .buildClient();
        this.secretClient = new SecretClientBuilder()
                .vaultUrl(keyVaultUrl)
                .credential(new ManagedIdentityCredentialBuilder().build())
                .buildClient();

Expected behavior

No exceptions

Screenshots If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

  • OS: azure linux function app service
  • IDE: IntelliJ]
  • Library/Libraries:
com.microsoft.azure.functions:azure-functions-java-library:1.4.2
com.microsoft.azure:azure-servicebus:3.6.4
com.azure:azure-security-keyvault-secrets:4.2.8
com.azure:azure-security-keyvault-certificates:4.1.8
com.azure:azure-identity:1.3.1

maven dependencies tree:

[INFO] ± com.microsoft.azure.functions:azure-functions-java-library:jar:1.4.2:compile [INFO] ± com.microsoft.azure:azure-servicebus:jar:3.6.4:compile [INFO] | ± org.apache.qpid:proton-j:jar:0.33.4:compile [INFO] | ± com.microsoft.azure:qpid-proton-j-extensions:jar:1.2.3:compile [INFO] | ± org.slf4j:slf4j-api:jar:1.7.30:compile [INFO] | ± org.asynchttpclient:async-http-client:jar:2.12.1:compile [INFO] | | ± org.asynchttpclient:async-http-client-netty-utils:jar:2.12.1:compile [INFO] | | ± io.netty:netty-codec-socks:jar:4.1.48.Final:compile [INFO] | | ± io.netty:netty-handler-proxy:jar:4.1.48.Final:compile [INFO] | | ± io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.48.Final:compile [INFO] | | ± io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.48.Final:compile [INFO] | | ± org.reactivestreams:reactive-streams:jar:1.0.3:compile [INFO] | | ± com.typesafe.netty:netty-reactive-streams:jar:2.0.4:compile [INFO] | | - com.sun.activation:javax.activation:jar:1.2.0:compile [INFO] | - com.microsoft.azure:azure-client-authentication:jar:1.7.3:compile [INFO] | ± com.microsoft.azure:azure-client-runtime:jar:1.7.3:compile [INFO] | | - com.microsoft.rest:client-runtime:jar:1.7.3:compile [INFO] | | ± com.squareup.retrofit2:retrofit:jar:2.7.2:compile [INFO] | | ± com.squareup.okhttp3:okhttp:jar:3.12.6:compile [INFO] | | | - com.squareup.okio:okio:jar:1.15.0:compile [INFO] | | ± com.squareup.okhttp3:logging-interceptor:jar:3.12.2:compile [INFO] | | ± com.squareup.okhttp3:okhttp-urlconnection:jar:3.12.2:compile [INFO] | | ± com.squareup.retrofit2:converter-jackson:jar:2.7.2:compile [INFO] | | ± com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.10.1:compile [INFO] | | ± io.reactivex:rxjava:jar:1.3.8:compile [INFO] | | - com.squareup.retrofit2:adapter-rxjava:jar:2.7.2:compile [INFO] | ± com.microsoft.azure:adal4j:jar:1.6.4:compile [INFO] | | - com.google.code.gson:gson:jar:2.8.0:compile [INFO] | - com.microsoft.azure:azure-annotations:jar:1.10.0:compile [INFO] ± org.apache.httpcomponents.client5:httpclient5:jar:5.0.3:compile [INFO] | ± org.apache.httpcomponents.core5:httpcore5:jar:5.0.2:compile [INFO] | ± org.apache.httpcomponents.core5:httpcore5-h2:jar:5.0.2:compile [INFO] | - commons-codec:commons-codec:jar:1.13:compile [INFO] ± com.azure:azure-security-keyvault-secrets:jar:4.2.8:compile [INFO] | ± com.azure:azure-core:jar:1.16.0:compile [INFO] | | ± com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile [INFO] | | ± com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile [INFO] | | ± com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile [INFO] | | ± com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.2:compile [INFO] | | ± com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.12.2:compile [INFO] | | | ± com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.2:compile [INFO] | | | | ± jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile [INFO] | | | | - jakarta.activation:jakarta.activation-api:jar:1.2.1:compile [INFO] | | | ± org.codehaus.woodstox:stax2-api:jar:4.2.1:compile [INFO] | | | - com.fasterxml.woodstox:woodstox-core:jar:6.2.4:compile [INFO] | | ± io.projectreactor:reactor-core:jar:3.4.5:compile [INFO] | | - io.netty:netty-tcnative-boringssl-static:jar:2.0.38.Final:compile [INFO] | - com.azure:azure-core-http-netty:jar:1.9.2:compile [INFO] | ± io.netty:netty-codec-http2:jar:4.1.63.Final:compile [INFO] | ± io.netty:netty-transport-native-unix-common:jar:4.1.63.Final:compile [INFO] | - io.projectreactor.netty:reactor-netty:jar:1.0.6:compile [INFO] | ± io.projectreactor.netty:reactor-netty-core:jar:1.0.6:compile [INFO] | | ± io.netty:netty-resolver-dns:jar:4.1.63.Final:compile [INFO] | | | - io.netty:netty-codec-dns:jar:4.1.63.Final:compile [INFO] | | - io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.63.Final:compile [INFO] | ± io.projectreactor.netty:reactor-netty-http:jar:1.0.6:compile [INFO] | - io.projectreactor.netty:reactor-netty-http-brave:jar:1.0.6:runtime [INFO] | - io.zipkin.brave:brave-instrumentation-http:jar:5.13.3:runtime [INFO] | - io.zipkin.brave:brave:jar:5.13.3:runtime [INFO] | - io.zipkin.reporter2:zipkin-reporter-brave:jar:2.16.3:runtime [INFO] | - io.zipkin.reporter2:zipkin-reporter:jar:2.16.3:runtime [INFO] | - io.zipkin.zipkin2:zipkin:jar:2.23.2:runtime [INFO] ± com.azure:azure-security-keyvault-certificates:jar:4.1.8:compile [INFO] ± com.azure:azure-identity:jar:1.3.1:compile [INFO] | ± com.microsoft.azure:msal4j:jar:1.10.0:compile [INFO] | | - com.nimbusds:oauth2-oidc-sdk:jar:9.4:compile [INFO] | | ± com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile [INFO] | | ± com.nimbusds:content-type:jar:2.1:compile [INFO] | | ± net.minidev:json-smart:jar:2.4.2:compile (version selected from constraint [1.3.2,2.4.2]) [INFO] | | | - net.minidev:accessors-smart:jar:2.4.2:compile [INFO] | | | - org.ow2.asm:asm:jar:8.0.1:compile [INFO] | | ± com.nimbusds:lang-tag:jar:1.5:compile [INFO] | | - com.nimbusds:nimbus-jose-jwt:jar:9.8.1:compile [INFO] | ± com.microsoft.azure:msal4j-persistence-extension:jar:1.1.0:compile [INFO] | | - net.java.dev.jna:jna:jar:5.5.0:compile [INFO] | ± net.java.dev.jna:jna-platform:jar:5.6.0:compile [INFO] | - org.linguafranca.pwdb:KeePassJava2:jar:2.1.4:compile [INFO] | ± org.linguafranca.pwdb:KeePassJava2-kdb:jar:2.1.4:compile [INFO] | | ± org.linguafranca.pwdb:database:jar:2.1.4:compile [INFO] | | | - com.madgag.spongycastle:core:jar:1.54.0.0:compile [INFO] | | - org.jetbrains:annotations:jar:15.0:compile [INFO] | ± org.linguafranca.pwdb:KeePassJava2-dom:jar:2.1.4:compile [INFO] | | - org.linguafranca.pwdb:KeePassJava2-kdbx:jar:2.1.4:compile [INFO] | ± org.linguafranca.pwdb:KeePassJava2-jaxb:jar:2.1.4:compile [INFO] | - org.linguafranca.pwdb:KeePassJava2-simple:jar:2.1.4:compile [INFO] | ± org.simpleframework:simple-xml:jar:2.7.1:compile [INFO] | | ± stax:stax-api:jar:1.0.1:compile [INFO] | | - stax:stax:jar:1.2.0:compile [INFO] | ± org.apache.httpcomponents:httpcore:jar:4.4.5:compile [INFO] | - com.fasterxml:aalto-xml:jar:1.0.0:compile [INFO] ± xpp3:xpp3:jar:1.1.4c:compile [INFO] ± org.slf4j:log4j-over-slf4j:jar:1.7.30:compile [INFO] ± ch.qos.logback:logback-classic:jar:1.2.3:compile [INFO] | - ch.qos.logback:logback-core:jar:1.2.3:compile [INFO] ± org.projectlombok:lombok:jar:1.18.20:provided [INFO] ± org.junit.jupiter:junit-jupiter:jar:5.7.1:test [INFO] | ± org.junit.jupiter:junit-jupiter-api:jar:5.7.1:test [INFO] | | ± org.apiguardian:apiguardian-api:jar:1.1.0:test [INFO] | | ± org.opentest4j:opentest4j:jar:1.2.0:test [INFO] | | - org.junit.platform:junit-platform-commons:jar:1.7.1:test [INFO] | ± org.junit.jupiter:junit-jupiter-params:jar:5.7.1:test [INFO] | - org.junit.jupiter:junit-jupiter-engine:jar:5.7.1:test [INFO] | - org.junit.platform:junit-platform-engine:jar:1.7.1:test [INFO] ± org.mockito:mockito-core:jar:3.9.0:test [INFO] | ± net.bytebuddy:byte-buddy:jar:1.10.20:test [INFO] | ± net.bytebuddy:byte-buddy-agent:jar:1.10.20:test [INFO] | - org.objenesis:objenesis:jar:3.2:test [INFO] ± org.mockito:mockito-junit-jupiter:jar:3.9.0:test [INFO] ± org.mock-server:mockserver-netty:jar:5.11.2:test [INFO] | ± org.mock-server:mockserver-core:jar:5.11.2:test [INFO] | | ± com.lmax:disruptor:jar:3.4.2:test [INFO] | | ± javax.servlet:javax.servlet-api:jar:4.0.1:test [INFO] | | ± com.jcraft:jzlib:jar:1.1.3:test [INFO] | | ± com.fasterxml.uuid:java-uuid-generator:jar:4.0.1:test [INFO] | | ± org.apache.velocity:velocity-engine-scripting:jar:2.2:test [INFO] | | ± org.apache.velocity:velocity-engine-core:jar:2.2:test [INFO] | | ± net.javacrumbs.json-unit:json-unit-core:jar:2.19.0:test [INFO] | | | - org.hamcrest:hamcrest-core:jar:2.2:test [INFO] | | | - org.hamcrest:hamcrest:jar:2.2:test [INFO] | | ± com.github.java-json-tools:json-schema-validator:jar:2.2.14:test [INFO] | | | ± com.github.java-json-tools:jackson-coreutils-equivalence:jar:1.0:test [INFO] | | | | - com.github.java-json-tools:jackson-coreutils:jar:2.0:test [INFO] | | | | - com.github.java-json-tools:msg-simple:jar:1.2:test [INFO] | | | | - com.github.java-json-tools:btf:jar:1.3:test [INFO] | | | ± com.github.java-json-tools:json-schema-core:jar:1.2.14:test [INFO] | | | | ± com.github.java-json-tools:uri-template:jar:0.10:test [INFO] | | | | - org.mozilla:rhino:jar:1.7.7.2:test [INFO] | | | ± com.sun.mail:mailapi:jar:1.6.2:test [INFO] | | | ± joda-time:joda-time:jar:2.10.5:compile [INFO] | | | ± com.googlecode.libphonenumber:libphonenumber:jar:8.11.1:test [INFO] | | | - net.sf.jopt-simple:jopt-simple:jar:5.0.4:test [INFO] | | ± com.jayway.jsonpath:json-path:jar:2.4.0:test [INFO] | | ± io.swagger.parser.v3:swagger-parser:jar:2.0.23:test [INFO] | | | ± io.swagger.parser.v3:swagger-parser-v2-converter:jar:2.0.23:test [INFO] | | | | ± io.swagger:swagger-core:jar:1.6.2:test [INFO] | | | | | ± io.swagger:swagger-models:jar:1.6.2:test [INFO] | | | | | | - io.swagger:swagger-annotations:jar:1.6.2:test [INFO] | | | | | - javax.validation:validation-api:jar:1.1.0.Final:test [INFO] | | | | ± io.swagger:swagger-parser:jar:1.0.52:test [INFO] | | | | | - org.slf4j:slf4j-ext:jar:1.7.28:test [INFO] | | | | ± io.swagger:swagger-compat-spec-parser:jar:1.0.52:test [INFO] | | | | | - org.apache.httpcomponents:httpclient:jar:4.5.2:test [INFO] | | | | | - commons-logging:commons-logging:jar:1.2:test [INFO] | | | | ± io.swagger.core.v3:swagger-models:jar:2.1.5:test [INFO] | | | | - io.swagger.parser.v3:swagger-parser-core:jar:2.0.23:test [INFO] | | | - io.swagger.parser.v3:swagger-parser-v3:jar:2.0.23:test [INFO] | | | ± io.swagger.core.v3:swagger-core:jar:2.1.5:test [INFO] | | | | ± io.swagger.core.v3:swagger-annotations:jar:2.1.5:test [INFO] | | | | - jakarta.validation:jakarta.validation-api:jar:2.0.2:test [INFO] | | | - com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.11.1:test [INFO] | | | - org.yaml:snakeyaml:jar:1.26:test [INFO] | | ± javax.xml.bind:jaxb-api:jar:2.4.0-b180830.0359:test [INFO] | | | - javax.activation:javax.activation-api:jar:1.2.0:test [INFO] | | ± org.xmlunit:xmlunit-core:jar:2.8.0:test [INFO] | | ± org.xmlunit:xmlunit-placeholders:jar:2.8.0:test [INFO] | | - org.apache.commons:commons-text:jar:1.9:test [INFO] | ± io.netty:netty-buffer:jar:4.1.53.Final:compile [INFO] | ± io.netty:netty-codec:jar:4.1.53.Final:compile [INFO] | ± io.netty:netty-codec-http:jar:4.1.53.Final:compile [INFO] | ± io.netty:netty-common:jar:4.1.53.Final:compile [INFO] | ± io.netty:netty-handler:jar:4.1.53.Final:compile [INFO] | | - io.netty:netty-resolver:jar:4.1.53.Final:compile [INFO] | ± io.netty:netty-transport:jar:4.1.53.Final:compile [INFO] | ± commons-io:commons-io:jar:2.8.0:test [INFO] | - com.google.guava:guava:jar:30.0-android:compile [INFO] | ± com.google.guava:failureaccess:jar:1.0.1:compile [INFO] | ± com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile [INFO] | ± com.google.code.findbugs:jsr305:jar:3.0.2:compile [INFO] | ± org.checkerframework:checker-compat-qual:jar:2.5.5:compile [INFO] | ± com.google.errorprone:error_prone_annotations:jar:2.3.4:compile [INFO] | - com.google.j2objc:j2objc-annotations:jar:1.3:compile [INFO] ± org.mock-server:mockserver-client-java:jar:5.11.2:test [INFO] | - org.apache.commons:commons-lang3:jar:3.11:compile [INFO] ± org.mock-server:mockserver-junit-jupiter:jar:5.11.2:test [INFO] ± com.github.stefanbirkner:system-lambda:jar:1.2.0:test [INFO] - uk.org.webcompere:system-stubs-jupiter:jar:1.2.0:test [INFO] - uk.org.webcompere:system-stubs-core:jar:1.2.0:test

Additional context

Deploy the function in a function app service, place a message in the queue, the function get triggered but fails to connect to the keyvault to fetch a certificate.

Tried to mitigate by adding reactor/netty dependencies with no success:

        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.63.Final</version>
        </dependency>

        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty-http</artifactId>
            <version>3.4.5</version>
        </dependency>

result in chasing other exceptions, with no luck so far:

2021-06-10T09:10:09.804094220Z Caused by: java.lang.NoClassDefFoundError: Could not initialize class reactor.netty.http.client.HttpClientConfig
2021-06-10T09:10:09.804097920Z  at reactor.netty.http.client.HttpClientConnect.<init>(HttpClientConnect.java:84)
2021-06-10T09:10:09.804101820Z  at reactor.netty.http.client.HttpClient.create(HttpClient.java:391)
2021-06-10T09:10:09.804352328Z  at com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)
2021-06-10T09:10:09.804373429Z  at com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)
2021-06-10T09:10:09.804377729Z  at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)
2021-06-10T09:10:09.804381429Z  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)
2021-06-10T09:10:09.804385030Z  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)
2021-06-10T09:10:09.804388630Z  at com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)
2021-06-10T09:10:09.804392230Z  at com.azure.security.keyvault.certificates.CertificateClientBuilder.buildAsyncClient(CertificateClientBuilder.java:167)
2021-06-10T09:10:09.804395930Z  at com.azure.security.keyvault.certificates.CertificateClientBuilder.buildClient(CertificateClientBuilder.java:109)

Information Checklist

Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 21 (5 by maintainers)

Most upvoted comments

@alzimmermsft DING DING DING! That was it, thank you a million times, who knew one little setting like that would do it? I’m going to do some more unit testing on my end but I was able to use the BOM definition and also it uses netty still (I removed all the exclusions and the okhttp implementation).

Hi @Paizo and @derekg52, which version of Java is being used by Azure Functions? If you are using Java 8 could you run your application with the FUNCTIONS_WORKER_JAVA_LOAD_APP_LIBS environment variable set to true.