micronaut-core: https client timeout error on native image
Declarative http client calls fails on any https url’s when built on native image with exception: io.micronaut.http.client.exceptions.HttpClientException: Connect Error: connection timed out
There is no such problem with http urls.
I am providing link to example application build from template mn create-app my-app --features aws-api-gateway-graal and following extensions:
ExampleControllerextended to callhttpsAPI by using declarative http client- Dockerfile was updated to include libsunec.so
- Dockerfile was updated with additional
native-imageoptions -H:EnableURLProtocols=http,https --enable-all-security-services -H:+JNI
Task List
- Steps to reproduce provided
- Stacktrace (if present) provided
- Example that reproduces the problem uploaded to Github
- Full description of the issue provided (see below)
Steps to Reproduce
- Create application by template:
mn create-app my-app --features aws-api-gateway-graal(found in docs underCustom GraalVM Native Runtimesin https://micronaut-projects.github.io/micronaut-aws/latest/guide/#customRuntimes) - Extend
ExampleControllerto use declarative http client for calling anyhttpsAPI/URL - Deploy and run
Expected Behaviour
No error should happen
Actual Behaviour
ExampleController https call results in error:
io.micronaut.http.client.exceptions.HttpClientException: Connect Error: connection timed out: www.boredapi.com/34.237.203.145:443
at io.micronaut.http.client.DefaultHttpClient.lambda$null$27(DefaultHttpClient.java:1048)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:263)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52)
at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: www.boredapi.com/34.237.203.145:443
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
... 13 more
Environment Information
- Operating System: macOS
- Micronaut Version: 1.2.3,1.2.5
- JDK Version: graal 19.2.0.1, 19.2.1
Example Application
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (10 by maintainers)
Commits related to this issue
- Initialize some Netty classes at run-time. Fixes #2335 — committed to micronaut-projects/micronaut-core by ilopmar 4 years ago
- Initialize some classes at runtime to improve GraalVM support (#9963) Motivation: Deploying a Micronaut application as GraalVM native image to AWS Lambda with custom runtime fails when using Micro... — committed to netty/netty by ilopmar 4 years ago
- Initialize some classes at runtime to improve GraalVM support (#9963) Motivation: Deploying a Micronaut application as GraalVM native image to AWS Lambda with custom runtime fails when using Micro... — committed to netty/netty by ilopmar 4 years ago
- Initialize some classes at runtime to improve GraalVM support (#9963) Motivation: Deploying a Micronaut application as GraalVM native image to AWS Lambda with custom runtime fails when using Micro... — committed to ihanyong/netty by ilopmar 4 years ago
@petermd I didn’t see your issue and PR, sorry about that. The Netty team already merged my PR.
The fix on Micronaut is included in 1.3.0.RC1 and once Netty 4.1.46 is released I’ll upgrade Micronaut to use it and remove my fix.
Thanks for your help! 💯
@petermd Thanks for digging into this, we will take a look at verifying what you are seeing on our side
I think I figured out what the issue is. The Netty class
io.netty.util.concurrent.ScheduledFutureTaskcontains a memoized static value fromSystem.nanoTime()that all task timers are normalized to (see ScheduledFutureTask.java#L28).Because Netty is defaulting to
initialize-at-build-time, this value is computed when the native image is created, butSystem.nanoTime()can return an arbitrary value not one that is consistent across environments. If the value memoized innative-imageis higher than the one returned bySystem.nanoTime()at runtime then the computed deltas go negative and get clamped to 0 causing the immediate timeout.On a local environment the values are consistent, which explains why local testing works fine, whereas the opposite is true for Lambda where the runtime is fresh on every function update, so Lambda would be more susceptible to the problem even though there doesn’t appear to be anything Lambda specific to it.
There is a straightforward fix to over-ride the initialization phase for this class (and two dependent classes) in the
native-imagecommand.if you can update your test to verify?