aws-sdk-java-v2: DynamoDb-enhanced NoClassDefFoundError with different ClassLoaders
The BeanTableSchema
of dynamodb-enhanced does not work correctly when this library and the bean class reside in different classloaders. A NoClassDefFoundError
for the bean class is thrown, when a call against a bean class/object is made (<init>, getter, setter).
Describe the bug
Internally, the BeanTableSchema creates “Accessor”-Lambdas for performant creation of bean objects, as well as getters and setters for the properties. An example would be the software.amazon.awssdk.enhanced.dynamodb.mapper.BeanTableSchema#newObjectSupplierForClass
(also #getterForProperty
and #setterForProperty
) responsible for creating a Supplier
that returns a new bean instance.
Internally, the LambdaToMethodBridgeBuilder
is then used to construct such lambda via the LambdaMetafactory
. The creation of the lambda itself is successful.
However, when this generated lambda is invoked, it crashes with the mentioned NoClassDefFoundError
. This is related to the fact that the Lambda (somehow) tries to resolve the bean class with the classloader of the SDK. But since the bean class is loaded in a different classloader, the classloader of the SDK cannot find the bean class.
More specifically, the problem arises when the SDK classloader is either a parent of or entirely unrelated to the classloader of the bean.
This issue is most likely related to issue #2198 since the error messages are identical. However that issue is specifically about usage with the play framework (which I do not know anything about). Furthermore, a usage of Class.forName
is mentioned but I did not come across usage of that. Therefore I decided to create a more focused issue for the core problem I (somewhat) have a solution for.
This issue should fix the NoClassDefFoundError
of https://github.com/quarkusio/quarkus/issues/12168.
Expected Behavior
Normal execution and same behavior when AWS SDK and bean class are in the same classloader.
Current Behavior
When such generated lambda is invoked, the following (sample) stack trace is thrown:
Caused by: java.lang.NoClassDefFoundError: org/acme/dynamodb/Fruit
at software.amazon.awssdk.enhanced.dynamodb.internal.mapper.ResolvedImmutableAttribute.lambda$create$0(ResolvedImmutableAttribute.java:48)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema.lambda$itemToMap$5(StaticImmutableTableSchema.java:502)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema.itemToMap(StaticImmutableTableSchema.java:500)
at software.amazon.awssdk.enhanced.dynamodb.mapper.WrappedTableSchema.itemToMap(WrappedTableSchema.java:64)
at software.amazon.awssdk.enhanced.dynamodb.mapper.WrappedTableSchema.itemToMap(WrappedTableSchema.java:64)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.generateRequest(PutItemOperation.java:71)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.generateRequest(PutItemOperation.java:40)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.CommonOperation.execute(CommonOperation.java:113)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.TableOperation.executeOnPrimaryIndex(TableOperation.java:59)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:180)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:188)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:193)
at org.acme.dynamodb.FruitSyncService.add(FruitSyncService.java:49)
at org.acme.dynamodb.FruitSyncService_ClientProxy.add(FruitSyncService_ClientProxy.zig:354)
at org.acme.dynamodb.FruitResource.add(FruitResource.java:40)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:476)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:434)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:408)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:69)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
... 15 more
Caused by: java.lang.ClassNotFoundException: org.acme.dynamodb.Fruit
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:421)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:397)
... 47 more
(where Fruit
is the bean class.)
The code that creates the dynamodb client looks like this:
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder().dynamoDbClient(ddb).build();
var table = enhancedClient.table("Fruits", TableSchema.fromBean(Fruit.class));
offending user-application code is:
table.putItem(fruit);
Steps to Reproduce
Loading a dynamodb-enhanced annotated bean in a child (or unrelated) classloader, pass it to TableSchema.fromBean(...)
and use any action like getItem
or putItem
.
A self-contained runnable example can be found here: https://github.com/Nithanim/awssdk2-dynamodb-enhanced-NDFE/tree/master/src/main/java
You can find a “crashing” example specifically using the problematic AWS SDK class. Additionally there is an isolated case without the AWS SDK only with plain java.
Possible Solution
When creating the Lambdas via the LambdaMetafactory
the Lookup
of the AWS SDK is used. This is (as far as I understand) the cause that the wrong classloader for the lookup is used.
A working solution would be to use the Lookup
of the bean class, which in turn will use the classloader of the bean class to resolve the bean class on invocation of the lambda.
In the same repository (https://github.com/Nithanim/awssdk2-dynamodb-enhanced-NDFE/tree/master/src/main/java) you can also find the “Fixed” classes, which have been patched to reslove the issue.
However, I am completely unsure if Lookup.privateLookupIn(...)
is the correct way.
Context
I wanted to use the dynamodb-enhanced in quarkus (and specifically later in native-image) but the Tests kept failing with the NoClassDefFoundError
. After some digging I found out that quarkus uses two different classloaders for tests. One for all libraries and then a child classloader for the tests. A bit of insight can be found here: https://quarkus.io/guides/class-loading-reference
Thank you for your help!
Your Environment
- AWS Java SDK version used: 2.16.81
- JDK version used: build 11.0.11+9-Ubuntu-0ubuntu2.20.04
- Operating System and version: Ubuntu 20.04.2 LTS
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 31
- Comments: 21 (1 by maintainers)
It does not work well with Spring Boot Devtools too.
Dev tools uses two class loaders, one for third party libraries, and the other for project classes.
So I get something like this :
com.example.myblog.model.dynamodb.User incompatible with com.example.myblog.model.dynamodb.User java.lang.ClassCastException: com.example.myblog.model.dynamodb.User incompatible with com.example.myblog.model.dynamodb.User at com.example.myblog.repository.dynamodb.UserDynamoDbRepository.findAll(UserDynamoDbRepository.java:96) at com.example.myblog.controller.MainController.index(MainController.java:29) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:853)
I noticed that my proposed solution only works for Java 9+ but this library is 8+. Sadly I have not found any other good solution.
I have not been able to find a workaround just using the MethodHandles/Lookups provided by Java 8 alone. A workaround would be to create child ClassLoader (CL) of the bean CL in which we can load a class from which we can fetch a Lookup for that context.
However, this is rather ugly because we would need the bytecode of such class and then load it in a classloader as shown here. My proof-of-concept with bytebuddy is already convoluted enough considering this is all just a dance-around for the simple Lookup.privateLookupIn() of Java 9+.
In short:
Yes right, let me try, my observation was couple of months old, just now i saw another thread where you guys are discussing about fix, will try that and also update on cold start with both the approach, thanks
Quarkus extension dont solve dynamodb enhanced client issues ( native issue, dev and test issue),
To solve native issue if someone need to build native - use static dynamodb table schema
To solve dev, test issue - use %dev.quarkus.class-loading.reloadable-artifacts=software.amazon.awssdk:dynamodb-enhanced
%test.quarkus.class-loading.reloadable-artifacts=software.amazon.awssdk:dynamodb-enhanced
edit Quarkus dynamodb extension is needed and on top of that we have to follow above mentioned steps to make dynamodb enhanced client work( just dynamodb client will work with just extension but to make enhanced client work we need to make some extra changes than just adding quarkus dynamodb extension)
Same issue here with 2.11.1, even setting quarkus.class-loading.reloadable-artifact
Sadly have been stuck on this and have to avoid enhanced dynamodb up to now in our native images. When this issue gets resolved it would simplify our business code a lot