quarkus: --initialize-at-build-time appears to be ignored | AWS XRay SDK

Describe the bug I’m trying to get the AWS XRay SDK working using Manual Instrumentation from this doc. When doing so, I get an error during mvn clean install -Dnative -Dquarkus.native.container-build=true :

Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a FileDescriptor in the image heap. File descriptors opened during image generation are no longer open at image run time, and the files might not even be present anymore at image run time.  To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-build-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace:  object java.net.PlainDatagramSocketImpl
        object java.net.DatagramSocket
        object com.amazonaws.xray.emitters.DefaultEmitter
        object com.amazonaws.xray.AWSXRayRecorder
        field com.amazonaws.xray.AWSXRay.globalRecorder

Setting --initialize-at-build-time=com.amazonaws.xray in quarkus.native.additional-build-args appears to have no effect. I notice that --initialize-at-build-time= is set twice per #5480.

Expected behavior Setting --initialize-at-build-time=com.amazonaws.xray would lead the static field with a file handle to resolve at build time, and the stack trace to go away.

Actual behavior It doesn’t happen, even though I’m following guidance from the stacktrace.

To Reproduce Steps to reproduce the behavior:

Add to your pom.xml

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-xray-recorder-sdk-aws-sdk-v2</artifactId>
            <version>2.4.0</version>
        </dependency>

Instrument the AWS SDK per Manual Instrumentation from this doc.
Update application.properties with --initialize-at-build-time=com.amazonaws.xray
mvn clean install -Dnative -Dquarkus.native.container-build=true

Configuration

quarkus.lambda.handler=myHandler
quarkus.ssl.native=true
quarkus.native.additional-build-args=\
  -H:ReflectionConfigurationFiles=reflection-config.json,\
  --initialize-at-build-time=com.amazonaws.xray,\
  --initialize-at-run-time=com.some.avro

Environment (please complete the following information):

Output of uname -a or ver: Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64
Output of java -version: openjdk version “1.8.0_212” OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b03) OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b03, mixed mode)
GraalVM version (if different from Java): 19.2.1-grl via sdkman
Quarkus version or git rev: 1.1.1.Final

Additional context If you need a reproducer, let me know. I could create one on Monday.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 34 (31 by maintainers)

Commits related to this issue

Merge pull request #7165 from patriot1burke/lambda-trace Support for AWS XRay Issue #6785 — committed to quarkusio/quarkus by gsmet 4 years ago

Most upvoted comments

Got reflection working, and we’ve run into what appears to be a more fundamental issue. At runtime, everything boots up correctly, but when tracing should begin, we get the following: WARN [com.ama.xra.con.LambdaSegmentContext] (Lambda Thread) _X_AMZN_TRACE_ID is missing a trace ID, parent ID, or sampling decision. Subsegment S3 discarded.

It appears the root cause here is that AWS Lambda Custom Runtimes are required to do several things, including propagate the _X_AMZN_TRACE_ID environment variable. The quarkus lambda extension handles these concerns in Java code, but does not handle pass on _X_AMZN_TRACE_ID from the /runtime/invocation/next invocation. Digging into the AWS XRay SDK, there does not appear to be a way to set the trace id in the recorder apart from a environment variable though I haven’t looked at this extensively.

So at this point, it appears that sending xray subsegments is not possible with the current aws lambda extension for quarkus, and that we have the following options forward:

update the quarkus extension with some approach to set the env var despite the fact that java doesn’t allow that by default.
change the strategy for the quarkus lambda extension to use a bootstrap script to set the env vars and then pass information to the quarkus process
ask if aws will allow another way of setting the trace ID, perhaps via a system property?

sherl0cks on Jan 28, 2020

@sherl0cks @christianThor I improved things a bit so there’s less boilerplate to do. Here’s what the config would look like.

quarkus.lambda.handler=test quarkus.ssl.native=true quarkus.native.additional-build-args=-H:DynamicProxyConfigurationFiles=dynamic-proxy-config.json

Eventually I want to be able to get rid of the wrapper shell script, and the need to package the lib.so file and cacert file, but not sure I’ll be able to do that with the first PR.

You tell me, but, I also don’t think the DynamicProxyConfig is related to Xray, and is more of a function of using apache. Need to think about how to remove that boilerplate too.

patriot1burke on Feb 12, 2020

@sherl0cks Actually instead of hacking System.getenv, which may or may not work, I think I’ll just code replace LambdaSegmentContext.getTraceHeaderFromEnvironment

patriot1burke on Feb 4, 2020

I should be able to look at it on Tuesday. Looks like a simple fix.

patriot1burke on Feb 3, 2020