apm-agent-java: JVM crash with Java 7 + Agent 1.3.0

Describe the bug We updated apm agent to 1.3.0 yesterday and then after about 18 hours two instances got JVM crash which look pretty similar.

# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x000000008fb018d8, pid=122146, tid=140043922573056

Both are using Java 7 (7.0_75 and 7.0_141). One is running on Glassfish 4.0 and the other Payara 4.1.1.171.1.

Debug logs

Current thread (0x00007f7ee0354800):  JavaThread "apm-reporter" daemon 

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x000000008fb018d8

Also there is co.elastic.apm.agent.report.serialize.DslJsonSerializer::sanitizeTagKey in both compilation events.

Full logs: hs_err_pid122146.log hs_err_pid43918.log

This might be related to elastic/apm-agent-java#444

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 47 (23 by maintainers)

Commits related to this issue

Use reflection rather than MethodHandles This is a workaround for a bug in the C2 compiler in Java 7 which leads to a segfault closes #458 — committed to felixbarny/apm-agent-java by felixbarny 5 years ago
Use reflection rather than MethodHandles (#487) This is a workaround for a bug in the C2 compiler in Java 7 which leads to a segfault closes #458 — committed to elastic/apm-agent-java by felixbarny 5 years ago

Most upvoted comments

About 18h uptime now with 1.3.1-SNAPSHOT without any compile exclusions so seems to work.

1.3.0 with exclusions still ended up crashing after 17h uptime. Last compiled method was java.lang.invoke.BoundMethodHandle$Species_LL::<init> and nothing else obvious to my eye. Really strange how well it breaks JVM 😃 hs_err_pid27244.log test-20190210_184348.log

Aketzu on Feb 12, 2019

I think we should use reflection instead of method handles if the agent is running on Java 7.

felixbarny on Feb 11, 2019

Now it has been running 14 hours without crashes (with metrics_interval=1s) so looks pretty good.

stdout shows

### Excluding compile: static java.lang.invoke.LambdaForm$NFI/991593213::invoke_J_D
### Excluding compile: static java.lang.invoke.LambdaForm$NFI/1305645237::invoke_L_J
### Excluding compile: static java.lang.invoke.LambdaForm$NFI/1234030171::invoke_D_D
### Excluding compile: static java.lang.invoke.LambdaForm$NFI/2013163400::invoke_L_D

Aketzu on Feb 11, 2019

Not yet. I don’t want to test many things at the same time.

TeroPihlaja on Feb 7, 2019

We are running 1.3.0 with Java 8. This issue seems to only happen on Java 7.

I will try your suggestions and see what happens.

TeroPihlaja on Feb 5, 2019