newrelic-java-agent: NR agent fails on IBM Semeru Runtimes

Is your feature request related to a problem? Please describe.

Ref discussion here: https://discuss.newrelic.com/t/feature-idea-java-new-relic-agent-doesnt-work-with-openj9-17/170671

When IBM Semeru Runtimes Java 11 recently changed its java.vendor string to “IBM Corporation”, the NR Java agent seems to have stopped working and apparently never supported Java 17 or any other recent IBM Java release. When I asked for more details about why the vendor string mattered, I was pointed at this repo to investigate for myself.

I discovered that the AttachLifecycleObserver is disabled for any JVM that sets java.vendor = "IBM Corporation" which includes all Semeru Runtimes (Open and Certified Edition) as well as IBM’s older IBM SDK for Java 7, 7.1, and 8. There isn’t any reasoning for it in the code, that I could find. I’m suspicious that the reason this change was made is no longer relevant.

Since the NR agent worked just fine for Java 11 before we changed the java.vendor string and the Eclipse OpenJ9 JVM built into IBM Semeru Runtimes uses the same source code across all Java releases (8,11,17,18), I think the NR agent should just work. That probably includes the IBM SDK for Java 8 (which also uses the Eclipse OpenJ9 JVM).

Feature Description

I believe all that’s needed is to remove isAgentSafe() from the AttachLifecycleObserver class. I have a pull request ready to go if the project would like me to submit it.

Describe Alternatives

A clear and concise description of any alternative solutions or features you’ve considered. Are there examples you could link us to?

(migrate to Jira)

Additional context

IBM Semeru Runtimes are the same binaries as used to be built by the AdoptOpenJDK community as “OpenJDK with Eclipse OpenJ9”. When AdoptOpenJDK moved to the Eclipse Foundation, they were prevented (by contract with Oracle) from continuing to produce the OpenJDK with OpenJ9 binaries. IBM stepped forward at the time and has produced the same binaries under the name “IBM Semeru Runtimes”: built with open source components and the binaries are available at no cost for everyone to use. As chief architect for IBM Java and Semeru Runtimes, I would love to see New Relic agents working smoothly with Semeru Runtimes for all our Java releases.

Priority

Really Want. We have customers who have been happily using NR with Semeru Runtimes 11 who are now broken and, while we have a workaround that involves an agent that mashes back the older java.vendor setting, it’s a super hack.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 6
Comments: 22 (11 by maintainers)

Most upvoted comments

@bogdanbrindusan and @mstoodle I’ve removed the IBM workaround logic. We were unable to test as fully as we’d liked. Gradle seems to have trouble with downloading IBM Semeru builds. We hoped to get our full suites running. I could tell in some of my tests that there are some special cases I’ll need to address in our integration tests. Not being able to download them locally with gradle made it really cumbersome to step through.

At any rate, this is going into our 7.11 release. There’s a snapshot build linked below. If you are able to run this agent and provide any feedback it would be greatly appreciated, even if a simple smoke test to confirm you don’t hit any issues. https://github.com/newrelic/newrelic-java-agent/suites/8940160106/artifacts/410261834

Thanks again for bringing this to our attention.

tbradellis on Oct 25, 2022

Thanks for the update, @kford-newrelic .

For the record, the change needed is very minimal from a coding perspective (See https://github.com/newrelic/newrelic-java-agent/compare/main...mstoodle:ibmjava-changes). But I can understand if you’ll want to introduce some additional testing on the free Semeru Runtimes binaries (available here: https://ibm.com/semeru-runtimes/downloads) that it would enable you to support, and that may be effort you aren’t willing to commit to in the next quarter.

It is a shame, though, that all those Fortune 500 and other companies that rely on Semeru Runtimes to run their businesses can’t use your product in the meantime. I guess not many of them are complaining about it, which means it could represent an untapped market segment for you… I would love to tell all of them how well New Relic “now” works out of the box with IBM Semeru Runtimes 😃 .

mstoodle on Jun 15, 2022

Thank you @tbradellis !! That’s fantastic news! On the gradle point, would you please take a few moments to summarize the problem you experienced in an issue at the Semeru Runtimes site so we have a record of it and can start investigating?

https://github.com/ibmruntimes/Semeru-Runtimes/issues/new

Thanks again!

mstoodle on Oct 25, 2022

@mstoodle and @bogdanbrindusan and @zalacmar I was able to replicate the java.lang.NoClassDefFoundError: com.newrelic.agent.bridge.datastore.RecordSql and spend some time troubleshooting this. In doing so I discovered that I gave the wrong argument to workaround this. Apologies!

tldr The system property to use is: -Dibm_iv25688_workaround=false It should be set false i.e. do not use the workaround. I confirmed and was able to use the Semeru 17 runtime without any issue.

@bogdanbrindusan You are right, we are pointlessly failing on some IBM runtimes. However, there are a few more places in the code that would need to be cleaned up that are related to what was put in your branch and we may need to take a slightly different approach. Also, I’ll need to do some testing and add some semeru et. al. runtimes to our integration tests.

A couple notes: As you were suggesting, this code stems from an old bug that has been resolved. https://www.ibm.com/support/pages/apar/IV25688. This bug only exists in Java 6 and 7 jdks anyway, so some of this code is completely obsolete (as our current Java Agent has dropped support for java 6 and 7).

I started working up an explainer for what the various code paths are doing and why we hit the error when we go down the workaround path. Since this is an older path that deals with changes in classloaders across jdks, I’ll need to give a thorough presentation of the control flow to the team for review before ripping it out. I’ll post again here for full visibility, because it’s pretty interesting. For now, setting -Dibm_iv25688_workaround=false should work.

The crux of the coding issue really was that when we go down the ibmworkaround path we never perform addBridgeJarToClassPath(inst, AGENT_BRIDGE_DATASTORE_JAR_NAME); hence NoClassDefFound for RecordSql which is archived in the agent-bridge-datastore.jar).

Thanks everyone for bringing this up!

For visibilty: The system property to use is: -Dibm_iv25688_workaround=false It should be set false i.e. do not use the workaround. I confirmed and was able to use the Semeru 17 runtime without any issue.

tbradellis on Aug 28, 2022

@mstoodle don’t give up on us yet!

If it’s possible to squeeze this into our next quarter’s work, we certainly will. Stay tuned!

kford-newrelic on Jun 22, 2022

@mstoodle Thank you for your efforts to determine what was wrong and summarizing that information here - we really appreciate it!

kford-newrelic on May 9, 2022