newrelic-php-agent: Large performance regression in 10.3.0

Description

We are seeing a very large hit to performance everywhere we are running New Relic after we updated the extension to 10.3.0.315. It seems like other users are also seeing this issue.

Simply switching between 10.2.0.314 and 10.3.0.315 causes some pages for us to be at least an order of magnitude slower (I say at least because the difference is a page taking 6 seconds to load vs. hitting the 60 second timeout we had).

We assumed the performance hit came from the new logging changes so we tried the following different config options, but so far we have not had any success in mitigating the issue:

newrelic.cross_application_tracer.enabled = false
newrelic.distributed_tracing_enabled = false
newrelic.span_events_enabled = false
newrelic.application_logging.enabled = false
newrelic.application_logging.forwarding.enabled = false
newrelic.application_logging.metrics.enabled = false

Here are some graphs from a couple of different sites on different hardware configurations and you can clearly see when the extension was updated and then reverted:

2022-11-04 09_55_54

2022-11-04 09_57_23

Steps to Reproduce

Install 10.3.0.315 extension version with some organic traffic.

Expected Behavior

No performance degradation.

Relevant Logs / Console output

N/A

Your Environment

All environments were Debian Linux with different PHP versions affected (7.4.32 and 8.0.25 at least).

Additional context

Reinstalling the older NR extension with no other changes to the environment immediately rectified the issue.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 6
Comments: 24 (7 by maintainers)

Most upvoted comments

Fixed in 10.5.0 release.

lavarou on Jan 18, 2023

Saw this over the past week as well. Except our WP site pages went from 0.5s load times to 15-30s load times if they would finish at all.

Troubleshot all kinds of potential plugin issues to track down the performance issue. Was using NR to find it, only to realize NR was the entire cause.

With all plugins active and NR off, back to <1s load times across the board.

danieliser on Nov 21, 2022

We are experiencing a significant degradation in performance after upgrading to PHP 8 and the newest agent. We can not use JIT, which is hurting us from a performance point of view. We do not use WordPress. We Use Laravel. Our solution was to revert to an old version - 9.18. We saw a 10 -20 % performance gain after that !!! And this is before using JIT. That is crazy. This is a workaround as we want to be on the latest. Does anyone know if Newrelic acknowledges the issue? What timeline to solve this?

nlitterat on Jan 11, 2023

hi @bduranleau-nr, we at hypernode (ecommerce hosting) have been monitoring this thread and this discussion thread following several messages from concerned users of our platform over the past couple of days and I just wanted to put in our 2 cents and mention that if there’s any way for you guys to prioritize addressing this issue with a new version of the new relic extension as soon as possible that would be great. I think we have a pretty good representative sample size of webshops and we have had significant trouble with woocommerce / wordpress based shops that belong to people who are newrelic customers and especially this close to black friday that’s really something that’s affecting people’s trust in new relic as a product. To the extent that we see this having caused issues on our hosting platform I can imagine that there’s a lot of users out there in the world who are running into the exact same problem but don’t feel the need to report it or haven’t been able to pinpoint it back to newrelic. Especially because newrelic is often the tool people rely on for visibility in matters of performance, now that newrelic is actually the issue that is catching people by surprise and that can turn in to a lot of wasted time and frustration.

vdloo on Nov 16, 2022

Thanks again for bringing this to the team’s attention @pgibson1-godaddy - we believe we’ve identified the issue, and we are currently working on implementing a fix that should resolve it. We also believe that impact is limited to Wordpress applications, and other frameworks should not see this kind of performance degradation.

In the meantime, downgrading to version 10.2.0.314 should resolve the negative performance impact while we work on putting together a release to formally address it.

bduranleau-nr on Nov 4, 2022