newrelic-php-agent: All transactions unknown & 100% error rate post PHP8.1 upgrade (laravel v8)

Hello everyone, as always we thank you for your invaluable feedback and inputs that help us build a better product. This is a known issue that our team is actively working to resolve and is part of our currently prioritized initiatives. We are hopeful to have a solution that resolves this as well as improves the PHP Agent soon. We will provide more information on completion as we hash out further details but would like to assure you that this is actively being worked on at present.

@ak-war, re:

we are expecting a release sometime in October 2022 that will address this issue.

Any updates on when in October the release will be?

New Relic PHP Agent v10.3.0.315 has been released with the fix: https://github.com/newrelic/newrelic-php-agent/releases.

10 days ago, september; and now, October. @ak-war , sorry but your product is basically useless right now. Most of PHP users are using laravel/symfony with PHP8+ and as a APM, NewRelic is simply a brick on our servers. No error reporting, no tracing, nothing.

That issue is created on March, I mean, like 6 months from now, half a year… Core feature is not working at all, but what I see from boards, you are developing other parts.

It is like, produce a car which drivers can’t drive it, but you are painting other colors and developing other mirror technologies.

Guys, this car is not going somewhere, there is issue on engine?!

If you need help, just tell us what’s wrong and what you need, and we’ll help you, otherwise, tell NewRelic to stop PHP part.

Hi all,

For those who need transaction data ASAP, I just released a Laravel New Relic package which bypasses these unknown transaction issues 🚀

Hopefully this helps anyone here who needs visibility into their data until a proper solution is found. (I haven’t mentioned anywhere in the README that the package only came about because of this issue, as it can be used standalone anyway, so I’ll keep that between us!)

The package is plug-and-play, it should work right out of the box. Most importantly I think it should bypass most of the issues being described here (it did for me anyway 😄).

The package identifies transaction names and manages their lifecycles “manually”, so New Relic doesn’t need to interact with the opcache to detect which methods are being called. Transactions are automatically recorded for:

• HTTP requests
• Artisan commands
• Queue workers
• Scheduled tasks

It filters out long-running processes and internal calls by default, but it’s configurable so you can tailor it to your liking.

I’ve been using this for about a week in production now. Without the package, all my web transactions were unknown (but they worked for CLI, speculatively due to opcache differences). With the package installed, everything is easily-identifiable across the board.

Hope this helps 👍

/cc @iquad @rjp-thijs @TheFrankman @jeffcaulfield-morphsites

@alitopaloglu we are using this middleware to workaround the issue

<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
use Illuminate\Http\Response;

class NewRelicTransactionName
{
    /**
     * Handle an incoming request.
     *
     * @param  Request  $request
     * @param Closure(\Illuminate\Http\Request): (\Illuminate\Http\Response|\Illuminate\Http\RedirectResponse)  $next
     * @return Response|RedirectResponse
     */
    public function handle(Request $request, Closure $next)
    {
        $response = $next($request);

        if (extension_loaded('newrelic')) { // Ensure PHP agent is available
            if ($route = $request->route()) {
                newrelic_name_transaction($route->getAction('controller'));
                // or alternatively:
                // newrelic_name_transaction($route->getName());
            }
        }

        return $response;

    }
}

Come on now @ak-war we pay a lot of money for this service which has been useless for months. No updates at all from your end. We deserve a reply that’s better than “We don’t know, we hope soon”.

Any update on this issue?

PHP 7.4 is nearing end of life in 2 months, and a lot of folks are working on PHP 8.1 upgrades as a result. We’re about to be forced into a version of PHP which will break NewRelic. Please update!

The lack of updates continue 😠

Same issue here with Laravel v9.16.0 and PHP v8.1.6

All transactions are “unknown”

I just noticed a PR was opend about 12 hours ago: https://github.com/newrelic/newrelic-php-agent/pull/419

@ZNeumann this is disappointing response. “It’s in the roadmap” offers no assurances to anyone, that could mean anything or just not be true.

To lend my voice to everyone else, NewRelic was functioning on Laravel 8.x prior to the PHP 8.1 upgrade. I am sure that you will have thousands of customers using Laravel who are soon to find out that when they upgrade to PHP8.1 the product will be broken. At this point if not supported you will have an exodus on your hands.

Support for PHP8.1 was around 3 months late and and support for Laravel8 is a year too late (laravel is actually on v9 now)

This isn’t a threat but just a reality, I (and i’m sure everyone else here) simply have to have monitoring. As such I will have to look at using something else. NewRelic should also know that competitors have seen this post and I have had two different APM services reach out to me.

At the end of the day Laravel is built on Symphony, its routing is simple. Perhaps just get one of your people to have a quick look into it. Creating a VM with laravel 8.x installed with NewRelic should take your capable selves about an hour and should be part of your testing process anyway. For myself, it’s probably too late, i need to get monitoring in place and i need to know it’s going to be updated in reliable timeframes but you might be able to maintain your laravel customers if you act swiftly.

P.s Disabling Opcache should not even be up for discussion.

Awesome team! I’m able to confirm that this update fix the problem! No other issues for now.

NOTE: For other doing the update right now, we have 4 apps running the APM Agent. In 3 of them, make an apt upgrade was enough, but one of the servers have requested the license key again to update, get your licence key before start the process. Regards!

@ak-war perfect! is tomorrow still the release day ?

If this service doesn’t work, then why do we pay for it?

Tbh, we are planning on moving to Sentry in the next few days (our release_candidate is already using it)

@trothwell-carsguide We’ve been using the following in our app\Exceptions\Handler.php to handle error reporting:

public function register()
{
    $this->reportable(function (Throwable $e) {
        if (function_exists('newrelic_notice_error')) {
            newrelic_notice_error($e);
        }
    });
}

@raulamoretti @quickbutik-ronflorax We’re using this plugin https://github.com/newrelic/newrelic-php-agent/issues/387#issuecomment-1133747566 and it is working fine for us so far.

@thomasdavis Yes, we were able to get errors reporting again with this function in the Handler:

public function report(Throwable $exception)
{
    if ($this->shouldReport($exception) && App::environment('production')) {
        newrelic_notice_error($exception);
    }
    
    parent::report($exception);
}

I haven’t tested the following yet, but it is a bit cleaner and more in line with Laravels documentation for exception reporting:

public function register()
{
    $this->reportable(function (Throwable $e) {
        newrelic_notice_error($e);
    });
}

@crishoj

So we’ve upgraded to v9.21.0.311 and PHP 8.1.6. Saldy we see no improvement and still almost all transactions are marked as Unknown.

Just to chime in here (as I’ve been chasing my tail over weird opcache bugs for the past 4 days) — it appears there’s a known bug with PHP’s opcache, which Laravel and Monolog seem to be triggering (though other frameworks and packages have also reported it too).

The bug is supposed to be fixed in PHP 8.1.6, which I believe is due to be released on May 12th. More info here: https://github.com/php/php-src/issues/8164

I mention this because of @zsistla’s comment the other week:

If opcache is required, the recommendation would be to use PHP 8.0 with Laravel 8+ since with Laravel 8+ and PHP 8.1 we observed frequent apache segfaults and frequent cases where the opcache simply stopped reporting information about classes.

At least some portion of the issue appears to be something to bring up with the laravel and/or PHP and/or apache teams as we observed the issues even with the PHP Agent disabled.

I actually ended up installing New Relic for the first time to try and debug what was going on (so confirmed, it’s happening with or without the agent disabled). My users were getting blank HTTP 500 pages, and Laravel was logging absolutely nothing because the opcache is refusing to invalidate a faulty compilation. Reloading PHP-FPM fixes it ~30% of the time (in my case, anyway), rebuilding the opcache ~50%, but it’s totally random and very unpredictable.

Hopefully the PHP 8.1.6 release fixes this 🤞

Hi and thanks for your continued engagement.

High level summary

We did identify a problem on the New Relic side and were able to roll that in to the most recent release 9.21.0.311. However, there are other issues occurring that we continue to investigate.

If opcache is required, the recommendation would be to use PHP 8.0 with Laravel 8+ since with Laravel 8+ and PHP 8.1 we observed frequent apache segfaults and frequent cases where the opcache simply stopped reporting information about classes.

At least some portion of the issue appears to be something to bring up with the laravel and/or PHP and/or apache teams as we observed the issues even with the PHP Agent disabled.

While there was a fix for an error on our side that was rolled into the latest release, there does seem to be some instability with Laravel and another process which is not related to NR as the following events were observed with NR disabled. If the underlying zend opcache is no longer to properly report on classes, the agent will be unable to function correctly. The remaining “unknown” issues were because the agent queried the opcache for a class and was told the class did not exist. Once in this state, for proper naming, the opcache needs to be reset.

Again, we’ve observed these issues with the NR agent disabled.

Details

1. apache segfaults: The segfaults leave the opcache in a state that the php agent was unable to query about classes anymore and without the ability to query for classes in the opcache, we are unable to name the transaction properly (although the URI shows correctly and all children segments are properly labeled) and hence it shows as unknown. Once in this state, the opcache needs to be reset to provide the agent with the proper classes for naming again. This instability was also observed with the agent disabled. We tried with php 8.1.1, 8.1.2, 8.1.3, and 8.1.4 and while all exhibited segfaults, the frequency of segfaults decreased as the PHP version increased.

Sometimes only one apache process will segfault at a time, sometimes all of them will:

laravel_web    | [Mon Apr 18 21:11:26.760412 2022] [core:notice] [pid 1] AH00052: child pid 33 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:29.764021 2022] [core:notice] [pid 1] AH00052: child pid 55 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:31.766644 2022] [core:notice] [pid 1] AH00052: child pid 56 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:34.772210 2022] [core:notice] [pid 1] AH00052: child pid 27 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:36.775568 2022] [core:notice] [pid 1] AH00052: child pid 28 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:38.778298 2022] [core:notice] [pid 1] AH00052: child pid 29 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:45.793312 2022] [core:notice] [pid 1] AH00052: child pid 63 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:47.798363 2022] [core:notice] [pid 1] AH00052: child pid 64 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:11:50.768259 2022] [core:notice] [pid 1] AH00052: child pid 65 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:12:06.805142 2022] [core:notice] [pid 1] AH00052: child pid 66 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:12:08.808704 2022] [core:notice] [pid 1] AH00052: child pid 67 exit signal Segmentation fault (11)
laravel_web    | [Mon Apr 18 21:12:10.810495 2022] [core:notice] [pid 1] AH00052: child pid 68 exit signal Segmentation fault (11)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.3 216704 31472 ?        Ss   21:08   0:00 apache2 -DFOREGROUND
root        25  0.0  0.1 1157788 10676 ?       Sl   21:08   0:00 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/
root        37  0.8  0.3 1382056 28448 ?       Sl   21:08   0:01 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/
root        47  0.0  0.0   2420   528 pts/0    Ss   21:08   0:00 /bin/sh
www-data    64  3.0  0.3 222088 28928 ?        R    21:11   0:00 apache2 -DFOREGROUND
www-data    65  0.0  0.0 216744  8356 ?        S    21:11   0:00 apache2 -DFOREGROUND
www-data    66  0.0  0.0 216744  8356 ?        S    21:11   0:00 apache2 -DFOREGROUND
www-data    67  0.0  0.0 216744  8356 ?        S    21:11   0:00 apache2 -DFOREGROUND
www-data    68  0.0  0.0 216744  8356 ?        S    21:11   0:00 apache2 -DFOREGROUND
www-data    69  0.0  0.0 216744  8356 ?        S    21:11   0:00 apache2 -DFOREGROUND
root        70  0.0  0.0   6700  2988 pts/0    R+   21:11   0:00 ps aux

1. There is another process (laravel? apache?) also dealing with the opcache in the same areas we are. This is apparent in the zend_get_resource_handle handle the agent gets. For all php version 8 and under, we get the same handle of 0 meaning no other entities requested a handle before us. For PHP 8.1, we get a higher numbered handle indicating the presence of another entity or profiler manipulating the opache in the same way the agent does. The agent and this entity appear to be clashing.

If another entity (including xdebug, APMs, etc.) is modifying the framework’s default dispatching method, this causes incompatibilities and New Relic may not be able to detect or hook into the framework’s dispatcher, and it will not be able to provide a meaningful transaction naming structure.

1. Laravel also exhibited instability under these conditions:

Fatal error: Uncaught Illuminate\Contracts\Container\BindingResolutionException: Target [Illuminate\Contracts\Http\Kernel] is not instantiable. in /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:1089 Stack trace: #0 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(886): Illuminate\Container\Container->notInstantiable('Illuminate\\Cont...') #1 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(758): Illuminate\Container\Container->build('Illuminate\\Cont...') #2 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(851): Illuminate\Container\Container->resolve('Illuminate\\Cont...', Array, true) #3 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(694): Illuminate\Foundation\Application->resolve('Illuminate\\Cont...', Array) #4 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(836): Illuminate\Container\Container->make('Illuminate\\Cont...', Array) #5 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(1419): Illuminate\Foundation\Application->make('Illuminate\\Cont...') #6 /var/www/app/bootstrap/app.php(15): Illuminate\Container\Container->offsetGet('Illuminate\\Cont...') #7 /var/www/app/public/index.php(47): require_once('/var/www/app/bo...') #8 {main} thrown in /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php on line 1089

Possible related issues:

https://github.com/php/php-src/issues/8064
https://bugs.php.net/bug.php?id=81380
https://bugs.php.net/bug.php?id=81678

We will continue to investigate this issue.

Hi all, we’d like to take a moment to thank you all for your civility (and even the brutal honesty) with this understandably frustrating issue.

@rjp-thijs Thanks so much for the logs!!! They were extremely helpful in guiding us to narrow down the debugging strategies.

We believe we have a fix for this issue which was related to how Laravel 8+ handles their opcache preloading with PHP 8.1. They do preloading with autoload.php and autoload_real.php even if they don’t explicitly set preloading on as an ini setting and they do it slightly differently for PHP 8.1 possibly due to the opcache changes that went into PHP 8.1.

Snapshot using PHP 8.1 w/Laravel 8.83.6:

We’re going to be testing out the fix in the coming week to ensure we covered the edge cases, then hope to get it into our upcoming release.

I have confirmed that we’ve successfully updated the agent using yum info newrelic-php5 and newrelic-daemon -v. However, running Magento 2.4.4-p2, which listed as explicitly compatible with the PHP agent, we are still seeing everything clumped together under index.php in APM Transctions.

Are there any other steps we need to take other than installing the new agent, or is this issue not really fixed?

Thank you everyone for your constructive feedback. A release will be made around November 3 that will be fixing this issue.

Any update here? We will be forced to switch to another product in a few weeks if we don’t get a solution.

Using the middleware from above that contains newrelic_name_transaction worked but our transaction are coming through as

WebTransaction/Custom/OurAccountName\SearchController@search instead of WebTransaction/Action/OurAccountName\SearchController@search

So just be aware of that.

Upgrading to New Relic 10 alone didn’t fix this for us FYI

This middleware works for me in Lumen

<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
use Illuminate\Http\Response;

class NewRelicTransactionName
{
    /**
     * Handle an incoming request.
     *
     * @param  Request  $request
     * @param Closure(\Illuminate\Http\Request): (\Illuminate\Http\Response|\Illuminate\Http\RedirectResponse)  $next
     * @return Response|RedirectResponse
     */
    public function handle(Request $request, Closure $next)
    {
        $response = $next($request);

        if (extension_loaded('newrelic')) { // Ensure PHP agent is available
            newrelic_name_transaction($this->getTransactionName($request));
        }

        return $response;

    }

    /**
     * Builds the transaction name. It will return the assigned controller action first, then the route name before
     * falling back to just "index.php"
     *
     * @param Request $request
     *
     * @return string
     */
    public function getTransactionName(Request $request)
    {
        $route = $request->route();

        if (is_array($route)) {
            // Try the assigned controller action
            if (isset($route[1]) && isset($route[1]['uses'])) {
                return $route[1]['uses'];
            } // Try named routes
            elseif (isset($route[1]) && isset($route[1]['as'])) {
                return $route[1]['as'];
            }
        }

        return 'index.php';
    }
}

@JackWH This issue:

Unfortunately /var/log/newrelic/php_agent.log is filling up rapidly, with a new line logged for every single request:

2022-05-17 21:56:35.851 +0000 (13379 13379) warning: User instrumentation from opcache: opcache status information is not an array

Is addressed by https://github.com/newrelic/newrelic-php-agent/pull/447, which will be included in the next release.

@brettraminGTR Did that solution work?

Has anyone got errors reporting again?

@JackWH Thanks for the link! We’ll try it out with 8.1.6 and see what happens 🤞🤞🤞🤞

Hi All,

I have only just stumbled upon this issue, however it appears that the latest release (9.21) of your PHP agent breaks PHP 7.4 reporting of transactions in Laravel 8, when it was working totally fine in version 9.19

We’ve only just discovered this today, and have had to go through and roll back all of our API webservers to previous version of the New Relic agent as every transaction was showing up as unknown.

The one that monitores nginx has the problem with naming almost all transactions as unknown and has opcache enabled. The one that monitors console commands doesn’t name anything unknown.

Exactly what I’m observing as well.

Frankly, New Relic is testing my patience as a paying customer. Might be time to consider other options.

We’re really looking for a sollution on this as NewRelic now basically is unusable and it has been for a while. Can someone maybe try out my statement in https://github.com/newrelic/newrelic-php-agent/issues/392? That this appears to be Opcache related by disabling opcache?

I understand that that’s not something you’re able to do on a production environment, but maybe someone has this issue on a non-prod environment.

This knowledge won’t fix this bug, but maybe help someone with knowledge fixing things 😃

Any update on this?

We are facing the same problem, also there is a related discussion with no answer