monolog: ErrorException: Caught ErrorException (HTTP/1.0 500 Internal Server Error): fwrite(): write of X bytes failed with errno=32 Broken pipe
Monolog version 2
Got these lot of errors when sudden burst traffic hit the service. Even though the request end up success with HTTP 200, unfortunately this make additional at least ~2s latency for every processed requests thus slowing down the service.
Stacktrace:
ErrorException: Caught ErrorException (HTTP/1.0 500 Internal Server Error): fwrite(): write of 111 bytes failed with errno=32 Broken pipe in /var/www/app/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php:178
#0 [internal function]: Laravel\Lumen\Application->Laravel\Lumen\Concerns\{closure}()
#1 /var/www/app/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php(178): fwrite()
#2 /var/www/app/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php(162): Monolog\Handler\StreamHandler->streamWrite()
#3 /var/www/app/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(48): Monolog\Handler\StreamHandler->write()
#4 /var/www/app/vendor/monolog/monolog/src/Monolog/Logger.php(327): Monolog\Handler\AbstractProcessingHandler->handle()
#5 /var/www/app/vendor/monolog/monolog/src/Monolog/Logger.php(565): Monolog\Logger->addRecord()
#6 /var/www/app/vendor/illuminate/log/Logger.php(174): Monolog\Logger->error()
#7 /var/www/app/vendor/illuminate/log/Logger.php(87): Illuminate\Log\Logger->writeLog()
#8 /var/www/app/vendor/illuminate/log/LogManager.php(543): Illuminate\Log\Logger->error()
#9 /var/www/app/vendor/laravel/lumen-framework/src/Exceptions/Handler.php(56): Illuminate\Log\LogManager->error()
#10 /var/www/app/app/Exceptions/Handler.php(51): Laravel\Lumen\Exceptions\Handler->report()
#11 /var/www/app/vendor/laravel/lumen-framework/src/Concerns/RegistersExceptionHandlers.php(104): App\Exceptions\Handler->report()
#12 /var/www/app/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(174): Laravel\Lumen\Application->sendExceptionToHandler()
#13 /var/www/app/vendor/laravel/lumen-framework/src/Concerns/RoutesRequests.php(109): Laravel\Lumen\Application->dispatch()
#14 /var/www/app/public/index.php(28): Laravel\Lumen\Application->run()
#15 {main}
I use stdout and stderr for channels. It’s then collected by filebeat then sent to ES. This is standard EFK stack in kubernetes cluster.
I thought that the disk is full, but I was wrong. The disk is still far from full. Then I suspect two things:
- the log may be too big to handle (2.2 million logs per day) so doing some filters become necessary, or
- there’s a misconfiguration on my config
'stderr' => [
'driver' => 'monolog',
'handler' => StreamHandler::class,
'with' => [
'stream' => 'php://stderr',
],
],
'stdout' => [
'driver' => 'monolog',
'handler' => StreamHandler::class,
'formatter' => Monolog\Formatter\LineFormatter::class,
'formatter_with' => [
'format' => "[%datetime%] ".env('APP_NAME', 'app-name').".%level_name%: %message%\n",
'dateFormat' => 'Y-m-d H:i:s',
],
'with' => [
'stream' => 'php://stdout',
],
],
I was wondering why it says HTTP 500 internal server error? It’s from php error message right? 🤔
I just think this kind of monolog error doesn’t affect the application.
Wondering if anyone face the same issue. Any help is appreciated.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 7
- Comments: 26 (10 by maintainers)
Commits related to this issue
- Close file handle after each write, refs #1862, refs #1860, refs #1634 — committed to Seldaek/monolog by Seldaek 3 years ago
- Close file handle after each write, refs #1862, refs #1860, refs #1634 — committed to Seldaek/monolog by Seldaek 3 years ago
- Close file handle after each write, refs #1862, refs #1860, refs #1634 — committed to Seldaek/monolog by Seldaek 3 years ago
- Close and reopen file handles if a write fails (#1882) * Close file handle after each write, refs #1862, refs #1860, refs #1634 * Modify patch to retry writes once if they fail, then throw if not ... — committed to Seldaek/monolog by Seldaek 2 years ago
Unfortunately not yet @xuandung38.
I upgraded from php 8.0 to 8.2 along with Laravel from 8 to 10, which consecutively upgraded monolog library from 2.9 to 3.5.
And I tried a couple of solutions to fix the issue, but none of them worked:
I ran into this issue this week upon upgrading Monolog from 2.9 to 3.5, which is part of upgrading Lumen from 8 to 10.
This issue only showed up on production, not on local dev environment, nor on staging.
In all environments the log channel is configured to go to stderr
LOG_CHANNEL=stderr, and I managed to confirm the log really is going to stderr, and not to a file.What’s also most frightening is that I got the same error @ehginanjar and @Atantares returned in the endpoint’s response to end users with a 200 http status, and wasn’t logged in the application’s log, apparently because the stderr stream pipe got broken, and fwrite couldn’t write anything.
We stream it via stout and stderr which then goes out to Datadog.
That is likely what can happen, to prove it either way you can try playing around with sysctl and pipe/open files limits:
sysctl -w fs.file-max=xxx
If the library does not recycle the stream handlers at large scale the process can locally run out of the open files limit easily. OS can close the pipe at any time really (even if its local) due to the kernel limits.
What I found in debugging this is that
is_resourcehere https://github.com/Seldaek/monolog/blob/479c936d2c230d8c467bdb3882afab45a6e6b8ad/src/Monolog/Handler/StreamHandler.php#L118 doesn’t detect if the resource STDERR/STDOUT, that was previously opened on this line https://github.com/Seldaek/monolog/blob/479c936d2c230d8c467bdb3882afab45a6e6b8ad/src/Monolog/Handler/StreamHandler.php#L127, is writable. So althoughis_resource($this->stream)is true,fwritestill fails to write to it.I don’t know what causes the pipe to break only in production, or why only with the latest version of illuminate/log, but my suspicion is that in production multiple php-fpm processes are constantly trying to open new handlers, leading the system to overload and break previously opened ones.
So the solution that we went with for now is replacing
StreamHandlerwithErrorLogHandlerin Laravel’s bootstrap, which useserror_log()instead offwrite()I’m not entirely sure but it might be if that’s indeed what caused it… My gut feeling says no tho
Ok, that might be due to this change which would have made it swallow errors in some cases before? https://github.com/Seldaek/monolog/compare/2.9.1...2.9.2#diff-a1ddc5c4ead6773b8670f9be5007cbe0239638aedbcf12166acf075a9b8742a2 see also https://github.com/Seldaek/monolog/pull/1815