swoole-src: Alpine + Swoole + PgSQL on the host = `could not send SSL negotiation packet: Resource temporarily unavailable`
Since the issue is so mysterious, I was able to and made a reproducible. Please try it yourself.
https://github.com/flexchar/pdo-pgsql-issue
Please answer these questions before submitting your issue.
- What did you do? If possible, provide a simple script for reproducing the error.
I attempt to connect to the pgsql running on my host mac machine from inside docker container running Laravel. It would fail with
SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailableerror.
It does work if I would connect using psql cli terminal. It also works if I used pg_connect instead php-pdo.
I cannot test with php83 since the latest swoole pecl upstream has been broken for months now. I am not sure what’s going on.
-
What did you expect to see? It should connect.
-
What did you see instead?
SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable -
What version of Swoole are you using (show your
php --ri swoole)?
Swoole => enabled
Author => Swoole Team <team@swoole.com>
Version => 5.1.1
Built => Nov 28 2023 19:24:57
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.1.4 24 Oct 2023
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
c-ares => 1.27.0
zlib => 1.3
brotli => E16781312/D16781312
mutex_timedlock => enabled
pthread_barrier => enabled
mysqlnd => enabled
async_redis => enabled
coroutine_pgsql => enabled
coroutine_sqlite => enabled
Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
as installed from https://pkgs.alpinelinux.org/packages?name=php82-pecl-swoole&branch=v3.19&repo=community&arch=x86_64&maintainer=
- What is your machine environment used (show your
uname -a&php -v&gcc -v) ?
$ uname -a
Linux 031104a4b74c 6.6.22-linuxkit #1 SMP Fri Mar 29 12:21:27 UTC 2024 aarch64 Linux
$ php -v
PHP 8.2.18 (cli) (built: Apr 11 2024 14:41:12) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.18, Copyright (c) Zend Technologies
$ gcc -v
ash: gcc: not found
It may also be that pdo-pgsql is broken but I am not sure where to report it. I would appreciate the guidance. Thank you. 😃
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 9
- Comments: 47 (14 by maintainers)
Commits related to this issue
- Fix bug #5313 — committed to NathanFreeman/swoole-src by NathanFreeman 2 years ago
- Fix bug #5313 — committed to NathanFreeman/swoole-src by NathanFreeman 2 years ago
- Fix bug #5313 (#5394) — committed to swoole/swoole-src by NathanFreeman 2 years ago
- Fix bug #5313 (#5395) — committed to swoole/swoole-src by NathanFreeman 2 years ago
- Fix bug #5313 — committed to NathanFreeman/swoole-src by NathanFreeman 2 years ago
- Fix bug #5313 — committed to NathanFreeman/swoole-src by NathanFreeman 2 years ago
- [5.1]Fix bug #5313 (#5397) * Fix bug #5313 * optimize code — committed to swoole/swoole-src by NathanFreeman 2 years ago
- [6.0]Fix bug #5313 (#5396) * Fix bug #5313 * optimize code — committed to swoole/swoole-src by NathanFreeman 2 years ago
After some debugging, I found the problem. Swoole sets the PostgreSQL connection to non-blocking mode and does not use functions like select to listen for the connection status in non-coroutine mode. This causes the PQconnectPoll function to return immediately.
It is not possible to reproduce this problem between containers because the latency between them is small. When the PQconnectPoll function returns, the socket has been sent. This problem only occurs when trying to connect to an external server with a delay of >2ms.
Try to rebuild Swoole using the following patches, and the problem is temporarily resolved.
@EliasElimah see https://github.com/swoole/swoole-src/releases/tag/v5.1.4
@flexchar @myxingkong Here, it should be necessary to distinguish between coroutine and non-coroutine situations. Let me make the modification.
@NathanFreeman I understand you cannot reproduce the issue, because it only happens sometimes. The bug doesn’t always occur on every environment and I have absolutely no idea why.
I’ve spend the good part of the day trying to debug this issue, because we run into this issue on 2 of our 4 DTAP environments.
In our environments, everything is running great up until the acceptance and production environments, where the ‘SSL negotiation packet’ error is thrown. Each environment is rolled out using the same IaC codebase. I have confirmed they are all using the same software, tools, versions and OS’s.
The same image - based on your Dockerfile - is being used on all environments during my tests. I’ve also tested it with a Debian based image, which gives the exact same results.
Connecting from within the image to the Postgres database using the
psqlclient works as expected, which rules out potential firewall issues.Disabling the SSL mode - for testing only obviously - results in a
could not send startup packeterror message.All things together; it looks like the PHP process just can’t seem to connect with the Postgres database, even though TCP packets are send to and from the database when these errors occur.
I’m currently out of ideas how to debug this issue and provide you with the information you need. If there is anything that I missed here or anything I can do to help you out, let me know.
@NathanFreeman You are not gonna believe me. Yesterday we upgraded our server from
t4g.largetot4g.2xlargeand the bug suddenly disappeared:The disk image was exactly the same, so it wasn’t something related to software. Maybe the server was under a heavy load and that caused the bug? I am not sure.
I will be keeping you up to date in case the bug appears again.
But thank you a lot for your help, and thanks for making Swoole. Thanks to Swoole our app runs super fast. Thanks!
@jhm-ciberman I’m very sorry; I missed fixing one more thing in the previous update. I will likely submit a new PR tonight or tomorrow.
@matyhtf @NathanFreeman Hi! We have been getting this same error in our production environment after upgrading to
v5.1.4Chart: Number of occurrences of
could not send SSL negotiation packet: Resource temporarily unavailablein our live production server.Last sunday we upgraded from Swoole
v5.1.1to Swoolev5.1.3to fix this issue related to cookies. Then on monday we discovered this bug (SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable) and proceeded to upgrade from swoolev5.1.3to swoolev5.1.4.The number of occurrences of this error went down from 22,000 per day to “only” 200 per day. But still, 200 lost connections to our database is a lot and we are losing important customer data so I am still worried.
We are using an Ubuntu server hosted on AWS, Laravel Forge for deployment and nginx.
Any idea what can it be the cause?
Edit reason: I mistyped the version numbers. The correct version numbers are good now.
Edit2: Swoole is installed from PECL.
Hello @NathanFreeman When will the fix be released? We’ve been waiting for a while.
Hi, I have the same problem with all versions starting from version 5.1, with debian and alpine, with php 8.2 and 8.3. I’m using official swoole docker images. With official docker images of swoole 5.0 everything works correctly.
@flexchar I am more of a VPS server guy. I like docker, but I prefer the simplicity of working with our own machine. Maybe in a future I change my mind, so thanks for the suggestion.
Definitely no. Our server handles 1700 requests per minute. We cannot afford losing (more) customer data.
Server is a AWS EC2 instance. Database is a AWS Postgres RDS instance.
@jhm-ciberman It may be that the PHP pgsql extension is dependent on an older version of the PostgreSQL library. If the PostgreSQL library version is <= 16.1 and SSL connections are enabled, there may be issues.
@jhm-ciberman If your project does not use coroutines, then it doesn’t matter if you disable
--enable-swoole-pgsql. Swoole pgsql is primarily designed for coroutine-based usage with PostgreSQL, and it hooks into some functions of pdo-pgsql.Send to build https://github.com/alpinelinux/aports/commit/ee4d00d63cf5531bc86e444add144345653657e6 https://github.com/alpinelinux/aports/commit/d89ba0092d54d54064a330b0f25b0a4330396b0b
For anyone running into this problem with a Laravel Forge provisioned server, here are the steps I took to disable Swoole. Forge, by default, installs the Swoole extension so users can easily run Laravel Octane, but its there whether or not you decide to use it.
Then comment out the extension line like this
Restart PHP
Obviously if not using 8.3, update the command to the particular version, but this fixed the
could not send SSL negotiation packeterror.Not sure if this is of help, otherwise just another +1 here.
We have the same with a Laravel application which we try to migrate from OpenSwoole to Swoole. Our phpinfo output looks similar to the mentions above.
Our image is built
FROM alpine:3.20, packages are installed as needed viaapk add php82-XXX, swoole viaapk add php82-pecl-swoole. That’s essentially the way to reproduce it.On start a PostgreSQL db migration on an AWS RDS instance via
php artisan migrateis necessary for our app. While OpenSwoole succeeds, Swoole fails at this stage withWe hunt this problem for 3 months now and cannot find anything, so we’re stuck with OpenSwoole currently.
Maybe you have an idea how we can debug this further from here in our app and provide you with infos?