swoole-src: Alpine + Swoole + PgSQL on the host = `could not send SSL negotiation packet: Resource temporarily unavailable`

Since the issue is so mysterious, I was able to and made a reproducible. Please try it yourself.

https://github.com/flexchar/pdo-pgsql-issue


Please answer these questions before submitting your issue.

  1. What did you do? If possible, provide a simple script for reproducing the error. I attempt to connect to the pgsql running on my host mac machine from inside docker container running Laravel. It would fail with SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable error.

It does work if I would connect using psql cli terminal. It also works if I used pg_connect instead php-pdo.

I cannot test with php83 since the latest swoole pecl upstream has been broken for months now. I am not sure what’s going on.

  1. What did you expect to see? It should connect.

  2. What did you see instead? SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable

  3. What version of Swoole are you using (show your php --ri swoole)?

Swoole => enabled
Author => Swoole Team <team@swoole.com>
Version => 5.1.1
Built => Nov 28 2023 19:24:57
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.1.4 24 Oct 2023
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
c-ares => 1.27.0
zlib => 1.3
brotli => E16781312/D16781312
mutex_timedlock => enabled
pthread_barrier => enabled
mysqlnd => enabled
async_redis => enabled
coroutine_pgsql => enabled
coroutine_sqlite => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

as installed from https://pkgs.alpinelinux.org/packages?name=php82-pecl-swoole&branch=v3.19&repo=community&arch=x86_64&maintainer=

  1. What is your machine environment used (show your uname -a & php -v & gcc -v) ?
$ uname -a
Linux 031104a4b74c 6.6.22-linuxkit #1 SMP Fri Mar 29 12:21:27 UTC 2024 aarch64 Linux

$ php -v
PHP 8.2.18 (cli) (built: Apr 11 2024 14:41:12) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.18, Copyright (c) Zend Technologies

$ gcc -v
ash: gcc: not found

It may also be that pdo-pgsql is broken but I am not sure where to report it. I would appreciate the guidance. Thank you. 😃

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 9
  • Comments: 47 (14 by maintainers)

Commits related to this issue

Most upvoted comments

After some debugging, I found the problem. Swoole sets the PostgreSQL connection to non-blocking mode and does not use functions like select to listen for the connection status in non-coroutine mode. This causes the PQconnectPoll function to return immediately.

It is not possible to reproduce this problem between containers because the latency between them is small. When the PQconnectPoll function returns, the socket has been sent. This problem only occurs when trying to connect to an external server with a delay of >2ms.

Try to rebuild Swoole using the following patches, and the problem is temporarily resolved.

diff --git a/ext-src/swoole_pgsql.cc b/ext-src/swoole_pgsql.cc
index 7c8ee1649..fade6827f 100644
--- a/ext-src/swoole_pgsql.cc
+++ b/ext-src/swoole_pgsql.cc
@@ -86,7 +86,28 @@ PGconn *swoole_pgsql_connectdb(const char *conninfo) {

     PQsetnonblocking(conn, 1);

+    fd_set readfds, writefds;
+    timeval timeout = {0, 10000};
+    int retval = 0;
+    if (swoole_pgsql_blocking) {
+        FD_ZERO(&readfds);
+        FD_ZERO(&writefds);
+        FD_SET(fd, &readfds);
+        FD_SET(fd, &writefds);
+    }
+
     SW_LOOP {
+        if (swoole_pgsql_blocking) {
+            retval = select(fd + 1, &readfds, &writefds, nullptr, &timeout);
+            if (retval == -1) {
+                if (errno == EINTR) {
+                    continue;
+                }
+                return nullptr;
+            } else if (retval == 0) {
+                continue;
+            }
+        }
         int r = PQconnectPoll(conn);
         if (r == PGRES_POLLING_OK || r == PGRES_POLLING_FAILED) {
             break;

@flexchar @myxingkong Here, it should be necessary to distinguish between coroutine and non-coroutine situations. Let me make the modification.

@NathanFreeman I understand you cannot reproduce the issue, because it only happens sometimes. The bug doesn’t always occur on every environment and I have absolutely no idea why.

I’ve spend the good part of the day trying to debug this issue, because we run into this issue on 2 of our 4 DTAP environments.

In our environments, everything is running great up until the acceptance and production environments, where the ‘SSL negotiation packet’ error is thrown. Each environment is rolled out using the same IaC codebase. I have confirmed they are all using the same software, tools, versions and OS’s.

The same image - based on your Dockerfile - is being used on all environments during my tests. I’ve also tested it with a Debian based image, which gives the exact same results.

Connecting from within the image to the Postgres database using the psql client works as expected, which rules out potential firewall issues.

Disabling the SSL mode - for testing only obviously - results in a could not send startup packet error message.

All things together; it looks like the PHP process just can’t seem to connect with the Postgres database, even though TCP packets are send to and from the database when these errors occur.

I’m currently out of ideas how to debug this issue and provide you with the information you need. If there is anything that I missed here or anything I can do to help you out, let me know.

@NathanFreeman You are not gonna believe me. Yesterday we upgraded our server from t4g.large to t4g.2xlarge and the bug suddenly disappeared:

image

The disk image was exactly the same, so it wasn’t something related to software. Maybe the server was under a heavy load and that caused the bug? I am not sure.

I will be keeping you up to date in case the bug appears again.

But thank you a lot for your help, and thanks for making Swoole. Thanks to Swoole our app runs super fast. Thanks!

@jhm-ciberman I’m very sorry; I missed fixing one more thing in the previous update. I will likely submit a new PR tonight or tomorrow.

@matyhtf @NathanFreeman Hi! We have been getting this same error in our production environment after upgrading to v5.1.4

Chart: Number of occurrences of could not send SSL negotiation packet: Resource temporarily unavailable in our live production server. image

Last sunday we upgraded from Swoole v5.1.1 to Swoole v5.1.3 to fix this issue related to cookies. Then on monday we discovered this bug (SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable) and proceeded to upgrade from swoole v5.1.3 to swoole v5.1.4.

The number of occurrences of this error went down from 22,000 per day to “only” 200 per day. But still, 200 lost connections to our database is a lot and we are losing important customer data so I am still worried.

We are using an Ubuntu server hosted on AWS, Laravel Forge for deployment and nginx.

Any idea what can it be the cause?

Edit reason: I mistyped the version numbers. The correct version numbers are good now.

Edit2: Swoole is installed from PECL.

Hello @NathanFreeman When will the fix be released? We’ve been waiting for a while.

Hi, I have the same problem with all versions starting from version 5.1, with debian and alpine, with php 8.2 and 8.3. I’m using official swoole docker images. With official docker images of swoole 5.0 everything works correctly.

@jhm-ciberman I could totally recommend hosting on your own server, with docker & cloudflare tunnels.

@flexchar I am more of a VPS server guy. I like docker, but I prefer the simplicity of working with our own machine. Maybe in a future I change my mind, so thanks for the suggestion.

That being said, if you’re feeling lucky, you could also try Swoole 6.0.0. beta. Don’t run it if you risk loosing revenue but do try. I tried and found no issues whatsoever, even though it is a beta version.

Definitely no. Our server handles 1700 requests per minute. We cannot afford losing (more) customer data.

Also, where is your database hosted at?

Server is a AWS EC2 instance. Database is a AWS Postgres RDS instance.

@jhm-ciberman It may be that the PHP pgsql extension is dependent on an older version of the PostgreSQL library. If the PostgreSQL library version is <= 16.1 and SSL connections are enabled, there may be issues.

@jhm-ciberman If your project does not use coroutines, then it doesn’t matter if you disable --enable-swoole-pgsql. Swoole pgsql is primarily designed for coroutine-based usage with PostgreSQL, and it hooks into some functions of pdo-pgsql.

For anyone running into this problem with a Laravel Forge provisioned server, here are the steps I took to disable Swoole. Forge, by default, installs the Swoole extension so users can easily run Laravel Octane, but its there whether or not you decide to use it.

vi /etc/php/8.3/cli/conf.d/25-swoole.ini

Then comment out the extension line like this

;extension=swoole.so

Restart PHP

sudo service php8.3-fpm restart

Obviously if not using 8.3, update the command to the particular version, but this fixed the could not send SSL negotiation packet error.

Not sure if this is of help, otherwise just another +1 here.

We have the same with a Laravel application which we try to migrate from OpenSwoole to Swoole. Our phpinfo output looks similar to the mentions above.

Our image is built FROM alpine:3.20, packages are installed as needed via apk add php82-XXX, swoole via apk add php82-pecl-swoole. That’s essentially the way to reproduce it.

On start a PostgreSQL db migration on an AWS RDS instance via php artisan migrate is necessary for our app. While OpenSwoole succeeds, Swoole fails at this stage with

   Illuminate\Database\QueryException

  SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable (Connection: pgsql, SQL: select * from information_schema.tables where table_catalog = <dbname> and table_schema = public and table_name = migrations and table_type = 'BASE TABLE')

  at vendor/laravel/framework/src/Illuminate/Database/Connection.php:829
    825▕                     $this->getName(), $query, $this->prepareBindings($bindings), $e
    826▕                 );
    827▕             }
    828▕
  ➜ 829▕             throw new QueryException(
    830▕                 $this->getName(), $query, $this->prepareBindings($bindings), $e
    831▕             );
    832▕         }
    833▕     }

      +35 vendor frames

  36  artisan:35
      Illuminate\Foundation\Console\Kernel::handle()

We hunt this problem for 3 months now and cannot find anything, so we’re stuck with OpenSwoole currently.

Maybe you have an idea how we can debug this further from here in our app and provide you with infos?