mongo-php-driver: extension started causing segfaults

Bug Report

I have 3 servers running php with the mongo extensions installed. On Feb 13, all of them encountered the same error at the same time. The error being a segfault coming from libcrypto.so.1.0.2k.

The odd part(aside from all the servers experiencing the same thing at the same time) is that the code i’m running is not even using the mongo extension. We use JWT tokens for auth. This crash seems to come up only on api requests that are interacting with a crypto algorithm, to either encode or decode a JWT token. I disabled the mongo extension, restarted php-fpm, and things started working normally again. Thankfully i’m just using mongo to store xhprof profiles on demand but i have no idea what changed on all 3 of my servers at the same time to cause this.

In my debugging, i wrote a simple test script that generates a JWT token using the same library. i was able to run this script fine from the command line with the mongo extension enabled. It appears a magic combination of the following causes the error:

• php-fpm handling the request
• mongo extension being enabled
• php interacting with a crypto lib

I was able to make requests against my public api endpoints(that don’t require any crypto) with the mongo extension enabled and those responded fine.

I develop in docker and i’m unable to reproduce the issue in my development container however i can still reproduce it in my actual servers.

Environment

• Centos 7
• PHP v7.33.3
• mongodb extension 1.10.0(i’ve also tried updating to v1.12.0)
• OpenSSL 1.0.2k-fips 26 Jan 2017

$ php -i | grep -E 'mongodb|libmongoc|libbson'
/etc/php.d/mongodb.ini,
mongodb
libbson bundled version => 1.20.0
libmongoc bundled version => 1.20.0
libmongoc SSL => enabled
libmongoc SSL library => OpenSSL
libmongoc crypto => enabled
libmongoc crypto library => libcrypto
libmongoc crypto system profile => disabled
libmongoc SASL => disabled
libmongoc ICU => disabled
libmongoc compression => enabled
libmongoc compression snappy => disabled
libmongoc compression zlib => enabled
libmongoc compression zstd => disabled
libmongocrypt bundled version => 1.3.0
libmongocrypt crypto => enabled
libmongocrypt crypto library => libcrypto
mongodb.debug => /srv/mongodebug => /srv/mongodebug
mongodb.mock_service_id => Off => Off

Expected and Actual Behavior

Debug Log

[root@server mongodebug]# cat PHONGO-1644934589G9Hllq
[2022-02-15T14:16:29.910968+00:00]     mongoc: TRACE   > ENTRY: _mongoc_linux_distro_scanner_get_distro():389
[2022-02-15T14:16:29.911028+00:00]     mongoc: TRACE   > ENTRY: _mongoc_linux_distro_scanner_read_key_value_file():154
[2022-02-15T14:16:29.911331+00:00]     mongoc: TRACE   > ENTRY: _process_line():93
[2022-02-15T14:16:29.911338+00:00]     mongoc: TRACE   > TRACE: _process_line():121 Found name: CentOS Linux
[2022-02-15T14:16:29.911341+00:00]     mongoc: TRACE   >  EXIT: _process_line():128
[2022-02-15T14:16:29.911344+00:00]     mongoc: TRACE   > ENTRY: _process_line():93
[2022-02-15T14:16:29.911346+00:00]     mongoc: TRACE   >  EXIT: _process_line():128
[2022-02-15T14:16:29.911349+00:00]     mongoc: TRACE   > ENTRY: _process_line():93
[2022-02-15T14:16:29.911352+00:00]     mongoc: TRACE   >  EXIT: _process_line():128
[2022-02-15T14:16:29.911354+00:00]     mongoc: TRACE   > ENTRY: _process_line():93
[2022-02-15T14:16:29.911357+00:00]     mongoc: TRACE   >  EXIT: _process_line():128
[2022-02-15T14:16:29.911360+00:00]     mongoc: TRACE   > ENTRY: _process_line():93
[2022-02-15T14:16:29.911363+00:00]     mongoc: TRACE   > TRACE: _process_line():125 Found version: 7
[2022-02-15T14:16:29.911366+00:00]     mongoc: TRACE   >  EXIT: _process_line():128
[2022-02-15T14:16:29.911381+00:00]     mongoc: TRACE   >  EXIT: _mongoc_linux_distro_scanner_read_key_value_file():205
[2022-02-15T14:16:29.911386+00:00]     mongoc: TRACE   >  EXIT: _mongoc_linux_distro_scanner_get_distro():398