mysql2: Segmentation fault in `connect` when password provided on macOS 10.15 running MySQL 5.6

I get the following segmentation fault in Mysql2::Client.new when running on macOS 10.15 (19A602), against MySQL 5.6.43 installed via Homebrew, if and only if a password is provided for the user:

tthomas@Todds-Mac mysql2_test % ruby mysql2-test.rb                    
/Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:90: [BUG] Segmentation fault at 0x00007ffeee651000
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-darwin19]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:                    
     * ~/Library/Logs/DiagnosticReports                                     
     * /Library/Logs/DiagnosticReports                                      
   for more details.                                                        
Don't forget to include the above Crash Report log file in bug reports.     

-- Control frame information -----------------------------------------------
c:0005 p:---- s:0039 e:000038 CFUNC  :connect
c:0004 p:0612 s:0027 e:000026 METHOD /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:90 [FINISH]
c:0003 p:---- s:0013 e:000012 CFUNC  :new
c:0002 p:0025 s:0008 E:0023d0 EVAL   mysql2-test.rb:3 [FINISH]
c:0001 p:0000 s:0003 E:001ee0 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
mysql2-test.rb:3:in `<main>'
mysql2-test.rb:3:in `new'
/Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:90:in `initialize'
/Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:90:in `connect'

-- Machine register context ------------------------------------------------
 rax: 0x00007ffeee64f25d rbx: 0x0000000101be5568 rcx: 0x0000000101be3cc8
 rdx: 0x0000000101be5568 rdi: 0x00007ffeee650afd rsi: 0x00007ffeee651000
 rbp: 0x00007ffeee64f1f0 rsp: 0x00007ffeee64f1f0  r8: 0x00007ff3c0e98600
  r9: 0xffffffff807ff7ff r10: 0x0000000101be5568 r11: 0xfffffffffffffafd
 r12: 0x0000ffff55387f57 r13: 0x00007ff3c1067200 r14: 0x00007ffeee64f760
 r15: 0x00007ffff02347c5 rip: 0x00007fff65148d29 rfl: 0x0000000000010202

-- C level backtrace information -------------------------------------------
0                                       0x00000001017b0e18 rb_vm_bugreport + 136
1                                       0x000000010162a9b6 rb_bug_context + 470
2                                       0x000000010171ef51 sigsegv + 81
3   libsystem_platform.dylib            0x00007fff6514bb1d _sigtramp + 29
4   libsystem_platform.dylib            0x00007fff65148d29 _platform_memmove$VARIANT$Haswell + 41

-- Other runtime information -----------------------------------------------

* Loaded script: mysql2-test.rb

* Loaded features:

    0 enumerator.so
    1 thread.rb
    2 rational.so
    3 complex.so
    4 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/x86_64-darwin19/enc/encdb.bundle
    5 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/x86_64-darwin19/enc/trans/transdb.bundle
    6 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/x86_64-darwin19/rbconfig.rb
    7 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/compatibility.rb
    8 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/defaults.rb
    9 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/deprecate.rb
   10 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/errors.rb
   11 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/version.rb
   12 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/requirement.rb
   13 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/platform.rb
   14 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/basic_specification.rb
   15 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/stub_specification.rb
   16 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/util/list.rb
   17 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/x86_64-darwin19/stringio.bundle
   18 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/rfc2396_parser.rb
   19 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/rfc3986_parser.rb
   20 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/common.rb
   21 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/generic.rb
   22 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/ftp.rb
   23 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/http.rb
   24 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/https.rb
   25 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/ldap.rb
   26 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/ldaps.rb
   27 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri/mailto.rb
   28 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/uri.rb
   29 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/specification.rb
   30 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/exceptions.rb
   31 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/dependency.rb
   32 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_gem.rb
   33 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/monitor.rb
   34 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb
   35 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems.rb
   36 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/path_support.rb
   37 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/version.rb
   38 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/core_ext/name_error.rb
   39 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/levenshtein.rb
   40 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/jaro_winkler.rb
   41 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/spell_checker.rb
   42 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/delegate.rb
   43 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/spell_checkers/name_error_checkers/class_name_checker.rb
   44 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/spell_checkers/name_error_checkers/variable_name_checker.rb
   45 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/spell_checkers/name_error_checkers.rb
   46 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/spell_checkers/method_name_checker.rb
   47 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/spell_checkers/key_error_checker.rb
   48 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/spell_checkers/null_checker.rb
   49 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean/formatters/plain_formatter.rb
   50 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/did_you_mean-1.2.0/lib/did_you_mean.rb
   51 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/bundler_version_finder.rb
   52 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/util.rb
   53 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/x86_64-darwin19/date_core.bundle
   54 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/date.rb
   55 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/x86_64-darwin19/bigdecimal.bundle
   56 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/version.rb
   57 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/error.rb
   58 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/mysql2.bundle
   59 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/result.rb
   60 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb
   61 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/field.rb
   62 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/statement.rb
   63 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/2.5.0/timeout.rb
   64 /Users/tthomas/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2.rb

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

[IMPORTANT]
Don't forget to include the Crash Report log file under
DiagnosticReports directory in bug reports.

zsh: abort      ruby mysql2-test.rb

I initially saw this with mysql2 v. 0.4.10, but it persisted after updating to 0.5.2.

The test program that produced that output was

require 'mysql2'
  
client = Mysql2::Client.new(:host => "localhost", :username => "pwd", :password => "pwd")

results = client.query("SELECT * FROM reviews")
puts results

(I realize the query in that program would have failed because no database was selected, but the segfault occurs on line 3 before the query. I’ve used various forms of that test program, including ones where a :database key-value pair was passed to new, but the segfault always and only occurs when the :password key-value pair is passed.)

MySQL was installed via

brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/f171f1c74/Formula/mysql@5.6.rb

That version of the formula was used in order to get MySQL 5.6.43, instead of 5.6.46, which is what the latest version of the formula installs. Currently mysql2 can’t be built against 5.6.46 due to https://github.com/Homebrew/homebrew-core/issues/45397.

A more interesting C stack trace can be obtained from lldb:

tthomas@Todds-Mac mysql2_test % bundle exec lldb -- ruby mysql2-test.rb                                    
(lldb) target create "ruby"
Current executable set to 'ruby' (x86_64).
(lldb) settings set -- target.run-args  "mysql2-test.rb"
(lldb) run
Process 5673 launched: '/Users/tthomas/.rbenv/versions/2.5.5/bin/ruby' (x86_64)
Process 5673 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7ffeefc00000)
    frame #0: 0x00007fff65148d29 libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 41
libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell:
->  0x7fff65148d29 <+41>: rep    movsb	(%rsi), %es:(%rdi)
    0x7fff65148d2b <+43>: popq   %rbp
    0x7fff65148d2c <+44>: retq   
    0x7fff65148d2d <+45>: cmpq   %rdi, %rsi
Target 0: (ruby) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7ffeefc00000)
  * frame #0: 0x00007fff65148d29 libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 41
    frame #1: 0x0000000102009372 libmysqlclient.18.dylib`write_length_encoded_string4 + 54
    frame #2: 0x000000010200befb libmysqlclient.18.dylib`send_client_reply_packet + 1115

I think this is a problem with MySQL, not the mysql2 gem, but I’m not certain. I wrote a C program that uses the MySQL client lib to try to reproduce the problem outside of the context of the mysql2 gem and the Ruby interpreter.

//
//  main.c
//  mysql_test
//
//  Build like so: `clang $(mysql_config --cflags) $(mysql_config --libs) -L/usr/local/opt/openssl/lib main.c`
//
//  If `mysql_options4` is called, *and* a password is passed to `mysql_real_connect`, a segfault will occur
//  during the call to `mysql_real_connect`.
//
//  For example, if the database has a user named "nopassword", which requires no password, and a user named
//  "password", which requires the password "password", then the program yields the following results:
//
//  ./a.out -unopassword                     # success
//  ./a.out -unopassword --options4          # success
//  ./a.out -upassword -ppassword            # success
//  ./a.out -upassword -ppassword --options4 # segfault
//

#include <mysql.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, const char * argv[]) {
    const char* host = "localhost";
    const char* user = "";
    const char* password = NULL;
    const char * const k_options4_arg_name = "--options4";
    int call_mysql_options4 = 0;

    // I know this is not the most efficient, idiomatic C command-line processing. Sorry, it's been a long time since I wrote C for a living.
    // This is only here so testing with and without a password can be done without recompiling.
    for (int i = 1; i < argc; i++) {
        const char* arg = argv[i];
        if (strlen(arg) > 2 && strncmp("-h", arg, 2) == 0) {
            host = arg + 2;
        } else if (strlen(arg) > 2 && strncmp("-u", arg, 2) == 0) {
            user = arg + 2;
        } else if (strlen(arg) > 2 && strncmp("-p", arg, 2) == 0) {
            password = arg + 2;
        } else if (strncmp(k_options4_arg_name, arg, strlen(k_options4_arg_name)) == 0) {
            call_mysql_options4 = 1;
        } else {
            fprintf(stderr, "Usage: %s -u<username> [-p<password> [default: NULL]] [-h<hostname> [default: localhost]] [%s [default: false]]\n", argv[0], k_options4_arg_name);
            exit(1);
        }
    }
    if (strlen(user) == 0) {
        fprintf(stderr, "must specify -u<username>\n");
        exit(1);
    }

    MYSQL mysql;

    mysql_init(&mysql);

    int err;
    if (call_mysql_options4) {
        // Segfault in `mysql_real_connect` only occurs if `mysql_options4` is called and a password is passed to `mysql_real_connect`.
        err = mysql_options4(&mysql, MYSQL_OPT_CONNECT_ATTR_ADD, "program_name", "/Users/tthomas/src/mysql_test/main.c");
        printf("mysql_options4 returned %d\n", err);
    } else {
        // If `mysql_options` is called instead of `mysql_options4`, then the segfault does not occur, regardless of whether a password is
        // passed to `mysql_real_connect`.
        err = mysql_options(&mysql, MYSQL_READ_DEFAULT_GROUP, "/Users/tthomas/src/mysql_test/main.c");
        printf("mysql_options returned %d\n", err);
    }

    // Setting a connect timeout doesn't seem to have any effect on whether the segfault occurs.
    //int timeout = 20;
    //err = mysql_options(&mysql, MYSQL_OPT_CONNECT_TIMEOUT, &timeout);
    //printf("mysql_options returned %d\n", err);

    // Set the same connect flags as the mysql2 gem does by default.
    unsigned long flags = CLIENT_PROTOCOL_41 | CLIENT_TRANSACTIONS | CLIENT_SECURE_CONNECTION | CLIENT_LONG_FLAG | CLIENT_CONNECT_ATTRS | CLIENT_LONG_PASSWORD | CLIENT_REMEMBER_OPTIONS;
    printf("Using connect flags %lu\n", flags);

    // Segfault occurs when the next line is executed if `mysql_options4` was called previously, and password is passed.
    if (!mysql_real_connect(&mysql, host, user, password, NULL, 0, NULL, flags)) {
        fprintf(stderr, "Failed to connect to database: Error: %s\n", mysql_error(&mysql));
        exit(1);
    }

    // The rest of this program just exercises the connection, if we were able to obtain one.
    unsigned long version = mysql_get_server_version(&mysql);
    printf("MySQL version is %lu\n", version);

    mysql_close(&mysql);

    return 0;
}

When the segfault happens with this C program, the backtrace is the same as with the mysql2 program:

tthomas@Todds-Mac mysql_test % lldb -- a.out -upwd -ppwd --options4
(lldb) target create "a.out"
Current executable set to 'a.out' (x86_64).
(lldb) settings set -- target.run-args  "-upwd" "-ppwd" "--options4"
(lldb) run
Process 10967 launched: '/Users/tthomas/src/mysql_test/a.out' (x86_64)
mysql_options4 returned 0
Using connect flags 2148573701
Process 10967 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7ffeefc00000)
    frame #0: 0x00007fff65148d29 libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 41
libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell:
->  0x7fff65148d29 <+41>: rep    movsb	(%rsi), %es:(%rdi)
    0x7fff65148d2b <+43>: popq   %rbp
    0x7fff65148d2c <+44>: retq   
    0x7fff65148d2d <+45>: cmpq   %rdi, %rsi
Target 0: (a.out) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7ffeefc00000)
  * frame #0: 0x00007fff65148d29 libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 41
    frame #1: 0x000000010010e372 libmysqlclient.18.dylib`write_length_encoded_string4 + 54
    frame #2: 0x0000000100110efb libmysqlclient.18.dylib`send_client_reply_packet + 1115

Given that the call to mysql_options4 is essential for the segfault in the above program, I thought I could work around the problem in mysql2 by omitting its call to mysql_options4 with this change:

diff --git a/ext/mysql2/client.c b/ext/mysql2/client.c
index 2926571..2cd4c45 100644
--- a/ext/mysql2/client.c
+++ b/ext/mysql2/client.c
@@ -429,7 +429,7 @@ static VALUE rb_mysql_connect(VALUE self, VALUE user, VALUE pass, VALUE host, VA
   args.mysql       = wrapper->client;
   args.client_flag = NUM2ULONG(flags);
 
-#ifdef CLIENT_CONNECT_ATTRS
+#ifdef CLIENT_CONNECT_ATTRS_NOPE
   mysql_options(wrapper->client, MYSQL_OPT_CONNECT_ATTR_RESET, 0);
   rb_hash_foreach(conn_attrs, opt_connect_attr_add_i, (VALUE)wrapper);
 #endif

But that didn’t work–I still got the segfault when using a gem built from that code.

I verified all of the above using a fresh install of macOS 10.15 in a VMware Fusion virtual machine, and in a virtual machine that started as a fresh install of macOS 10.14 and then was upgraded to 10.15. In the latter case, I could not reproduce the problem on 10.14, nor could I reproduce it immediately after upgrading to 10.15. I had to do brew reinstall https://raw.githubusercontent.com/Homebrew/homebrew-core/f171f1c74/Formula/mysql@5.6.rb after the upgrade to 10.15 before I could reproduce the problem. That mirrors how I initially encountered this. The mysql2 gem had been working fine after I upgraded my dev machine’s 10.14 installation to 10.15. It was only after a brew upgrade upgraded MySQL from 5.6.43 to 5.6.46 that I saw this problem. I then had to revert to 5.6.43 due to the linking issue mentioned above, but this segfault issue persisted.