openssl: Rare and random crash in bn_sqr8x_internal

Since few months we are getting some crash dumps in our Instant Messenger application (GaduGadu - www.gg.pl). It happens rarely - average like once per 2 weeks - so it is not so big problem and we tried to wait it out hoping for fix in a new version, but there isn’t any. We have really huge traffic (50 logins per second, 100 000 simultaneous connections) and it happens so rarely, that it looks like some race condition or strange data, which can randomly triggers it. It is hard to say what change caused begin of these crashes, because we changed few things at once (like: OpenSSL version 1.0 -> 1.1.1, new ciphers and support for TLS 1.2, our new certificate) and sometimes interval between crashes is one month or longer, so it is difficult to debug and it is in production environment, so we can’t afford to undo all changes and wait for months. We are using OpenSSL for 10 years or more and never had such problems. It started +/- about 10 months ago, and looking at all what I have written, I’m sure that it isn’t a problem in our code.

So now some details about our environment. We are using manually built Debian package in version 1.1.1h (built with symbols and full debug information for debugging this issue, but crash happens on earlier versions of 1.1.1 - we started using 1.1.1 from 1.1.1d) and our application run on Jessie Debian distribution. Kernel version: Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u1 (2016-10-10) Cipher string for old versions of our application: ECDHE;AES;!EXP;!eNULL;!aNULL Cipher string for new versions of our application: ECDHE;ECDH;AES;ECDSA;DHE;AESGCM;CHACHA20;!eNULL;!aNULL;!SHA1;!SHA256;!RSA It looks like crash is occuring on both cipher strings.

And here is stacktrace from dump:

#0  0x00007f952c754067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f952c755448 in __GI_abort () at abort.c:89
#2  0x00007f95309caea2 in signal_core (signum=<optimized out>, si=<optimized out>, arg=0x6) at /var/SVN/server-side/engine/branches/stable/c/linux/signal.c:144
#3  <signal handler called>
#4  bn_sqr8x_internal () at crypto/bn/x86_64-mont5.s:1762
#5  0x00007f952e7ef579 in bn_power5 () at crypto/bn/x86_64-mont5.s:1177
#6  0x00007f952e7d6a35 in BN_mod_exp_mont_consttime (rr=0x7f94f6244878, a=0x7f94f6244878, p=0x28b1010, m=0x28b0eb0, ctx=0x7f94f7ebda30, in_mont=0x7f94e8014af0) at crypto/bn/bn_exp.c:1007
#7  0x00007f952e8fb576 in rsa_ossl_mod_exp (r0=0x7f94f6244848, I=0x7f94f6244830, rsa=0x28b0310, ctx=0x7f94f7ebda30) at crypto/rsa/rsa_ossl.c:677
#8  0x00007f952e8fa594 in rsa_ossl_private_encrypt (flen=51, from=0x7f94f5a322a0 "010\r\006\t`\206H\001e\003\004\002\001\005", 
    to=0x7f94f7f8719d "\003U\004\n\023\031Unizeto Technologies S.A.1'0%\006\003U\004\v\023\036Certum Certification Authority1)0'\006\003U\004\003\023 Certum Domain Validation CA SHA20\036\027\r200930000000Z\027\r210930000000Z0\031\061\027\060\025\006\003U\004\003\f\016*.gadu-gadu.pl0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", rsa=0x28b0310, padding=1) at crypto/rsa/rsa_ossl.c:316
#9  0x00007f952e8f616b in RSA_private_encrypt (flen=51, from=0x7f94f5a322a0 "010\r\006\t`\206H\001e\003\004\002\001\005", 
    to=0x7f94f7f8719d "\003U\004\n\023\031Unizeto Technologies S.A.1'0%\006\003U\004\v\023\036Certum Certification Authority1)0'\006\003U\004\003\023 Certum Domain Validation CA SHA20\036\027\r200930000000Z\027\r210930000000Z0\031\061\027\060\025\006\003U\004\003\f\016*.gadu-gadu.pl0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", rsa=0x28b0310, padding=1) at crypto/rsa/rsa_crpt.c:36
#10 0x00007f952e8ff9ae in RSA_sign (type=672, m=0x7f94c27f24b0 "\353\314\345Y\206\276\252R'\371\216w\307Y\276\236\023E\252ὧ\243\253\064\220meܙD3 %\177\302\224\177", m_len=32, 
    sigret=0x7f94f7f8719d "\003U\004\n\023\031Unizeto Technologies S.A.1'0%\006\003U\004\v\023\036Certum Certification Authority1)0'\006\003U\004\003\023 Certum Domain Validation CA SHA20\036\027\r200930000000Z\027\r210930000000Z0\031\061\027\060\025\006\003U\004\003\f\016*.gadu-gadu.pl0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", siglen=0x7f94c27f2400, rsa=0x28b0310) at crypto/rsa/rsa_sign.c:103
#11 0x00007f952e8fcfcf in pkey_rsa_sign (ctx=0x7f94f553da50, 
    sig=0x7f94f7f8719d "\003U\004\n\023\031Unizeto Technologies S.A.1'0%\006\003U\004\v\023\036Certum Certification Authority1)0'\006\003U\004\003\023 Certum Domain Validation CA SHA20\036\027\r200930000000Z\027\r210930000000Z0\031\061\027\060\025\006\003U\004\003\f\016*.gadu-gadu.pl0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", siglen=0x7f94c27f2588, tbs=0x7f94c27f24b0 "\353\314\345Y\206\276\252R'\371\216w\307Y\276\236\023E\252ὧ\243\253\064\220meܙD3 %\177\302\224\177", tbslen=32) at crypto/rsa/rsa_pmeth.c:163
#12 0x00007f952e8af5e6 in EVP_PKEY_sign (ctx=0x7f94f553da50, 
    sig=0x7f94f7f8719d "\003U\004\n\023\031Unizeto Technologies S.A.1'0%\006\003U\004\v\023\036Certum Certification Authority1)0'\006\003U\004\003\023 Certum Domain Validation CA SHA20\036\027\r200930000000Z\027\r210930000000Z0\031\061\027\060\025\006\003U\004\003\f\016*.gadu-gadu.pl0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", siglen=0x7f94c27f2588, tbs=0x7f94c27f24b0 "\353\314\345Y\206\276\252R'\371\216w\307Y\276\236\023E\252ὧ\243\253\064\220meܙD3 %\177\302\224\177", tbslen=32) at crypto/evp/pmeth_fn.c:66
#13 0x00007f952e8ab4d8 in EVP_DigestSignFinal (ctx=0x7f9465d76270, 
    sigret=0x7f94f7f8719d "\003U\004\n\023\031Unizeto Technologies S.A.1'0%\006\003U\004\v\023\036Certum Certification Authority1)0'\006\003U\004\003\023 Certum Domain Validation CA SHA20\036\027\r200930000000Z\027\r210930000000Z0\031\061\027\060\025\006\003U\004\003\f\016*.gadu-gadu.pl0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", siglen=0x7f94c27f2588) at crypto/evp/m_sigver.c:148
#14 0x00007f952e8ab605 in EVP_DigestSign (ctx=0x7f9465d76270, 
    sigret=0x7f94f7f8719d "\003U\004\n\023\031Unizeto Technologies S.A.1'0%\006\003U\004\v\023\036Certum Certification Authority1)0'\006\003U\004\003\023 Certum Domain Validation CA SHA20\036\027\r200930000000Z\027\r210930000000Z0\031\061\027\060\025\006\003U\004\003\f\016*.gadu-gadu.pl0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", siglen=0x7f94c27f2588, 
    tbs=0x7f9464bb3a00 "\a\003\017\035\062\060\346\066\235\353,33i\002ɤ\251\245\327\273\fСO*r4\306\327\217ŢxL2y\034\273\230\220R\203۾\035\231\376E\276\360\236\243\067\303KDOWNGRD\001\003", tbslen=133) at crypto/evp/m_sigver.c:170
#15 0x00007f952ecb60db in tls_construct_server_key_exchange (s=0x7f94ec75e090, pkt=0x7f94c27f26c0) at ssl/statem/statem_srvr.c:2825
#16 0x00007f952ec9f57b in write_state_machine (s=0x7f94ec75e090) at ssl/statem/statem.c:843
#17 0x00007f952ec9ea26 in state_machine (s=0x7f94ec75e090, server=1) at ssl/statem/statem.c:443
#18 0x00007f952ec9e50a in ossl_statem_accept (s=0x7f94ec75e090) at ssl/statem/statem.c:255
#19 0x00007f952ec66dd6 in ssl3_read_bytes (s=0x7f94ec75e090, type=23, recvd_type=0x0, buf=0x7f94c27f2a00 "", len=4096, peek=0, readbytes=0x7f94c27f29e0) at ssl/record/rec_layer_s3.c:1278
#20 0x00007f952ec735db in ssl3_read_internal (s=0x7f94ec75e090, buf=0x7f94c27f2a00, len=4096, peek=0, readbytes=0x7f94c27f29e0) at ssl/s3_lib.c:4473
#21 0x00007f952ec736b2 in ssl3_read (s=0x7f94ec75e090, buf=0x7f94c27f2a00, len=4096, readbytes=0x7f94c27f29e0) at ssl/s3_lib.c:4497
#22 0x00007f952ec81a15 in ssl_read_internal (s=0x7f94ec75e090, buf=0x7f94c27f2a00, num=4096, readbytes=0x7f94c27f29e0) at ssl/ssl_lib.c:1763
#23 0x00007f952ec81a72 in SSL_read (s=0x7f94ec75e090, buf=0x7f94c27f2a00, num=4096) at ssl/ssl_lib.c:1777
#24 0x00007f95309ce3d0 in ssl_handler_read (socket=0x7f947769cac0, current=<optimized out>) at /var/SVN/server-side/engine/branches/stable/c/cclib/ssl.c:197
#25 0x00007f95309c1cb6 in handler_call_stack_param (socket=0x7f947769cac0, aspect=<optimized out>, param=0x0) at /var/SVN/server-side/engine/branches/stable/c/cclib/handler.c:176
#26 0x00007f95309c71b7 in thread_worker_oper (e=<optimized out>, td=<optimized out>) at /var/SVN/server-side/engine/branches/stable/c/cclib/thread.c:577
#27 0x00007f95309c6460 in thread_worker (thread=<optimized out>, data=<optimized out>) at /var/SVN/server-side/engine/branches/stable/c/cclib/thread.c:774
#28 0x00007f952cad20a4 in start_thread (arg=0x7f94c27f4700) at pthread_create.c:309
#29 0x00007f952c80787d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

I can send more info if needed.

It will be nice if You can fix this crash, but eventually we can live without it. At least until it starts to happen more often…