openssl: ThreadSanitizer reports recursive lock overflow caused by impl_cache_flush_cache()

This is another error reported by the ThreadSanitizer. The setup is the same as in issue #9454.

It can be reproduced on current master (d0cf719efb4e) as follows:

./config -g -fsanitize=thread
make -j 16
TSAN_OPTIONS=second_deadlock_stack=1 OPENSSL_TEST_RAND_ORDER=1563972976 util/shlib_wrap.sh /home/msp/src/openssl/test/evp_extra_test

The recursive lock overflow is caused by the fact that

impl_cache_flush_cache() calls RAND_bytes() in frame 350

https://github.com/openssl/openssl/blob/d0cf719efb4e60364ee80d3d7c9c8f69c69cdb95/crypto/property/property.c#L400

although the master drbg is currently still being initialized

by drbg_ossl_ctx_new() in frame 367.

https://github.com/openssl/openssl/blob/d0cf719efb4e60364ee80d3d7c9c8f69c69cdb95/crypto/rand/drbg_lib.c#L175

This issue also reveals a fundamental problem: low level core library routines need to be very careful when calling higher level crypto api functions, otherwise difficult reentrancy problems like this locking problems might arise.

FATAL: ThreadSanitizer CHECK failed: /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/sanitizer_common/sanitizer_deadlock_detector.h:63 “((n_recursive_locks)) < (((sizeof(recursive_locks)/sizeof((recursive_locks)[0]))))” (0x40, 0x40) #0 __tsan::TsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_rtl_report.cc:46 (libtsan.so.0+0x83c9f) … … … #6 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1246 (libtsan.so.0+0x2d88b) #7 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495) #8 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f7059) #9 get_default_method_store crypto/evp/evp_fetch.c:65 (libcrypto.so.3+0x1d5b5c) #10 evp_generic_fetch crypto/evp/evp_fetch.c:165 (libcrypto.so.3+0x1d5b5c) #11 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d4993) #12 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x242720) #13 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2445b4) #14 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246242) #15 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246242) #16 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x24672d) #17 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x24672d) #18 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f6afe) #19 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1f9aeb) #20 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f7082) #21 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x246bba) #22 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (libcrypto.so.3+0x246bba) #23 rand_bytes_ex crypto/rand/rand_lib.c:840 (libcrypto.so.3+0x248c09) #24 RAND_bytes crypto/rand/rand_lib.c:850 (libcrypto.so.3+0x248c70) #25 impl_cache_flush_cache crypto/property/property.c:400 (libcrypto.so.3+0x23c7ed) #26 doall_util_fn crypto/lhash/lhash.c:205 (libcrypto.so.3+0x1f6004) #27 OPENSSL_LH_doall_arg crypto/lhash/lhash.c:220 (libcrypto.so.3+0x1f6004) #28 lh_QUERY_doall_IMPL_CACHE_FLUSH crypto/property/property.c:374 (libcrypto.so.3+0x23c721) #29 impl_cache_flush_one_alg crypto/property/property.c:418 (libcrypto.so.3+0x23c721) #30 sa_doall crypto/sparse_array.c:98 (libcrypto.so.3+0x20604c) … … … #324 RAND_bytes crypto/rand/rand_lib.c:850 (libcrypto.so.3+0x248c70) #325 impl_cache_flush_cache crypto/property/property.c:400 (libcrypto.so.3+0x23c7ed) #326 doall_util_fn crypto/lhash/lhash.c:205 (libcrypto.so.3+0x1f6004) #327 OPENSSL_LH_doall_arg crypto/lhash/lhash.c:220 (libcrypto.so.3+0x1f6004) #328 lh_QUERY_doall_IMPL_CACHE_FLUSH crypto/property/property.c:374 (libcrypto.so.3+0x23c721) #329 impl_cache_flush_one_alg crypto/property/property.c:418 (libcrypto.so.3+0x23c721) #330 sa_doall crypto/sparse_array.c:98 (libcrypto.so.3+0x20604c) #331 OPENSSL_SA_doall_arg crypto/sparse_array.c:152 (libcrypto.so.3+0x20604c) #332 ossl_sa_ALGORITHM_doall_arg crypto/property/property.c:66 (libcrypto.so.3+0x23d830) #333 ossl_method_cache_flush_some crypto/property/property.c:428 (libcrypto.so.3+0x23d830) #334 ossl_method_store_cache_set crypto/property/property.c:474 (libcrypto.so.3+0x23d830) #335 evp_generic_fetch crypto/evp/evp_fetch.c:221 (libcrypto.so.3+0x1d5da2) #336 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d4993) #337 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x242720) #338 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2445b4) #339 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246242) #340 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246242) #341 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x24672d) #342 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x24672d) #343 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f6afe) #344 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1f9aeb) #345 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f7082) #346 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x246bba) #347 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (libcrypto.so.3+0x246bba) #348 rand_bytes_ex crypto/rand/rand_lib.c:840 (libcrypto.so.3+0x248c09) #349 RAND_bytes crypto/rand/rand_lib.c:850 (libcrypto.so.3+0x248c70) #350 impl_cache_flush_cache crypto/property/property.c:400 (libcrypto.so.3+0x23c7ed) #351 doall_util_fn crypto/lhash/lhash.c:205 (libcrypto.so.3+0x1f6004) #352 OPENSSL_LH_doall_arg crypto/lhash/lhash.c:220 (libcrypto.so.3+0x1f6004) #353 lh_QUERY_doall_IMPL_CACHE_FLUSH crypto/property/property.c:374 (libcrypto.so.3+0x23c721) #354 impl_cache_flush_one_alg crypto/property/property.c:418 (libcrypto.so.3+0x23c721) #355 sa_doall crypto/sparse_array.c:98 (libcrypto.so.3+0x20604c) #356 OPENSSL_SA_doall_arg crypto/sparse_array.c:152 (libcrypto.so.3+0x20604c) #357 ossl_sa_ALGORITHM_doall_arg crypto/property/property.c:66 (libcrypto.so.3+0x23d830) #358 ossl_method_cache_flush_some crypto/property/property.c:428 (libcrypto.so.3+0x23d830) #359 ossl_method_store_cache_set crypto/property/property.c:474 (libcrypto.so.3+0x23d830) #360 evp_generic_fetch crypto/evp/evp_fetch.c:221 (libcrypto.so.3+0x1d5da2) #361 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d4993) #362 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x242720) #363 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2445b4) #364 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246242) #365 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246242) #366 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x24672d) #367 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x24672d) #368 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f6afe) #369 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1f9aeb) #370 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f7082) #371 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x246bba) #372 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (libcrypto.so.3+0x246bba)

cc @paulidale @bernd-edlinger