liboauth2: Occasional segfaults

Hello,

I’m seeing occasional segfaults using liboauth2==1.4.5 and mod_oauth2==3.2.3 in a multithreaded environment. Apache info:

# httpd -V
Server version: Apache/2.4.37 (Red Hat Enterprise Linux)
Server built:   Jun 15 2022 08:27:14
Server's Module Magic Number: 20120211:83
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)

3 stack traces from 3 different child processes:

Thread 267 "httpd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fb19f7fe700 (LWP 3066)]
0x00007fb36a02376d in hashtable_do_clear () from /usr/lib64/libjansson.so.4
(gdb) bt
#0  0x00007fb36a02376d in hashtable_do_clear () from /usr/lib64/libjansson.so.4
#1  0x00007fb36a02383d in hashtable_close () from /usr/lib64/libjansson.so.4
#2  0x00007fb36a029009 in json_delete () from /usr/lib64/libjansson.so.4
#3  0x00007fb36a45f87f in json_decref () from /usr/lib64/liboauth2.so.0
#4  0x00007fb36a46264e in oauth2_jose_jwt_header_peek () from /usr/lib64/liboauth2.so.0
#5  0x00007fb36a462e83 in oauth2_jose_jwt_verify () from /usr/lib64/liboauth2.so.0
#6  0x00007fb36a46b2fb in _oauth2_metadata_verify_callback () from /usr/lib64/liboauth2.so.0
#7  0x00007fb36a46bc53 in oauth2_token_verify () from /usr/lib64/liboauth2.so.0
#8  0x00007fb36a88d683 in oauth2_request_handler () from /etc/httpd/modules/mod_oauth2.so
#9  0x00005647faf15ca8 in ap_run_check_user_id ()
#10 0x00005647faf188fc in ap_process_request_internal ()
#11 0x00005647faf37840 in ap_process_async_request ()
#12 0x00005647faf33ce0 in ap_process_http_connection ()
#13 0x00005647faf2a0c8 in ap_run_process_connection ()
#14 0x00007fb36f984a47 in process_socket () from /etc/httpd/modules/mod_mpm_event.so
#15 0x00007fb36f9853ea in worker_thread () from /etc/httpd/modules/mod_mpm_event.so
#16 0x00007fb37ac6117a in start_thread () from /lib64/libpthread.so.0
#17 0x00007fb37a78cdc3 in clone () from /lib64/libc.so.6
Thread 197 "httpd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f80f9ffb700 (LWP 2968)]
0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
#1  0x00007f8170c0e6c2 in oauth2_mem_alloc () from /usr/lib64/liboauth2.so.0
#2  0x00007f8170c0ff72 in oauth2_strndup () from /usr/lib64/liboauth2.so.0
#3  0x00007f8170c1c4a4 in oauth2_jose_jwt_header_peek () from /usr/lib64/liboauth2.so.0
#4  0x00007f8170c1ce83 in oauth2_jose_jwt_verify () from /usr/lib64/liboauth2.so.0
#5  0x00007f8170c252fb in _oauth2_metadata_verify_callback () from /usr/lib64/liboauth2.so.0
#6  0x00007f8170c25c53 in oauth2_token_verify () from /usr/lib64/liboauth2.so.0
#7  0x00007f8171047683 in oauth2_request_handler () from /etc/httpd/modules/mod_oauth2.so
#8  0x00005573b5c63ca8 in ap_run_check_user_id ()
#9  0x00005573b5c668fc in ap_process_request_internal ()
#10 0x00005573b5c85840 in ap_process_async_request ()
#11 0x00005573b5c81ce0 in ap_process_http_connection ()
#12 0x00005573b5c780c8 in ap_run_process_connection ()
#13 0x00007f817613ea47 in process_socket () from /etc/httpd/modules/mod_mpm_event.so
#14 0x00007f817613f3ea in worker_thread () from /etc/httpd/modules/mod_mpm_event.so
#15 0x00007f818141b17a in start_thread () from /lib64/libpthread.so.0
#16 0x00007f8180f46dc3 in clone () from /lib64/libc.so.6
Thread 298 "httpd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f7fdf7fe700 (LWP 3087)]
0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
#1  0x00007f817367590d in CRYPTO_zalloc () from /lib64/libcrypto.so.1.1
#2  0x00007f81736496d1 in EVP_DigestInit_ex () from /lib64/libcrypto.so.1.1
#3  0x00007f8170c19c3f in oauth2_jose_hash_bytes () from /usr/lib64/liboauth2.so.0
#4  0x00007f8170c1b901 in oauth2_jose_hash2s () from /usr/lib64/liboauth2.so.0
#5  0x00007f8170c2d7b9 in _oauth2_cache_hash_key () from /usr/lib64/liboauth2.so.0
#6  0x00007f8170c2daf9 in oauth2_cache_get () from /usr/lib64/liboauth2.so.0
#7  0x00007f8170c1ec59 in oauth2_jose_resolve_from_uri () from /usr/lib64/liboauth2.so.0
#8  0x00007f8170c1ee1f in _oauth2_jose_jwks_resolve_from_uri () from /usr/lib64/liboauth2.so.0
#9  0x00007f8170c1ee8d in oauth2_jose_jwks_uri_resolve () from /usr/lib64/liboauth2.so.0
#10 0x00007f8170c1cfa1 in oauth2_jose_jwt_verify () from /usr/lib64/liboauth2.so.0
#11 0x00007f8170c252fb in _oauth2_metadata_verify_callback () from /usr/lib64/liboauth2.so.0
#12 0x00007f8170c25c53 in oauth2_token_verify () from /usr/lib64/liboauth2.so.0
#13 0x00007f8171047683 in oauth2_request_handler () from /etc/httpd/modules/mod_oauth2.so
#14 0x00005573b5c63ca8 in ap_run_check_user_id ()
#15 0x00005573b5c668fc in ap_process_request_internal ()
#16 0x00005573b5c85840 in ap_process_async_request ()
#17 0x00005573b5c81ce0 in ap_process_http_connection ()
#18 0x00005573b5c780c8 in ap_run_process_connection ()
#19 0x00007f817613ea47 in process_socket () from /etc/httpd/modules/mod_mpm_event.so
#20 0x00007f817613f3ea in worker_thread () from /etc/httpd/modules/mod_mpm_event.so
#21 0x00007f818141b17a in start_thread () from /lib64/libpthread.so.0
#22 0x00007f8180f46dc3 in clone () from /lib64/libc.so.6

Seems oauth2_jose_jwt_verify() may be a commonality? Thank you in advance.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 20 (9 by maintainers)

Commits related to this issue

Most upvoted comments

I deployed the fix to our environment and no crashes for 7 hours! I think I can declare victory. Thanks again for you help!