liboauth2: Occasional segfaults
Hello,
I’m seeing occasional segfaults using liboauth2==1.4.5 and mod_oauth2==3.2.3 in a multithreaded environment. Apache info:
# httpd -V
Server version: Apache/2.4.37 (Red Hat Enterprise Linux)
Server built: Jun 15 2022 08:27:14
Server's Module Magic Number: 20120211:83
Server loaded: APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
3 stack traces from 3 different child processes:
Thread 267 "httpd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fb19f7fe700 (LWP 3066)]
0x00007fb36a02376d in hashtable_do_clear () from /usr/lib64/libjansson.so.4
(gdb) bt
#0 0x00007fb36a02376d in hashtable_do_clear () from /usr/lib64/libjansson.so.4
#1 0x00007fb36a02383d in hashtable_close () from /usr/lib64/libjansson.so.4
#2 0x00007fb36a029009 in json_delete () from /usr/lib64/libjansson.so.4
#3 0x00007fb36a45f87f in json_decref () from /usr/lib64/liboauth2.so.0
#4 0x00007fb36a46264e in oauth2_jose_jwt_header_peek () from /usr/lib64/liboauth2.so.0
#5 0x00007fb36a462e83 in oauth2_jose_jwt_verify () from /usr/lib64/liboauth2.so.0
#6 0x00007fb36a46b2fb in _oauth2_metadata_verify_callback () from /usr/lib64/liboauth2.so.0
#7 0x00007fb36a46bc53 in oauth2_token_verify () from /usr/lib64/liboauth2.so.0
#8 0x00007fb36a88d683 in oauth2_request_handler () from /etc/httpd/modules/mod_oauth2.so
#9 0x00005647faf15ca8 in ap_run_check_user_id ()
#10 0x00005647faf188fc in ap_process_request_internal ()
#11 0x00005647faf37840 in ap_process_async_request ()
#12 0x00005647faf33ce0 in ap_process_http_connection ()
#13 0x00005647faf2a0c8 in ap_run_process_connection ()
#14 0x00007fb36f984a47 in process_socket () from /etc/httpd/modules/mod_mpm_event.so
#15 0x00007fb36f9853ea in worker_thread () from /etc/httpd/modules/mod_mpm_event.so
#16 0x00007fb37ac6117a in start_thread () from /lib64/libpthread.so.0
#17 0x00007fb37a78cdc3 in clone () from /lib64/libc.so.6
Thread 197 "httpd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f80f9ffb700 (LWP 2968)]
0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
#1 0x00007f8170c0e6c2 in oauth2_mem_alloc () from /usr/lib64/liboauth2.so.0
#2 0x00007f8170c0ff72 in oauth2_strndup () from /usr/lib64/liboauth2.so.0
#3 0x00007f8170c1c4a4 in oauth2_jose_jwt_header_peek () from /usr/lib64/liboauth2.so.0
#4 0x00007f8170c1ce83 in oauth2_jose_jwt_verify () from /usr/lib64/liboauth2.so.0
#5 0x00007f8170c252fb in _oauth2_metadata_verify_callback () from /usr/lib64/liboauth2.so.0
#6 0x00007f8170c25c53 in oauth2_token_verify () from /usr/lib64/liboauth2.so.0
#7 0x00007f8171047683 in oauth2_request_handler () from /etc/httpd/modules/mod_oauth2.so
#8 0x00005573b5c63ca8 in ap_run_check_user_id ()
#9 0x00005573b5c668fc in ap_process_request_internal ()
#10 0x00005573b5c85840 in ap_process_async_request ()
#11 0x00005573b5c81ce0 in ap_process_http_connection ()
#12 0x00005573b5c780c8 in ap_run_process_connection ()
#13 0x00007f817613ea47 in process_socket () from /etc/httpd/modules/mod_mpm_event.so
#14 0x00007f817613f3ea in worker_thread () from /etc/httpd/modules/mod_mpm_event.so
#15 0x00007f818141b17a in start_thread () from /lib64/libpthread.so.0
#16 0x00007f8180f46dc3 in clone () from /lib64/libc.so.6
Thread 298 "httpd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f7fdf7fe700 (LWP 3087)]
0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f8180ecfbc6 in malloc () from /lib64/libc.so.6
#1 0x00007f817367590d in CRYPTO_zalloc () from /lib64/libcrypto.so.1.1
#2 0x00007f81736496d1 in EVP_DigestInit_ex () from /lib64/libcrypto.so.1.1
#3 0x00007f8170c19c3f in oauth2_jose_hash_bytes () from /usr/lib64/liboauth2.so.0
#4 0x00007f8170c1b901 in oauth2_jose_hash2s () from /usr/lib64/liboauth2.so.0
#5 0x00007f8170c2d7b9 in _oauth2_cache_hash_key () from /usr/lib64/liboauth2.so.0
#6 0x00007f8170c2daf9 in oauth2_cache_get () from /usr/lib64/liboauth2.so.0
#7 0x00007f8170c1ec59 in oauth2_jose_resolve_from_uri () from /usr/lib64/liboauth2.so.0
#8 0x00007f8170c1ee1f in _oauth2_jose_jwks_resolve_from_uri () from /usr/lib64/liboauth2.so.0
#9 0x00007f8170c1ee8d in oauth2_jose_jwks_uri_resolve () from /usr/lib64/liboauth2.so.0
#10 0x00007f8170c1cfa1 in oauth2_jose_jwt_verify () from /usr/lib64/liboauth2.so.0
#11 0x00007f8170c252fb in _oauth2_metadata_verify_callback () from /usr/lib64/liboauth2.so.0
#12 0x00007f8170c25c53 in oauth2_token_verify () from /usr/lib64/liboauth2.so.0
#13 0x00007f8171047683 in oauth2_request_handler () from /etc/httpd/modules/mod_oauth2.so
#14 0x00005573b5c63ca8 in ap_run_check_user_id ()
#15 0x00005573b5c668fc in ap_process_request_internal ()
#16 0x00005573b5c85840 in ap_process_async_request ()
#17 0x00005573b5c81ce0 in ap_process_http_connection ()
#18 0x00005573b5c780c8 in ap_run_process_connection ()
#19 0x00007f817613ea47 in process_socket () from /etc/httpd/modules/mod_mpm_event.so
#20 0x00007f817613f3ea in worker_thread () from /etc/httpd/modules/mod_mpm_event.so
#21 0x00007f818141b17a in start_thread () from /lib64/libpthread.so.0
#22 0x00007f8180f46dc3 in clone () from /lib64/libc.so.6
Seems oauth2_jose_jwt_verify() may be a commonality? Thank you in advance.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 20 (9 by maintainers)
I deployed the fix to our environment and no crashes for 7 hours! I think I can declare victory. Thanks again for you help!