redis-client: hiredis-client Segfault on pubsub

From time to time I have a segfault in the hiredis gem. It seems to happen only on my local computer.

Here’s the error :

 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95: [BUG] Segmentation fault at 0x0000000000000000
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95: [BUG] Segmentation fault at 0x0000000000000000
 ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [x86_64-darwin22]
 ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [x86_64-darwin22]


 -- Crash Report log information --------------------------------------------
 -- Crash Report log information --------------------------------------------
    See Crash Report log file in one of the following locations:
    See Crash Report log file in one of the following locations:
      * ~/Library/Logs/DiagnosticReports
      * ~/Library/Logs/DiagnosticReports
      * /Library/Logs/DiagnosticReports
      * /Library/Logs/DiagnosticReports
    for more details.
    for more details.
 Don't forget to include the above Crash Report log file in bug reports.
 Don't forget to include the above Crash Report log file in bug reports.


 -- Control frame information -----------------------------------------------
 -- Control frame information -----------------------------------------------
 c:0007 p:---- s:0036 e:000035 CFUNC  :_read
 c:0007 p:---- s:0036 e:000035 CFUNC  :_read
 c:0006 p:0028 s:0032 e:000031 METHOD /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95
 c:0006 p:0028 s:0032 e:000031 METHOD /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95
 c:0005 p:0023 s:0025 e:000024 METHOD /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/redis-client-0.17.0/lib/redis_client.rb:458
 c:0005 p:0023 s:0025 e:000024 METHOD /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/redis-client-0.17.0/lib/redis_client.rb:458
 c:0004 p:0007 s:0020 e:000019 BLOCK  /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:99 [FINISH]
 c:0004 p:0007 s:0020 e:000019 BLOCK  /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:99 [FINISH]
 c:0003 p:---- s:0011 e:000010 CFUNC  :loop
 c:0003 p:---- s:0011 e:000010 CFUNC  :loop
 c:0002 p:0024 s:0007 E:0004d8 BLOCK  /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:98 [FINISH]
 c:0002 p:0024 s:0007 E:001d08 BLOCK  /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:98 [FINISH]
 c:0001 p:---- s:0003 e:000002 DUMMY  [FINISH]
 c:0001 p:---- s:0003 e:000002 DUMMY  [FINISH]


 -- Ruby level backtrace information ----------------------------------------
 -- Ruby level backtrace information ----------------------------------------
 /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:98:in `block in listen_to_changes'
 /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:98:in `block in listen_to_changes'
 /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:98:in `loop'
 /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:98:in `loop'
 /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:99:in `block (2 levels) in listen_to_changes'
 /Users/mathieu/Documents/rbean/rbean/app/models/feature_flags.rb:99:in `block (2 levels) in listen_to_changes'
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/redis-client-0.17.0/lib/redis_client.rb:458:in `next_event'
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/redis-client-0.17.0/lib/redis_client.rb:458:in `next_event'
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95:in `read'
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95:in `read'
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95:in `_read'
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.rb:95:in `_read'


 -- Machine register context ------------------------------------------------
 -- Machine register context ------------------------------------------------
  rax: 0x0002600000026000 rbx: 0x0000000000000002 rcx: 0x0000000000000000
  rax: 0x00000001104f38c0 rbx: 0x0000000000000000 rcx: 0x0000000000000000
  rdx: 0x0000000000000000 rdi: 0x000000012e3cc030 rsi: 0x000000012aaf1b28
  rdx: 0xffffffffe5894855 rdi: 0x0000000000000000 rsi: 0xffffffffe5894855
  rbp: 0x0000700004a87180 rsp: 0x0000700004a87150  r8: 0x0000000000004002
  rbp: 0x0000700004422140 rsp: 0x0000700004422110  r8: 0x0000000000004002
   r9: 0x000000012e3d5088 r10: 0x0000000000001604 r11: 0x000000012e3cf205
   r9: 0x00000001244a8088 r10: 0x0000000000001604 r11: 0x00000001244a3205
  r12: 0x0000000000000000 r13: 0x0000700004a872a8 r14: 0x0000000000000000
  r12: 0x0000000000000000 r13: 0x0000700004422208 r14: 0x0000000126430d88
  r15: 0x000000012aaf1b28 rip: 0x000000011eb4afed rfl: 0x0000000000010206
  r15: 0x0000000126430d88 rip: 0x00000001102acd07 rfl: 0x0000000000010246


 -- C level backtrace information -------------------------------------------
 -- C level backtrace information -------------------------------------------
 /Users/mathieu/.rbenv/versions/3.2.2/lib/libruby.3.2.dylib(rb_vm_bugreport+0x7c4) [0x10c8af444]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/libruby.3.2.dylib(rb_vm_bugreport+0x7c4) [0x11055b444]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/libruby.3.2.dylib(rb_bug_for_fatal_signal+0x1d9) [0x10c6adf59]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/libruby.3.2.dylib(rb_bug_for_fatal_signal+0x1d9) [0x110359f59]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/libruby.3.2.dylib(sigsegv+0x5b) [0x10c7fc4ab]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/libruby.3.2.dylib(sigsegv+0x5b) [0x1104a84ab]
 /usr/lib/system/libsystem_platform.dylib(_sigtramp+0x1d) [0x7ff8180d9c1d]
 /usr/lib/system/libsystem_platform.dylib(_sigtramp+0x1d) [0x7ff8180d9c1d]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(reply_append+0x1d) [0x11eb4afed]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/libruby.3.2.dylib(rb_ary_store+0x17) [0x1102acd07]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(redisReaderGetReply+0xb4d) [0x11eb565fd]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(reply_append+0x73) [0x1228eb043]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(redisGetReplyFromReader+0x19) [0x11eb4e629]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(redisReaderGetReply+0xb4d) [0x1228f65fd]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(hiredis_read+0x1a9) [0x11eb4a259]
 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(redisGetReplyFromReader+0x19) [0x1228ee629]

 /Users/mathieu/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/hiredis-client-0.17.0/lib/redis_client/hiredis_connection.bundle(hiredis_read+0x1a9) [0x1228ea259]

and then thousand of lines of loaded files.

Here’s the listen_to_changes function :

def self.listen_to_changes
  Thread.new do
    conn = RedisClient.new(reconnect_attempts: [0.2, 0.8, 2]).pubsub

    conn.call('subscribe', 'feature_flags')
    loop do
      next unless message = conn.next_event(20)

      [... activerecord stuff ...]
    end
  end
end

This code used to work correctly.

I recently update my ruby version to enable YJIT and Jemalloc on my local computer, so it could be related, but the bug is not easily reproducible.

Tell me if you need more information and thanks for your help.

About this issue

Original URL
State: closed
Created 10 months ago
Comments: 29 (2 by maintainers)

Commits related to this issue

Add a GC_GUARD for the reader stack in hiredis_read_internal Potential fix for https://github.com/redis-rb/redis-client/issues/141 I suspect the reference isn't always kept on the stack, and GC trig... — committed to redis-rb/redis-client by byroot 10 months ago
Add some debug statements for #141 — committed to redis-rb/redis-client by byroot 10 months ago
Add some debug statements for #141 — committed to redis-rb/redis-client by byroot 10 months ago

Most upvoted comments

No problem!

My repro script didn’t segfault yet. In the meantime I will try without YJIT and without Jemalloc (the last 2 things I changed) see if I can find any differencies.

Thanks for your help and I’ll keep you updated

mathieumahe on Sep 12, 2023