google-cloud-ruby: Subscriptions on threads segfault on exit

Minimal repro:

ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin15]
google-cloud-core (0.21.1)
google-cloud-pubsub (0.23.2)
google-protobuf (3.2.0.1)
googleapis-common-protos (1.3.5)
grpc (1.1.2)

require 'google/cloud/pubsub'
s = Google::Cloud::Pubsub::Service.new("czi-dev", Google::Cloud::Credentials.default)
t = Thread.new { puts s.pull("test.walter.pull").inspect }
t.join
# Hit ctrl-C while waiting for pull to return

Crash:

Exception Type:        EXC_BAD_ACCESS (SIGABRT)
Exception Codes:       KERN_INVALID_ADDRESS at 0x00007000089af808

VM Regions Near 0x7000089af808:
    Stack                  00007000088ac000-00007000088b2000 [   24K] rw-/rwx SM=COW  thread 1
--> 
    Stack                  0000700008b39000-0000700008b3b000 [    8K] rw-/rwx SM=PRV  

...
5   libruby.2.3.0.dylib           	0x000000010f9fe754 sigsegv + 68
6   libsystem_platform.dylib      	0x00007fffb5e50bba _sigtramp + 26
7   ???                           	000000000000000000 0 + 0
8   grpc_c.bundle                 	0x0000000110405c54 grpc_exec_ctx_flush + 84
9   grpc_c.bundle                 	0x0000000110414313 grpc_call_cancel_with_status + 323
10  grpc_c.bundle                 	0x00000001104140e8 grpc_call_destroy + 376
11  grpc_c.bundle                 	0x00000001103f3ddb 0x1103f1000 + 11739
12  libruby.2.3.0.dylib           	0x000000010f953ff0 finalize_list + 64 (gc.c:2678)
13  libruby.2.3.0.dylib           	0x000000010f947ef8 rb_gc_call_finalizer_at_exit + 1272 (gc.c:2842)
14  libruby.2.3.0.dylib           	0x000000010f931c61 ruby_cleanup + 721 (eval.c:224)

It appears that c->cb points to unallocated address space here: https://github.com/grpc/grpc/blob/46357c882df1afc28f7a5228c40fde522093fa32/src/core/lib/iomgr/exec_ctx.c#L76

Perhaps a cleanup issue similar to #1096?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 21 (8 by maintainers)

Most upvoted comments

@apolcyn 1.2.1.pre2 no longer faults in my test program or in my actual program. 😃 👍

wrs on Mar 27, 2017

@hxiong388 @blowmage the grpc-1.2.1-pre2 was just published with the fix for this (fix the client side segfault)

apolcyn on Mar 27, 2017

This bug indeed looks within grpc, and it does looks similar to #1096.

This repros every time on OS X with this plain grpc repro:

tweaking the greeter_client.rb(https://github.com/grpc/grpc/blob/master/examples/ruby/greeter_client.rb) example main to:

def main
  stub = Helloworld::Greeter::Stub.new('localhost:50051', :this_channel_is_insecure)
  user = ARGV.size > 0 ?  ARGV[0] : 'world'
  t = Thread.new do
    message = stub.say_hello(Helloworld::HelloRequest.new(name: user)).message
    p "Greeting: #{message}"
  end
  t.join
end

and change the greeter_server.rb “greeter handler” to:

# GreeterServer is simple server that implements the Helloworld Greeter server.
class GreeterServer < Helloworld::Greeter::Service
  # say_hello implements the SayHello rpc method.
  def say_hello(hello_req, _unused_call)
    sleep 10
    Helloworld::HelloReply.new(message: "Hello #{hello_req.name}")
  end
end

and then ctrl-c client while the server handler.

This sort of thing should be probably be handled within grpc though - can’t be easily avoided by joining threads. cc @murgatroid99

(again, it looks like thread is getting garbage collected before the call on it completes. then references on that thread’s stack get invalidly uses it when call objects GC function is called).

apolcyn on Mar 20, 2017