grpc: absl: Potential Mutex deadlock - when using signal handler

What version of gRPC and what language are you using?

1.33.2 / C++

What operating system (Linux, Windows,…) and version?

Ubuntu 18.04

What runtime / compiler are you using (e.g. python version or version of gcc)

gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

What did you do?

Tried to implement shutdown of a grpc::Server with a signal handler when pressing ctrl+c (SIGINT). See repro case below.

#include <csignal>

static std::unique_ptr<grpc::Server> _server;

void shutdown(int)
{
    _server->Shutdown();
}

int main()
{
    signal(SIGINT, &shutdown);

    grpc::ServerBuilder builder;
    builder.AddListeningPort("127.0.0.1:5001", grpc::InsecureServerCredentials());
    testing::EchoTestService::Service service;
    builder.RegisterService(&service);
    _server = builder.BuildAndStart();
    _server->Wait();
    return 0;
}

What did you expect to see?

Normal shutdown and exit.

What did you see instead?

When built with dbg

^C[mutex.cc : 1356] RAW: Potential Mutex deadlock: 
	@ 0x55fa744bdc5a absl::lts_2020_02_25::DebugOnlyDeadlockCheck()
	@ 0x55fa744bdf30 absl::lts_2020_02_25::Mutex::Lock()
	@ 0x55fa744a5dc2 gpr_mu_lock
	@ 0x55fa742f677e grpc::CoreCodegen::gpr_mu_lock()
	@ 0x55fa74124077 grpc::internal::MutexLock::MutexLock()
	@ 0x55fa742da51b grpc::Server::ShutdownInternal()
	@ 0x55fa741216d2 grpc::ServerInterface::Shutdown()
	@ 0x55fa7412140f shutdown()
	@ 0x7f15c2bb3f20 
	@ 0x55fa744c435b absl::lts_2020_02_25::synchronization_internal::Waiter::Wait()
	@ 0x55fa744c4097 AbslInternalPerThreadSemWait
	@ 0x55fa744c2a92 absl::lts_2020_02_25::synchronization_internal::PerThreadSem::Wait()
	@ 0x55fa744c2e30 absl::lts_2020_02_25::Mutex::DecrementSynchSem()
	@ 0x55fa744c1c11 absl::lts_2020_02_25::CondVar::WaitCommon()
	@ 0x55fa744c1db9 absl::lts_2020_02_25::CondVar::Wait()
	@ 0x55fa744a5e9c gpr_cv_wait
	@ 0x55fa742f6822 grpc::CoreCodegen::gpr_cv_wait()
	@ 0x55fa742dc11b grpc::internal::CondVar::Wait()
	@ 0x55fa742dc0b5 grpc::internal::CondVar::Wait()
	@ 0x55fa742da9f6 grpc::Server::Wait()
	@ 0x55fa74121552 main
	@ 0x7f15c2b96b97 __libc_start_main

[mutex.cc : 1366] RAW: Acquiring 0x55fa75cabe78    Mutexes held:  0x55fa75cabe78
[mutex.cc : 1367] RAW: Cycle: 
[mutex.cc : 1381] RAW: mutex@0x55fa75cabe78 stack: 
	@ 0x55fa744bdc5a absl::lts_2020_02_25::DebugOnlyDeadlockCheck()
	@ 0x55fa744bdf30 absl::lts_2020_02_25::Mutex::Lock()
	@ 0x55fa744a5dc2 gpr_mu_lock
	@ 0x55fa742f677e grpc::CoreCodegen::gpr_mu_lock()
	@ 0x55fa74124077 grpc::internal::MutexLock::MutexLock()
	@ 0x55fa742da51b grpc::Server::ShutdownInternal()
	@ 0x55fa741216d2 grpc::ServerInterface::Shutdown()
	@ 0x55fa7412140f shutdown()
	@ 0x7f15c2bb3f20 
	@ 0x55fa744c435b absl::lts_2020_02_25::synchronization_internal::Waiter::Wait()
	@ 0x55fa744c4097 AbslInternalPerThreadSemWait
	@ 0x55fa744c2a92 absl::lts_2020_02_25::synchronization_internal::PerThreadSem::Wait()
	@ 0x55fa744c2e30 absl::lts_2020_02_25::Mutex::DecrementSynchSem()
	@ 0x55fa744c1c11 absl::lts_2020_02_25::CondVar::WaitCommon()
	@ 0x55fa744c1db9 absl::lts_2020_02_25::CondVar::Wait()
	@ 0x55fa744a5e9c gpr_cv_wait
	@ 0x55fa742f6822 grpc::CoreCodegen::gpr_cv_wait()
	@ 0x55fa742dc11b grpc::internal::CondVar::Wait()
	@ 0x55fa742dc0b5 grpc::internal::CondVar::Wait()
	@ 0x55fa742da9f6 grpc::Server::Wait()
	@ 0x55fa74121552 main
	@ 0x7f15c2b96b97 __libc_start_main

[mutex.cc : 1386] RAW: dying due to potential deadlock
Aborted (core dumped)

When built with opt


^C[mutex.cc : 2043] RAW: Check waitp == nullptr || waitp->thread->waitp == nullptr || waitp->thread->suppress_fatal_errors failed: detected illegal recursion into Mutex code
Aborted (core dumped)

Anything else we should know about your project / environment?

Works as expected on grpc 1.28.1 which was the previous version used before upgrading to 1.33.2.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (9 by maintainers)

Most upvoted comments

For the record, here is what I do, and it works with recent grpc (1.43.0).

bool shutdown_required = false;
void handle_signal(int sig)
{
    std::cout << "Got signal: " << strsignal(sig) << std::endl;
    shutdown_required = true;
}
void thread_check_shutdown(void)
{
    while (!shutdown_required) sleep(1);
    server->Shutdown();
}

int main(int, char *[])
{
    signal(SIGINT, handle_signal); // Interrupt from keyboard ^C
    signal(SIGQUIT, handle_signal); // Dump core
    signal(SIGTERM, handle_signal); // 'termination signal'... ??
    std::thread t(thread_check_shutdown);
    RunServer();
    t.join();
    return 0;
}

Your handler for SIGINT should just set some global variable, let’s call it sigint_received . When you enter main, spawn a thread that does nothing but check the value of sigint_received (maybe sleep for 1 second between checks) and do your shutdown processing once it becomes true. You can also check for any other exit conditions that you want in that thread. And then remember to join that thread before exit’ing main. Or if possible, just do all that work in main itself, if main doesn’t do any blocking operations. Those are all operations that are async-safe from within the signal handler, and then you synchronize it against the main thread with that check and the join.

@vjpai @orzel Running a sort of infinite while loop seems very inefficient. The task can be easily achieved by conditional variables. Here is my more efficient approach…

bool shutdown_required = false;
std::mutex mutex;
std::condition_variable cv;

void handleSignal(int sig)
{
    std::cout << "Got signal: " << strsignal(sig) << std::endl;
    shutdown_required = true;
    cv.notify_one();
}
void shutdownCheckingThread(void)
{
    std::unique_lock<std::mutex> lock(mutex);
    cv.wait(lock, []()
            { return shutdown_required; });
    server->Shutdown();
}

int main(int, char *[])
{
    signal(SIGINT, handleSignal); 
    signal(SIGQUIT, handleSignal); 
    signal(SIGTERM, handleSignal);
    std::thread t(shutdownCheckingThread);
    RunServer();
    t.join();
    return 0;
}

HasanMdArik on Oct 9, 2022

For the record, here is what I do, and it works with recent grpc (1.43.0).

bool shutdown_required = false;
void handle_signal(int sig)
{
    std::cout << "Got signal: " << strsignal(sig) << std::endl;
    shutdown_required = true;
}
void thread_check_shutdown(void)
{
    while (!shutdown_required) sleep(1);
    server->Shutdown();
}

int main(int, char *[])
{
    signal(SIGINT, handle_signal); // Interrupt from keyboard ^C
    signal(SIGQUIT, handle_signal); // Dump core
    signal(SIGTERM, handle_signal); // 'termination signal'... ??
    std::thread t(thread_check_shutdown);
    RunServer();
    t.join();
    return 0;
}

orzel on Apr 9, 2022