grpc: Flake: ASAN memory leak for ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires test

ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires/2 test sometimes fails with the following ASAN failure.
Repro: Do the following at grpc root:
tools/run_tests/run_tests.py -lc++ -casan -r ".*RpcDeadlineExpires/2" --force_use_pollers epollex -n inf -S
After a few minutes (somtimes takes as many as 3000 runs or so, it fails with the following ASAN failure)
=================================================================
==165659==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 320 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x8a87b8 in grpc_raw_compressed_byte_buffer_create /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/byte_buffer.cc:36:26
    #3 0x8a863f in grpc_raw_byte_buffer_create /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/byte_buffer.cc:27:10
    #4 0x8c0a70 in process_data_after_md(batch_control*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:1388:33
    #5 0x8be463 in receiving_stream_ready(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:1412:5
    #6 0x8bb917 in receiving_stream_ready_in_call_combiner(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:1424:3
    #7 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #8 0x721550 in grpc_closure_run(char const*, int, grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/./src/core/lib/iomgr/closure.h:258:5
    #9 0xd97234 in recv_message_ready(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/message_size/message_size_filter.cc:135:3
    #10 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #11 0x721550 in grpc_closure_run(char const*, int, grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/./src/core/lib/iomgr/closure.h:258:5
    #12 0xd2a071 in hs_recv_message_ready(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/http/server/http_server_filter.cc:305:5
    #13 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #14 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #15 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #16 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #17 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #18 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #19 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #20 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #21 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #22 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #23 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #24 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #25 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #26 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #27 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #28 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #29 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13

Objects leaked above:
0x612000047d40 (320 bytes)

Indirect leak of 17015 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x875c26 in gpr_malloc_aligned /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:92:13
    #3 0x9c05a4 in zalloc_aligned(unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/arena.cc:48:15
    #4 0x9c099b in gpr_arena_alloc(gpr_arena*, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/arena.cc:82:23
    #5 0x9c0c4d in gpr_arena_alloc(gpr_arena*, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/arena.cc:94:12
    #6 0x8ab326 in grpc_call_create(grpc_call_create_args const*, grpc_call**) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:333:22
    #7 0x8e4c20 in accept_stream(void*, grpc_transport*, void const*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/server.cc:780:23
    #8 0xc93b4e in grpc_chttp2_parsing_accept_stream(grpc_chttp2_transport*, unsigned int) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:789:3
    #9 0xd0a9b2 in init_header_frame_parser(grpc_chttp2_transport*, int) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/parsing.cc:593:9
    #10 0xd08682 in init_frame_parser(grpc_chttp2_transport*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/parsing.cc:289:14
    #11 0xd0695e in grpc_chttp2_perform_read(grpc_chttp2_transport*, grpc_slice) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/parsing.cc:174:13
    #12 0xcb6b3d in read_action_locked(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2416:19
    #13 0x7716fe in grpc_combiner_continue_exec_ctx() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/combiner.cc:257:5
    #14 0x711a97 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #15 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #16 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #17 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #18 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #19 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #20 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #21 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #22 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #23 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #24 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #25 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #26 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #27 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #28 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #29 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13

Objects leaked above:
0x62900000f200 (17015 bytes)

Indirect leak of 16520 byte(s) in 1 object(s) allocated from:
    #0 0x4cdf10 in calloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdf10)
    #1 0x875d56 in zalloc_with_calloc(unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:27:53
    #2 0x875812 in gpr_zalloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:67:7
    #3 0xca9d0f in grpc_create_chttp2_transport(grpc_channel_args const*, grpc_endpoint*, bool) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:3100:31
    #4 0x95f17a in on_handshake_done(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc:130:11
    #5 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #6 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #7 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #8 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #9 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #10 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #11 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #12 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #13 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #14 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #15 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #16 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #17 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #18 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #19 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #20 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #21 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13
    #22 0x7f46d961be6e  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb9e6e)

...    #26 0x7f46d961be6e  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb9e6e)

Objects leaked above:
0x61100004ff00 (224 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x702785 in grpc_error_create(char const*, int, grpc_slice, grpc_error**, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/error.cc:321:34
    #3 0xcb1da2 in write_action_end_locked(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:1028:14
    #4 0x7716fe in grpc_combiner_continue_exec_ctx() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/combiner.cc:257:5
    #5 0x711a97 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #6 0x7ba573 in run_closures(grpc_closure_list) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:80:5
    #7 0x7b9e72 in executor_thread(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:177:22
    #8 0x8814f6 in thread_body(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/thd_posix.cc:67:3
    #9 0x7f46d9b11493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

Objects leaked above:
0x60f000006fd0 (168 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x784708 in fd_create(int, char const*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:341:24
    #3 0x70dbb1 in grpc_fd_create(int, char const*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:193:10
    #4 0xc7630a in on_read(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/tcp_server_posix.cc:247:22
    #5 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #6 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #7 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #8 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #9 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #10 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #11 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #12 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #13 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #14 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #15 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #16 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #17 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #18 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #19 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #20 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #21 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13
    #22 0x7f46d961be6e  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb9e6e)

Objects leaked above:
0x60f00001df50 (168 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x702785 in grpc_error_create(char const*, int, grpc_slice, grpc_error**, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/error.cc:321:34
    #3 0x99230c in timer_callback(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/deadline/deadline_filter.cc:65:9
    #4 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #5 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #6 0x7403e1 in run_some_timers() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/timer_manager.cc:128:3
    #7 0x73fcba in timer_main_loop() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/timer_manager.cc:230:9
    #8 0x73fa34 in timer_thread(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/timer_manager.cc:277:3
    #9 0x8814f6 in thread_body(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/thd_posix.cc:67:3
    #10 0x7f46d9b11493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

Objects leaked above:
0x60f000016570 (168 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x702785 in grpc_error_create(char const*, int, grpc_slice, grpc_error**, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/error.cc:321:34
    #3 0xc991da in close_transport_locked(grpc_chttp2_transport*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:594:13
    #4 0xcb1db2 in write_action_end_locked(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:1027:7
    #5 0x7716fe in grpc_combiner_continue_exec_ctx() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/combiner.cc:257:5
    #6 0x711a97 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #7 0x7ba573 in run_closures(grpc_closure_list) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:80:5
    #8 0x7b9e72 in executor_thread(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:177:22
    #9 0x8814f6 in thread_body(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/thd_posix.cc:67:3
    #10 0x7f46d9b11493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

Objects leaked above:
0x60f000006ee0 (168 bytes)

Indirect leak of 137 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x753aa1 in grpc_slice_intern /usr/local/google/home/sreek/workspace/grpc/src/core/lib/slice/slice_intern.cc:240:33
    #3 0xff1ba3 in user_agent_from_args(grpc_channel_args const*, char const*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/http/client/http_client_filter.cc:513:12
    #4 0xfebce2 in init_channel_elem(grpc_channel_element*, grpc_channel_element_args*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/http/client/http_client_filter.cc:530:7
    #5 0xc6829e in grpc_channel_stack_init(int, void (*)(void*, grpc_error*), void*, grpc_channel_filter const**, unsigned long, grpc_channel_args const*, grpc_transport*, char const*, grpc_channel_stack*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/channel/channel_stack.cc:120:25
    #6 0x8a31fb in grpc_channel_stack_builder_finish(grpc_channel_stack_builder*, unsigned long, int, void (*)(void*, grpc_error*), void*, void**) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/channel/channel_stack_builder.cc:283:23
    #7 0x985e3a in publish_transport_locked(grpc_subchannel*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/client_channel/subchannel.cc:573:23
    #8 0x9809a6 in on_subchannel_connected(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/client_channel/subchannel.cc:628:7
    #9 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #10 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #11 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #12 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #13 0x8d6855 in cq_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1163:9
    #14 0x8d022b in grpc_completion_queue_pluck /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1190:10
    #15 0x8038c1 in grpc::CoreCodegen::grpc_completion_queue_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/core_codegen.cc:70:10
    #16 0x602eed in grpc::CompletionQueue::Pluck(grpc::internal::CompletionQueueTag*) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:296:15
    #17 0x69442d in grpc::internal::BlockingUnaryCallImpl<grpc::testing::EchoRequest, grpc::testing::EchoResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc::testing::EchoRequest const&, grpc::testing::EchoResponse*) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/client_unary_call.h:72:9
    #18 0x689e1b in grpc::Status grpc::internal::BlockingUnaryCall<grpc::testing::EchoRequest, grpc::testing::EchoResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc::testing::EchoRequest const&, grpc::testing::EchoResponse*) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/client_unary_call.h:41:10
    #19 0x684347 in grpc::testing::EchoTestService::Stub::Echo(grpc::ClientContext*, grpc::testing::EchoRequest const&, grpc::testing::EchoResponse*) /usr/local/google/home/sreek/workspace/grpc/gens/src/proto/grpc/testing/echo.grpc.pb.cc:42:10
    #20 0x5826b0 in grpc::testing::(anonymous namespace)::ProxyEnd2endTest_RpcDeadlineExpires_Test::TestBody() /usr/local/google/home/sreek/workspace/grpc/test/cpp/end2end/end2end_test.cc:1299:14
    #21 0x160296b in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2402:10
    #22 0x15d1000 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2438:14
    #23 0x159b609 in testing::Test::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2474:5
    #24 0x159d511 in testing::TestInfo::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2656:5
    #25 0x159e6a1 in testing::TestCase::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2774:5
    #26 0x15ae2cc in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:4649:11
    #27 0x160bd46 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2402:10
    #28 0x15d6620 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2438:14
    #29 0x15ad8c1 in testing::UnitTest::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:4257:10
...

SUMMARY: AddressSanitizer: 67327 byte(s) leaked in 111 allocation(s).
FAILED: bins/asan/end2end_test --gtest_filter=ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires/2  GRPC_POLL_STRATEGY=epollex [ret=1, pid=165659, time=13.2sec]
FLAKE: bins/asan/end2end_test --gtest_filter=ProxyEnd2end
About this issue

Original URL
State: closed
Created 6 years ago
Comments: 35 (35 by maintainers)
Most upvoted comments

We mentioned many issues here.
The memory leak issue is very similar to #14549. Since @ncteisen already took that one, I will also assign this one to him. If this bug needs immediate attention, please re-assign both to me.
The refcount issue rootcaused by @jtattermusch was fixed by #14681.
The other less frequent refcount issue was filed in #14703, as mentioned above.
#12917 happens again (-c dbg). So we may need to re-open it or file a new issue? Or should we increase set_server_sleep_us?
AspirinSJL on Mar 14, 2018