grpc: segfault in python `1.31.0`

It looks like the default polling strategy changed from epoll1 in v1.30 to epollex in v1.31. Unfortunately epollex doesn’t have fork support so it doesn’t work with Celery: Fork support is only compatible with the epoll1 and poll polling strategies. We were able to work around this issue by setting GRPC_POLL_STRATEGY=epoll1 on our Celery workers.

We have had this issue too and this thread was helpful for us. We have a workaround, so I thought I would provide details, in hopes it may help.

What we did was as suggested, to set GRPC_POLL_STRATEGY=epoll1 on our workers that use multiprocessing with fork on linux based images. As a longterm strategy, we have been gradually getting rid of using fork in general (as it was the cause for many other issues). This happened on workers running python 3.7.6.

We have also found that workers running on python 3.8.6 do not see this issue. However, we have seen errors like these instead:

2021-02-01 12:16:49.489 EST "Corruption detected."
2021-02-01 12:16:49.490 EST "error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT"
2021-02-01 12:16:49.490 EST "error:1000008b:SSL routines:OPENSSL_internal:DECRYPTION_FAILED_OR_BAD_RECORD_MAC"
2021-02-01 12:16:49.490 EST " Decryption error: TSI_DATA_CORRUPTED"
2021-02-01 12:16:49.490 EST "SSL_write failed with error SSL_ERROR_SSL."

which is similar to what @cpaulik is seeing.

If I had to guess, I would naively suspect that what is occurring is that both the parent and child processes are attempting to use the same socket connection (as fork ing provides a copy) resulting in data corruption. Not sure if this helps, or if this is the wrong direction.

Hi, 2022 checking in. It appears that gRPC doesn’t work with any kind of pre-fork (eg uWSGI) model system unless you specify the environment variables noted above.

Perhaps a solution would be to add some documentation?

We’re using Flask application and Google PubSub emulator and Google Datastore emulator (both communicating via grpc through client libraries) to run tests. With grpcio==1.31.0 we are getting segmentation fault errors but fortunately, with 1.30.0 it works ok. Not sure what could be the issue. We’re Python 3.7.5

This is happening to me too with grpcio-1.33.2-cp36-cp36m-manylinux2014_x86_64 on Ubuntu 18.04.4 LTS (GNU/Linux 5.3.0-1030-aws x86_64). I get constant segfaults. Downgrading to 1.30.0 solves the problem.

I have the same problem. After upgrading python dependencies in project we started to get kubernetes pods restart with celery workers. There were no logs, no tracebacks, no exceptions. We enabled python faulthandler https://blog.richard.do/2018/03/18/how-to-debug-segmentation-fault-in-python/ and got different tracebacks not connected with grpc in different parts of code. By excluding last updates in working branch step by step we found that it was grpc package source of segfaults. Downgrading to 1.30.0 solved this issue

Same here. Python multiprocessing.Process died randomly, leaving segfault message in dmesg like [6558262.243117] grpc_global_tim[42632]: segfault at 7f172819d038 ip 00007f1789caff59 sp 00007f1739bb6c40 error 4 in cygrpc.cpython-36m-x86_64-linux-gnu.so[7f1789a60000+5e8000]. Downgrading to 1.30.0 solves.

install grpcio-1.33.2-cp36-cp36m-manylinux2010_x86_64.whl does not help, still getting same error.

After deploying 1.30.0 on our production systems, the number of segfaults went down to 0.

So, although I was able to create a segfault with 1.30.0 on my own machines, it does seem 1.30.0 is more stable than 1.31.0

Note: the epollex poller has been removed in gRPC 1.46.0 released in May 2022. So I don’t think that the solution above to force the poller to epoll1 is still relevant today, as this is now the default.

We have had this issue too and this thread was helpful for us. We have a workaround, so I thought I would provide details, in hopes it may help.

What we did was as suggested, to set GRPC_POLL_STRATEGY=epoll1 on our workers that use multiprocessing with fork on linux based images. As a longterm strategy, we have been gradually getting rid of using fork in general (as it was the cause for many other issues). This happened on workers running python 3.7.6.

We have also found that workers running on python 3.8.6 do not see this issue. However, we have seen errors like these instead:

2021-02-01 12:16:49.489 EST "Corruption detected."
2021-02-01 12:16:49.490 EST "error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT"
2021-02-01 12:16:49.490 EST "error:1000008b:SSL routines:OPENSSL_internal:DECRYPTION_FAILED_OR_BAD_RECORD_MAC"
2021-02-01 12:16:49.490 EST " Decryption error: TSI_DATA_CORRUPTED"
2021-02-01 12:16:49.490 EST "SSL_write failed with error SSL_ERROR_SSL."

which is similar to what @cpaulik is seeing.

If I had to guess, I would naively suspect that what is occurring is that both the parent and child processes are attempting to use the same socket connection (as fork ing provides a copy) resulting in data corruption. Not sure if this helps, or if this is the wrong direction.

thank you so much @jrmlhermitte I’ve been trying find a solution for segfault issue for a very long time. This worked like a charm.

We faced the exact same issue - multiprocessing with grpc leading to segmentation faults. We were using 1.34.0 in our case but downgrading to 1.30.0 resolved the issue. Haven’t tried it with 1.32.0 and 1.33.0 yet.