ray: [Bug] Upgrade to gRPC 1.46 once it's released (which fixes fork support)

@HeyItsBethany3 I just downgraded to grpcio==1.43, which is a workaround until there’s a more fundamental fix.

For some reason, this is one of the top viewed pages on github:

Thank you so much! - I downgraded the version and it worked

@rkooo567 I’m still experiencing this issue - what is the solution? I have tried setting GRPC_FORK_SUPPORT_ENABLED as an environmental variable using os.environ["GRPC_FORK_SUPPORT_ENABLED"]="0" in python and export GRPC_FORK_SUPPORT_ENABLED=0 in the terminal but I’m still getting these error messages.

@vrazdalovschi It seems the logspam fix is available in gRPC 1.48, so we can try using that. Btw only Python grpcio should need to be upgraded. I can send out a PR.

@pcmoritz I think that’s because the issue stems from grpcio 1.44, so it affected much more than just Ray. I’m not a Ray user but ended up here because I’ve got the same issue on my own program. The latest issue regarding this message on the gRPC GitHub is closed and 4 years old, so people naturally end up here.

Is it possible to update to 1.47+? We need 1.47+, because previous versions have issues with mac m1 wheels and need to do unpleasant interactions to support it. Thank you! PS: for both grpc and grpcio

The issue should have been fixed on gRPC master. We can wait for the release containing the fix.

Btw it is not the long term fix. I think we need to mark it P1 and find the long term fix to allow us to use 1.44 (according to philipp this is the first version that works well with M1 Mac)

Is Ray going to publish patch releases for older versions that include the <=1.43 version range limiter?

This is an absolutely god awful work-around, but it works:

Basically, it intercepts the stdout out stream and drops the grpc error msg. Use at your own risk:

stdout_fileno = sys.stdout.fileno()
stdout_save = os.dup(stdout_fileno)
stdout_pipe = os.pipe()
os.dup2(stdout_pipe[1], stdout_fileno)
os.close(stdout_pipe[1])

def drain_pipe():
    EOL = '\n'.encode('utf-8')[0]
    GRPC_ERR_MSG = 'Fork support is only compatible'.encode('utf-8')

    def readline():
        line = b''
        while True:
            line += os.read(stdout_pipe[0], 1)
            if line[-1] == EOL:
                break 
        return line


    while True:
        line = readline()
        if GRPC_ERR_MSG in line:
            continue 
        os.write(stdout_save, line)

t = threading.Thread(target=drain_pipe)
t.start()

Seems like we can easily solve it by setting GRPC_FORK_SUPPORT_ENABLED =0 in python? or GRPC_FORK_SUPPORT_ENABLED =1 in core

Yes, fork+exec is always supported. There’s a mismatch between the default values of GRPC_FORK_SUPPORT_ENABLED in core, where it defaults to true, and in the Python layer, where it defaults to false. There is no signal in core that could let us distinguish beforehand between fork without exec and fork with exec. This is why the earlier error log message (here) is not “log spam” by default. I am fine changing the “Fork support is only compatible…” log statement in fork_posix.cc to INFO level, but we still want an error to surface to Python users that have explicitly set GRPC_FORK_SUPPORT_ENABLED=1. This can probably be done by checking grpc_get_poll_strategy_name() in fork_posix.pyx.pxi.