ray: [Core] [Error Message] Possible unhandled error from worker

When you have a remote function that handles an exception, a Possible unhandled error message is still printed on the driver side. For example,

import time
import ray

ray.init()

@ray.remote
def foo():
    raise Exception("oops")

@ray.remote
def main():
    i = 0
    while True:
        i += 1
        time.sleep(1)
        print(i)
        try:
            ray.get(foo.remote())
        except:
            pass

ray.get(main.remote())

Shows the following message on the driver

2020-11-13 18:03:16,514	ERROR worker.py:1036 -- Possible unhandled error from worker: ray::foo() (pid=68380, ip=192.168.2.228)
  File "python/ray/_raylet.pyx", line 482, in ray._raylet.execute_task
  File "<stdin>", line 3, in foo
Exception: oops

even though the exception is actually being properly handled in the remote function main. This is very confusing for the user because it implies that an error actually occurs even though it doesn’t. At the very least we should emphasize that it is a Possible unhandled error and that it can be safe to ignore this message if it is being handled.

See https://github.com/ray-project/ray/issues/5240 and https://github.com/ray-project/ray/pull/11849#issuecomment-727111327 for more examples.