ray: `ray.get` hangs on `ObjectRef`s passed out-of-band

What is the problem?

Tested on Ray 0.8.6.

The problem is that it seems I cannot store and retrieve arbitrary objects outside of tasks/actors anymore. Documentation for ray.put and ray.get seems to reflect that this is still possible, but it is not really. See the reproduction.

For us this is now a major blocker. We used Ray (and before Plasma directly) to pass objects between user interface and workers. On user interface user would upload data and we would call ray.put and then over a custom API send the hex object ID to the main server which would then call Ray task. Now, that task simply hangs, ray.get never to return.

I understand that there is now fancy reference counting and things like that, but simple low-level ray.get and ray.put should still be possible. I think semantics should be that when ray.get calls an object ID not otherwise known to be used by this task, the task blocks, object is send to the task, and then task unblocks. This is how it was before. Even more, before during the time the task was blocked, the worker was in fact freed and other tasks could run there.

So, why does ray.get blocks? Why is not object eventually retrieved?

I would also be OK if those objects are simply eventually removed from memory, when there is pressure on storage. And once removed, if you call ray.get then, an exception is thrown. That is all OK. (While reference counting can help not remove something when it is known that it is still needed.) But that it just blocks without any error, instead of pulling the object to the worker, this is really strange and makes it useless and prevents out-of-band ray.put and ray.get.

Reproduction

import ray
from ray import utils

ray.init()

@ray.remote
def foo(object_id_hex):
    print(ray.get(utils.binary_to_object_id(utils.hex_to_binary(object_id_hex))))

object_id = ray.put(42)
ray.get(foo.remote(object_id.hex()))
print("finished")

finished is newer printed.