ray: [Ray Core] failed to serialize namedtuple in py3.9

What happened + What you expected to happen

ray failed to serialize namedtuple in py3.9, which looks like a error from cloudpickle: https://github.com/cloudpipe/cloudpickle/issues/460

Versions / Dependencies

python 3.9.7 ray 1.13.0

not tested with other versions

Reproduction script

from typing import NamedTuple
import ray
ray.init()

class T(NamedTuple):
    a:str
    
@ray.remote
def test():
    return T('a')

ray.get(test.remote())
Traceback (most recent call last):
  File "/home/auderson/miniconda3/envs/py3.9/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3361, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-7f1d37acd5e0>", line 1, in <cell line: 1>
    ray.get(test.remote())
  File "/home/auderson/miniconda3/envs/py3.9/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/auderson/miniconda3/envs/py3.9/lib/python3.9/site-packages/ray/worker.py", line 1831, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::test() (pid=353630, ip=...)
RuntimeError: The remote function failed to import on the worker. This may be because needed library dependencies are not installed in the worker environment:
ray::test() (pid=353630, ip=...)
  File "/home/auderson/miniconda3/envs/py3.9/lib/python3.9/site-packages/ray/cloudpickle/cloudpickle.py", line 872, in _make_skeleton_class
    skeleton_class = types.new_class(
  File "/home/auderson/miniconda3/envs/py3.9/lib/python3.9/types.py", line 77, in new_class
    return meta(name, resolved_bases, ns, **kwds)
  File "/home/auderson/miniconda3/envs/py3.9/lib/python3.9/typing.py", line 1852, in __new__
    module=ns['__module__'])
KeyError: '__module__'
ray.util.inspect_serializability(T)
================================================
Checking Serializability of <class '__main__.T'>
================================================
Out[9]: (True, set())

Issue Severity

Low: It annoys or frustrates me.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (13 by maintainers)

Commits related to this issue

Most upvoted comments

@auderson Yes, this is indeed an error from cloudpickle. Currently a workaround is to move

class T(NamedTuple):
    a:str

to another python module and import it (as you mentioned).

@rkooo567 @scv119 I think we should wait for the upstream (cloudpickle) to fix it if this is not a major blocker.

ray.util.inspect_serializability(T) failed to give the right hints because this is an unusual abnormality in cloudpickle (with python3.9) and common checks would not catch it. I think we should just fix this case in cloudpickle w/o adding special rules for inspect_serializability.

I can also help improving inspect_serializability in a general way to possibly figure out more issues about this kind of objects.

@moriyoshi we are bringing the upstream changes to the latest ray now

I don’t know the implementation detail of serialization in ray. Because ray.util.inspect_serializability reports True, so I think maybe I should report this in case there are other potential bugs.

I didn’t try their suggestion, but I guess that can work.

For your last question, using typing.NamedTuple is just my personal preference.

Hi @auderson, thank you for reporting. As you found out, this is not really an issue with ray but with cloudpickle. I am afraid there is not much to be done here. Did you try what they suggested in the issue with moving the class in its module?

Out of curiosity, why not using a dataclass or collections.namedtuple?