ray: bad warning - AttributeError: 'NoneType' object has no attribute 'global_worker'

What is the problem?

After launching the dcgan sample on local machine, the sample finishes all epochs and failed with the following log from each worker:

Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'

Full log:

$ python dcgan.py --num-workers 44
2020-04-13 16:48:16,768 INFO resource_spec.py:204 -- Starting Ray with 120.17 GiB memory available for workers and up to 55.5 GiB for objects. You can adjust these settings with ray.init(mem
ory=<bytes>, object_store_memory=<bytes>).
2020-04-13 16:48:17,201 INFO services.py:1146 -- View the Ray dashboard at localhost:8265
  num_samples    epoch    batch_count    loss_g    last_loss_g    loss_d    last_loss_d    inception    last_inception
-------------  -------  -------------  --------  -------------  --------  -------------  -----------  ----------------
        60016        1            124   2.50279        2.34154   1.69439        1.71655      1.85531           2.09533
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:17<00:00, 77.08s/epoch, loss_g=2.5, loss_d=1.69]
1/1e: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 124/124 [01:17<00:00,  1.61batch/sE
xception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x7fc8e9ffae50>
Traceback (most recent call last):
  File "/rpovelik/installed/miniconda3/envs/ray/lib/python3.8/site-packages/ray/actor.py", line 657, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'

Ray version and other system information (Python version, TensorFlow version, OS): ray 0.8.4 Torch 1.4.0 Ubuntu 18.04

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

$ python dcgan.py --num-workers 44

If we cannot run your script, we cannot fix your issue.

  • I have verified my script runs in a clean environment and reproduces the issue.
  • I have verified the issue also occurs with the latest wheels.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (4 by maintainers)

Most upvoted comments

I can reproduce this in Python 3.8.6 and ray 1.4.0 with simple code like below

import ray

ray.init()


@ray.remote
class Counter(object):
    def __init__(self):
        self.n = 0

    def increase(self):
        self.n += 1

    def read(self):
        return self.n


counters = [Counter.remote() for i in range(4)]
[c.increase.remote() for c in counters]
futures = [c.read.remote() for c in counters]

print(ray.get(futures))

log below

Exception ignored in: <function ActorHandle.__del__ at 0x114c71a60>
Traceback (most recent call last):
  File "/Users/ivan/.pyenv/versions/ray-side/lib/python3.8/site-packages/ray/actor.py", line 823, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x114c71a60>
Traceback (most recent call last):
  File "/Users/ivan/.pyenv/versions/ray-side/lib/python3.8/site-packages/ray/actor.py", line 823, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x114c71a60>
Traceback (most recent call last):
  File "/Users/ivan/.pyenv/versions/ray-side/lib/python3.8/site-packages/ray/actor.py", line 823, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
Exception ignored in: <function ActorHandle.__del__ at 0x114c71a60>
Traceback (most recent call last):
  File "/Users/ivan/.pyenv/versions/ray-side/lib/python3.8/site-packages/ray/actor.py", line 823, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'
+3
email2liyang on Jun 13, 2021

Yeah this is a harmless error. However, it is misleading and is due to a race condition on our side.

+3
richardliaw on Feb 7, 2021

Is this issue solved? I struggling with same issue

import ray
from ray import serve
from fairseq.models.transformer import TransformerModel
import requests

class FairseqModel:
    def __init__(self):
        self.model = TransformerModel.from_pretrained("""My model""")
    def __call__(self, request):        
        return self.model.translate(str(request.json["data"]))

ray.init()
client = serve.start(http_host="0.0.0.0", detached=True)

client.create_backend("fairseq_model", FairseqModel)
client.create_endpoint(
    "my_endpoint",
    backend="fairseq_model",
    route="/translate",
    methods=["POST"])

# Query it!
sample_request_input = {"data": "sample input"}
response = requests.post("http://0.0.0.0:8000/translate", json=sample_request_input)
print(response.text)

Full log

2021-02-03 06:10:43,121 INFO services.py:1090 -- View the Ray dashboard at http://127.0.0.1:8265
2021-02-03 06:10:43,125 WARNING services.py:1552 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67104768 bytes available. This may slow down performance! You may be able to free up space by deleting files in /dev/shm or terminating any running plasma_store_server processes. If you are inside a Docker container, you may need to pass an argument with the flag '--shm-size' to 'docker run'.
(pid=23234) 2021-02-03 06:10:44,630     INFO controller.py:191 -- Starting router with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-node:172.17.0.5-0' on node 'node:172.17.0.5-0' listening on '0.0.0.0:8000'
(pid=23253) 2021-02-03 06:10:44,766     WARNING worker.py:1428 -- Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(pid=23253) INFO:     Started server process [23253]
(pid=23243) normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.
(pid=23234) 2021-02-03 06:10:50,393     INFO controller.py:702 -- Registering route '/translate' to endpoint 'my_endpoint' with methods '['POST']'.
#translation of sample input
Exception ignored in: <function ActorHandle.__del__ at 0x7efc3083d0d0>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/ray/actor.py", line 753, in __del__
AttributeError: 'NoneType' object has no attribute 'global_worker'

Ray version and other system information (Python version, TensorFlow version, OS): ray 1.1.0 fairseq 0.10.2 python 3.8.3 Ubuntu 18.0.4

+1
dg4271 on Feb 3, 2021
Read more comments on GitHub
← ray: Bad performance of LSTM policies with PPO
ray: [Bug] [Autoscaler] Autoscaler can start unnecessary nodes due to poor bin packing heuristic ordering →