django-q: TypeError: can't pickle _thread.lock objects

Django 2.2.11 python 3.7.0 django-q 1.2.1 windows 10

Hello, when i run manage.py qcluster i get error, does somebody know what could be source of it and how to resolve it?

  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "C:\Users\Mateusz\Desktop\project\env\lib\site-packages\django\core\management\__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "C:\Users\Mateusz\Desktop\project\env\lib\site-packages\django\core\management\__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\Mateusz\Desktop\project\env\lib\site-packages\django\core\management\base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Users\Mateusz\Desktop\project\env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "C:\Users\Mateusz\Desktop\project\env\lib\site-packages\django_q\management\commands\qcluster.py", line 22, in handle
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Mateusz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Mateusz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
    q.start()
  File "C:\Users\Mateusz\Desktop\project\env\lib\site-packages\django_q\cluster.py", line 65, in start
    self.sentinel.start()
  File "C:\Users\Mateusz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\Mateusz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Mateusz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\Mateusz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Mateusz\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 48 (7 by maintainers)

Commits related to this issue

Merge pull request #482 from ihuk/master Fix for #424 — committed to Koed00/django-q by Koed00 4 years ago
Install Django Q directly from GitHub This fixes Koed00/django-q#424. As soon as the PR (#482) is incorporated in a new release, the official release can be used again — committed to eckynde/sprinklercontrol by jannikbook 4 years ago
Install Django Q directly from GitHub This fixes Koed00/django-q#424. As soon as the PR (#482) is incorporated in a new release, the official release can be used again — committed to eckynde/sprinklercontrol by jannikbook 4 years ago

Most upvoted comments

I didn’t see issue #389 before, so I’m also commenting in this thread as it may help others. Mentioned issue provided a workaround for me. I ended up adding the following snippet to my manage.py and it works 😃

import platform
if platform.system() != "Linux":
    from multiprocessing import set_start_method
    set_start_method("fork")

My setup is macOS 10.15.6, django-q 1.3.3 with redis, python 3.8.5

jeroenbrouwer on Sep 26, 2020

I tested this out on Windows Py 3.8, it doesn’t work. The issue is that the broker connection object is unpickleable, possibly due to the network connection being process local. In spawn context multiprocessing (MacOS/Windows) new processes are not child processes like in Linux, they’re separate. You should be able to test this environment in Linux with multiprocessing.set_start_method('spawn')

from django_q.brokers import get_broker
import pickle
pickle.dumps(get_broker().connection)
# TypeError: cannot pickle '_thread.lock' object

Solanar on Mar 18, 2020

Awesome job - I still have some PR’s to review but I’ll try to get this out soon.

Koed00 on Oct 21, 2020

Merged to master - does anyone want to give the master branch a try to see if it fixes the issue for them?

Koed00 on Oct 20, 2020

The problem lies with the shared Value object the workers keep around as timers. Those were made to be ‘process safe’ by adding locks. In theory this should just work fine on all platforms, but I guess some minor chages were added in the newer Pythons.

It’s very hard for me to test, cause I have neither MacOs or Windows as a development environment, and this will probably one of those cases where the python documentation hasn’t kept up yet and I will just have to try stuff.

That said; there is some inidication that setting the lock type explicitly on the ctype instance might help. I’ve done this in the lock branch here : a1f211d

Unless someone wants to help out, it will be a while before I can test this on one of the afflicted OS’s.

Hello, @Koed00 , found interesting tool for Linux users which let you test Django-Q on Mac, maybe you will be interested in - https://github.com/fastai/fastmac

NotSoSmartDev on Sep 12, 2020

Workaround: Develop in WSL, it works good there: https://code.visualstudio.com/docs/remote/wsl

09:34:39 [Q] INFO Q Cluster dakota-kilo-zebra-monkey starting.
09:34:39 [Q] INFO Process-1:1 ready for work at 663
09:34:39 [Q] INFO Process-1:2 ready for work at 664
09:34:39 [Q] INFO Process-1:3 ready for work at 665
09:34:39 [Q] INFO Process-1:4 ready for work at 666
09:34:39 [Q] INFO Process-1:5 monitoring at 667
09:34:39 [Q] INFO Process-1 guarding cluster dakota-kilo-zebra-monkey
09:34:39 [Q] INFO Q Cluster dakota-kilo-zebra-monkey running.
09:34:39 [Q] INFO Process-1:6 pushing tasks at 668

faulander on Jun 17, 2020