doit: 'get_var' fails while the multiprocess execution if any task uses delayed creation.

@schettino72 , hey! ๐Ÿ˜ƒ

So, i found strange behaviour while trying to implement some stuff.

#! /usr/bin/doit -f
# -*- coding: utf-8 -*-

from doit import get_var
from doit import create_after

get_var( 'A', None )


def task_foo() :
  return {
    'actions': [ 'echo foo' ],
    'task_dep': [ 'bar' ],
  }
  
@create_after( executed = 'baz' )
def task_bar() :
  for i in range( 10 ) :
    yield {
      'name': 'bar_{}'.format( i ),
      'actions': [ 'echo bar_{}'.format( i ) ]
    }
    
def task_baz() :
  for i in range( 10 ) :
    yield {
      'name': 'baz_{}'.format( i ),
      'actions': [ 'echo baz_{}'.format( i ) ]
    }

This code works fine while being executed in single-process mode. But it fails in multi process mode.

/root>GetVar.py -n 2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/py3.4/lib/multiprocessing/spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "/py3.4/lib/multiprocessing/spawn.py", line 116, in _main
    self = pickle.load(from_parent)
  File "/root/GetVar.py", line 7, in <module>
    get_var( 'A', None )
  File "/py3.4/lib/site-packages/doit/doit_cmd.py", line 35, in get_var
    return _CMDLINE_VARS.get(name, default)
AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/py3.4/lib/multiprocessing/spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "/py3.4/lib/multiprocessing/spawn.py", line 116, in _main
    self = pickle.load(from_parent)
  File "/root/GetVar.py", line 7, in <module>
    get_var( 'A', None )
  File "/py3.4/lib/site-packages/doit/doit_cmd.py", line 35, in get_var
    return _CMDLINE_VARS.get(name, default)
AttributeError: 'NoneType' object has no attribute 'get'

But if i remove the create_after decorator, all became works fine even in multi process mode. So, i found it ambiguous. I had seen no notifications in docs about such behaviour.

Can you confirm this is the doit error or not? Currently i use the last (0.30.0) version of doit.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Ah, yes, that makes sense, thanks ๐Ÿ˜ƒ See the PR. Not sure if this is the optimal way to test itโ€ฆ

The OPโ€™s test file works now with multiple processes.