TensorNetwork: conflict between pickle and thus multiprocessing with backend module assignment
In the current implementations of backends, we have the following thing:
class SomeBackend():
def __init__(self):
self.np = numpy
Note numpy is a module, and of course such backend is included as one attr of tn.Node.
The effect of such design is you will fail when you try:
from multiprocessing import pool
def f(x):
return tn.Node(np.eye(x))
with Pool(3) as p:
print(p.map(f, [1, 2, 3]))
the error is can't pickle module objects which is induced ultimately by numpy attr in backend class. This is due to the lack of ability to pickle modules in python. To verify this, one can either comment self.backend=backend in network_components or change all self.numpy to plain numpy in numpy_backend py. There will be no error for both modifications.
It is worth noting that other multiprocessing libraries such as pathos has no problem to interact with TensorNetwork as its core pickle library is dill which is more powerful. However, you may want to interact TN with other libraries with python builtin multiprocessing baked in. In such cases, TN fails the task.
Maybe we should just use plain np, tf in backend files instead of self.np to overcome this?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 23 (18 by maintainers)
Yes as of the latest release that I literally just pushed lol. https://github.com/google/TensorNetwork/releases/tag/0.3.1