fiftyone: [BUG] Failed to bind to port with PyTorch DistributedDataParallel
System information
- Ubuntu 18.04:
- FiftyOne installed from pip:
- FiftyOne version v0.13.2:
- Python 3.6.9 (virtual env):
Commands to reproduce
python -u -m torch.distributed.launch --nproc_per_node=4 test.py
Describe the problem
FiftyOne can’t bind ports after calling the script with PyTorch DistributedDataParallel. There are no problems with a single GPU, but with 2 and more GPUs script terminating instantly and very rare works without issues. Probably it’s directly related to the multi-process parallelism of DistributedDataParallel.
Other info / logs
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Process 16911 (service/main.py --51-service db --multi) did not respond
{"t":{"$date":"2021-09-27T06:21:42.600Z"},"s":"I", "c":"CONTROL", "id":20697, "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log","newLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log.2021-09-27T06-21-42"}}
Subprocess ['/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/db/bin/mongod', '--dbpath', '/home/ubuntu/.fiftyone/var/lib/mongo', '--logpath', '/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log', '--port', '0', '--nounixsocket'] exited with error 100:
{"t":{"$date":"2021-09-27T06:21:42.609Z"},"s":"I", "c":"CONTROL", "id":20697, "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log","newLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log.2021-09-27T06-21-42"}}
Subprocess ['/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/db/bin/mongod', '--dbpath', '/home/ubuntu/.fiftyone/var/lib/mongo', '--logpath', '/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log', '--port', '0', '--nounixsocket'] exited with error 100:
{"t":{"$date":"2021-09-27T06:21:42.627Z"},"s":"I", "c":"CONTROL", "id":20697, "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log","newLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log.2021-09-27T06-21-42"}}
Subprocess ['/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/db/bin/mongod', '--dbpath', '/home/ubuntu/.fiftyone/var/lib/mongo', '--logpath', '/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log', '--port', '0', '--nounixsocket'] exited with error 100:
Uncaught exception
Traceback (most recent call last):
File "train_test.py", line 7, in <module>
from data_loader_fletch import create_data_loaders
File "/home/ubuntu/LaboroTrainingOpenImages/data_loader_fletch.py", line 1, in <module>
import fiftyone as fo
File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/__init__.py", line 25, in <module>
from fiftyone.__public__ import *
File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/__public__.py", line 14, in <module>
foo.establish_db_conn(config)
File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/odm/database.py", line 77, in establish_db_conn
port = _db_service.port
File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/service.py", line 295, in port
return self._wait_for_child_port()
File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/service.py", line 179, in _wait_for_child_port
return find_port()
File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 212, in call
raise attempt.get()
File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/home/ubuntu/OID/lib/python3.6/site-packages/six.py", line 719, in reraise
raise value
File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/service.py", line 177, in find_port
raise ServiceListenTimeout(etau.get_class_name(self), port)
fiftyone.core.service.ServiceListenTimeout: fiftyone.core.service.DatabaseService failed to bind to port
Willingness to contribute
The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?
- Yes. I can contribute a fix for this bug independently.
- Yes. I would be willing to contribute a fix for this bug with guidance from the FiftyOne community.
- No. I cannot contribute a bug fix at this time.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 23 (15 by maintainers)
Putting this here now for those that need it. An easy way of getting around this issue is by sequentially opening database connections after launching the processes. Here is a pseudocode snippet:
In a standard single node, multi proc distributed data parallel run instantiated with
torchrun, the environment variablesWORLD_SIZEandRANKwill have the values corresponding tonum_processesandrank.All of this will come in a neat and handy package (allowing you to turn any FiftyOne Dataset into a torch dataset with just 1 function) with https://github.com/voxel51/fiftyone/pull/5321. Stay tuned 😃
@jacobsela perfect, just add
Resolves #1302to the PR description of #5321 and it will automatically close this issue when the PR is merged ✅No update yet.
import fiftyonecurrently always opens a database connection in the main process, and those connections can’t be serialized. It seems we’d have to find a way to omit the connection from the serialization.Let’s leave it open to remind us to fix it properly 😃