fiftyone: [BUG] Failed to bind to port with PyTorch DistributedDataParallel

System information

  • Ubuntu 18.04:
  • FiftyOne installed from pip:
  • FiftyOne version v0.13.2:
  • Python 3.6.9 (virtual env):

Commands to reproduce

python -u -m torch.distributed.launch --nproc_per_node=4 test.py

Describe the problem

FiftyOne can’t bind ports after calling the script with PyTorch DistributedDataParallel. There are no problems with a single GPU, but with 2 and more GPUs script terminating instantly and very rare works without issues. Probably it’s directly related to the multi-process parallelism of DistributedDataParallel.

Other info / logs

*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Process 16911 (service/main.py --51-service db --multi) did not respond
{"t":{"$date":"2021-09-27T06:21:42.600Z"},"s":"I",  "c":"CONTROL",  "id":20697,   "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log","newLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log.2021-09-27T06-21-42"}}
Subprocess ['/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/db/bin/mongod', '--dbpath', '/home/ubuntu/.fiftyone/var/lib/mongo', '--logpath', '/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log', '--port', '0', '--nounixsocket'] exited with error 100:
{"t":{"$date":"2021-09-27T06:21:42.609Z"},"s":"I",  "c":"CONTROL",  "id":20697,   "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log","newLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log.2021-09-27T06-21-42"}}
Subprocess ['/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/db/bin/mongod', '--dbpath', '/home/ubuntu/.fiftyone/var/lib/mongo', '--logpath', '/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log', '--port', '0', '--nounixsocket'] exited with error 100:
{"t":{"$date":"2021-09-27T06:21:42.627Z"},"s":"I",  "c":"CONTROL",  "id":20697,   "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log","newLogPath":"/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log.2021-09-27T06-21-42"}}
Subprocess ['/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/db/bin/mongod', '--dbpath', '/home/ubuntu/.fiftyone/var/lib/mongo', '--logpath', '/home/ubuntu/.fiftyone/var/lib/mongo/log/mongo.log', '--port', '0', '--nounixsocket'] exited with error 100:
Uncaught exception
Traceback (most recent call last):
  File "train_test.py", line 7, in <module>
    from data_loader_fletch import create_data_loaders
  File "/home/ubuntu/LaboroTrainingOpenImages/data_loader_fletch.py", line 1, in <module>
    import fiftyone as fo
  File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/__init__.py", line 25, in <module>
    from fiftyone.__public__ import *
  File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/__public__.py", line 14, in <module>
    foo.establish_db_conn(config)
  File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/odm/database.py", line 77, in establish_db_conn
    port = _db_service.port
  File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/service.py", line 295, in port
    return self._wait_for_child_port()
  File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/service.py", line 179, in _wait_for_child_port
    return find_port()
  File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/home/ubuntu/OID/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
  File "/home/ubuntu/OID/lib/python3.6/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/ubuntu/OID/lib/python3.6/site-packages/fiftyone/core/service.py", line 177, in find_port
    raise ServiceListenTimeout(etau.get_class_name(self), port)
fiftyone.core.service.ServiceListenTimeout: fiftyone.core.service.DatabaseService failed to bind to port

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

  • Yes. I can contribute a fix for this bug independently.
  • Yes. I would be willing to contribute a fix for this bug with guidance from the FiftyOne community.
  • No. I cannot contribute a bug fix at this time.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 23 (15 by maintainers)

Most upvoted comments

Putting this here now for those that need it. An easy way of getting around this issue is by sequentially opening database connections after launching the processes. Here is a pseudocode snippet:

for i in range(num_processes):
    if rank == i:
        # load the dataset
        dataset = fo.load_dataset(dataset_name)
    torch.distributed.barrier()

In a standard single node, multi proc distributed data parallel run instantiated with torchrun, the environment variables WORLD_SIZE and RANK will have the values corresponding to num_processes and rank.

All of this will come in a neat and handy package (allowing you to turn any FiftyOne Dataset into a torch dataset with just 1 function) with https://github.com/voxel51/fiftyone/pull/5321. Stay tuned 😃

@jacobsela perfect, just add Resolves #1302 to the PR description of #5321 and it will automatically close this issue when the PR is merged ✅

No update yet. import fiftyone currently always opens a database connection in the main process, and those connections can’t be serialized. It seems we’d have to find a way to omit the connection from the serialization.

Let’s leave it open to remind us to fix it properly 😃