dvc: dvc: pretty slow on Mac

I experience around 10 seconds before and another 5 to 10 after dvc version output (below) on my Macbook Air.

DVC version: 0.61.2
Python version: 3.7.3
Platform: Darwin-18.7.0-x86_64-i386-64bit
Binary: False
Cache: reflink - True, hardlink - True, symlink - True
Filesystem type (cache directory): ('apfs', '/dev/disk1s1')
Filesystem type (workspace): ('apfs', '/dev/disk1s1')

Attached are a couple logs of python3 -mcProfile -s cumtime -m dvc version, the second one after opting out from analytics:

python3 -mcProfile -s cumtime -m dvc version (with analytics).log python3 -mcProfile -s cumtime -m dvc version (without analytics).log

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (16 by maintainers)

Commits related to this issue

lock: monkeypatch socket.getfqdn used by flufl.lock Workaround for slow and obsoleted gethostbyaddr used in `socket.getfqdn()`. See [1], [2] and [3] for more info. [1] https://bugs.python.org/issue5... — committed to efiop/dvc by efiop 5 years ago
lock: monkeypatch socket.getfqdn used by flufl.lock (#2628) Workaround for slow and obsoleted gethostbyaddr used in `socket.getfqdn()`. See [1], [2] and [3] for more info. [1] https://bugs.python... — committed to iterative/dvc by efiop 5 years ago
lock: skip using slow getaddrinfo This is basically Lock.__init__ copy-paste, except that instead of using `socket.getfqdn()` we use `socket.gethostname()` to speed this up. We've seen [1] `getfqdn()... — committed to efiop/dvc by efiop 5 years ago
lock: skip using slow getaddrinfo (#2642) This is basically Lock.__init__ copy-paste, except that instead of using `socket.getfqdn()` we use `socket.gethostname()` to speed this up. We've seen [1] ... — committed to iterative/dvc by efiop 5 years ago

Most upvoted comments

Ran that (with pip3 in my case) and it’s fast now! Under 1 sec to output dvc version and immediately back to the prompt. The installation did run into these errors though, pasting here in case it’s relevant:

ERROR: awscli 1.16.215 has requirement colorama<=0.3.9,>=0.2.5, but you'll have colorama 0.4.1 which is incompatible.
ERROR: docker-compose 1.23.2 has requirement requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1, but you'll have requests 2.22.0 which is incompatible.
ERROR: awsebcli 3.15.3 has requirement colorama<0.4.0,>=0.3.9, but you'll have colorama 0.4.1 which is incompatible.
ERROR: awsebcli 3.15.3 has requirement future<0.17.0,>=0.16.0, but you'll have future 0.18.1 which is incompatible.
ERROR: awsebcli 3.15.3 has requirement pathspec==0.5.9, but you'll have pathspec 0.6.0 which is incompatible.
ERROR: awsebcli 3.15.3 has requirement requests<2.21,>=2.20.1, but you'll have requests 2.22.0 which is incompatible.
ERROR: awsebcli 3.15.3 has requirement six<1.12.0,>=1.11.0, but you'll have six 1.12.0 which is incompatible.
ERROR: awsebcli 3.15.3 has requirement urllib3<1.25,>=1.24.1, but you'll have urllib3 1.25.6 which is incompatible.

jorgeorpinel on Oct 21, 2019

@jorgeorpinel Ok, this time this seems to be OS X fault for not caching dns requests https://github.com/rust-lang/rust/issues/31665 . Will prepare a workaround ASAP.

efiop on Oct 20, 2019

Nice! It’s fast now before output but still around 13 seconds to get back to the console prompt AFTER the command output, whether with or without analytics… Here’s the profiler output: https://pastebin.com/isBDYC4E

jorgeorpinel on Oct 20, 2019

@jorgeorpinel 0.63.3 is out. Please upgrade 🙂

efiop on Oct 18, 2019

For the record, submitted a PR https://gitlab.com/warsaw/flufl.lock/merge_requests/12 . @jorgeorpinel confirmed that it works for him now.

efiop on Oct 14, 2019

Thanks for looking at this Ruslan! Glad there’s a workaround we can use in lock.py so Mac users don’t have a bad experience with DVC (I guess most people don’t enable Remote login) even if that cpython bug doesn’t get addressed soon.

jorgeorpinel on Oct 14, 2019

Ok, so turns out on Mac “Computer name” that you set in Settings -> Sharing is not recognisable by anyone until you enable “Remote Login” service. So even ping Ruslans-MacBook-Pro.local will hang and then fail. The moment you enable that service, everything starts to work instantly. /etc/hosts is not updated by that operation, so not quite sure yet about the precise mechanism. But I am able to reproduce now and I am investigating.

efiop on Oct 14, 2019