dvc: get/import: could not perform a HEAD request

https://requests.kennethreitz.org/en/master/user/advanced/ says that session is using a connection pool by default. Chaning to using session instead of requests.request directly made everything work for me and I no longer see fluctuations in fd numbers. Will send a patch ASAP. Kudos @pared 🎉

Reproduction steps for Linux: script

#!/bin/bash

rm -rf repo
mkdir repo

cd repo

dvc init --no-scm
dvc import --rev 0547f58 \
           git@github.com:iterative/dataset-registry.git \
           use-cases/cats-dogs

Number of max connections here needs to be changed to some big amount. For me 10k worked.

@jorgeorpinel oops, sorry, wrong issue.

Little summary so far:

• The problem is too many file descriptors open upon remote/base.cache_exists
• Need to check why fetch/pull does not have the same problems as import/get

Possible way of handling the problem: The problem might be triggered because requests.sessions.Session object is created upon each requests.request calls. Maybe we could solve that by creating our own Session object, mounting proper HTTPAdapters and reusing this session, instead of calling requests.request each time.

It seems that the problem is that, with every request send, we are reserving socket “through” requests API, which is taking open file descriptor slot. In this particular case, in method RemoteLOCAL.cache_exists we try to paralelly do a lot of HEAD calls which leads to overcoming open file descriptors limit.

example: ulimit -n 16

and run:

from requests import sessions
from requests import head 
from concurrent.futures import ThreadPoolExecutor
import time
def run_session(i):
    try:
        head("https://www.google.com")
    except Exception as e:
        print(e)

with ThreadPoolExecutor(max_workers=24) as executor:
    args = [i for i in range(24)]
    executor.map(run_session, args)

print("finished")

Can reproduce on my mac, but not on linux

ERROR: failed to import 'use-cases/cats-dogs' from 'git@github.com:iterative/dataset-registry.git'. - could not perform a HEAD request
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x116945310>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='remote.dvc.org', port=443): Max retries exceeded with url: /dataset-registry/61/5bb7cebf1779b530f33b100d1f14b5 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x116945310>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/http.py", line 87, in _request
    return requests.request(method, url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='remote.dvc.org', port=443): Max retries exceeded with url: /dataset-registry/61/5bb7cebf1779b530f33b100d1f14b5 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x116945310>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/command/imp.py", line 20, in run
    rev=self.args.rev,
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/imp.py", line 6, in imp
    return self.imp_url(path, out=out, erepo=erepo, locked=True)
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/__init__.py", line 33, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/imp_url.py", line 25, in imp_url
    stage.run()
  File "/usr/local/lib/python3.7/site-packages/dvc/stage.py", line 861, in run
    self.deps[0].download(self.outs[0])
  File "/usr/local/lib/python3.7/site-packages/dvc/dependency/repo.py", line 77, in download
    out = self.fetch()
  File "/usr/local/lib/python3.7/site-packages/dvc/dependency/repo.py", line 72, in fetch
    repo.cloud.pull(out.get_used_cache())
  File "/usr/local/lib/python3.7/site-packages/dvc/data_cloud.py", line 81, in pull
    show_checksums=show_checksums,
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/local/__init__.py", line 412, in pull
    download=True,
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/local/__init__.py", line 376, in _process
    download=download,
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/local/__init__.py", line 301, in status
    md5s, jobs=jobs, name=str(remote.path_info)
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/base.py", line 738, in cache_exists
    ret = list(itertools.compress(checksums, in_remote))
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/base.py", line 731, in exists_with_progress
    ret = self.exists(path_info)
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/http.py", line 50, in exists
    return bool(self._request("HEAD", path_info.url))
  File "/usr/local/lib/python3.7/site-packages/dvc/remote/http.py", line 89, in _request
    raise DvcException("could not perform a {} request".format(method))
dvc.exceptions.DvcException: could not perform a HEAD request
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

@jorgeorpinel yes, please open a new UI issue!

@jorgeorpinel should it be use-cases/cats-dogs?