selenium: Regression in all versions >= 3.8.1 Non-deterministic ConnectionResetError with chrome and chromedriver

I’ve set up a repo to stress test the Python driver to reproduce it: https://github.com/kevlened/python-webdriver-stress-test

I used ChromeDriver 2.38.552518 and Chrome 66.0.3359.139, but I think the error is seen due to a change in the python client.

Between 3.8.0 and 3.8.1, this change went in. I haven’t dug into why this causes the issue (it should only open a socket briefly), but reverting it fixes the problem.

Reverting it also fixes these other errors I was seeing in chromedriver:

Starting ChromeDriver 2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011) on port 9515
Only local connections are allowed.
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)

That makes me think that common_utils.is_connectable() isn’t properly closing sockets.

I’ve got exactly same issue as @bySabi , using Selenium 3.13.0 with chromedriver seems to work fine on my windows machine, but when I use docker it has [error 104]connection reset by peer problem, and downgrading to 3.8.0 resolves this problem. Thanks to the solution, since I got stuck on this problem for hours.

Here is my dockerfile of installing chromedriver

FROM python:3.6-alpine

# update apk repo
RUN echo "http://dl-4.alpinelinux.org/alpine/v3.7/main" >> /etc/apk/repositories && \
    echo "http://dl-4.alpinelinux.org/alpine/v3.7/community" >> /etc/apk/repositories

# install chromedriver
RUN apk update
RUN apk add chromium chromium-chromedriver

and complete error code

 Traceback (most recent call last):
   File "/usr/local/bin/celery", line 11, in <module>
     sys.exit(main())
   File "/usr/local/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
     _main()
   File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
     cmd.execute_from_commandline(argv)
   File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
     super(CeleryCommand, self).execute_from_commandline(argv)))
   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 273, in execute_from_commandline
     argv = self.setup_app_from_commandline(argv)
   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 479, in setup_app_from_commandline
     self.app = self.find_app(app)
   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 501, in find_app
     return find_app(app, symbol_by_name=self.symbol_by_name)
   File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 359, in find_app
     sym = symbol_by_name(app, imp=imp)
   File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 504, in symbol_by_name
     return imports.symbol_by_name(name, imp=imp)
   File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
     module = imp(module_name, package=package, **kwargs)
   File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 104, in import_from_cwd
     return imp(module, package=package)
   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
   File "/app/xiaohongshu_sel.py", line 9, in <module>
     browser = webdriver.Chrome(options=options)
   File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
     desired_capabilities=desired_capabilities)
   File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 156, in __init__
     self.start_session(capabilities, browser_profile)
   File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 251, in start_session
     response = self.execute(Command.NEW_SESSION, parameters)
   File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 318, in execute
     response = self.command_executor.execute(driver_command, params)
   File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 472, in execute
     return self._request(command_info[0], url, body=data)
   File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 496, in _request
     resp = self._conn.getresponse()
   File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
     response.begin()
   File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
     version, status, reason = self._read_status()
   File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
   File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
     return self._sock.recv_into(b)
 ConnectionResetError: [Errno 104] Connection reset by peer

same problem for:

• selenium 3.11.0
• chromium (64.0.3282.168-r0)
• chromium-chromedriver (64.0.3282.168-r0)

for a code:

from selenium import webdriver

BEHAVE_DEBUG_ON_ERROR = False


def before_all(context):
    options = webdriver.ChromeOptions()
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    context.browser = webdriver.Chrome(chrome_options=options)

here are the logs:

Exception ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/bin/behave", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/behave/__main__.py", line 109, in main
    failed = runner.run()
  File "/usr/lib/python3.6/site-packages/behave/runner.py", line 672, in run
    return self.run_with_paths()
  File "/usr/lib/python3.6/site-packages/behave/runner.py", line 693, in run_with_paths
    return self.run_model()
  File "/usr/lib/python3.6/site-packages/behave/runner.py", line 471, in run_model
    self.run_hook('before_all', context)
  File "/usr/lib/python3.6/site-packages/behave/runner.py", line 405, in run_hook
    self.hooks[name](context, *args)
  File "features/environment.py", line 10, in before_all
    context.browser = webdriver.Chrome(chrome_options=options)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
    desired_capabilities=desired_capabilities)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 310, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 466, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 490, in _request
    resp = self._conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

With Selenium 3.13.0 and chromedriver its working fine on macos host but fail inside a docker container. Downgrading Selenium to 3.8.0 in docker seems its working

Turns out that under this stress test, the errors are also raised with 3.8.0, so I think the errors I’m seeing may be a different issue.

The Node webdriver implementation retries requests 3 times on these types of connection errors. I can open a PR to handle the errors in a similar way, but it would simply mask the error.

@chadawagner I’m curious what happens if instead of a delay, you add socket_.shutdown(socket.SHUT_RDWR) before it. The docs here say:

Note: close() releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call shutdown() before close().