pyrollbar: SSL/TLS error in Django connecting to api.rollbar.com on exceptions

This error happens for about 2 out of every 3 requests to send an exception to rollbar. It occasionally works, but mostly doesn’t. The following stack trace is printed only occasionally. Sometimes it will fail with no exception stack trace.

Traceback (most recent call last):
   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
     chunked=chunked)
   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
     self._validate_conn(conn)
   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
     conn.connect()
   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
     ssl_context=context)
   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
     return context.wrap_socket(sock, server_hostname=server_hostname)
   File "/usr/lib/python3.6/ssl.py", line 401, in wrap_socket
     _context=self, _session=session)
   File "/usr/lib/python3.6/ssl.py", line 808, in __init__
     self.do_handshake()
   File "/usr/lib/python3.6/ssl.py", line 1061, in do_handshake
     self._sslobj.do_handshake()
   File "/usr/lib/python3.6/ssl.py", line 683, in do_handshake
     self._sslobj.do_handshake()
 ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:749)

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
     timeout=timeout
   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
     _stacktrace=sys.exc_info()[2])
   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
     raise MaxRetryError(_pool, url, error or ResponseError(cause))
 urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.rollbar.com', port=443): Max retries exceeded with url: /api/1/item/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:749)'),))

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/usr/lib/python3.6/site-packages/rollbar/__init__.py", line 1209, in _send_payload
     _post_api('item/', payload, access_token=access_token)
   File "/usr/lib/python3.6/site-packages/rollbar/__init__.py", line 1255, in _post_api
     verify=SETTINGS.get('verify_https', True))
   File "/usr/lib/python3.6/site-packages/rollbar/lib/transport.py", line 7, in post
     return _session.post(*args, **kw)
   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 555, in post
     return self.request('POST', url, data=data, json=json, **kwargs)
   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='api.rollbar.com', port=443): Max retries exceeded with url: /api/1/item/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:749)'),))

Python 3.6.1 Django 1.11.1 uwsgi 2.0.14 requests 2.18.4 open-ssl 1.0.2k 26 Jan 2017 Docker Alpine Linux

Please let me know if I can provide more info. It seems like some kind of race condition, but not sure, since it sometimes works. Maybe it’s a thread safety issue?

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 2
Comments: 17 (2 by maintainers)

Most upvoted comments

Mentioned in #180, but we’re running django version 2.1.2 and rollbar version 0.14.5 and are still seeing a very similar traceback as mentioned above:

[ERROR]	2019-09-17T04:19:55.506Z	d8024161-f0a6-4e42-b421-dbdf8ad4cb6c	Exception while posting item SSLError(MaxRetryError("HTTPSConnectionPool(host='api.rollbar.com', port=443): Max retries exceeded with url: /api/1/item/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))",),)
Traceback (most recent call last):
  File "/var/task/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/var/task/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/var/task/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/var/task/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/var/task/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/var/lang/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/var/lang/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/var/lang/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/var/lang/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/var/task/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/var/task/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.rollbar.com', port=443): Max retries exceeded with url: /api/1/item/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/rollbar/__init__.py", line 1336, in _send_payload
    _post_api('item/', payload_str, access_token=access_token)
  File "/var/task/rollbar/__init__.py", line 1385, in _post_api
    proxy_password=SETTINGS.get('http_proxy_password'))
  File "/var/task/rollbar/lib/transport.py", line 33, in post
    return _session().post(*args, proxies=proxies, **kw)
  File "/var/task/requests/sessions.py", line 555, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/var/task/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/task/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/var/task/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.rollbar.com', port=443): Max retries exceeded with url: /api/1/item/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))

Other useful context might be that this code is executing in an AWS lambda function environment

mjhanke on Sep 18, 2019