google-api-python-client: socket.timeout with will cause api client to become unuseable

We recently ran into a socket.timeout issue using the google api client, and noticed that then all our requests would fail in the same way. I later find that connections are cached here: https://github.com/httplib2/httplib2/blob/master/python3/httplib2/__init__.py#L1759

however there is no code to remove cached connections that are bad. This is a major oversight.

Here’s the callstack we hit:

    batch_entry.batch.execute(http=http, num_retries=num_retries)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 1465, in execute
    self._execute(http, self._order, self._requests, num_retries)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 1394, in _execute
    body=body, headers=headers)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 185, in _retry_request
    raise exception
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1514, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1264, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1217, in _conn_request
    response = conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
File "/usr/local/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

actually no, it’s post 0.12.0 (806de52a2939d95800e43b1ab96be45a33b882d1 vs 806de52a2939d95800e43b1ab96be45a33b882d1 for 0.12.0), supposedly 12.1. This should be re-opened

thehesiod on Dec 6, 2018

Hello, this error is happening multiple times a day.

The read operation timed out Traceback (most recent call last): File “”, line 197, in File “”, line 34, in get_results File “/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py”, line 130, in positional_wrapper return wrapped(*args, **kwargs) File “/usr/local/lib/python3.8/site-packages/googleapiclient/http.py”, line 849, in execute resp, content = _retry_request( File “/usr/local/lib/python3.8/site-packages/googleapiclient/http.py”, line 184, in _retry_request raise exception File “/usr/local/lib/python3.8/site-packages/googleapiclient/http.py”, line 165, in _retry_request resp, content = http.request(uri, method, *args, **kwargs) File “/usr/local/lib/python3.8/site-packages/oauth2client/transport.py”, line 173, in new_request resp, content = request(orig_request_method, uri, method, body, File “/usr/local/lib/python3.8/site-packages/oauth2client/transport.py”, line 280, in request return http_callable(uri, method=method, body=body, headers=headers, File “/usr/local/lib/python3.8/site-packages/httplib2/init.py”, line 1982, in request (response, content) = self._request( File “/usr/local/lib/python3.8/site-packages/httplib2/init.py”, line 1650, in _request (response, content) = self._conn_request( File “/usr/local/lib/python3.8/site-packages/httplib2/init.py”, line 1589, in _conn_request response = conn.getresponse() File “/usr/local/lib/python3.8/http/client.py”, line 1322, in getresponse response.begin() File “/usr/local/lib/python3.8/http/client.py”, line 303, in begin version, status, reason = self._read_status() File “/usr/local/lib/python3.8/http/client.py”, line 264, in _read_status line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”) File “/usr/local/lib/python3.8/socket.py”, line 669, in readinto return self._sock.recv_into(b) File “/usr/local/lib/python3.8/ssl.py”, line 1241, in recv_into return self.read(nbytes, buffer) File “/usr/local/lib/python3.8/ssl.py”, line 1099, in read return self._sslobj.read(len, buffer) socket.timeout: The read operation timed out Script failed.

Any suggestions how to get rid of it

Thanks

After hours of hacking api code, found the solution.

import socket
socket.setdefaulttimeout(4000) 


import argparse
import googleapiclient.discovery
from google.oauth2 import service_account
import mimetypes
import json
import sys
import os

Make note the order - socket.setdefaulttimeout(timeoutInSec) needs to be before any import of googleapiclient. No magic happens instead 😉

nulld on Dec 3, 2020

v0.12.1 of httplib2 has been released and contains the update to clear dead connections.

busunkim96 on Jun 10, 2019