google-api-python-client: Google Drive Batch Request Error: socket.timeout: The read operation timed out
Possibly related to #563
Occasionally when executing a batch request using the Python client library callingdrive.files().copy
the operation fails with the error socket.timeout: The read operation timed out
Environment details
Running Python 3.7 on Google Cloud Functions environment
Relevant Code
def copy_files(access_token, file_ids, folder_ids, labels):
def copy_callback(request_id, response, exception):
if exception:
print(exception)
else:
print(response)
creds = AccessTokenCredentials(access_token, 'my-user-agent/1.0')
drive_service = build('drive', 'v3', credentials=creds)
move_files = drive_service.new_batch_http_request(callback=copy_callback)
for label, id in zip(labels, file_ids):
move_files.add(
drive_service.files().copy(fileId=id, body={"parents": [folder_ids[label]]})
)
move_files.execute()
Error
Traceback (most recent call last):
File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 313, in run_http_function
result = _function_handler.invoke_user_function(flask.request)
File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 215, in invoke_user_function
return call_user_function(request_or_event)
File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 208, in call_user_function
return self._user_function(request_or_event)
File "/user_code/main.py", line 19, in sort
request_json['labels'])
File "/user_code/drive.py", line 51, in copy_files
move_files.execute()
File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/http.py", line 1450, in execute
self._execute(http, self._order, self._requests)
File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/http.py", line 1382, in _execute
headers=headers)
File "/env/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
redirections, connection_type)
File "/env/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1926, in request
cachekey,
File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1595, in _request
conn, request_uri, method, body, headers
File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1533, in _conn_request
response = conn.getresponse()
File "/opt/python3.7/lib/python3.7/http/client.py", line 1321, in getresponse
response.begin()
File "/opt/python3.7/lib/python3.7/http/client.py", line 296, in begin
version, status, reason = self._read_status()
File "/opt/python3.7/lib/python3.7/http/client.py", line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/opt/python3.7/lib/python3.7/ssl.py", line 1052, in recv_into
return self.read(nbytes, buffer)
File "/opt/python3.7/lib/python3.7/ssl.py", line 911, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
Thanks!
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (6 by maintainers)
Commits related to this issue
- Moving away from batch requests due to https://github.com/googleapis/google-api-python-client/issues/632 — committed to ilpersi/drive-folder-copier by ilpersi 5 years ago
- Update: 1.0.7 - Fix "The read operation timed out" Source: https://github.com/googleapis/google-api-python-client/issues/632 Workaround: https://github.com/googleapis/google-api-python-client/issues/... — committed to jagrit007/Telegram-CloneBot by jagrit007 4 years ago
Thank you @AlJohri for linking that issue! 😃
The default timeout 60 seconds, which is probably too short for many batch requests. It makes sense that the behavior would go away when batches were made smaller (smaller batches -> faster requests).
https://github.com/googleapis/google-api-python-client/blob/f0b025b95b2fb5fd0369d7233614e35b9c86af30/googleapiclient/http.py#L1777-L1792
I am closing this issue now, but please do open a new issue if you are still experiencing problems.
Hiya, I have been seeing this error when I run reports for Google Analytics, a lot during 11am to 3pm PDT yesterday & just today again so far at 3am PDT. Is there a solution to this so far? I also updated to the 1.7.9 version on all forms.
There are a few other support channels that might be worth reaching out to:
https://cloud.google.com/functions/docs/getting-support
@hsyyid I agree with you and in the I came to the same conclusion: my current program is not using batch requests anymore.
The size hint is a nice one as it makes it easier to reproduce this issue.