google-api-python-client: Broken Pipe exceptions raised after idle periods
I’m using long-lived client objects like so:
client = discovery.build(....)
task = client.get_something(....).execute(num_retries=5)
some_long_running_process(task)
client.do_another_thing().execute(num_retries=5)
And running into exceptions on the calls to do_another_thing():
Traceback (most recent call last):
File "/opt/app/gcp_helper.py", line 168, in get_size
instanceGroupManager=instance_group).execute(num_retries=5)
File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/util.py", line 137, in positional_wrapper
return wrapped(*args, **kwargs)
File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/client.py", line 633, in new_request
self._refresh(request_orig)
File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/contrib/gce.py", line 129, in _refresh
META, headers={'Metadata-Flavor': 'Google'})
File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1273, in _conn_request
conn.request(method, request_uri, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1001, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 826, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe"
I would expect the num_retries argument to handle these exceptions in https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L145, but for the time being I have to implement my own retry logic around all API calls to overcome this.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 2
- Comments: 49 (17 by maintainers)
Commits related to this issue
- fix refresh bug + BrokenPipe fix attempt - was getting py3status crashes again with trying to refresh after losing an internet connection. fixed by moving self.last_update to _get_eve... — committed to igorgrebenkov/py3status by igorgrebenkov 7 years ago
- use httplib2shim The default connection pooling for httplib2 breask ling-running clients like ourselves. See: https://github.com/googleapis/google-api-python-client/issues/218 — committed to rjw57/winterpool-experiment by rjw57 6 years ago
- Fix for oauth2client exception, use google-auth library. Fix for exception: See also https://github.com/googleapis/google-api-python-client/issues/218#issuecomment-257918269 Traceback (most recent c... — committed to google/fuzzbench by inferno-chromium 4 years ago
- Fix for oauth2client exception, use google-auth library. (#742) Fix for exception: See also https://github.com/googleapis/google-api-python-client/issues/218#issuecomment-257918269 Traceback (mos... — committed to google/fuzzbench by inferno-chromium 4 years ago
- Make get_job() retry on transient errors for CAIP/uCAIP training. Unfortunately, this cannot completely replace the outer retry loop, which addresses an issue for long running client due to https://g... — committed to tensorflow/tfx by deleted user 3 years ago
- Make get_job() retry on transient errors for CAIP/uCAIP training. Unfortunately, this cannot completely replace the outer retry loop, which addresses an issue for long running client due to https://g... — committed to tensorflow/tfx by deleted user 3 years ago
- Make get_job() retry on transient errors for CAIP/uCAIP training. Unfortunately, this cannot completely replace the outer retry loop, which addresses an issue for long running client due to https://g... — committed to tensorflow/tfx by deleted user 3 years ago
I’m experiencing this issue as well…
Using:
Python 3.6.7Stack trace:
It’s something that happens now and then, not super common. Constantly recurring though.
As an example in a sample of 7,741,334 publishes I have 6 of those errors.
@Bobspadger probably not, but we’re soon replacing oauth2client in this library with google-auth, which will allow us to use urllib3 and requests without needing httplib2shim.
This is now happening to me via a Cloud Run execution that happens every 30 minutes. What’s the status of this fix?
Why was the issue closed when there is no solution or fix in place?
We have the same issue when periodically call status of long ai platform training job from CE instance. Pipe is broken after 1 hour (+/- minute) of using discovery for ml.v1 google-api-python-client==1.9.1
p.s. update to google.auth (1.24.0) did not help. The workaround is to recreate api client and retry on broken pipe exception
We are also experiencing this issue after upgrading to the latest version.
Would it be appropriate to re-open this ticket, or should I introduce a new one?
I get this with the Google drive API as well 😕
The same for me with ml engine train
Hi,
I’m having the same issue via Cloud Run also. I randomly get “Broken Pipe” or “connection reset by peer” errors. I am using the latest version of the Google API client so it seems this has not been fixed yet.
@nharpalani it’s looking unlikely that we’ll be able to replace httplib2 in this library without massive breaking changes. We’re working on a similar effort over on the google-cloud-python package and there’s a small chance that work might be transferrable here, but no promises.
Closing for now.
Sure