google-auth-library-python: Intermittent DefaultCredentialsError on GCE

I would love to see a fix for this issue

I’m still seeing this error fairly regularly running on GAE flex for Python 3.6 with google-auth==1.6.3. Here’s the full stack trace:

  ...
  File "/env/lib/python3.6/site-packages/google/auth/transport/requests.py", line 205, in request
    self._auth_request, method, url, request_headers)
  File "/env/lib/python3.6/site-packages/google/auth/credentials.py", line 122, in before_request
    self.refresh(request)
  File "/env/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 102, in refresh
    six.raise_from(new_exc, caught_exc)
  File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/myst-ai-crystal@appspot.gserviceaccount.com/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9964d74080>: Failed to establish a new connection: [Errno 111] Connection refused',))

@GEverding the recommendation right now is to use a service account keyfile instead of relying on the GCE metadata service.

It’s possible that we could make retry failed connections to the metadata service, but I’m unsure on that at the moment.

We see this issue in our setup as well. Python script connecting to BQ and running hundreds of queries in rapid fire sequence. We’ll occasionally see: A INFO:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable.

followed by a crash: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credential and re-run the application.

Also from @dmho418 (may be unrelated to this issue, but is probably the root cause):

...
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/upload.py", line 97, in transmit
    retry_strategy=self._retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
    func, RequestsMixin._get_status_code, retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
    response = func()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/compute_engine/credentials.py", line 93, in refresh
    raise exceptions.RefreshError(exc)
RuntimeError: RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true

@mike-seekwell Thanks for the call out! #323 merged a fix to retry the ping to the metadata server.

If you’re seeing this error, please upgrade to version 1.6.3 or greater.

Is this something that I’ll need to push up a keyfile with my flex deployment (as @vanpelt mentioned), or will it be fixed in the PR that was just linked?

@vanpelt yep, that is a completely acceptable approach.