google-cloud-python: Getting 401 errors after one hour of client usage

So I was wrong at the bottom of https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3259 , this problem has not gone away for me. I’m starting a new Issue to track my particular problem here, instead of conflating things.

After my GAE Flex server (and datastore client) has been running for an hour, I get the following error message from my datastore usage (and the errors continue until I re-push my server and re-create clients):

Unauthorized: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

I am running google-auth==1.0.2 and google-cloud==0.27.0, though it’s the same error that’s happened on older versions of this client.

The exact error is:

  File "/home/vmagent/app/services/ip_geolocation.py", line 47, in _get_cache
    obj = client.get(_dbkey(ip))
  File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/client.py", line 309, in get
    deferred=deferred, transaction=transaction)
  File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/client.py", line 356, in get_multi
    transaction_id=transaction and transaction.id,
  File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/client.py", line 138, in _extended_lookup
    project, read_options, key_pbs)
  File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.py", line 115, in lookup
    return super(GAPICDatastoreAPI, self).lookup(*args, **kwargs)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.py", line 82, in _catch_remap_gax_error
    six.reraise(error_class, new_exc, sys.exc_info()[2])
  File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
    yield
  File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.py", line 115, in lookup
    return super(GAPICDatastoreAPI, self).lookup(*args, **kwargs)
  File "/env/local/lib/python2.7/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 204, in lookup
    return self._lookup(request, options)
  File "/env/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 452, in inner
    return api_caller(api_call, this_settings, request)
  File "/env/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 438, in base_caller
    return api_call(*args)
  File "/env/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 376, in inner
    return a_func(*args, **kwargs)
  File "/env/local/lib/python2.7/site-packages/google/gax/retry.py", line 127, in inner
    ' classified as transient', exception)
Unauthorized: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."   

The ip_geolocation.py code is here if you are curious. I basically construct a datastore.Client('dancedeets-hrd') at app startup, and re-use it on all requests thereafter. It works for an hour before getting it “breaks” and starts spewing 401s on every usage.

I had a previous workaround attempt, to construct a new client on every request. But this caused latency issues for me, as the overhead of the first db query seemed to take awhile.

I suppose the best workaround is to recreate the client everytime I time I get a 401, though obviously this feels like something the client should handle for me.