google-cloud-python: Invalid JWT Token when using Service Account JSON

  1. OS type and version

macOS Sierra running a Debian Jessie Docker Container

  1. Python version and virtual environment information python --version

CPython 3.5.0, no virtual environment

  1. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze

google-cloud-bigquery==0.23.0

  1. Stacktrace if available
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/app/warehouse/celery.py", line 53, in __call__
    return super().__call__(pyramid_env["request"], *args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/app/warehouse/packaging/tasks.py", line 58, in compute_trending
    query.run()
  File "/usr/local/lib/python3.5/site-packages/google/cloud/bigquery/query.py", line 364, in run
    method='POST', path=path, data=self._build_resource())
  File "/usr/local/lib/python3.5/site-packages/google/cloud/_http.py", line 299, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/local/lib/python3.5/site-packages/google/cloud/_http.py", line 193, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/local/lib/python3.5/site-packages/google/cloud/_http.py", line 223, in _do_request
    body=data)
  File "/usr/local/lib/python3.5/site-packages/google_auth_httplib2.py", line 187, in request
    self._request, method, uri, request_headers)
  File "/usr/local/lib/python3.5/site-packages/google/auth/credentials.py", line 116, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/service_account.py", line 318, in refresh
    request, self._token_uri, assertion)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/_client.py", line 109, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/usr/local/lib/python3.5/site-packages/google/oauth2/_client.py", line 59, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{\n  "error" : "invalid_grant",\n  "error_description" : "Invalid JWT Signature."\n}')
  1. Steps to reproduce

Try to query anything in BigQuery using a service account with “Viewer” permissions and GOOGLE_APPLICATION_CREDENTIALS pointed to a JSON file downloaded when creating the service account.

  1. Code example
bq = bigquery.Client()
query = bq.run_sync_query(
        """ SELECT project,
                   IF(
                        STDDEV(downloads) > 0,
                        (todays_downloads - AVG(downloads))/STDDEV(downloads),
                        NULL
                    ) as zscore
            FROM (
                SELECT project,
                       date,
                       downloads,
                       FIRST_VALUE(downloads) OVER (
                            PARTITION BY project
                            ORDER BY DATE DESC
                            ROWS BETWEEN UNBOUNDED PRECEDING
                                AND UNBOUNDED FOLLOWING
                        ) as todays_downloads
                FROM (
                    SELECT file.project as project,
                           DATE(timestamp) AS date,
                           COUNT(*) as downloads
                    FROM `the-psf.pypi.downloads*`
                    WHERE _TABLE_SUFFIX BETWEEN
                        FORMAT_DATE(
                            "%Y%m%d",
                            DATE_ADD(CURRENT_DATE(), INTERVAL -31 day))
                        AND
                        FORMAT_DATE(
                            "%Y%m%d",
                            DATE_ADD(CURRENT_DATE(), INTERVAL -1 day))
                    GROUP BY file.project, date
                )
            )
            GROUP BY project, todays_downloads
            HAVING SUM(downloads) >= 5000
            ORDER BY zscore DESC
        """
    )
query.use_legacy_sql = False
query.run()

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 5
  • Comments: 28 (7 by maintainers)

Commits related to this issue

Most upvoted comments

@liangjacky can you check your system clock? This is a very strange error and is difficult to reproduce.

As per email conversation with @dstufft, closing for now.

Ok, I found as issue with Docker for Mac https://github.com/docker/for-mac/issues/2076 (a couple related issues actually, but this one pretty clearly covers the problem). The time of the VM that runs containers drifts from system time, and doesn’t appear to reset itself properly. Stopping and restarting Docker for Mac fixed the issue for me.

First, thanks for all the work you do on packaging @dstufft!

@jonparrott Any ideas? @dstufft This probably belongs in https://github.com/GoogleCloudPlatform/google-auth-library-python/, but let’s discuss here first.

I’ve seen an issue where an invalid grant was caused by the clock on the local machine being out of sync by 4+ hours with the real world, but even being wrong by 65 minutes (in either direction) would make one of the “issued at” and “expires” timestamps in the JWT be impossible.

I have this issue but my code is deployed as a cloud function. What should i do?

Really a good catch!!! https://github.com/googleapis/google-cloud-python/issues/3100#issuecomment-323210685

With some reason my system clock was 2 hours ahead and setting it back to correct time, all worked well.

I just synced my time with time.apple.com and and it didn’t seem to change anything. Thanks for the suggestion though. To answer your question, I’m getting the exact same error.

invalid_grant + invalid JWT signature. Authenticating using the application-default, and a json key on disk.

I’m running into this issue, and I know for certain that the time being used by the SDK is the correct time. I extracted the exact JWT that was being posted and verified that the exp being posted was in the future. The iat timestamp was set a few minutes in the past.

It was the time that was causing this error. I took my battery out and the laptop lost track of time. Now after reading your comments I just updated the time and everything worked well. Thank you guys.