dbt-bigquery: [CT-1094] [Bug] BigQuery `oauth-secrets` refresh token profile not authenticating

Describe the bug

I’ve tried to setup a oauth-secrets profile to connect with my BigQuery warehouse following the instruction on https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#oauth-token-based.

However, I’m unable to authneticate when I try to do a dbt run -m <some model>

Steps To Reproduce

Here’s my profile,

bigquery_oauth:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth-secrets
      project: myproject
      threads: 1
      schema: dbt_varun
      refresh_token: "myrefreshtoken"
      client_id: myclientid
      client_secret: myclientsecret
      token_uri: https://www.googleapis.com/oauth2/v4/token

and I’ve specified this profile in my project yml and I’m running with dbt run -m <some model>.

Expected behavior

I would expect the model to run without issue.

Screenshots and log output

I get the following error

$ dbt run -m ./models/example/my_first_dbt_model
dbt run -m my_first_dbt_model
19:42:57  Running with dbt=1.2.1
19:42:57  Unable to do partial parsing because of a dbt version mismatch. Saved manifest version: 1.2.0. Current version: 1.2.1.
19:42:58  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.bigquery-integration.example

19:42:58  Found 2 models, 4 tests, 0 snapshots, 0 analyses, 285 macros, 0 operations, 12 seed files, 0 sources, 0 exposures, 0 metrics
19:42:58
19:43:00  Encountered an error:
Runtime Error
  Unable to generate access token, if you're using impersonate_service_account, make sure your initial account has the "roles/iam.serviceAccountTokenCreator" role on the account you are trying to impersonate.

  ('invalid_scope: Bad Request', {'error': 'invalid_scope', 'error_description': 'Bad Request'})
19:43:00  Traceback (most recent call last):
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/adapters/bigquery/connections.py", line 194, in exception_handler
    yield
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/adapters/bigquery/connections.py", line 611, in _retry_and_handle
    return retry.retry_target(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/adapters/bigquery/impl.py", line 186, in query_schemas
    return [ds.dataset_id for ds in all_datasets]
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/adapters/bigquery/impl.py", line 186, in <listcomp>
    return [ds.dataset_id for ds in all_datasets]
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/api_core/page_iterator.py", line 208, in _items_iter
    for page in self._page_iter(increment=False):
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/api_core/page_iterator.py", line 244, in _page_iter
    page = self._next_page()
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/api_core/page_iterator.py", line 373, in _next_page
    response = self._get_next_page_response()
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/api_core/page_iterator.py", line 432, in _get_next_page_response
    return self.api_request(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 440, in api_request
    return self._call_api(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 482, in api_request
    response = self._make_request(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 341, in _make_request
    return self._do_request(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 379, in _do_request
    return self.http.request(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/auth/transport/requests.py", line 545, in request
    self.credentials.before_request(auth_request, method, url, request_headers)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/oauth2/credentials.py", line 302, in refresh
    ) = reauth.refresh_grant(
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/oauth2/reauth.py", line 347, in refresh_grant
    _client._handle_error_response(response_data)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/google/oauth2/_client.py", line 62, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_data)
google.auth.exceptions.RefreshError: ('invalid_scope: Bad Request', {'error': 'invalid_scope', 'error_description': 'Bad Request'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/main.py", line 129, in main
    results, succeeded = handle_and_check(args)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/main.py", line 191, in handle_and_check
    task, res = run_from_args(parsed)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/main.py", line 238, in run_from_args
    results = task.run()
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/task/runnable.py", line 470, in run
    result = self.execute_with_hooks(selected_uids)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/task/runnable.py", line 432, in execute_with_hooks
    self.before_run(adapter, selected_uids)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/task/run.py", line 442, in before_run
    self.create_schemas(adapter, required_schemas)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/task/runnable.py", line 555, in create_schemas
    existing_schemas_lowered.update(ls_future.result())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/utils.py", line 480, in connected
    return func(*args, **kwargs)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/task/runnable.py", line 532, in list_schemas
    for s in adapter.list_schemas(database_quoted)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/adapters/bigquery/impl.py", line 188, in list_schemas
    return self.connections._retry_and_handle(msg="list dataset", conn=conn, fn=query_schemas)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/adapters/bigquery/connections.py", line 610, in _retry_and_handle
    with self.exception_handler(msg):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/varun/dbt/dbt-integration-projects/bigquery-integration/venv/lib/python3.10/site-packages/dbt/adapters/bigquery/connections.py", line 213, in exception_handler
    raise RuntimeException(message)
dbt.exceptions.RuntimeException: Runtime Error
  Unable to generate access token, if you're using impersonate_service_account, make sure your initial account has the "roles/iam.serviceAccountTokenCreator" role on the account you are trying to impersonate.

  ('invalid_scope: Bad Request', {'error': 'invalid_scope', 'error_description': 'Bad Request'})

System information

The output of dbt --version:

dbt 1.2.1

The operating system you’re using: Linux

The output of python --version: 3.10.5

Additional context

I’m using a script to get the refresh token. I can confirm the refresh token is in fact valid – I’ve used another nodejs script with the nodejs bigquery library to connect with BigQuery without any issues. The script I’m using to get the refresh token is https://github.com/googleapis/nodejs-bigquery/blob/main/samples/auth-user-sample/authUserFlow.js with some very minor tweaks.

I’ve also use this script to get the access_token and I’ve setup a oauth-secrets temporary token profile. That profile works without any issues and I’m able to run dbt run -m <some model> on the same BigQuery warehouse and same dbt project.

bigquery_oauth_temp_token:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth-secrets
      project: myproject
      threads: 1
      schema: dbt_varun
      token: "myaccesstoken"