astronomer-cosmos: GCP BigQuery service account json not supported in v0.7
We ran into this problem upgrading from v0.6.X -> v0.7.X using the default google_cloud_platform airflow connection that uses a keyfile json and not the keyfile path that is currently supported in astronomer cosmos.
All of the profile var mapping of the json service account can be found in an older tag before the refactor of profiles in 0.7 here: https://github.com/astronomer/astronomer-cosmos/blob/astronomer-cosmos-v0.6.8/cosmos/providers/dbt/core/profiles/bigquery.py as a place to start from.
When I have time I’d be happy to take a stab at this, but opening up this issue in case others also run into this during an upgrade.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 16 (9 by maintainers)
Commits related to this issue
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to astronomer/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support for BQ SA keyfile_dict Related issue: https://github.com/astronomer/astronomer-cosmos/issues/350 — committed to JoeSham/astronomer-cosmos by JoeSham a year ago
- Add support to GCP connections that define `keyfile_dict` instead of `keyfile` (#352) Add support to Google Cloud Platform connections that define `keyfile_dict` (actual value) instead of `keyfile` ... — committed to astronomer/astronomer-cosmos by JoeSham a year ago
Hey @tatiana @jlaneve 👋 May I ask what is the current status, please?
That’s a great start, @JoeSham ! Thank you very much. Please, do let us know once you’d like reviews - or if you need any support, e.g for writing the tests
Ok great @jlaneve 😃 Draft PR opened
@jlaneve yea you are right, it was just a quick first draft. I now made another quick 2nd draft, the way you recommended. Without tests so far. Not sure if I will have more time for it in the upcoming days though…
@tatiana Hi, yes, correct. The GoogleCloudServiceAccountFileProfileMapping seems to only accept key_path as a path to SA keyfile, but there is no keyfile_dict or similar option. I also tested passing the SA keyfile_dict into the key_path param, in which case it passed in the ProfileMapping but then failed later on the fact that the keyfile (as given) does not exist