dbt-bigquery: [ADAP-695] [Regression] Error when use REF inside COPY materialization

Is this a regression in a recent version of dbt-bigquery?

I believe this is a regression in dbt-bigquery functionality
I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

if you use REF in model with copy materialization, then you receive following error:

Compilation Error in model bla-bla
Model 'bla-bla' (models/model_copy.sql) depends on a node named 'name' which was not found

> in macro materialization_copy_bigquery (macros/materializations/copy.sql)
> called by model model_copy (models/model_copy.sql)

I’ve checked all previous versions and there is no issue at all. the issue is on 1.5 only if you use SOURCE then all works fine only REF usage leads above issue

Expected/Previous Behavior

both REF and SOURCE works fine within COPY materialization

Steps To Reproduce

use copy materialization with any of your source or model. config exampe:

{{ config(
    materialized='copy'
)}}

select * from {{ source('ANY_YOUR_SOURCE') }}, {{ ref('ANY_YOUR_MODEL') }}

if you run this model on 1.5.x then you receive an error then delete REF (or change it to any other SOURCE) from this model and run. All works fine

Relevant log output

No response

Environment

- OS: dockerized env using `python:3.11-slim` image
- Python: 3.11
- dbt-core (working version): 1.4.6
- dbt-bigquery (working version): 1.4.3
- dbt-core (regression version): 1.5.2 (all 1.5.x)
- dbt-bigquery (regression version): 1.5.3 (all 1.5.x)

Additional Context

No response

About this issue

Original URL
State: closed
Created a year ago
Reactions: 2
Comments: 21 (19 by maintainers)

Most upvoted comments

here’s the current plan for engineering

review #910, add deprecation warning, and merge
patch bugfix back to 1.5
make PR to remove copy materialization for 1.8 and onwards

However, we will not be able to address this work until October 23 at the earliest, given that the release cut of 1.7.0 is this week, and our annual user conference, Coalesce, is next week.

dataders on Oct 11, 2023

Model versioning is a new feature in 1.5 (as per https://docs.getdbt.com/docs/collaborate/govern/model-versions) so the keyword argument is new as well.

Similarly to @dbeatty10 I’m currently using

{{ source_array.append(ref(ref_table.get('package'), ref_table.name, version=ref_table.get('version'))) }}

as a hot fix. I have not confirmed though that it really works with packages and versions.

A hint to @xemuliam about why lines 12 and 16 are similar, yet one does not work, is that while model.sources is an array of arrays in 1.5, model.refs is an array of dictionaries (with keys name and optionally package and version). I’m not sure how it was in older versions than 1.5 but I suspect model.refs structure was different.

m-sche on Aug 16, 2023

@xemuliam I can take a look at this 👀

dbeatty10 on Aug 8, 2023

@mikealfare and @dataders I didn’t notice pre-existing issues, so opened these up:

dbeatty10 on Oct 11, 2023

I’m the engineer who wrote dbt clone and am not super familiar with the copy materialization in dbt-bigquery. Can you please enumerate why it won’t work for your use case?

Thanks for raising this issue, and looking forward to coming to a good resolution soon.

Hi @aranke ,

BigQuery copy materialization is the implementation of following API: https://cloud.google.com/bigquery/docs/managing-tables#copying_multiple_source_tables

The main advantages of it (please pay an attention that there can be more than one source table):

creation of fully independent, writeable copy of base table(s)
data and metadata in target table can be: 2.1 just appended from source table(s) 2.3 fully replaced from source table(s)
above operations are performed on storage layer, w/o involving SQL engine at all
whereas (3) is in place, then those operations are fast and free of charge
whereas (1) you can rewrite base table unlike with clone or snapshot from it

Hope it was clear enough

xemuliam on Aug 17, 2023

Env 1.5 spin up and run copy model:

$ docker run -it --rm -v $PWD:/usr/app -v $PWD/profiles.yml:/root/.dbt/profiles.yml -v $PWD/dev.json:/opt/secrets/secret.json -w /usr/app python:3.11-slim bash

root@eaad28485f90:/usr/app# pip install --upgrade dbt-bigquery==1.5.3
...

root@eaad28485f90:/usr/app# dbt --version
Core:
  - installed: 1.5.2
  - latest:    1.5.2 - Up to date!

Plugins:
  - bigquery: 1.5.3 - Up to date!

root@eaad28485f90:/usr/app# dbt run --select model_copy
13:36:01  Running with dbt=1.5.2
13:36:01  Registered adapter: bigquery=1.5.3
13:36:02  Found 4 models, 0 tests, 0 snapshots, 0 analyses, 358 macros, 0 operations, 0 seed files, 3 sources, 0 exposures, 0 metrics, 0 groups
13:36:02
13:36:03  Concurrency: 2 threads (target='poc')
13:36:03
13:36:03  1 of 1 START sql copy model poc.edw-fact-web_search_t_api_copy ................. [RUN]
13:36:03  1 of 1 ERROR creating sql copy model poc.edw-fact-web_search_t_api_copy ........ [ERROR in 0.05s]
13:36:03
13:36:03  Finished running 1 copy model in 0 hours 0 minutes and 1.50 seconds (1.50s).
13:36:03  
13:36:03  Completed with 1 error and 0 warnings:
13:36:03
13:36:03  Compilation Error in model model_copy (models/model_copy.sql)
13:36:03    Model 'model.rb_dbt_poc.model_copy' (models/model_copy.sql) depends on a node named 'name' which was not found
13:36:03
13:36:03    > in macro materialization_copy_bigquery (macros/materializations/copy.sql)
13:36:03    > called by model model_copy (models/model_copy.sql)
13:36:03
13:36:03  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

root@eaad28485f90:/usr/app#

==============================

Install dbt-bigquery 1.4 and run copy model inside the same container:

root@eaad28485f90:/usr/app# pip install --upgrade dbt-bigquery==1.4.3
...

root@eaad28485f90:/usr/app# dbt --version
Core:
  - installed: 1.4.6
  - latest:    1.5.2 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - bigquery: 1.4.3 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

root@eaad28485f90:/usr/app# dbt run --select model_copy
13:39:48  Running with dbt=1.4.6
13:39:48  Unable to do partial parsing because of a version mismatch
13:39:50  Found 4 models, 0 tests, 0 snapshots, 0 analyses, 338 macros, 0 operations, 0 seed files, 3 sources, 0 exposures, 0 metrics
13:39:50
13:39:51  Concurrency: 2 threads (target='poc')
13:39:51
13:39:51  1 of 1 START sql copy model poc.edw-fact-web_search_t_api_copy ................. [RUN]
13:39:56  1 of 1 OK created sql copy model poc.edw-fact-web_search_t_api_copy ............ [COPY TABLE with materialization: table in 5.40s]
13:39:56  
13:39:56  Finished running 1 copy model in 0 hours 0 minutes and 6.60 seconds (6.60s).
13:39:56  
13:39:57  Completed successfully
13:39:57
13:39:57  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

root@eaad28485f90:/usr/app#

as you can see, the same model works fine in 1.4 but raises an error in 1.5

xemuliam on Jul 26, 2023