In Spark when you set spark.conf.set(“spark.sql.sources.partitionOverwriteMode”,“dynamic”) and then do an insert into a partitioned table in overwrite mode. The newly inserted partitions would overwrite only partitions being inserted. Other partitions that were not part of that group would also stick around untouched. When writing to BigQuery with this connector the entire table in BigQuery gets wiped out and only the new partitions inserted show up. Can the connector be updated to support dynamic partition overwrites?

I am testing with gs://spark-lib/bigquery/spark-bigquery-latest.jar.

Thanks!

Example setup of this scenario:

Ran this on BigQuery directly:

CREATE OR REPLACE TABLE gcp-project.dev.wiki_page_views_spark_write ( wiki_project STRING, wiki_page STRING, wiki_page_views INT64, date DATE ) PARTITION BY date OPTIONS ( partition_expiration_days=999999 )

spark.conf.set(“spark.sql.sources.partitionOverwriteMode”,“dynamic”)

Saving the data to BigQuery

wiki.write.format(‘bigquery’)
.option(‘table’, ‘gcp-project.dev.wiki_page_views_spark_write’)
.option(‘project’,‘gcp-project’)
.option(‘temporaryGcsBucket’,‘gcp-project/tmp/bq_staging’)
.mode(‘overwrite’)
.save()