airflow: airflow jinja template render error

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

version 2.6.2

An error occurs when *.json is included in the parameters of BigQueryInsertJobOperator.

to_gcs_task = BigQueryInsertJobOperator(
        dag=dag,
        task_id='to_gcs',
        gcp_conn_id='xxxx',
        configuration={
            "extract": {
                 # The error occurred at this location.          
                "destinationUris": ['gs://xxx/yyy/*.json'],


                "sourceTable": {
                    "projectId": "abc",
                    "datasetId": "def",
                    "tableId": "ghi"
                },
                "destinationFormat": "NEWLINE_DELIMITED_JSON"
            }
        }
    )

error log

jinja2.exceptions.TemplateNotFound: gs://xxx/yyy/*.json

What you think should happen instead

According to the airflow.template.templater source : https://github.com/apache/airflow/blob/main/airflow/template/templater.py#L152

      if isinstance(value, str):
            if any(value.endswith(ext) for ext in self.template_ext):  # A filepath.
                template = jinja_env.get_template(value)
            else:
                template = jinja_env.from_string(value)
            return self._render(template, context)

In the Jinja template source, if the value ends with .json or .sql, an attempt is made to read the resource file by calling jinja_env.get_template.

How to reproduce

just call BigQueryInsertJobOperator with configuration what i added

Operating System

m2 mac

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project’s Code of Conduct

About this issue

Original URL
State: closed
Created 10 months ago
Comments: 16 (13 by maintainers)

Most upvoted comments

Maybe the easiest way out would be to add a Literally wrapper that disables rendering? So you can write

BigQueryInsertJobOperator(
    ...,
    configuration={
        "extract": {
            "destinationUris": [Literally("gs://xxx/yyy/*.json")],
            ...,
        },
    },
)

Please do bikeshed on the name. I’m intentionally avoiding Literal to avoid confusion with typing.Literal.

uranusjr on Sep 6, 2023

Didn’t we have the exact same bug when we added json as templated ext for KubernetesPodOperator ? https://github.com/apache/airflow/issues/16922

eladkal on Aug 25, 2023

I have absolutely No objections, even more, I would love that as an “official” way of modifying templated/template_ext fields on-the-flight - as long as we get a few more people say “well, indeed that does not seem to have some side effects” 😄

potiuk on Aug 24, 2023