airflow: PowerBIDatasetRefreshOperator task fails whereas dataset refresh succeeds

Apache Airflow Provider(s)

microsoft-azure

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-azure==11.1.0

Apache Airflow version

2.10.2

Operating System

linux

Deployment

Google Cloud Composer

Deployment details

No response

What happened

We use the operator PowerBIDatasetRefreshOperator to refresh our PowerBI datasets. Sometimes, the task quickly fails with the following error:

airflow.exceptions.AirflowException: An error occurred: Unable to fetch the details of dataset refresh with Request Id: <request id>
image

However, on PowerBI side, the dataset is refreshed (timezone GMT+1, both the time in the refresh and the log are about the same): image

If I understand correctly, this corresponds to an error in this function, which may mean that there was an error when trying to fetch the refresh status, even though the refresh was running. Maybe the error comes from the request to the Microsoft API, which could be improved with a retry (but we did not test for the moment).

What you think should happen instead

The task should succeed. (since the dataset refresh suceeded)

How to reproduce

On our side, we use the following configuration.

refresh_powerbi_dataset_success = PowerBIDatasetRefreshOperator(
        task_id="refresh_powerbi_dataset_success",
        conn_id="our-conn-id",
        dataset_id="our-dataset-id",
        group_id="our-group-id",
    )

The dataset ID and group ID correspond to valid PowerBI dataset and workspace.

Anything else

This bug occurs about once every 3 task runs (the task fails but the refresh succeeds). Sometimes if fails several times in a row then works again, other times it works on the first run. I think this should mean that the configuration is good (since the operator often works fine). I haven’t been able to identify any specific pattern.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (23 by maintainers)

Most upvoted comments

Hi @dabla I opened a PR with the following changes: the separation between the 2 flows and the retry on the func fetch_refresh_status_and_error. I’d be happy to have your feedbacks on this PR when you have some time !

Hi @dabla I quickly tested what we said, so that the triggers return to execute_complete method, which itself checks the message and if it detects that there was an issue fetching the refresh Id, it redirects to a retry_execution method that counts the retries and re-executes the execute method. Of course the code is not very clean, but here is an overview:

def execute(self, context: Context):
    """Refresh the Power BI Dataset."""
    if self.wait_for_termination:
        self.defer(
            trigger=PowerBITrigger(
                conn_id=self.conn_id,
                group_id=self.group_id,
                dataset_id=self.dataset_id,
                timeout=self.timeout,
                proxies=self.proxies,
                api_version=self.api_version,
                check_interval=self.check_interval,
                wait_for_termination=self.wait_for_termination,
            ),
            method_name=self.get_refresh_status.__name__,
        )

def get_refresh_status(self, context: Context, event: dict[str, str] | None = None):
    """Push the refresh Id to XCom then runs the Triggers to wait for refresh completion."""

    if event:
        if event["status"] == "error" and "Unable to fetch the details of dataset refresh with Request Id" not in event["message"] and "not found" not in event["message"]:
            raise AirflowException(event["message"])

        self.xcom_push(context=context, key="powerbi_dataset_refresh_Id", value=event["dataset_refresh_id"])

    dataset_refresh_id = self.xcom_pull(context=context, key="powerbi_dataset_refresh_Id")
    if dataset_refresh_id:
        self.defer(
            trigger=PowerBITrigger(
                conn_id=self.conn_id,
                group_id=self.group_id,
                dataset_id=self.dataset_id,
                dataset_refresh_id=dataset_refresh_id,
                timeout=self.timeout,
                proxies=self.proxies,
                api_version=self.api_version,
                check_interval=self.check_interval,
                wait_for_termination=self.execute_complete,
            ),
            method_name=self.execute_complete.__name__,
        )

def retry_execution(self, context: Context):
    retries = self.xcom_pull(context=context, key="retries")
    if retries and retries >= self.max_retries:
        raise AirflowException("Max number of retries reached!")

    if not retries:
        retries = 0
    self.xcom_push(context=context, key="retries", value=retries+1)

    self.get_refresh_status(context)

def execute_complete(self, context: Context, event: dict[str, str]) -> Any:
    """
    Return immediately - callback for when the trigger fires.

    Relies on trigger to throw an exception, otherwise it assumes execution was successful.
    """
    if event:
        if event["status"] == "error":
            if "Unable to fetch the details of dataset refresh with Request Id" in event["message"] or "not found" in event["message"]:
                self.retry_execution(context)
            else:
                raise AirflowException(event["message"])

        self.xcom_push(context=context, key="powerbi_dataset_refresh_status", value=event["status"])

Note: in addition to these changes, we should add a new way to handle the refresh cancellation. By default, if the trigger encounters an exception, it cancels the refresh (which is not compatible with the retry made by the operator). If we keep this solution, we have to change this behavior.

I think this solution can work but might add a little too much complexity to the operator compared to a simple retry, though I think that this separation between the trigger refresh and the status fetch is nice. What’s your opinion?

However, it should be possible to retry the operator “execute” function without retrying the task, as done in DmsStartReplicationOperator

You’re right about the Xcoms being flushed when a task fails, I like the idea/above of approach, by define a dedicated retry method which is passed as argument of the next_method of the Trigger, that could be an option. If it’s too difficult, then I would do what @ambika-garg proposed and I think you also proposed at the beginning of this issue.

For the second case, in the operator, you should do in the execute method an xcom_pull to see if there is an existing dataset_refresh_id or not, and if not you know you have to execute the whole flow, if it’s there then you know you should only trigger the second part.

After a bit a investigation, I think this won’t be possible to pass XCom messages between different task executions of the same operator (if this was what you meant). According to Airflow XComs documentation, “If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.”. From what I understand, it is not possible for a 2nd operator (in a task retry) to access the 1st task XCom message. However, it should be possible to retry the operator “execute” function without retrying the task, as done in DmsStartReplicationOperator: https://github.com/apache/airflow/blob/9ba279d15f26088400b15281b3cc346e2d7a0e30/providers/src/airflow/providers/amazon/aws/operators/dms.py#L720-L723

Though I’m not sure if this is what you’d prefer.

After looking at the codebase, I see retry mechanisms in some hooks (ex:BaseDatabricksHook, LivyHook, GlueJobHook, EmrContainerHook…), handlers (ex: S3TaskHandler) and in some operators as well (ex: the Kubernetes operators as your colleague pointed out, or some AWS operators such as BedrockCustomizeModelOperator, DmsDeleteReplicationConfigOperator…). I didn’t look at all the files, there must be others. The retry can be used to wait for a status (ex: hook ElastiCacheReplicationGroupHook, method wait_for_availability, link) or to wait for an API request to succeed. In our case, I think that we can fall in these cases: We could add a retry in PowerBIHook.get_refresh_details_by_refresh_id before raising the PowerBIDatasetRefreshException?

Not sure which solution is best to be honest. What do you think?

  • Should I set the retry within the operator? with a retry_execution method? or a while loop in the execute method? (or else?)
  • Should I handle the retry in the existing while loop from the trigger?
  • Or should I set a retry in the hook directly?

Btw, I just (locally) implemented your refactor and the separation between the 2 flows (refresh trigger and get refresh status) in 2 deferrable triggers seems to let enough time between the refresh creation and the first “get refresh status” request. It seems that I don’t get the error anymore (not that the bug is really fixed, but in my environment, the duration between both events is enough to prevent the error).

the operator can then retry the second case directly instead of redoing the whole flow

Regarding this, how/where would you retry the second case?

Indeed, that’s also a consequence of using the trigger twice, as more time will pass automatically between both invocations, the error will probably not occurs anymore and everything will succeed in one attempt.

For the second case, in the operator, you should do in the execute method an xcom_pull to see if there is an existing dataset_refresh_id or not, and if not you know you have to execute the whole flow, if it’s there then you know you should only trigger the second part.

Btw, I just (locally) implemented your refactor and the separation between the 2 flows (refresh trigger and get refresh status) in 2 deferrable triggers seems to let enough time between the refresh creation and the first “get refresh status” request. It seems that I don’t get the error anymore (not that the bug is really fixed, but in my environment, the duration between both events is enough to prevent the error).

the operator can then retry the second case directly instead of redoing the whole flow

Regarding this, how/where would you retry the second case?

Hello! Thank you very much for taking the time to look at the code and making this suggestion. I tried to reduce the check_interval to 5, unfortunately it did not seem to fix the issue on our side.

Also, I thought that the default timeout was 1 week because of this line: https://github.com/apache/airflow/blob/608de6b97ef04307559d79b2044899adfb9b2a93/providers/src/airflow/providers/microsoft/azure/triggers/powerbi.py#L57

But maybe I got this wrong. Am I mistaken?

Regarding the bug, I think the issue may come from the very first request to get the refresh status, at this line: https://github.com/apache/airflow/blob/f7da5e48a2740a2bfb1c3dd57bb463ae29599a26/providers/src/airflow/providers/microsoft/azure/triggers/powerbi.py#L116

This request is made by the PowerBi hook (using the method get_refresh_details_by_refresh_id) and I think that, sometimes, the API is not yet “up-to-date” with the latest refresh IDs and can’t find the status of our refresh ID. So, we fall in this case: https://github.com/apache/airflow/blob/f7da5e48a2740a2bfb1c3dd57bb463ae29599a26/providers/src/airflow/providers/microsoft/azure/hooks/powerbi.py#L159

and the hook raises the error “Unable to fetch the details of dataset refresh with Request Id” (which is also what I find in my Airflow logs). Do you think it makes sense?

I did try to add a retry on this request (and it seemed to fix the bug), but I understand that you don’t want to introduce this type of custom retry mechanism. Maybe another option could be to wait “check_interval” seconds BEFORE sending the first request. What do you think?

OK I see, so the retry is not specific to the “get status” request. From what I understand, the issue seems to be that the very first request (to get the refresh status) fails (maybe because the refresh id is not yet available for this endpoint) and the task exits. If I understand correctly, the retry on the operator allows us to re-run the whole process (so re-trigger a refresh), instead of retrying the request to get the refresh status already running.