airflow: PowerBIDatasetRefreshOperator task fails whereas dataset refresh succeeds
Apache Airflow Provider(s)
microsoft-azure
Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure==11.1.0
Apache Airflow version
2.10.2
Operating System
linux
Deployment
Google Cloud Composer
Deployment details
No response
What happened
We use the operator PowerBIDatasetRefreshOperator to refresh our PowerBI datasets. Sometimes, the task quickly fails with the following error:
airflow.exceptions.AirflowException: An error occurred: Unable to fetch the details of dataset refresh with Request Id: <request id>
However, on PowerBI side, the dataset is refreshed (timezone GMT+1, both the time in the refresh and the log are about the same):
If I understand correctly, this corresponds to an error in this function, which may mean that there was an error when trying to fetch the refresh status, even though the refresh was running. Maybe the error comes from the request to the Microsoft API, which could be improved with a retry (but we did not test for the moment).
What you think should happen instead
The task should succeed. (since the dataset refresh suceeded)
How to reproduce
On our side, we use the following configuration.
refresh_powerbi_dataset_success = PowerBIDatasetRefreshOperator(
task_id="refresh_powerbi_dataset_success",
conn_id="our-conn-id",
dataset_id="our-dataset-id",
group_id="our-group-id",
)
The dataset ID and group ID correspond to valid PowerBI dataset and workspace.
Anything else
This bug occurs about once every 3 task runs (the task fails but the refresh succeeds). Sometimes if fails several times in a row then works again, other times it works on the first run. I think this should mean that the configuration is good (since the operator often works fine). I haven’t been able to identify any specific pattern.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (23 by maintainers)
Hi @dabla I opened a PR with the following changes: the separation between the 2 flows and the retry on the func
fetch_refresh_status_and_error. I’d be happy to have your feedbacks on this PR when you have some time !Hi @dabla I quickly tested what we said, so that the triggers return to
execute_completemethod, which itself checks the message and if it detects that there was an issue fetching the refresh Id, it redirects to aretry_executionmethod that counts the retries and re-executes theexecutemethod. Of course the code is not very clean, but here is an overview:Note: in addition to these changes, we should add a new way to handle the refresh cancellation. By default, if the trigger encounters an exception, it cancels the refresh (which is not compatible with the retry made by the operator). If we keep this solution, we have to change this behavior.
I think this solution can work but might add a little too much complexity to the operator compared to a simple retry, though I think that this separation between the trigger refresh and the status fetch is nice. What’s your opinion?
You’re right about the Xcoms being flushed when a task fails, I like the idea/above of approach, by define a dedicated retry method which is passed as argument of the next_method of the Trigger, that could be an option. If it’s too difficult, then I would do what @ambika-garg proposed and I think you also proposed at the beginning of this issue.
After a bit a investigation, I think this won’t be possible to pass XCom messages between different task executions of the same operator (if this was what you meant). According to Airflow XComs documentation, “If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.”. From what I understand, it is not possible for a 2nd operator (in a task retry) to access the 1st task XCom message. However, it should be possible to retry the operator “execute” function without retrying the task, as done in
DmsStartReplicationOperator: https://github.com/apache/airflow/blob/9ba279d15f26088400b15281b3cc346e2d7a0e30/providers/src/airflow/providers/amazon/aws/operators/dms.py#L720-L723Though I’m not sure if this is what you’d prefer.
After looking at the codebase, I see retry mechanisms in some hooks (ex:
BaseDatabricksHook,LivyHook,GlueJobHook,EmrContainerHook…), handlers (ex:S3TaskHandler) and in some operators as well (ex: the Kubernetes operators as your colleague pointed out, or some AWS operators such asBedrockCustomizeModelOperator,DmsDeleteReplicationConfigOperator…). I didn’t look at all the files, there must be others. The retry can be used to wait for a status (ex: hookElastiCacheReplicationGroupHook, methodwait_for_availability, link) or to wait for an API request to succeed. In our case, I think that we can fall in these cases: We could add a retry inPowerBIHook.get_refresh_details_by_refresh_idbefore raising the PowerBIDatasetRefreshException?Not sure which solution is best to be honest. What do you think?
retry_executionmethod? or awhileloop in theexecutemethod? (or else?)whileloop from the trigger?Indeed, that’s also a consequence of using the trigger twice, as more time will pass automatically between both invocations, the error will probably not occurs anymore and everything will succeed in one attempt.
For the second case, in the operator, you should do in the execute method an xcom_pull to see if there is an existing dataset_refresh_id or not, and if not you know you have to execute the whole flow, if it’s there then you know you should only trigger the second part.
Btw, I just (locally) implemented your refactor and the separation between the 2 flows (refresh trigger and get refresh status) in 2 deferrable triggers seems to let enough time between the refresh creation and the first “get refresh status” request. It seems that I don’t get the error anymore (not that the bug is really fixed, but in my environment, the duration between both events is enough to prevent the error).
Regarding this, how/where would you retry the second case?
Hello! Thank you very much for taking the time to look at the code and making this suggestion. I tried to reduce the
check_intervalto 5, unfortunately it did not seem to fix the issue on our side.Also, I thought that the default timeout was 1 week because of this line: https://github.com/apache/airflow/blob/608de6b97ef04307559d79b2044899adfb9b2a93/providers/src/airflow/providers/microsoft/azure/triggers/powerbi.py#L57
But maybe I got this wrong. Am I mistaken?
Regarding the bug, I think the issue may come from the very first request to get the refresh status, at this line: https://github.com/apache/airflow/blob/f7da5e48a2740a2bfb1c3dd57bb463ae29599a26/providers/src/airflow/providers/microsoft/azure/triggers/powerbi.py#L116
This request is made by the PowerBi hook (using the method
get_refresh_details_by_refresh_id) and I think that, sometimes, the API is not yet “up-to-date” with the latest refresh IDs and can’t find the status of our refresh ID. So, we fall in this case: https://github.com/apache/airflow/blob/f7da5e48a2740a2bfb1c3dd57bb463ae29599a26/providers/src/airflow/providers/microsoft/azure/hooks/powerbi.py#L159and the hook raises the error “Unable to fetch the details of dataset refresh with Request Id” (which is also what I find in my Airflow logs). Do you think it makes sense?
I did try to add a retry on this request (and it seemed to fix the bug), but I understand that you don’t want to introduce this type of custom retry mechanism. Maybe another option could be to wait “check_interval” seconds BEFORE sending the first request. What do you think?
OK I see, so the retry is not specific to the “get status” request. From what I understand, the issue seems to be that the very first request (to get the refresh status) fails (maybe because the refresh id is not yet available for this endpoint) and the task exits. If I understand correctly, the retry on the operator allows us to re-run the whole process (so re-trigger a refresh), instead of retrying the request to get the refresh status already running.