dolphinscheduler: [Bug] [TaskExecutionRunnable] Sub workflow task always repeat run in master-server failover

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

Normally, FailoverCoordinator.getFailoverWorkflowsForMaster() finds all workflows that need failover. When the sub-workflow task’s TaskExecutionRunnable.failover() method is called, takeOverTaskFromExecutor() now returns false if the task is a logic task. This results in the creation of a new sub-workflow task instance and the publication of its TaskStartLifecycleEvent, causing the sub-workflow to run again during the failover process.

https://github.com/apache/dolphinscheduler/blob/071994933b05850e7dd5f7bccbf45d867640e244/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/task/runnable/TaskExecutionRunnable.java#L120-L135

https://github.com/apache/dolphinscheduler/blob/071994933b05850e7dd5f7bccbf45d867640e244/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/task/runnable/TaskExecutionRunnable.java#L160-L164

so i think it would be better to check the sub-workflow instance properly(by dao or server communicate) and take it over, instead of creating a whole new task instance.

What you expected to happen

Take over sub-workflow task if the sub-workflow instance is in good status.

How to reproduce

execute a workflow with sub-workflow, restart the master-server, query the database

Anything else

No response

Version

dev

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (14 by maintainers)

Commits related to this issue

Most upvoted comments

@ruanwenjun In dev branch now, there’s a better way to takeover sub-workflow task in failover, i made a branch to implement: https://github.com/apache/dolphinscheduler/compare/dev...reele:fix-takeover-workflow-in-failover?expand=1 after refactored TaskExecutorClient, takeover is called before initializeTaskExecutionContext, so there should only check workflow instance in takeover, and put the TaskExecutor to taskExecutorRepository in TaskEngine, and it will work for good.

Can we move the logic to SubWorkflowLogicTask ? when we failover a task instance(e.g. sub workflow task instance), we directly generate a new task instance which contains the pre runtime context, once the task instance started, it will check if it is a failover task instance, and if it can take-over. Since different logic task instances might have its own logic to judge whether it can take-over. You can find a todo at SubWorkflowLogicTask#initializeSubWorkflowInstance

it’s indeed a simpler approach - my method impacted architecture design, i tried.

Also, during failover, the master directly starts sub-workflows, which creates a split from SubWorkflowLogicTask’s processing flow, i think it would be better to have SubWorkflowLogicTask fully handle triggering fault-tolerant sub-workflows.

@reele Hi, the subworkflow take-over logic is under SubWorkflowLogicTask, once we failover a subworkflow task, will generate a new logic task instance, the new task instance will contains the origin SubWorkflowLogicTaskRuntimeContext and then can track the origin sub workflow instance.

I will test this.