st2: Workflows with action policies can deadlock

Given the attached pack (useful for meditation), the system deadlocks when the action _dispatcher_pool (50 entries) gets filled with workflows, and all workflow actions are subsequently held in the _work_buffer queue indefinitely.

One possible solution would be to limit the number of workflows in the _dispatcher_pool to one less than half the total entries, in order to leave room for action execution.

deadlock.tar.gz

Deadlock pack

This pack contains actions for testing policy deadlock and other behaviors

Actions

  • delay2 - a two second delay
  • delay20 - a twenty second delay
  • delay2test - a workflow with delay2 followed by delay20

Policies

  • delay20 - limit concurrency to 1
  • delay2test - limit concurrency to 10

Testing

Launch multiple instances of the workflow:

  • for i in seq 1 100; do st2 action execute deadlock.delay2test; done

Without the delay2test policy, the actions deadlock

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 22 (21 by maintainers)

Most upvoted comments

@bri365 Thanks for the feedback.

I will also wait on @lakshmi-kannan and others to chime in and if they agree with my proposed approach, I will go ahead and implement it 😃