sdk-go: Canceling workflow can cause infinite replay attempts

Expected Behavior

Canceling a workflow should cancel the workflow without side effects or replays.

Actual Behavior

Canceling workflows works the majority of the time, but for some cases it returns this error:

Task processing failed with error Namespace project TaskQueue taskqueue WorkerID 19252@hung-MacBook-Pro.local@ WorkerType WorkflowWorker Error BadRequestCancelActivityAttributes: invalid history builder state for action: add-activitytask-cancel-requested-event

This causes Temporal to keep attempting replays with Attempt 1 until the workflow is terminated manually. The workflow is marked as running when it should be canceled. In the logs, the workflow is canceled at the same activity step (WaitQueryCompleteActivity), but one succeeds and other fails with the error and keeps replaying.

Steps to Reproduce the Problem

1. ExecuteQueryWorkflow (workflow.ActivityOptions{WaitForCancellation: true})

• workflow.ExecuteLocalActivity(CreateQueryActivity)
• workflow.ExecuteLocalActivity(RunQueryActivity)
• f(ctx, data) – workflow.ExecuteActivity(ctx, WaitQueryCompleteActivity, engine).Get(ctx, nil)…

1. client.CancelWorkflow(context.Background(), getQueryWorkflowID(queryId), “”)

possibly related?: https://github.com/temporalio/sdk-go/issues/469

Specifications

• Version: – go.temporal.io/api v1.4.1-0.20210420220407-6f00f7f98373 – go.temporal.io/sdk v1.7.0
• Platform: – Mac

Logs: query 118 success- 8546f839-3403-4841-af11-a25ba8ca91eb.json.txt query 119 succes - 5c3e2e65-44b7-4899-ab0d-fbf29ab21106.json.txt