azure-webjobs-sdk: Singleton Listeners causing 409 (Conflict) error

Please provide a succinct description of the issue.

When you run timer jobs with many worker roles (50+) one of those worker roles gets the 409 error from storage service and gets recycled. When this happens the exception message says “StatusMessage:The lease ID specified did not match the lease ID for the blob. ErrorCode:LeaseIdMismatchWithLeaseOperation”.

Provide the steps required to reproduce the problem

10 timer jobs with 50-150 worker role instances. Every 30 minutes also one of roles will start recycling due to 409 (conflict) error.

Expected behavior

No conflict error from storage service.

Provide a description of the expected behavior.

Actual behavior

Getting 409 error.

Known workarounds

None.

Related information

Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (409) Conflict. —> System.Net.WebException: The remote server returned an error: (409) Conflict. at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase1 cmd, Exception ex) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Shared\Protocol\HttpResponseParsers.Common.cs:line 50 at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.<>c__DisplayClass33.<RenewLeaseImpl>b__32(RESTCommand1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 3319 at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 306 — End of inner exception stack trace — at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 60 at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 115 — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.WebJobs.Host.SingletonManager.RenewLeaseCommand.<ExecuteAsync>d__36.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() Request Information RequestID:d8d712e5-0001-012c-651a-0dec95000000 RequestDate:Mon, 12 Sep 2016 17:26:41 GMT StatusMessage:The lease ID specified did not match the lease ID for the blob. ErrorCode:LeaseIdMismatchWithLeaseOperation

About this issue

Original URL
State: open
Created 8 years ago
Comments: 50 (22 by maintainers)

Most upvoted comments

seeing this 409 LeaseIdMismatchWithLeaseOperation regularly with our blob trigger. We have very low volume and only (2) instances.

2017-11-28T03:28:58.536 Host lock lease acquired by instance ID '66525d2801dbf933e2e11518b127855f'.
info: Host.General[0]
      Host lock lease acquired by instance ID '66525d2801dbf933e2e11518b127855f'.
2017-11-28T03:29:06.300 A ScriptHost error has occurred
2017-11-28T03:29:06.300 System.InvalidOperationException : Singleton lock renewal failed for blob 'locks/my-function/host' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 0001-01-01T00:00:00Z (-2147483648 milliseconds ago) with a duration of 0 milliseconds. The lease period was 15000 milliseconds.
2017-11-28T03:29:06.309 	 'locks/my-function/host' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 0001-01-01T00:00:00Z (-2147483648 milliseconds ago) with a duration of 0 milliseconds. The lease period was 15000 milliseconds.
2017-11-28T03:29:06.312 Failed to renew host lock lease: Another host has acquired the lease. The last successful renewal completed at 2017-11-28T03:28:54.229Z (12080 milliseconds ago) with a duration of 780 milliseconds.

Seems to freeze up our host and we have to restart the function to get it working again.

alohaninja on Nov 28, 2017

I’m facing the same error when debug locally a webjob having several scheduled singleton functions.

abatishchev on Oct 16, 2016