azure-webjobs-sdk: Function host fails for Blob Trigger watching a ADLSv2 account

Repro steps

Create an ADLSv2 account
Create a Function App (Python 3.7, blob trigger) (vs code template func is sufficient):
Configure bindings to use the ALDSv2 account

Expected behavior The blobs should be processed.

Actual behavior Sometimes function host fails with the stack trace like this (may happen both locally and in Azure):

assembly,level,line,method
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",0,0,"System.ThrowHelper.ThrowKeyNotFoundException"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",1,0,"System.Collections.Generic.Dictionary`2.get_Item"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",2,183,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener+<GetLogsWithPrefixAsync>d__16.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",3,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",4,154,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener+<ListRecentLogFilesAsync>d__15.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",5,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",6,0,"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",7,55,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener+<GetRecentBlobWritesAsync>d__12.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",8,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",9,110,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.PollLogsStrategy+<ExecuteAsync>d__12.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",10,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",11,0,"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",12,126,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.ScanBlobScanLogHybridPollingStrategy+<ExecuteAsync>d__13.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",13,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",14,147,"Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer+<RunAsync>d__14.MoveNext"

Known workarounds recreate storage account

Related information Related issue - https://github.com/microsoft/vscode-azurefunctions/issues/1725

Root cause: no check for the existence of the dictionary key. (I assume a presence of a metadata property is not guaranteed.) https://github.com/Azure/azure-webjobs-sdk/blob/fca811638dd3c7dcf55905dea6837be5b30f5f15/src/Microsoft.Azure.WebJobs.Extensions.Storage/Blobs/Listeners/BlobLogListener.cs#L181

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 15 (4 by maintainers)

Most upvoted comments

I’m trying to work through this now – we’ve had a series of deployment issues that have slowed a lot our progress down. But I’ll see if i can get a release out soon.

brettsam on Apr 16, 2020

This will be a part of the Microsoft.Azure.WebJobs.Extensions.Storage 3.0.11 release, which will likely happen in the next couple of weeks.

However, it’s already a part of the 4.0.0-preview1 release, which I’ve just pushed to Nuget this morning. That moves our storage extension up to newer Storage SDK nugets, so there are some breaking changes around that for C# functions (if directly using the Storage types), which you can see here: https://github.com/Azure/azure-webjobs-sdk/releases/tag/storage-v4.0.0-preview1

brettsam on Jan 24, 2020