azure-functions-host: Microsoft.Azure.WebJobs.Script.WebHost: Could not create BlobContainerClient.

Hi,

During the last couple of days we noticed that our function apps are not starting with the following error: Microsoft.Azure.WebJobs.Script.WebHost: Could not create BlobContainerClient.

We managed to find a workaround by setting FUNCTIONS_EXTENSION_VERSION = ~3.0.15885. If it is set to ~3, the function might start or might not start. Or might start and eventually stop working. We are using Consumption Plans.

The log stream provides this information: 2021-06-14T07:35:26.619 [Information] Initializing Warmup Extension. 2021-06-14T07:35:26.635 [Error] A host error has occurred during startup operation ‘bbe85313-1d1b-4435-a581-34f942e2274e’.System.InvalidOperationException : Could not create BlobContainerClientat async Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.ReadAsync(ScriptSecretsType type,String functionName) at D:\a\1\s\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs : 103at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.LoadSecretsAsync(??) at D:\a\1\s\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs : 392at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.LoadSecretsAsyncT at D:\a\1\s\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs : 385at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.GetHostSecretsAsync() at D:\a\1\s\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs : 92at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at async Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetOrCreateExtensionKey(String extensionName) at D:\a\1\s\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs : 74at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetExtensionWebHookRoute(String extensionName) at D:\a\1\s\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs : 66at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetUrl(IExtensionConfigProvider extension) at D:\a\1\s\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs : 51at Microsoft.Azure.WebJobs.Host.Config.ExtensionConfigContext.GetWebhookHandler() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Config\ExtensionConfigContext.cs : 55at Microsoft.Azure.WebJobs.Extensions.EventGrid.EventGridExtensionConfigProvider.Initialize(ExtensionConfigContext context)at Microsoft.Azure.WebJobs.Host.DefaultExtensionRegistryFactory.Create() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\DefaultExtensionRegistryFactory.cs : 38at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_0(IServiceProvider p) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs : 56at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs : 156at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 4550at System.Linq.Expressions.Interpreter.FuncCallInstruction3.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.LightLambda.Run(Object[] arguments)at Thunk(Func2 ,IResolverContext )at DryIoc.Factory.<>c__DisplayClass26_0.<ApplyReuse>b__2() at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6605at DryIoc.Scope.TryGetOrAdd(ImMap1 items,Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7849at DryIoc.Scope.GetOrAdd(Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7834at DryIoc.Factory.ApplyReuse(Expression serviceExpr,Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6604at DryIoc.Factory.GetExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6564at DryIoc.Factory.GetDelegateOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6634at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7740at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,Object serviceKey,IfUnresolved ifUnresolved,Type requiredServiceType,Request preResolveParent,Object[] args) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 291at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow(TargetInvocationException exception)at System.Linq.Expressions.Interpreter.MethodInfoCallInstruction.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.LightLambda.Run(Object[] arguments)at Thunk(Func2 ,IResolverContext )at DryIoc.Factory.<>c__DisplayClass26_0.<ApplyReuse>b__2() at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6605at DryIoc.Scope.TryGetOrAdd(ImMap1 items,Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7849at DryIoc.Scope.GetOrAdd(Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7834at DryIoc.Factory.ApplyReuse(Expression serviceExpr,Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6604at DryIoc.Factory.GetExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6564at DryIoc.Factory.GetDelegateOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6634at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType,IfUnresolved ifUnresolved) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 211at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,IfUnresolved ifUnresolved) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 196at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType,IfUnresolved ifUnresolved) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs : 99at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs : 82at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider,Type serviceType)at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_4(IServiceProvider p) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs : 91at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs : 156at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 4550at System.Linq.Expressions.Interpreter.FuncCallInstruction3.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.LightLambda.Run(Object[] arguments)at Thunk(Func2 ,IResolverContext )at DryIoc.Factory.<>c__DisplayClass26_0.<ApplyReuse>b__2() at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6605at DryIoc.Scope.TryGetOrAdd(ImMap1 items,Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7849at DryIoc.Scope.GetOrAdd(Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7834at DryIoc.Factory.ApplyReuse(Expression serviceExpr,Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6604at DryIoc.Factory.GetExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6564at DryIoc.Factory.GetDelegateOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6634at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7740at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,Object serviceKey,IfUnresolved ifUnresolved,Type requiredServiceType,Request preResolveParent,Object[] args) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 291at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow(TargetInvocationException exception)at System.Linq.Expressions.Interpreter.MethodInfoCallInstruction.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame frame)at System.Linq.Expressions.Interpreter.LightLambda.Run(Object[] arguments)at Thunk(Func2 ,IResolverContext )at DryIoc.Factory.<>c__DisplayClass26_0.<ApplyReuse>b__2() at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6605at DryIoc.Scope.TryGetOrAdd(ImMap1 items,Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7849at DryIoc.Scope.GetOrAdd(Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7834at DryIoc.Factory.ApplyReuse(Expression serviceExpr,Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6604at DryIoc.Factory.GetExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6564at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7083at DryIoc.Factory.GetExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6554at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7083at DryIoc.Factory.GetExpressionOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6554at DryIoc.Factory.GetDelegateOrDefault(Request request) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6634at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType,IfUnresolved ifUnresolved) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 211at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,IfUnresolved ifUnresolved) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 196at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType,IfUnresolved ifUnresolved) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs : 99at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType) at D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs : 77at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)at async Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation,Int32 attemptCount,JobHostStartupMode startupMode) at D:\a\1\s\src\WebJobs.Script.WebHost\WebJobsScriptHostService.cs : 248

Timestamp: 2021-06-14T07:35:26.619
Function App version: v2
Function App name: e4tkxrdwdqk37ezd
Region: West Europe

Programming language used c#
Bindings used Queue, EventGrid

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21

Most upvoted comments

For me, I had to set FUNCTIONS_EXTENSION_VERSION back to 3.0.15733 and the function app started working again.

Note: some of our function apps appear fine on 3.0.15960 while others appear to have this issue until I pin them back to 3.0.15733

steve-sharman on Jun 15, 2021

Thank you for the information, team is already investigating on this issue. Any information will be updated shortly.

v-anvari on Jun 15, 2021

I just sent email to @v-anvari with reproduction steps. I spent a lot of time trying to narrow it down. Indeed if you play around with application settings, it may temporary help out, but it gets back to error (maybe in 5 minutes, maybe in an hour…). The only stable workaround I know, is to set FUNCTIONS_EXTENSION_VERSION = ~3.0.15885. With this setting we have ~70 functionapps running smoothly “for now” for a whole day. During my simplified reproduction steps, it looks to me that it is not related to infrastructure or to function app settings, but rather to nuget packages referenced in project. I hope @v-anvari can pinpoint the problem. Here are my packages:

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="4.0.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid" Version="2.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Http" Version="3.0.12" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="4.0.4" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Host.Storage" Version="4.0.1" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.13" />
  </ItemGroup>

Kravca on Jun 15, 2021

We are also experiencing the same error since friday. We deploy multiple functions the same way (az cli deploy (create+config) -> stop -> dotnet publish -> start), but the error is happening only to some of them (maybe some dll hell because they both use Microsoft.Azure.WebJobs.Extensions.EventGrid).

Now, I noticed… when I go to portal -> failing function -> Configuration -> + New application setting -> whathever/whatever -> Save -> The App fixes itselfs

First I suspected dll hell due to event grid conflict with something else, but I am not sure now. Is the configuration set by az cli (az functionapp config appsettings set) broken and saving the configuration in the portal fixes it? I am confused about this.

PetrPospisil on Jun 15, 2021