azure-storage-net: StorageException thrown in 4.2.1
Using the NuGet package “WindowsAzure.Storage” version 4.2.1, the following C# code…
try
{
int maxRetryCount = 3;
var blobRequestOptions = new BlobRequestOptions
{
RetryPolicy = new ExponentialRetry(TimeSpan.FromSeconds(5), maxRetryCount),
MaximumExecutionTime = TimeSpan.FromMinutes(60),
ServerTimeout = TimeSpan.FromMinutes(60)
};
using (var fileStream = File.Create(localPath))
{
blockBlob.DownloadToStream(fileStream, null, blobRequestOptions);
}
}
catch (Exception e)
{
logger.Error(string.Format("Downloading {0}: {1}", blobName, e.Message));
}
…sometimes throws the following exception (mainly when downloading a big blob, ~500 MB, from a storage account that is far away from the website):
[06/29/2015 17:53:44 > 324d4d: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. —> System.TimeoutException: The client could not finish the operation within specified timeout. [06/29/2015 17:53:44 > 324d4d: ERR ] — End of inner exception stack trace — [06/29/2015 17:53:44 > 324d4d: ERR ] at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() [06/29/2015 17:53:44 > 324d4d: ERR ] at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult) [06/29/2015 17:53:44 > 324d4d: ERR ] at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar) [06/29/2015 17:53:44 > 324d4d: ERR ] — End of stack trace from previous location where exception was thrown — [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) [06/29/2015 17:53:44 > 324d4d: ERR ] at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<TryExecuteAsync>d__9.MoveNext() [06/29/2015 17:53:44 > 324d4d: ERR ] — End of stack trace from previous location where exception was thrown — [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) [06/29/2015 17:53:44 > 324d4d: ERR ] at Microsoft.Azure.WebJobs.Host.Timers.RecurrentTaskSeriesCommand.<ExecuteAsync>d__0.MoveNext() [06/29/2015 17:53:44 > 324d4d: ERR ] — End of stack trace from previous location where exception was thrown — [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) [06/29/2015 17:53:44 > 324d4d: ERR ] at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext() [06/29/2015 17:53:44 > 324d4d: ERR ] — End of stack trace from previous location where exception was thrown — [06/29/2015 17:53:44 > 324d4d: ERR ] at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0() [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Threading.ThreadHelper.ThreadStart_Context(Object state) [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) [06/29/2015 17:53:44 > 324d4d: ERR ] at System.Threading.ThreadHelper.ThreadStart()
How can I prevent this??
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 17 (6 by maintainers)
Is there yet a solution to this ? I have a continuous webjob that fails after about 8 minutes with the following:
An unhandled exception of type ‘Microsoft.WindowsAzure.Storage.StorageException’ occurred in Microsoft.Azure.WebJobs.Host.dll
Additional information. The client could not finish the operation within specified timeout.
Hey guys, I’m from the WebJobs team. For the persons on this thread seeing timeouts with WebJobs SDK code in the stack trace (e.g. Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand), when your job function includes a TextWriter/TraceWriter parameter and writes logs to it, the SDK will periodically flush those logs to blob storage, so you can see the output in the WebJobs Dashboard. Generally this is not much data, so there shouldn’t be any timeouts. Hopefully you’re not trying to log huge amounts of data to the function log. I haven’t heard other WebJobs SDK users complain of this, so it might be that you’re doing something unusual, or you’re saturating your network/connections causing the timeouts?
There is a way to configure the CloudBlobClient that the SDK uses for this - you can set the CloudBlobClient.DefaultRequestOptions.ServerTimeout to a larger value (the SDK just accepts whatever the Azure Storage default is). You can configure this by providing your own StorageClientFactory via JobHostConfiguration.StorageClientFactory, overriding the creation of blob clients setting whatever options you need. However, I’m not yet sure increasing that is the right thing to do - it might be that you need to address underlying bandwith issues with your job function.
It looks like you are using WebJobs, based on the stack trace: Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand
So the error is coming from a WebJobs call to storage. Could you describe a little about your setup, in particular how you are using WebJobs?