azure-sdk-for-net: Azure.Storage.Blobs.BlobClient.UploadAsync throws System.Net.Sockets.SocketException

I have also stuck with this issue. Its fix is very simple, just set your stream position as 0.

fileStream.Position = 0;

All set, it will work.

100 seconds is the HttpClient Timeout default. You need to tweak your Transport if you want a longer timeout. You can do that via BlobClientOptions when creating your client:

new BlobClientOptions
{
    Transport = new HttpClientTransport(
        new HttpClient { Timeout = TimeSpan.FromSeconds(102) })
};

The best practice would be to create a static HttpClient instance with a longer Timeout somewhere that you share across all your Azure clients.

100 seconds is the HttpClient Timeout default. You need to tweak your Transport if you want a longer timeout. You can do that via BlobClientOptions when creating your client:

new BlobClientOptions
{
    Transport = new HttpClientTransport(
        new HttpClient { Timeout = TimeSpan.FromSeconds(102) })
};

The best practice would be to create a static HttpClient instance with a longer Timeout somewhere that you share across all your Azure clients.

Unfortunately, 100 seconds also happens to be the default RetryOptions.NetworkTimeout value, which meant for me that I was still timing out with the above even when I set Timeout.InfiniteTimeSpan. This did the trick though:

new DataLakeClientOptions // or whatever options class suitable for your client
{
    Transport = new HttpClientTransport( new HttpClient { Timeout = Timeout.InfiniteTimeSpan } ),
    Retry = { NetworkTimeout = Timeout.InfiniteTimeSpan } // <== this was the key
}

@jaschrep-msft - Yes, the data was successfully loaded. I have found the file in storage with the same ETag value reported in the response above. I have also confirmed that the Content-MD5 reported in the response above matches the MD5 hash of my local file.

To recap. A request is made to upload the file (from local system drive) using the following code.

var blobClient = new Azure.Storage.Blobs.BlobClient(new Uri("https://<storageAcct>.blob.core.windows.net/<container>/<blobName>?<sas>"));

await using (var fileStream = System.IO.File.OpenRead("<path>"))
{
    await blobClient.UploadAsync(fileStream, true, CancellationToken.None);
}

In Fiddler, I see the following session (request only) with storageAcct.blob.core.windows.net

PUT https://<storageAcct>.blob.core.windows.net/<container>/<blobName>?<sas> HTTP/1.1
Host: <storageAcct>.blob.core.windows.net
x-ms-blob-type: BlockBlob
x-ms-version: 2019-02-02
x-ms-client-request-id: e81bc21c-5331-4684-8907-ffd6fa1f2208
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.1.0+ee51b9a6328321b8bc491824116ec9038f2fed5b (.NET Core 3.1.1; Microsoft Windows 10.0.17134)
Content-Length: 38771467

100 seconds later (give or take a tenth) the application logs the following error.

System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
 ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.

The application moves to upload more files. 7 minutes and 25 seconds later (my application is still executing as it continues to upload files), Fiddler logs the following 201 Created response in the storageAcct.blob.core.windows.net session.

HTTP/1.1 201 Created
Content-Length: 0
Content-MD5: WHHcDGfrHi9e8zJpVQ8sCw==
Last-Modified: Wed, 15 Jan 2020 02:32:25 GMT
ETag: "0x8D79963285CD733"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: c3479df1-501e-0131-0b4a-cb8caa000000
x-ms-client-request-id: e81bc21c-5331-4684-8907-ffd6fa1f2208
x-ms-version: 2019-02-02
x-ms-content-crc64: xsaeQ/qoKB4=
x-ms-request-server-encrypted: true
Date: Wed, 15 Jan 2020 02:32:24 GMT

Why is the Task being cancelled after 100 seconds when clearly the upload is still being carried out even after the System.Threading.Tasks.TaskCanceledException is raised?

The fact that the upload continues and succeeds is great. That is exactly what I want.

The fact that a System.Threading.Tasks.TaskCanceledException is raised is bad, as the application (and the end user) thinks the upload has failed.

100 seconds is the HttpClient Timeout default. You need to tweak your Transport if you want a longer timeout. You can do that via BlobClientOptions when creating your client:

new BlobClientOptions
{
    Transport = new HttpClientTransport(
        new HttpClient { Timeout = TimeSpan.FromSeconds(102) })
};

The best practice would be to create a static HttpClient instance with a longer Timeout somewhere that you share across all your Azure clients.

Is there a way to achieve the same when using the HttpWebRequest-based transport which is the default on .NET Framework since Azure.Core 1.5.0?

I can verify that using BlobClientOptions to pass a singleton HttpClient with an extended timeout period resolves my issue.

Thanks to everyone involved in helping resolve this issue.

@tg-msft - Thank you. Once I observed that System.Threading.Tasks.TaskCanceledExceptions were being raised right at 100 seconds, I figured that this was down to the underlying HttpClient used to interact with the Blob Service REST API. Due to time constraints, I hadn’t been able to dig further into what options are available configure the blob client. Thanks for pointing me in the right direction.

@jaschrep-msft @seanmcc-msft - Give me chance to try using BlobClientOptions to pass an extended timeout value and I’ll let you know how it goes.

I should add, that I am working with a pretty slow connection with sub 5 Mbps upload speeds.

It very well could be that this issue could be ameliorated by having a faster connection. But I do still wonder what is required to force a file to be uploaded in small chunks rather than in one go.

@jaschrep-msft, I can reliably recreate this issue on my system.

I have traced the request/response with Fiddler. Here is a sample request/response scenario.

REQUEST

	PUT https://<storageAcct>.blob.core.windows.net/<container>/<blobName>?<sas> HTTP/1.1
	Host: <storageAcct>.blob.core.windows.net
	x-ms-blob-type: BlockBlob
	x-ms-version: 2019-02-02
	x-ms-client-request-id: e81bc21c-5331-4684-8907-ffd6fa1f2208
	x-ms-return-client-request-id: true
	User-Agent: azsdk-net-Storage.Blobs/12.1.0+ee51b9a6328321b8bc491824116ec9038f2fed5b (.NET Core 3.1.1; Microsoft Windows 10.0.17134)
	Content-Length: 38771467

RESPONSE - Overall Elapsed:	0:07:25.119

	HTTP/1.1 201 Created
	Content-Length: 0
	Content-MD5: WHHcDGfrHi9e8zJpVQ8sCw==
	Last-Modified: Wed, 15 Jan 2020 02:32:25 GMT
	ETag: "0x8D79963285CD733"
	Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
	x-ms-request-id: c3479df1-501e-0131-0b4a-cb8caa000000
	x-ms-client-request-id: e81bc21c-5331-4684-8907-ffd6fa1f2208
	x-ms-version: 2019-02-02
	x-ms-content-crc64: xsaeQ/qoKB4=
	x-ms-request-server-encrypted: true
	Date: Wed, 15 Jan 2020 02:32:24 GMT

Note that the response captured/returned is 201, but the BlobClient returned a “System.Threading.Tasks.TaskCanceledException: The operation was canceled.” after exactly 100 seconds.

When attempting to copy this file (stored locally on system SSD drive), the attempt fails with the same error I reported every single time after exactly 100 seconds.

When attempting to copy this file (stored remotely on SharePoint and accessed via WebDAV protocol), the attempt fails with the same error I reported every single time but with variation (E.g. 206s, 152s, 229s, 163s).

As long as the process that is uploading files is still running, it appears that the socket for these requests stays open and eventually the file uploads with the Blob service returning 201 after the client upload was cancelled.

I suspected that Fiddler (and the SSL Tunnel) may be influencing the results of these tests. But…

With Fiddler, requests to upload local files on system SSD drive always fail after exactly 100 seconds (despite continuing to upload).

Without Fiddler, requests to upload local files on system SSD, the error becomes more intermittent, but nevertheless still occurs. And it appears that all failures occur after exactly 100 seconds. All successful attempts take less than 100 seconds.

So it would seem that I am hitting a default timeout for web requests.

NOTE: I see that the request to upload the file is being done in one single PUT operation. Under what conditions should I expect to see the file being chunked?