azure-storage-net: (400) Bad Request - The specified block list is invalid. ErrorCode:InvalidBlockList

We are also facing the same issue and have done a little bit of an investigation in the application insights. Here is what we have found.

Trivia

The Put Block operation creates a new block to be committed as part of a blob.

The Put Block List operation writes a blob by specifying the list of block IDs that make up the blob. In order to be written as part of a blob, a block must have been successfully written to the server in a prior Put Block operation.

Successful upload

Consists of 1-n Put Block operations and final Put Block List operation.

Failed upload

In the failed upload the final Put Block List fails with message The specified block list is invalid.

Possible cause

It seems that storage SDK is executing the Put Block List in parallel with the Put Block and does not wait for the Put Block to finish.

Successful upload timeline Upload #1 - 1 block 2020-12-07T12:53:09.8442922Z + 0.01 2020-12-07T12:53:09.854617Z

Upload #2 - many blocks 2020-12-07T13:17:51.3556283Z + 0.371 2020-12-07T13:17:51.7276951Z

Failed upload timeline Upload #3 - 2 blocks 2020-11-27T13:27:44.3833279Z + 0.03 2020-11-27T13:27:44.3835717Z

Upload #4 - 2 blocks 2020-11-27T13:38:11.1370163Z + 0.105 2020-11-27T13:38:11.1372083Z

We are also using OpenWriteAsync() as was @angelcalvasp in their example, so my guess is the problem lies in the BlobWriteStream.

CommitAsync is calling PutBlockListAsync (the final operation) and before that it runs FlushAsync. It appears the FlushAsync should block until all pending block writes are finished, but I guess it doesn’t work properly.

Edited: SDK Version: 11.2.2 Platform: .Net Framework 4.7.2

The fix is available on Nuget. Version 11.2.3.

To provide update. I was able to establish a repro here https://github.com/kasobol-msft/azure-storage-net/commit/32f243196be7fc013e02da7e2f895ae32d848700 . (a bit messy but repros the issue consistently within minutes). And it seems that the culprit is in this line: https://github.com/Azure/azure-storage-net/blob/b0d47bfd4c1c0dcbe60eaf373e2977cd7405aa5f/Lib/Common/Core/Util/AsyncExtensions.Common.cs#L230-L233 I.e. the task scheduled in constructor may complete later and unlock the AsyncManualResetEvent in the middle of writting to stream.

We’ll be working on the fix.

Here’s sample run where this repro’ed:

Hi,

This issue is intermittently breaking the upload availability in our service. Since V12 includes lots of breaking changes, which will cost explicit time and effort to complete the upgrade, due to the emergency of this issue and our development schedule, upgrading to V12 is not an option as the fix of this issue.

According to the business announcement on https://github.com/Azure/azure-storage-net, • We will be making only fixes related data integrity and security for 11.x. • We will not be adding new storage service version support for this SDK. • We will not be back porting fixes and features added to the current version to the versions in this repo • We will not be making any changes to the performance characteristics of this SDK.

Apparently, this issue falls into the category of “related data integrity and security for 11.x”, it should be fixed in V11.

According to https://docs.microsoft.com/en-us/dotnet/api/overview/azure/storage?view=azure-dotnet, the V11 .net SDK is not out of support:

If you cannot update existing applications to version 12.x, then Microsoft recommends using version 11.x.

I will request a fix on this through the support channel of Azure blob storage service

I have a feeling that this issue may be caused by two tasks writing to the same blob at the same time…

It also appears to be the conclusion of those who reported issue #456