runtime: HttpRequestException when creating empty key in AWS S3

Hello,

I’m getting a System.Net.Http.HttpRequestException after an AWS S3 PUT operation. I’m trying to create an S3 “folder” (ie: an empty object) usign the AWS SDK for S3. The folder itself is successfully created. The error happens on every request. Environment is ASP.NET Core 2.1, Windows 10 (dev computer).

I’m currently working around this by catching the exception and re-querying the S3 key to confirm creation. But something looks very out of place. And surely AWS is so heaveily used that this bug is would be rampent! (Which makes me wonder if this is something special or unusual). The stack dump is pointing at HttpConnection.SendAsyncCore(), so I don’t think this is Amazon’s fault.

I’ve checked in HttpConnection.SendAsyncCore() but that method is… complex. And the places where ThrowInvalidHttpResponse() is called don’t obviously correspond to anythign wrong with the HTTP response.

From the HTTP response, the only error I can see is it’s 200 OK rather than 204 Empty. Other corefx issues raised with this exception have missing headers or extra whitespace or missing colons, but the headers all look OK to me.

It would also be nice for the exception to “tag” which site it is thrown from, given there are ~15 possible places in HttpConnection.cs. Even if its an error number or short code, anything to narrow down the exact throw!

Regards Murray

HTTP Request

PUT https://circuitlink-bt.s3.ap-southeast-2.amazonaws.com/_Unprocessed/BTR12357/FIRMWARE/ HTTP/1.1
Expect: 100-continue
User-Agent: aws-sdk-dotnet-coreclr/3.3.18.3 aws-sdk-dotnet-core/3.3.24.0 .NET_Core/4.6.26515.07 OS/Microsoft_Windows_10.0.16299 ClientAsync
Host: circuitlink-bt.s3.ap-southeast-2.amazonaws.com
X-Amz-Date: 20180612T023113Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=[redacted]/20180612/ap-southeast-2/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=[redacted]
Content-Length: 0
Content-Type: text/plain

HTTP Response

HTTP/1.1 200 OK
x-amz-id-2: VECzyxSK/10lyVcQQickFaiZxvDpNRQiUKhUXN2IHQJD6OpJ+pJWGXvimA0YjeZlcKFwGJisoEU=
x-amz-request-id: 7CB4D3B7B2438CA0
Date: Tue, 12 Jun 2018 02:31:15 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Content-Length: 0
Server: AmazonS3

Client Code

public async Task CreateDirectory(string relativePath)
{
	// relativePath = "FIRMWARE"
    var putRequest = new PutObjectRequest();
    putRequest.BucketName = _Config.BucketName;		// = "circuitlink-bt"
    putRequest.Key = (String.IsNullOrEmpty(_Config.BasePath) ? "" : _Config.BasePath + "/")		// BasePath = "_Unprocessed"
                + relativePath + "/";       // Need trailing slash to create a folder.
    putRequest.ContentBody = "";
    Logger.Debug("Creating directory '{0}' in S3 bucket '{1}'.", putRequest.Key, putRequest.BucketName);
	// Logs: Creating directory '_Unprocessed/BTR12357/FIRMWARE/' in S3 bucket 'circuitlink-bt'.
    PutObjectResponse response;
    try
    {
        response = await _S3Client.PutObjectAsync(putRequest);
		// Next line is never hit.
        Logger.Debug("Created directory '{0}' in S3 bucket '{1}' successfully.", putRequest.Key, putRequest.BucketName);
    }
    catch (System.Net.Http.HttpRequestException ex)
    {
        Logger.Error(ex, "Exception while creating S3 directory '{0}' in S3 bucket '{1}'.", putRequest.Key, putRequest.BucketName);
		// Logs: Exception while creating S3 directory '_Unprocessed/BTR12357/FIRMWARE/' in S3 bucket 'circuitlink-bt'.
		// Stack dump below.
        throw;
    }
}

Exception and Stack Dump

13:57:36.6|DEBUG| | 4: | CL.BTCloud.Services.AWS.DeviceServerStorage.DeviceServerStorageOnS3 | http://localhost/api/register/BTC| HttpRequestException while creating S3 directory '_Unprocessed/BTR12357/FIRMWARE/' in S3 bucket 'circuitlink-bt'. System.Net.Http.HttpRequestException: The server returned an invalid or unrecognized response.
   at System.Net.Http.HttpConnection.ThrowInvalidHttpResponse()
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_mobile\HttpRequestMessageFactory.cs:line 428
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 175
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CredentialsRetriever.cs:line 98
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 137
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at CL.BTCloud.Services.AWS.DeviceServerStorage.DeviceServerStorageOnS3.CreateDirectory(String relativePath) in C:\Projects\CircuitLink\Intelligo\Braketesta\Shared\BTCloud.Services.AWS\DeviceServerStorage\DeviceServerStorageOnS3.cs:line 47