async-http-client: Error when using HTTP2 for AWS S3 request

Hello! I found an issue while using Vapor 4 and the Soto library to upload and delete objects from a DigitalOcean S3-compatible bucket. After some debugging you can follow in https://github.com/soto-project/soto/issues/608 we arrived at the conclusion that the issue must be in the AsyncHttpClient library.

You can follow the discussion in the linked issue but to sum up:

Using HTTP2 the delete object request fails with NotImplemented error meaning there is some unknown header according to the docs.
Using HTTP1 the request works fine. Also using HTTP2 and proxying it through Charles and even manually doing the cURL it also works fine.

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 1
Comments: 19 (3 by maintainers)

Most upvoted comments

Hi! One of the DigitalOcean Spaces engineers here.

Thanks for the reproduction case in https://github.com/pballart/soto-s3-digitalocean. It was really helpful.

The HTTP/2 stack in use is almost-stock Envoy Proxy. But the problem is a few layers deeper and involves the intersection: we use Ceph for S3 (this has been published and we’ve given talks about it).

Envoy converts HTTP/2 to transfer-encoding chunked for talking to HTTP/1.1 systems. The empty DATA frame is converted to a similar empty chunk.

With that in mind, I produced a HTTP/1.1 version of the bug:

curl 'https://REDACTED.nyc3.digitaloceanspaces.com/empty' -X DELETE -H 'x-amz-date: 20220818T235940Z' -H 'Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20220818/nyc3/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=REDACTED' -H 'x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'  -v -H "Transfer-Encoding: chunked" --http1.1

AWS gets it correct, Ceph gets it slightly wrong 😦.

This behavior has the potential to affect S3 operations that have no payloads. However, some of them DO work…

On the plus side, fixing Ceph looks like it would be easy, just a few operations are missing here: https://github.com/ceph/ceph/blob/main/src/rgw/rgw_rest_s3.cc#L5639-L5691

As a workaround for now, can you send Content-Length: 0 for these operations in your Soto system? (I’m not familiar with Swift, so I don’t know how hard that would be).

robbat2 on Aug 19, 2022

Hello! Sorry I forgot about this one 😓 Here you have the sample project: https://github.com/pballart/soto-s3-digitalocean You just need to configure the access to the bucket and try the deleteImage endpoint.

pballart on Aug 3, 2022