S3Mock: AWS CLI and boto3 file uploads using HTTP are failing in 4.0.0 due to erroneous 100-continue handling in botocore

Server fails to map AWS CLI request to on aws s3 cp:

o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 400
o.s.web.servlet.DispatcherServlet        : PUT "/backups/README.md", parameters={}
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.adobe.testing.s3mock.ObjectController#putObject(String, ObjectKey, List, String, String, StorageClass, HttpHeaders, InputStream)
o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Found 'Content-Type:application/xml' in response
o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [ErrorResponse[code=UnexpectedContent, message=This request contains unsupported content., resource=n (truncated)...]

The request is logged as:

Making request for OperationModel(name=PutObject) with params: {'url_path': '/README.md', 'query_string': {}, 'method': 'PUT', 'headers': {'Content-Type': 'text/markdown', 'User-Agent': 'aws-cli/2.15.37 Python/3.11.8 Linux/6.1.0-31-amd64 exe/x86_64.debian.12 prompt/off command/s3.cp', 'Content-MD5': 'rLtqOfcZH0Ny51fAf5RNrg==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7f0a3bd34f90>, 'auth_path': '/backups/README.md', 'url': 'http://localhost:9098/backups/README.md', 'context': {'client_region': 'eu-west-3', 'client_config': <botocore.config.Config object at 0x7f0a3c1ca410>, 'has_streaming_input': True, 'auth_type': 'v4', 'signing': {'region': 'eu-west-3', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'eu-west-3'}]}, 's3_redirect': {'redirected': False, 'bucket': 'backups', 'params': {'Bucket': 'backups', 'Key': 'README.md', 'Body': <s3transfer.utils.ReadFileChunk object at 0x7f0a3bd34f90>, 'ContentType': 'text/markdown'}}, 'S3Express': {'bucket_name': 'backups'}}}

Git bisect points at commit “Use Tomcat instead of Jetty”: https://github.com/adobe/S3Mock/commit/a3274149b6e9d43d0d1b445c547fc69a296137a3

Works with Docker Hub image 3.12.0 and fails with Docker hub image 4.0.0. Replicated locally by building commits with ./mvnw clean package -pl server,docker -am -DskipTests -Dcheckstyle.skip and starting server with docker run --rm -p 9098:9090 -e initialBuckets=backups -e debug=true adobe/s3mock:latest (not changing port mapping and using -p 9090:9090 yields to the same results): server.log.

CLI executed with aws --debug --endpoint-url http://localhost:9098 s3 cp README.md s3://backups/README.md (from the root of this project, hence project’s README being uploaded): cli.log

Fails with AWS CLI version 2.15.37 installed from Debian packages and also with version 2.22.10 running in an Alpine Docker container (FROM alpine:3.21.3 + apk add aws-cli).

aws --version
aws-cli/2.15.37 Python/3.11.8 Linux/6.1.0-31-amd64 exe/x86_64.debian.12 prompt/off

Other endpoints work fine:

aws --endpoint http://localhost:9098 s3api list-buckets
{
    "Buckets": [
        {
            "Name": "backups",
            "CreationDate": "2025-04-09T09:13:16.392000+00:00"
        }
    ],
    "Owner": {
        "DisplayName": "s3-mock-file-store",
        "ID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be"
    }
}

Writing files works from curl:

curl -v --request PUT --upload-file ./README.md http://localhost:9098/backups/README.md
> PUT /backups/README.md HTTP/1.1
> Host: localhost:9098
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 42443
> Expect: 100-continue
> 
< HTTP/1.1 100 
* We are completely uploaded and fine
< HTTP/1.1 200 
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Last-Modified: Wed, 09 Apr 2025 09:13:22 GMT
< ETag: "acbb6a39f7191f4372e757c07f944dae"
< Content-Length: 0
< Date: Wed, 09 Apr 2025 09:13:22 GMT

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 4
  • Comments: 18 (8 by maintainers)

Commits related to this issue

Most upvoted comments

dug around some more. This is a discrepancy between Tomcat “100-continue” implementation and Boto3 expectations when making HTTP requests.

Jetty always sends a HTTP/1.1 100 Continue: https://github.com/jetty/jetty.project/blob/jetty-12.0.x/jetty-core/jetty-http/src/main/java/org/eclipse/jetty/http/HttpStatus.java#L106 S3Mock used Jetty in 1.x, 2.x and 3.x The AWS CRT client does not work any more with Jetty 12.0.3+, see #2136

Tomcat only sends a HTTP/1.1 100: https://github.com/apache/tomcat/blob/main/java/org/apache/coyote/http11/Constants.java#L84 S3Mock uses Tomcat in 4.x

Boto3 / Botocore expects the status to consist of 3 parts: https://github.com/boto/botocore/blob/1.37.34/botocore/awsrequest.py#L225 So only a HTTP/1.1 100 Continue is detected as a correct response, even though HTTP/1.1 100 is also correct.

When using HTTPS in Boto3, the 100 continue is not sent, so no response is expected. This is why HTTPS with Tomcat and Boto3 is working, while HTTP is not.

RFC 9110 states that the reason phrase is optional: https://www.rfc-editor.org/rfc/rfc9110.html#name-control-data

Messages start with control data that describe its primary purpose. Request message control data includes a request method (Section 9), request target (Section 7.1), and protocol version (Section 2.5). Response message control data includes a status code (Section 15), optional reason phrase, and protocol version.

https://www.rfc-editor.org/rfc/rfc9110.html#section-15.1

The status codes listed below are defined in this specification. The reason phrases listed here are only recommendations – they can be replaced by local equivalents or left out altogether without affecting the protocol.

I’ll contribute a fix to the botocore project.

Hi @afranken

I have the same issue with boto3 1.35.40 in Python.

The usage looks like this:

import boto3

session = boto3.session.Session()
client = session.client("s3", endpoint_url="http://localhost:9090")
client.create_bucket(Bucket="my-bucket")
with open("file.txt", "rb") as data:
    client.upload_fileobj(data, "my-bucket", "my-key")

Bucket is successfully getting created, but the last command raises an exception:

botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "http://localhost:9090/my-bucket/my-key".

Similar to others, it works on 3.12.0.

Thank you for the clarification @afranken. Just to understand – is this the recommendation (using https) going forward or are there any plans to make the new major release work with http endpoints like it used to?

for now, the recommendation is to use HTTPS with 4.0.0, as it works just fine.

I have tried multiple hours to get HTTP with Tomcat working, no success as of yet. I can’t say when or if HTTP support will be fixed.

I found the cause: for as of yet unknown reasons, the Tomcat HTTP connector does not respond correctly to the “100 continue” challenge of the boto3 client (it’s used in aws cli as well).

Example output from aws cli with --debug and S3Mock 3.12.0:

2025-04-10 01:08:14,598 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <bound method UserAgentString.rebuild_and_replace_user_agent_handler of <botocore.useragent.UserAgentString object at 0x104d1f410>>
2025-04-10 01:08:14,598 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://localhost:9090/x/y.txt, headers={'Content-Type': b'text/plain', 'x-amz-sdk-checksum-algorithm': b'CRC32', 'User-Agent': b'aws-cli/2.25.13 md/awscrt#0.25.4 ua/2.1 os/macos#24.4.0 md/arch#arm64 lang/python#3.12.10 md/pyimpl#CPython m/N cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.put-object', 'Expect': b'100-continue', 'x-amz-checksum-crc32': b'NGSOug==', 'X-Amz-Date': b'20250409T230814Z', 'X-Amz-Content-SHA256': b'65dbc69741c062892a2842cc5d6099bbf02eefd4628d352d446c6c37cb0967da', 'Authorization': b'AWS4-HMAC-SHA256 Credential=NONE/20250409/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=37285ce19b27e82759690e90862796326813642064fbc021088488d477bf2f1e', 'Content-Length': '42443'}>
2025-04-10 01:08:14,598 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): localhost:9090
2025-04-10 01:08:14,599 - MainThread - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2025-04-10 01:08:14,613 - MainThread - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body.
2025-04-10 01:08:14,675 - MainThread - urllib3.connectionpool - DEBUG - http://localhost:9090 "PUT /x/y.txt HTTP/1.1" 200 0
2025-04-10 01:08:14,675 - MainThread - botocore.hooks - DEBUG - Event before-parse.s3.PutObject: calling handler <function _handle_200_error at 0x1029d4180>
2025-04-10 01:08:14,675 - MainThread - botocore.hooks - DEBUG - Event before-parse.s3.PutObject: calling handler <function handle_expires_header at 0x1029bff60>
2025-04-10 01:08:14,675 - MainThread - botocore.parsers - DEBUG - Response headers: {'Server': 'Jetty(12.0.12)', 'Date': 'Wed, 09 Apr 2025 23:08:14 GMT', 'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'x-amz-checksum-crc32': 'NGSOug==', 'Last-Modified': 'Wed, 09 Apr 2025 23:08:14 GMT', 'ETag': '"44a97f35fa58f36daa1cb42955a2bea0"', 'Content-Length': '0'}
2025-04-10 01:08:14,675 - MainThread - botocore.parsers - DEBUG - Response body:
b''

Same with S3Mock 4.0.0:

2025-04-10 01:41:22,320 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <bound method UserAgentString.rebuild_and_replace_user_agent_handler of <botocore.useragent.UserAgentString object at 0x1052779b0>>
2025-04-10 01:41:22,320 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://localhost:9090/x/y.txt, headers={'Content-Type': b'text/plain', 'x-amz-sdk-checksum-algorithm': b'CRC32', 'User-Agent': b'aws-cli/2.25.13 md/awscrt#0.25.4 ua/2.1 os/macos#24.4.0 md/arch#arm64 lang/python#3.12.10 md/pyimpl#CPython m/N cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.put-object', 'Expect': b'100-continue', 'x-amz-checksum-crc32': b'NGSOug==', 'X-Amz-Date': b'20250409T234122Z', 'X-Amz-Content-SHA256': b'65dbc69741c062892a2842cc5d6099bbf02eefd4628d352d446c6c37cb0967da', 'Authorization': b'AWS4-HMAC-SHA256 Credential=NONE/20250409/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=e6a74e4aafab3b1c03e282cf2064d65dbb22af4396cb34460f69439d180f36d3', 'Content-Length': '42443'}>
2025-04-10 01:41:22,320 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): localhost:9090
2025-04-10 01:41:22,321 - MainThread - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2025-04-10 01:41:22,324 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <function _update_status_code at 0x101d1c2c0>
2025-04-10 01:41:22,324 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x105277c80>>
2025-04-10 01:41:22,324 - MainThread - botocore.retries.standard - DEBUG - Retry needed, retrying request after delay of: 0.2322557477907542

A simple workaround is to use HTTPS, Tomcat is correctly responding and all requests go through successfully.

Example for aws cli:

aws s3api put-object \
    --bucket x \
    --key y.txt \
    --content-type text/plain \
    --endpoint https://localhost:9191 \
    --body README.md \
    --no-verify-ssl \
    --debug

@afranken I’m uploading objects to the mock S3 instance as part of a one-off docker-compose task. And yes, I’m using the aws-cli for uploading these objects. The task looks like:

  init-s3:
    image: amazon/aws-cli:latest
    env_file: .env.integration-tests
    volumes:
      - ./integration/s3_init_data:/s3_init_data
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        aws s3 cp --endpoint-url=$${S3_ENDPOINT} /s3_init_data/somefile ://my_custom_bucket/somefile
        // and so on

upload failed: /some/local/path to s3://some/s3/path Connection was closed before we received a valid response from endpoint URL: "http://s3:9090/some/s3/path.

That is the same error message. Updated issue title to be clearer.

I’m not sure if it’s related but we had our CI integration test builds suddenly failing with the below error when using 4.0.0 docker tag:

upload failed: /some/local/path to s3://some/s3/path Connection was closed before we received a valid response from endpoint URL: "http://s3:9090/some/s3/path.

Downgrading to the previous version made it go away, just FYI.