S3Mock: AWS CLI and boto3 file uploads using HTTP are failing in 4.0.0 due to erroneous 100-continue handling in botocore
Server fails to map AWS CLI request to on aws s3 cp:
o.s.web.servlet.DispatcherServlet : Exiting from "ERROR" dispatch, status 400
o.s.web.servlet.DispatcherServlet : PUT "/backups/README.md", parameters={}
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.adobe.testing.s3mock.ObjectController#putObject(String, ObjectKey, List, String, String, StorageClass, HttpHeaders, InputStream)
o.s.w.s.m.m.a.HttpEntityMethodProcessor : Found 'Content-Type:application/xml' in response
o.s.w.s.m.m.a.HttpEntityMethodProcessor : Writing [ErrorResponse[code=UnexpectedContent, message=This request contains unsupported content., resource=n (truncated)...]
The request is logged as:
Making request for OperationModel(name=PutObject) with params: {'url_path': '/README.md', 'query_string': {}, 'method': 'PUT', 'headers': {'Content-Type': 'text/markdown', 'User-Agent': 'aws-cli/2.15.37 Python/3.11.8 Linux/6.1.0-31-amd64 exe/x86_64.debian.12 prompt/off command/s3.cp', 'Content-MD5': 'rLtqOfcZH0Ny51fAf5RNrg==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7f0a3bd34f90>, 'auth_path': '/backups/README.md', 'url': 'http://localhost:9098/backups/README.md', 'context': {'client_region': 'eu-west-3', 'client_config': <botocore.config.Config object at 0x7f0a3c1ca410>, 'has_streaming_input': True, 'auth_type': 'v4', 'signing': {'region': 'eu-west-3', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'eu-west-3'}]}, 's3_redirect': {'redirected': False, 'bucket': 'backups', 'params': {'Bucket': 'backups', 'Key': 'README.md', 'Body': <s3transfer.utils.ReadFileChunk object at 0x7f0a3bd34f90>, 'ContentType': 'text/markdown'}}, 'S3Express': {'bucket_name': 'backups'}}}
Git bisect points at commit “Use Tomcat instead of Jetty”: https://github.com/adobe/S3Mock/commit/a3274149b6e9d43d0d1b445c547fc69a296137a3
Works with Docker Hub image 3.12.0 and fails with Docker hub image 4.0.0. Replicated locally by building commits with ./mvnw clean package -pl server,docker -am -DskipTests -Dcheckstyle.skip and starting server with docker run --rm -p 9098:9090 -e initialBuckets=backups -e debug=true adobe/s3mock:latest (not changing port mapping and using -p 9090:9090 yields to the same results): server.log.
CLI executed with aws --debug --endpoint-url http://localhost:9098 s3 cp README.md s3://backups/README.md (from the root of this project, hence project’s README being uploaded): cli.log
Fails with AWS CLI version 2.15.37 installed from Debian packages and also with version 2.22.10 running in an Alpine Docker container (FROM alpine:3.21.3 + apk add aws-cli).
aws --version
aws-cli/2.15.37 Python/3.11.8 Linux/6.1.0-31-amd64 exe/x86_64.debian.12 prompt/off
Other endpoints work fine:
aws --endpoint http://localhost:9098 s3api list-buckets
{
"Buckets": [
{
"Name": "backups",
"CreationDate": "2025-04-09T09:13:16.392000+00:00"
}
],
"Owner": {
"DisplayName": "s3-mock-file-store",
"ID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be"
}
}
Writing files works from curl:
curl -v --request PUT --upload-file ./README.md http://localhost:9098/backups/README.md
> PUT /backups/README.md HTTP/1.1
> Host: localhost:9098
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 42443
> Expect: 100-continue
>
< HTTP/1.1 100
* We are completely uploaded and fine
< HTTP/1.1 200
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Last-Modified: Wed, 09 Apr 2025 09:13:22 GMT
< ETag: "acbb6a39f7191f4372e757c07f944dae"
< Content-Length: 0
< Date: Wed, 09 Apr 2025 09:13:22 GMT
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 18 (8 by maintainers)
Commits related to this issue
- Pin down adobe/s3mock s3mock 4.0.0 came out with a bug: https://github.com/adobe/S3Mock/issues/2321 Should be fixed soon but in the meantime we downgrade to 3.12.0. Also we add a healthcheck for th... — committed to co-cddo/domains-register-a-govuk-domain by deleted user 2 years ago
- Pin down adobe/s3mock s3mock 4.0.0 came out with a bug: https://github.com/adobe/S3Mock/issues/2321 Should be fixed soon but in the meantime we downgrade to 3.12.0. Also we add a healthcheck for th... — committed to co-cddo/domains-register-a-govuk-domain by deleted user 2 years ago
- Pin down adobe/s3mock s3mock 4.0.0 came out with a bug: https://github.com/adobe/S3Mock/issues/2321 Should be fixed soon but in the meantime we downgrade to 3.12.0. Also we add a healthcheck for th... — committed to co-cddo/domains-register-a-govuk-domain by deleted user 2 years ago
- Pin down adobe/s3mock s3mock 4.0.0 came out with a bug: https://github.com/adobe/S3Mock/issues/2321 Should be fixed soon but in the meantime we downgrade to 3.12.0. Also we add a healthcheck for th... — committed to co-cddo/domains-register-a-govuk-domain by deleted user 2 years ago
- Pin down adobe/s3mock s3mock 4.0.0 came out with a bug: https://github.com/adobe/S3Mock/issues/2321 Should be fixed soon but in the meantime we downgrade to 3.12.0. Also we add a healthcheck for th... — committed to co-cddo/domains-register-a-govuk-domain by deleted user 2 years ago
See https://github.com/boto/botocore/pull/3455 https://github.com/boto/botocore/pull/3456
dug around some more. This is a discrepancy between Tomcat “100-continue” implementation and Boto3 expectations when making HTTP requests.
Jetty always sends a
HTTP/1.1 100 Continue: https://github.com/jetty/jetty.project/blob/jetty-12.0.x/jetty-core/jetty-http/src/main/java/org/eclipse/jetty/http/HttpStatus.java#L106 S3Mock used Jetty in 1.x, 2.x and 3.x The AWS CRT client does not work any more with Jetty 12.0.3+, see #2136Tomcat only sends a
HTTP/1.1 100: https://github.com/apache/tomcat/blob/main/java/org/apache/coyote/http11/Constants.java#L84 S3Mock uses Tomcat in 4.xBoto3 / Botocore expects the status to consist of 3 parts: https://github.com/boto/botocore/blob/1.37.34/botocore/awsrequest.py#L225 So only a
HTTP/1.1 100 Continueis detected as a correct response, even thoughHTTP/1.1 100is also correct.When using HTTPS in Boto3, the 100 continue is not sent, so no response is expected. This is why HTTPS with Tomcat and Boto3 is working, while HTTP is not.
RFC 9110 states that the reason phrase is optional: https://www.rfc-editor.org/rfc/rfc9110.html#name-control-data
https://www.rfc-editor.org/rfc/rfc9110.html#section-15.1
I’ll contribute a fix to the botocore project.
Hi @afranken
I have the same issue with
boto31.35.40 in Python.The usage looks like this:
Bucket is successfully getting created, but the last command raises an exception:
Similar to others, it works on 3.12.0.
for now, the recommendation is to use HTTPS with 4.0.0, as it works just fine.
I have tried multiple hours to get HTTP with Tomcat working, no success as of yet. I can’t say when or if HTTP support will be fixed.
I found the cause: for as of yet unknown reasons, the Tomcat HTTP connector does not respond correctly to the “100 continue” challenge of the boto3 client (it’s used in aws cli as well).
Example output from aws cli with
--debugand S3Mock 3.12.0:Same with S3Mock 4.0.0:
A simple workaround is to use HTTPS, Tomcat is correctly responding and all requests go through successfully.
Example for aws cli:
@afranken I’m uploading objects to the mock S3 instance as part of a one-off docker-compose task. And yes, I’m using the aws-cli for uploading these objects. The task looks like:
That is the same error message. Updated issue title to be clearer.
I’m not sure if it’s related but we had our CI integration test builds suddenly failing with the below error when using
4.0.0docker tag:Downgrading to the previous version made it go away, just FYI.