aws-sdk-java-v2: SSLPeerUnverifiedException on S3 actions

Hi @debora-ito , I’ve observed endpoints .s3.amazonaws.com and .s3.us-east-1.amazonaws.com return different certificates.

Global endpoint:

$ true | openssl s_client -connect some-bucket.s3.amazonaws.com:443 2>/dev/null |  openssl x509 -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            08:2d:f6:8e:e9:c6:93:15:be:bf:72:07:9b:38:10:fd
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Baltimore CA-2 G2
        Validity
            Not Before: Nov  9 00:00:00 2019 GMT
            Not After : Mar 12 12:00:00 2021 GMT
        Subject: C=US, ST=Washington, L=Seattle, O=Amazon.com, Inc., CN=*.s3.amazonaws.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:*.s3.amazonaws.com, DNS:s3.amazonaws.com
...

Regional endpoint:

$ true | openssl s_client -connect some-bucket.s3.us-east-1.amazonaws.com:443 2>/dev/null |  openssl x509 -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0d:64:50:6b:45:f3:0c:e3:5a:6c:2d:df:2c:18:b4:37
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Baltimore CA-2 G2
        Validity
            Not Before: Aug  4 00:00:00 2020 GMT
            Not After : Aug  9 12:00:00 2021 GMT
        Subject: C=US, ST=Washington, L=Seattle, O=Amazon.com, Inc., CN=s3.amazonaws.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:s3.amazonaws.com, DNS:*.s3.amazonaws.com, DNS:*.s3.dualstack.us-east-1.amazonaws.com, DNS:s3.dualstack.us-east-1.amazonaws.com, DNS:*.s3.us-east-1.amazonaws.com, DNS:s3.us-east-1.amazonaws.com, DNS:*.s3-control.us-east-1.amazonaws.com, DNS:s3-control.us-east-1.amazonaws.com, DNS:*.s3-control.dualstack.us-east-1.amazonaws.com, DNS:s3-control.dualstack.us-east-1.amazonaws.com, DNS:*.s3-accesspoint.us-east-1.amazonaws.com, DNS:*.s3-accesspoint.dualstack.us-east-1.amazonaws.com, DNS:*.s3.us-east-1.vpce.amazonaws.com
....

So, this code:

import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest
import software.amazon.awssdk.services.s3.model.DeleteObjectsRequest
import software.amazon.awssdk.services.s3.model.GetObjectRequest
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request
import software.amazon.awssdk.services.s3.model.ObjectIdentifier
import software.amazon.awssdk.services.s3.model.S3Object
import software.amazon.awssdk.services.s3.presigner.S3Presigner

fun main() {
        val bucket = awsConfiguration.bucket
        val prefix = "foo/bar"

        val listObjectsV2PaginatorResult = s3Client.listObjectsV2Paginator(
            ListObjectsV2Request
                .builder()
                .bucket(bucket)
                .prefix(prefix)
                .build()
        )

        val keys: List<String> = listObjectsV2PaginatorResult
            .contents()
            .stream()
            .map { it.key() }
            .toList()

        logger.info { "==================>>>>> KEYS: $keys" }
}

Worked in EU-CENTRAL-1 , but returned SdkClientException: Unable to execute HTTP request: Certificate for <some-bucket.s3.amazonaws.com> doesn’t match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] is US-EAST-1.

Full stack trace:

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Certificate for <some-bucket.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.handleThrownException(RetryableStage.java:137)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:95)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:63)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:43)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:81)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:61)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:198)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:122)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:148)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:102)
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
    at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:4926)
    at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:147)
    at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:138)
    at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)
    at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable$ItemsIterator.<init>(PaginatedItemsIterable.java:58)
    at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable.iterator(PaginatedItemsIterable.java:48)
    at java.lang.Iterable.spliterator(Iterable.java:101)
    at software.amazon.awssdk.core.pagination.sync.SdkIterable.stream(SdkIterable.java:34)

Setting AWS_S3_US_EAST_1_REGIONAL_ENDPOINT to regional fixed it.

But I believe this is a bug because the exception is raised even if the region is being explicitly passed to the S3Client builder:

        S3Client  s3Client = S3Client.builder()
                .region(Region.of(region))
                .credentialsProvider(DefaultCredentialsProvider.create())
                .build();

@pwannenmacher from my comment (https://github.com/aws/aws-sdk-java-v2/issues/1786#issuecomment-904148428),

One workaround is to force a very specific Apache version: 4.5.10. … As people noted here, the issue happens only when using recent Apache versions. That is, 4.5.10 works but 4.5.12 or 4.5.13 fails. … do note that using an older version prior to 4.5.10 may fail with the same error, so make sure you enforce the right version.

Our customers don’t use Java SDK at all, so the issue is fundamentally unrelated to the SDK.

Back then, I meticulously checked the code of all the Apache versions. However, nothing seemed wrong, and I couldn’t explain how this could happen. I was very perplexed. Our customers (who don’t use the Java SDK) hit this issue only with Amazon S3.

Incase it helps anyone else, I was hitting this error as part of a spark job using org.apache.hadoop.fs.s3a.S3AFileSystem.

Setting these fixed the problem:

spark.hadoop.fs.s3a.endpoint=s3.us-east-1.amazonaws.com
spark.hadoop.fs.s3a.path.style.access=true

I’ve just had this very issue and can confirm that from the errors it’s quite obvious that it is linked to dots in the S3 bucket name. It is a standard practice to set your S3 bucket name like a reverse DNS name to ensure no collisions (due to the scope of S3 bucket names being global)… e.g. com.mydomain.mybucket.

This is causing this very error for me right now, and the error string makes it clear it can’t match the list of peer certificates, and this is not surprising since wildcard certificates only support a single level of sub-domains.

The simplest solution, therefore, is to create another bucket where the dots are hyphens which is also unlikely to clash with another global S3 bucket name.

I’m having the exact same issue as described here.

I have got it working using the UrlConnectionHttpClient instead, but would like to use the ApacheHttpClient for its performance.

@debora-ito, @raonitimo would this be the correct way to get the SDK to pick up the AWS_S3_US_EAST_1_REGIONAL_ENDPOINT setting:

AWS_S3_US_EAST_1_REGIONAL_ENDPOINT=regional java -jar my_app.jar

For a Spring app with a bean configuration like:

    @Bean
    public S3Client s3Client() {
        return S3Client.builder()
            .httpClient(ApacheHttpClient.builder().build())
            .build();
    }

This is my stack trace:

SdkClientException: Unable to execute HTTP request: 
Certificate for <my-app.s3.amazonaws.com> 
doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

The S3 bucket I’m getting the above error from is in US East (N. Virginia).

I also have made sure that the apache client i’ve installed is the version stated as having the fix:

[INFO] |  +- software.amazon.awssdk:apache-client:jar:2.15.0:compile
[INFO] |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile