boto3: SSLError: SSL validation failed ... EOF occurred in violation of protocol

Describe the bug

Python: 3.10.2 boto3/botocore: 1.24.17 OS: Alpine Linux 3.15

When attempting to upload a file to s3 I receive the following exception:

Traceback (most recent call last):
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 396, in _make_request
    conn.request_chunked(method, url, **httplib_request_kw)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 275, in request_chunked
    self.send(to_send)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 218, in send
    return super().send(str)
  File "/usr/local/lib/python3.10/http/client.py", line 998, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.10/ssl.py", line 1236, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/lib/python3.10/ssl.py", line 1205, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2384)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/httpsession.py", line 448, in send
    urllib_response = conn.urlopen(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 396, in _make_request
    conn.request_chunked(method, url, **httplib_request_kw)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 275, in request_chunked
    self.send(to_send)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 218, in send
    return super().send(str)
  File "/usr/local/lib/python3.10/http/client.py", line 998, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.10/ssl.py", line 1236, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/lib/python3.10/ssl.py", line 1205, in send
    return self._sslobj.write(data)
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2384)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app/.venv/lib/python3.10/site-packages/dramatiq/worker.py", line 485, in process_message
    res = actor(*message.args, **message.kwargs)
  File "/opt/app/.venv/lib/python3.10/site-packages/dramatiq/actor.py", line 145, in __call__
    return self.fn(*args, **kwargs)
  File "/opt/app/src/bona/tasks/actors/uploads.py", line 54, in move_upload_to_s3
    s3.upload_file(
  File "/opt/app/.venv/lib/python3.10/site-packages/boto3/s3/inject.py", line 143, in upload_file
    return transfer.upload_file(
  File "/opt/app/.venv/lib/python3.10/site-packages/boto3/s3/transfer.py", line 288, in upload_file
    future.result()
  File "/opt/app/.venv/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/opt/app/.venv/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/opt/app/.venv/lib/python3.10/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/opt/app/.venv/lib/python3.10/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/opt/app/.venv/lib/python3.10/site-packages/s3transfer/upload.py", line 758, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/client.py", line 898, in _make_api_call
    http, parsed_response = self._make_request(
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/client.py", line 921, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/opt/app/.venv/lib/python3.10/site-packages/botocore/httpsession.py", line 475, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://bucket.s3.us-east-2.amazonaws.com/key/parts.ext EOF occurred in violation of protocol (_ssl.c:2384)

Expected Behavior

File should be uploaded to s3

Current Behavior

getting the EOF exception

Reproduction Steps

This error is being generated by the following:

        s3: S3Client = boto3.client('s3')
        s3.upload_file(
            'local/path.ext',
            'bucket-name',
            'new/path.ext',
            ExtraArgs={'ChecksumAlgorithm': 'SHA256'},
        )

I’ve been able to reproduce a similar error against s3 and other ssl sites using the following.

from botocore.httpsession import create_urllib3_context
import urllib3
http = urllib3.PoolManager(ssl_context=create_urllib3_context(), retries=False)
http.urlopen('GET', 'https://www.howsmyssl.com/a/check')

Traceback (most recent call last):
  File "<input>", line 1, in <module>
    http.urlopen('GET', 'https://www.howsmyssl.com/a/check')
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/opt/app/.venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.10/ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.10/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.10/ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.24.17

Environment details (OS name and version, etc.)

Alpine Linux 3.15, Running in EKS