dvc: dvc push: unable to validate self-signed certificate from S3 service

Bug Report

Description

Attempting dvc push to s3 remote which uses SSL termination. Getting certificate validation error since we are using self-signed certificates

Reproduce

  1. dvc init
  2. modify remote to s3 endpoint with https scheme
  3. copy certificate bundle from remote se to local directory
  4. update AWS_CA_BUNDLE environment variable to point to local certificate bundle
  5. dvc add
  6. dvc push

Expected

data sent to s3 remote

Environment information

MacOs Homebrew DVC installation

Output of dvc doctor: DVC version: 2.0.18 (brew)

Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs Cache types: reflink, hardlink, symlink Cache directory: apfs on /dev/disk1s1 Caches: local Remotes: s3 Workspace directory: apfs on /dev/disk1s1 Repo: dvc, git

$ dvc doctor

Additional Information (if any): I placed the output of dvc push -v below along with relevant environment variables

johnathan@Johnathans-MacBook-Pro repo % dvc push -v
2021-05-05 13:56:31,064 DEBUG: Check for update is enabled.
+------------------------------------------+
|                                          |
|     Update available 2.0.18 -> 2.1.0     |
|          Run `brew upgrade dvc`          |
|                                          |
+------------------------------------------+

2021-05-05 13:56:31,329 DEBUG: Preparing to upload data to 's3://internal-test-files'
2021-05-05 13:56:31,330 DEBUG: Preparing to collect status from s3://internal-test-files
2021-05-05 13:56:31,330 DEBUG: Collecting information from local cache...
2021-05-05 13:56:31,332 DEBUG: Collecting information from remote cache...                                                                                                            
2021-05-05 13:56:31,332 DEBUG: Querying 1 hashes via object_exists
2021-05-05 13:56:43,907 ERROR: unexpected error - SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/main.py", line 55, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/command/data_sync.py", line 56, in run
    processed_files_count = self.repo.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/push.py", line 41, in push
    return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/data_cloud.py", line 66, in push
    return remote.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 56, in wrapper
    return f(obj, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 466, in push
    ret = self._process(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 323, in _process
    dir_status, file_status, dir_contents = self._status(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 169, in _status
    remote_exists.update(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 242, in _indexed_dir_hashes
    dir_exists.update(self.odb.list_hashes_exists(dir_md5s - dir_exists))
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 338, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 329, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 233, in exists
    return self.isfile(path_info) or self.isdir(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 263, in isfile
    return path_info.path in self._list_paths(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 284, in _list_paths
    for obj_summary in obj_summaries:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 166, in pages
    for page in pages:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 341, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
2021-05-05 13:56:44,478 DEBUG: Version info for developers:
DVC version: 2.0.18 (brew)
---------------------------------
Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-05-05 13:56:44,479 DEBUG: Analytics is enabled.
2021-05-05 13:56:44,717 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'
2021-05-05 13:56:44,722 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'

johnathan@Johnathans-MacBook-Pro repo % printenv
USER=johnathan
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
...
TERM=xterm-256color
AWS_CA_BUNDLE=</path/to/bundle.crt>
REQUESTS_CA_BUNDLE=REQUESTS_CA_BUNDLE=</path/to/bundle.crt>
_=/usr/bin/printenv
johnathan@Johnathans-MacBook-Pro repo % cat cat /Users/johnathan/.bash_profile             
cat: cat: No such file or directory
export PATH="/usr/local/bin:$PATH"
export PATH="/usr/local/opt/python@3.7/bin:$PATH"
export GODEBUG="x509ignoreCN=0"
export AWS_CA_BUNDLE="</path/to/bundle.crt>"
export REQUESTS_CA_BUNDLE="</path/to/bundle.crt>"
export SSL_CERT_FILE="</path/to/bundle.crt>"

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (10 by maintainers)

Most upvoted comments

@justusschock, it’ll take a few hours to get to the homebrew, for now, it’s only available through pip.

+2
skshetry on Jul 8, 2021

@justusschock, we’ll be releasing 2.5.2 with the fix in a few moments.

+2
skshetry on Jul 8, 2021

@mgoldchild just so you are aware, that will disable SSL (HTTPS) entirely. If you were looking to use SSL with a self-signed certificate, you should be using the ssl_verify config option, and not disabling use_ssl.

+2
pmrowla on Jun 8, 2021

PS: The install link didn’t work. Had to specify @kareemjano s repo and branch 😃

Really sorry about that. I did test in my machine and it worked, not sure why it didn’t on yours. 😦

+1
skshetry on Jul 8, 2021

@justusschock, use_ssl means not to use ssl connections at all (i.e. use http instead of https), which is not what you want. ssl_verify is server certificate validations which you want to turn off. Though I would not recommend you to set ssl_verify to false for security reasons, please set ssl_verify=/path/to/custom_certs if you can.

+1
skshetry on Jul 8, 2021

@justusschock, did you set the endpoint_url? Could you please try posting the error that you get ? Please try adding the -v flag to the command that you are using.

+1
skshetry on Jul 8, 2021

This issue should be resolved in 2.3.0

In 2.3.0 ssl_verify for S3 remotes can be used to point directly to a custom CA bundle, the same way you would use verify with boto. If ssl_verify is unset in your remote config, DVC will fall back (via boto) to the CA bundle specified in your ~/.aws config (previously leaving ssl_verify unset would force the default value to True)

+1
pmrowla on Jun 8, 2021

@JoaoDeDynam can you try to install dvc from the master, pip install "dvc[s3] @ git+https://github.com/iterative/dvc and try again? We recently rewrote all our s3 code, so it would be nice if you could check that version out and let us know whether you still experience an odd behavior.

+1
isidentical on May 6, 2021
Read more comments on GitHub
← cml: Instances intermittently fail to terminate
dvc: add --external: fails using Azure remote →